You are viewing a plain text version of this content. The canonical link for it is here.
Posted to server-dev@james.apache.org by Robert Burrell Donkin <ro...@gmail.com> on 2007/09/20 22:53:30 UTC
[Mime4J] Support For Non-Streaming Inputs
i'd like to gauge opinions about whether anyone else feels strongly
about supporting non-streaming inputs for MimeTokenStream. i've been
keeping this in mind whilst reviewing jochen's patches but this has
lead to conflicts about the design
(https://issues.apache.org/jira/browse/MIME4J-30). the price of
support will be greater complexity.
- robert
---------------------------------------------------------------------
To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
For additional commands, e-mail: server-dev-help@james.apache.org
RE: [Mime4J] Support For Non-Streaming Inputs
Posted by Norman Maurer <no...@apache.org>.
Am Freitag, den 21.09.2007, 15:30 -0400 schrieb Noel J. Bergman:
> Robert Burrell Donkin wrote:
>
> > it's really about objectives and aims rather than technical designs
>
> > i wonder whether it's worthwhile having what's probably going to be a
> > long technical discussion about a design for a unified parser for both
> > streams and non-streams. if no one else really feels strongly about
> > non-streaming implementations then it's probably best for me to drop
> > this goal, revert the changes i've made to allow support for efficient
> > non-streaming and let jochen optimize mime4j just for streams.
>
> I agree with you: let's talk about use cases before detailed design. As
> Serge asked, what are the use cases for both? What are the trade-offs in
> choosing to support or limit the exposed functionality?
>
> Please note: I may not be responding until Sunday evening.
>
> --- Noel
>
+1
bye
Norman
---------------------------------------------------------------------
To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
For additional commands, e-mail: server-dev-help@james.apache.org
RE: [Mime4J] Support For Non-Streaming Inputs
Posted by "Noel J. Bergman" <no...@devtech.com>.
Robert Burrell Donkin wrote:
> it's really about objectives and aims rather than technical designs
> i wonder whether it's worthwhile having what's probably going to be a
> long technical discussion about a design for a unified parser for both
> streams and non-streams. if no one else really feels strongly about
> non-streaming implementations then it's probably best for me to drop
> this goal, revert the changes i've made to allow support for efficient
> non-streaming and let jochen optimize mime4j just for streams.
I agree with you: let's talk about use cases before detailed design. As
Serge asked, what are the use cases for both? What are the trade-offs in
choosing to support or limit the exposed functionality?
Please note: I may not be responding until Sunday evening.
--- Noel
---------------------------------------------------------------------
To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
For additional commands, e-mail: server-dev-help@james.apache.org
Re: [Mime4J] Support For Non-Streaming Inputs
Posted by Robert Burrell Donkin <ro...@gmail.com>.
On 9/21/07, Noel J. Bergman <no...@devtech.com> wrote:
> Robert Burrell Donkin wrote:
> > Jochen Wiedmann wrote:
> > > robert burrell donkin-2 wrote:
> > > > i'd like to gauge opinions about whether anyone else feels strongly
> > > > about supporting non-streaming inputs for MimeTokenStream. i've been
> > > > keeping this in mind whilst reviewing jochen's patches but this has
> > > > lead to conflicts about the design
> > > > (https://issues.apache.org/jira/browse/MIME4J-30). the price of
> > > > support will be greater complexity.
> > >
> > > What are "non-streaming inputs"? If you are referring to java.nio: The
> > > patch I am developing (keep in mind: This patch is *not* the issue in
> > MIME4J-30) was developed specifically with java.nio in mind. However, my
> > > patch is not even proposed. Why do we discuss it *before* I post my
> > > proposal?
> >
> > because if no one but me cares then there's little point in me
> > obstructing your patches for what are design reasons
>
> Let's have some technical discussion, please. I'm not sure that I've seen
> enough to understand the dispute.
it's really about objectives and aims rather than technical designs
i wonder whether it's worthwhile having what's probably going to be a
long technical discussion about a design for a unified parser for both
streams and non-streams. if no one else really feels strongly about
non-streaming implementations then it's probably best for me to drop
this goal, revert the changes i've made to allow support for efficient
non-streaming and let jochen optimize mime4j just for streams.
- robert
---------------------------------------------------------------------
To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
For additional commands, e-mail: server-dev-help@james.apache.org
RE: [Mime4J] Support For Non-Streaming Inputs
Posted by "Noel J. Bergman" <no...@devtech.com>.
Robert Burrell Donkin wrote:
> Jochen Wiedmann wrote:
> > robert burrell donkin-2 wrote:
> > > i'd like to gauge opinions about whether anyone else feels strongly
> > > about supporting non-streaming inputs for MimeTokenStream. i've been
> > > keeping this in mind whilst reviewing jochen's patches but this has
> > > lead to conflicts about the design
> > > (https://issues.apache.org/jira/browse/MIME4J-30). the price of
> > > support will be greater complexity.
> >
> > What are "non-streaming inputs"? If you are referring to java.nio: The
> > patch I am developing (keep in mind: This patch is *not* the issue in
> MIME4J-30) was developed specifically with java.nio in mind. However, my
> > patch is not even proposed. Why do we discuss it *before* I post my
> > proposal?
>
> because if no one but me cares then there's little point in me
> obstructing your patches for what are design reasons
Let's have some technical discussion, please. I'm not sure that I've seen
enough to understand the dispute.
--- Noel
---------------------------------------------------------------------
To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
For additional commands, e-mail: server-dev-help@james.apache.org
Re: [Mime4J] Support For Non-Streaming Inputs
Posted by Robert Burrell Donkin <ro...@gmail.com>.
On 9/20/07, Jochen Wiedmann <jo...@gmail.com> wrote:
>
>
>
> robert burrell donkin-2 wrote:
> >
> > i'd like to gauge opinions about whether anyone else feels strongly
> > about supporting non-streaming inputs for MimeTokenStream. i've been
> > keeping this in mind whilst reviewing jochen's patches but this has
> > lead to conflicts about the design
> > (https://issues.apache.org/jira/browse/MIME4J-30). the price of
> > support will be greater complexity.
> >
>
> Robert, I am completely lost, what we are discussing here.
>
> What are "non-streaming inputs"? If you are referring to java.nio: The patch
> I am developing (keep in mind: This patch is *not* the issue in MIME4J-30)
> was developed specifically with java.nio in mind. However, my patch is not
> even proposed. Why do we discuss it *before* I post my proposal?
because if no one but me cares then there's little point in me
obstructing your patches for what are design reasons
- robert
---------------------------------------------------------------------
To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
For additional commands, e-mail: server-dev-help@james.apache.org
Re: [Mime4J] Support For Non-Streaming Inputs
Posted by Jochen Wiedmann <jo...@gmail.com>.
robert burrell donkin-2 wrote:
>
> i'd like to gauge opinions about whether anyone else feels strongly
> about supporting non-streaming inputs for MimeTokenStream. i've been
> keeping this in mind whilst reviewing jochen's patches but this has
> lead to conflicts about the design
> (https://issues.apache.org/jira/browse/MIME4J-30). the price of
> support will be greater complexity.
>
Robert, I am completely lost, what we are discussing here.
What are "non-streaming inputs"? If you are referring to java.nio: The patch
I am developing (keep in mind: This patch is *not* the issue in MIME4J-30)
was developed specifically with java.nio in mind. However, my patch is not
even proposed. Why do we discuss it *before* I post my proposal?
Jochen
--
View this message in context: http://www.nabble.com/-Mime4J--Support-For-Non-Streaming-Inputs-tf4490104.html#a12805801
Sent from the James - Dev mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
For additional commands, e-mail: server-dev-help@james.apache.org
Re: [Mime4J] Support For Non-Streaming Inputs
Posted by Norman Maurer <no...@apache.org>.
Hi,
I don't think we need to support non-streaming solutions in mime4j.. But
that's only my "idea"..
bye
Norman
Am Donnerstag, den 20.09.2007, 21:53 +0100 schrieb Robert Burrell
Donkin:
> i'd like to gauge opinions about whether anyone else feels strongly
> about supporting non-streaming inputs for MimeTokenStream. i've been
> keeping this in mind whilst reviewing jochen's patches but this has
> lead to conflicts about the design
> (https://issues.apache.org/jira/browse/MIME4J-30). the price of
> support will be greater complexity.
>
> - robert
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
> For additional commands, e-mail: server-dev-help@james.apache.org
>
>
> !DSPAM:1,46f2ddf2283431127720120!
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
For additional commands, e-mail: server-dev-help@james.apache.org
Re: [Mime4J] Support For Non-Streaming Inputs
Posted by Robert Burrell Donkin <ro...@gmail.com>.
On 9/22/07, Robert Burrell Donkin <ro...@gmail.com> wrote:
> On 9/21/07, Jochen Wiedmann <jo...@gmail.com> wrote:
> >
> >
> >
> > Serge Knystautas-2 wrote:
> > >
> > > Thanks for raising this project scope issue since IMHO these are the
> > > bigger than technical decisions. If I can restate, currently we
> > > support both non-streaming and streaming uses, and have a cursor API
> > > out of that. The project scope question at hand is if we drop support
> > > for non-streaming (and thus the cursor API) and just optimize for the
> > > streaming use case.
> > >
> >
> > Could anybody please be so kind and start with an explanation what
> > "streaming", or "non-streaming" means? I have the definite impression, that
> > we are talking quite different things here. In particular, I am far from
> > understanding why they should be mutually exclusive.
>
> i'm not sure that they are necessarily mutally exclusive but i do
> believe that good support introduces complexity and leads to
> unnecessary debates about design
>
> some non-streaming use cases:
>
> 1 efficient parsing of nio data (primarily ByteBuffer)
> 1a (example) parse contents of single ByteBuffer (eg backed by byte
> array) without double buffering
> 1b (example) parse contents of a Channel (eg memory mapped file)
> without double buffering
>
> 2 efficient parsing of structured data
> 2a (example) parse MimeMessage without transfering raw data to an input stream
> 2b (example) parse mail message stored as fully or partially parsed
> records in a data source (eg database) without transfer of raw data to
> an input stream
streaming use case:
3 efficient parsing of data in a InputStream
3a (example) efficient parsing of data streamed from a (blocking) socket
3b (example) efficient parsing of data streamed from a file opened
with bio (blocking io eg FileInputStream)
- robert
---------------------------------------------------------------------
To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
For additional commands, e-mail: server-dev-help@james.apache.org
Re: [Mime4J] Support For Non-Streaming Inputs
Posted by Robert Burrell Donkin <ro...@gmail.com>.
On 9/23/07, Noel J. Bergman <no...@devtech.com> wrote:
> Robert Burrell Donkin wrote:
>
> > your patch would require double buffering when used with direct
> > buffers and would require convertion of structured data into bytes.
> > so, it's unsatisfactory for use cases 1 and 2. however, i'm not sure
> > whether this is something that is worthwhile arguing about.
>
> Since it effects use cases 1 and 2, both of which I consider important, the
> answer is YES! Emphatically and unequivocally.
i find it difficult to understand the intended design when reviewing
lots of small patches. yes, jochen's patches would regress the hooks
i'd added to support use cases 1 and 2 but it is more than possible
that he has an alternative design in mind which would also support
them.
but the key question is whether mime4j should aim to support these use cases
if so, then if we can determine a general strategy for 'how' then it
will be much easier to code and review these small patches
- robert
---------------------------------------------------------------------
To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
For additional commands, e-mail: server-dev-help@james.apache.org
Re: [Mime4J] Support For Non-Streaming Inputs
Posted by Robert Burrell Donkin <ro...@gmail.com>.
On 9/24/07, Stefano Bagnara <ap...@bago.org> wrote:
> Bernd Fondermann ha scritto:
> > On 9/23/07, Jochen Wiedmann <jo...@gmail.com> wrote:
> >>
> >>
> >> robert burrell donkin-2 wrote:
> >>> this is the problem with lots of small patches: i don't understand
> >>> where you are taking the design
> >>>
> >> In general, non-committers are expected to split their work up into smaller
> >> patches, because it helps the reviewers.
> >
> > Generally, yes. But if you'd like to present an alternative
> > experimental design as a whole, we could as well grant you a sandbox.
> > BTW, this would be the appropriate approach for such efforts even for
> > committers.
> >
> > Bernd
>
> As far as I've understood until now the "whole new design" was from
> Robert. Jochen patches are small performance improvement over the
> current design.
>
> The main problem is that Jochen patches make it more difficult to keep
> supporting the new design from Robert (or at least make the whole thing
> less "clean")
>
> I don't know what design is better for mime4j and I didn't analyze the
> patches in depth to judge their quality. What I can tell is that Jochen
> patches serve a specific concrete purpose (commons-upload integration
> and its needed capabilities) while I feel Robert ones more abstract. I
> guess Robert "design" is related to specific IMAP usage, so it would
> probably help (me) to understand what IMAP features are involved by this
> design decisions (and how).
i'm not too bothered about the current design: it was just a stepping stone
as long as jochen understands the additional use cases and either
takes them into account with his design or accepts that we may rewrite
later to take them into account then that's good enough for me.
- robert
---------------------------------------------------------------------
To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
For additional commands, e-mail: server-dev-help@james.apache.org
Re: [OT] Re: [Mime4J] Support For Non-Streaming Inputs
Posted by Stefano Bagnara <ap...@bago.org>.
Norman Maurer ha scritto:
> Am Montag, den 24.09.2007, 21:28 +0200 schrieb Stefano Bagnara:
>> Norman Maurer ha scritto:
>>> Ps: Serge maybe you allready knows... Your email client seems to set the
>>> reply-to field to your emailaddress. So if someone just use reply you
>>> are the only one which gets the answers..
>> I have no problems replying to Serge messages.
>> But the problem is that there are 2 Reply-To headers, so I think this
>> depends on the MIME parsing library used by the MUA (so we magically
>> make an OffTopic message InTopic again ;-) ).
>>
>> Stefano
>
> What you use ? Im using ubuntu gutsy with evolution ....
>
> bye
> Norman
Thunderbird 2.0.0.6
Stefano
---------------------------------------------------------------------
To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
For additional commands, e-mail: server-dev-help@james.apache.org
[OT] Re: [Mime4J] Support For Non-Streaming Inputs
Posted by Norman Maurer <no...@apache.org>.
Am Montag, den 24.09.2007, 21:28 +0200 schrieb Stefano Bagnara:
> Norman Maurer ha scritto:
> > Am Montag, den 24.09.2007, 14:58 -0400 schrieb Serge Knystautas:
> >> On 9/24/07, Stefano Bagnara <ap...@bago.org> wrote:
> >>> Ignoring the technical/design issue I think at this time it is better
> >>> for mime4j to increase its community/acceptance by helping other
> >>> projects (e.g. commons-upload) to use mime4j. IMHO collecting users that
> >>> depends on the library is the best thing to ensure that future design
> >>> choices will satisfy anyone needs.
> >> +0.6 New users/developers with energy to work on it are good.
> >>
> > +1
> > Norman
> >
> > Ps: Serge maybe you allready knows... Your email client seems to set the
> > reply-to field to your emailaddress. So if someone just use reply you
> > are the only one which gets the answers..
>
> I have no problems replying to Serge messages.
> But the problem is that there are 2 Reply-To headers, so I think this
> depends on the MIME parsing library used by the MUA (so we magically
> make an OffTopic message InTopic again ;-) ).
>
> Stefano
What you use ? Im using ubuntu gutsy with evolution ....
bye
Norman
---------------------------------------------------------------------
To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
For additional commands, e-mail: server-dev-help@james.apache.org
Re: [Mime4J] Support For Non-Streaming Inputs
Posted by Stefano Bagnara <ap...@bago.org>.
Norman Maurer ha scritto:
> Am Montag, den 24.09.2007, 14:58 -0400 schrieb Serge Knystautas:
>> On 9/24/07, Stefano Bagnara <ap...@bago.org> wrote:
>>> Ignoring the technical/design issue I think at this time it is better
>>> for mime4j to increase its community/acceptance by helping other
>>> projects (e.g. commons-upload) to use mime4j. IMHO collecting users that
>>> depends on the library is the best thing to ensure that future design
>>> choices will satisfy anyone needs.
>> +0.6 New users/developers with energy to work on it are good.
>>
> +1
> Norman
>
> Ps: Serge maybe you allready knows... Your email client seems to set the
> reply-to field to your emailaddress. So if someone just use reply you
> are the only one which gets the answers..
I have no problems replying to Serge messages.
But the problem is that there are 2 Reply-To headers, so I think this
depends on the MIME parsing library used by the MUA (so we magically
make an OffTopic message InTopic again ;-) ).
Stefano
---------------------------------------------------------------------
To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
For additional commands, e-mail: server-dev-help@james.apache.org
Re: [Mime4J] Support For Non-Streaming Inputs
Posted by Stefano Bagnara <ap...@bago.org>.
Serge Knystautas ha scritto:
> Norman Maurer wrote:
>> Ps: Serge maybe you allready knows... Your email client seems to set the
>> reply-to field to your emailaddress. So if someone just use reply you
>> are the only one which gets the answers..
>
> Thanks for letting me know. I'm just using GMail. Danny, I think you
> use GMail for ASF mail as well... do you have to do something special,
> or does everyone see this happen with email from Danny and me?
>
> (note I'm not sending this one from Gmail)
No, Danny's mail only show 1 Reply-To header (and 1 Sender header).
Instead in your previous mail you had a reply-to to your lokitech address.
Stefano
---------------------------------------------------------------------
To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
For additional commands, e-mail: server-dev-help@james.apache.org
Re: [Mime4J] Support For Non-Streaming Inputs
Posted by Serge Knystautas <se...@lokitech.com>.
Norman Maurer wrote:
> Ps: Serge maybe you allready knows... Your email client seems to set the
> reply-to field to your emailaddress. So if someone just use reply you
> are the only one which gets the answers..
Thanks for letting me know. I'm just using GMail. Danny, I think you
use GMail for ASF mail as well... do you have to do something special,
or does everyone see this happen with email from Danny and me?
(note I'm not sending this one from Gmail)
--
Serge Knystautas
Lokitech >> software . strategy . design >> http://www.lokitech.com
p. 301.656.5501
e. sergek@lokitech.com
---------------------------------------------------------------------
To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
For additional commands, e-mail: server-dev-help@james.apache.org
Re: [Mime4J] Support For Non-Streaming Inputs
Posted by Norman Maurer <no...@apache.org>.
Am Montag, den 24.09.2007, 14:58 -0400 schrieb Serge Knystautas:
> On 9/24/07, Stefano Bagnara <ap...@bago.org> wrote:
> > Ignoring the technical/design issue I think at this time it is better
> > for mime4j to increase its community/acceptance by helping other
> > projects (e.g. commons-upload) to use mime4j. IMHO collecting users that
> > depends on the library is the best thing to ensure that future design
> > choices will satisfy anyone needs.
>
> +0.6 New users/developers with energy to work on it are good.
>
+1
Norman
Ps: Serge maybe you allready knows... Your email client seems to set the
reply-to field to your emailaddress. So if someone just use reply you
are the only one which gets the answers..
---------------------------------------------------------------------
To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
For additional commands, e-mail: server-dev-help@james.apache.org
Re: [Mime4J] Support For Non-Streaming Inputs
Posted by Serge Knystautas <sk...@gmail.com>.
On 9/24/07, Stefano Bagnara <ap...@bago.org> wrote:
> Ignoring the technical/design issue I think at this time it is better
> for mime4j to increase its community/acceptance by helping other
> projects (e.g. commons-upload) to use mime4j. IMHO collecting users that
> depends on the library is the best thing to ensure that future design
> choices will satisfy anyone needs.
+0.6 New users/developers with energy to work on it are good.
--
Serge Knystautas
Lokitech >> software . strategy . design >> http://www.lokitech.com
p. 301.656.5501
e. sergek@lokitech.com
---------------------------------------------------------------------
To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
For additional commands, e-mail: server-dev-help@james.apache.org
Re: [Mime4J] Support For Non-Streaming Inputs
Posted by Niklas Therning <ni...@trillian.se>.
Stefano Bagnara wrote:
> Bernd Fondermann ha scritto:
>
>> On 9/23/07, Jochen Wiedmann <jo...@gmail.com> wrote:
>>
>>> robert burrell donkin-2 wrote:
>>>
>>>> this is the problem with lots of small patches: i don't understand
>>>> where you are taking the design
>>>>
>>>>
>>> In general, non-committers are expected to split their work up into smaller
>>> patches, because it helps the reviewers.
>>>
>> Generally, yes. But if you'd like to present an alternative
>> experimental design as a whole, we could as well grant you a sandbox.
>> BTW, this would be the appropriate approach for such efforts even for
>> committers.
>>
>> Bernd
>>
>
> As far as I've understood until now the "whole new design" was from
> Robert. Jochen patches are small performance improvement over the
> current design.
>
> The main problem is that Jochen patches make it more difficult to keep
> supporting the new design from Robert (or at least make the whole thing
> less "clean")
>
> I don't know what design is better for mime4j and I didn't analyze the
> patches in depth to judge their quality. What I can tell is that Jochen
> patches serve a specific concrete purpose (commons-upload integration
> and its needed capabilities) while I feel Robert ones more abstract. I
> guess Robert "design" is related to specific IMAP usage, so it would
> probably help (me) to understand what IMAP features are involved by this
> design decisions (and how).
>
> Ignoring the technical/design issue I think at this time it is better
> for mime4j to increase its community/acceptance by helping other
> projects (e.g. commons-upload) to use mime4j. IMHO collecting users that
> depends on the library is the best thing to ensure that future design
> choices will satisfy anyone needs.
>
> At the same time, we should try to be our first users with JAMES Server
> but we currently don't use it. If Robert design is a prerequisite to be
> able to use mime4j in the IMAP implementation then this worth discussing
> more.
>
> I'd also like to know what Niklas Therning thinks about this issue. He
> is one of the original authors and probably one of the few current users
> of this library.
>
We're using MimeStreamParser only and as long as the way that works
doesn't change too much we will be happy. In our POP3, IMAP and SMTP
proxies we're using NIO (Apache MINA) and ByteBuffers all over the place
so I guess we could maybe some day in the future have use for
non-streaming parsing. But I think that's quite unlikely.
I must admit I haven't had the time to understand this discussion
completely. I would certainly like to see commons-fileupload use mime4j
so I think we should try to accommodate Jochen's needs. It appears to me
that those needs would be satisfied by resolving MIME4J-27 and
MIME4J-30. As Robert says in one of the first comments on MIME4J-30 he
could look into producing an alternative patch which resolves that issue
while still keeping the current design. IIUC that would both solve
Jochen's problems and still let Robert use mime4j the way he wants.
Am I missing something here?
--
Niklas Therning
www.spamdrain.net
---------------------------------------------------------------------
To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
For additional commands, e-mail: server-dev-help@james.apache.org
Re: [Mime4J] Support For Non-Streaming Inputs
Posted by Stefano Bagnara <ap...@bago.org>.
Bernd Fondermann ha scritto:
> On 9/23/07, Jochen Wiedmann <jo...@gmail.com> wrote:
>>
>>
>> robert burrell donkin-2 wrote:
>>> this is the problem with lots of small patches: i don't understand
>>> where you are taking the design
>>>
>> In general, non-committers are expected to split their work up into smaller
>> patches, because it helps the reviewers.
>
> Generally, yes. But if you'd like to present an alternative
> experimental design as a whole, we could as well grant you a sandbox.
> BTW, this would be the appropriate approach for such efforts even for
> committers.
>
> Bernd
As far as I've understood until now the "whole new design" was from
Robert. Jochen patches are small performance improvement over the
current design.
The main problem is that Jochen patches make it more difficult to keep
supporting the new design from Robert (or at least make the whole thing
less "clean")
I don't know what design is better for mime4j and I didn't analyze the
patches in depth to judge their quality. What I can tell is that Jochen
patches serve a specific concrete purpose (commons-upload integration
and its needed capabilities) while I feel Robert ones more abstract. I
guess Robert "design" is related to specific IMAP usage, so it would
probably help (me) to understand what IMAP features are involved by this
design decisions (and how).
Ignoring the technical/design issue I think at this time it is better
for mime4j to increase its community/acceptance by helping other
projects (e.g. commons-upload) to use mime4j. IMHO collecting users that
depends on the library is the best thing to ensure that future design
choices will satisfy anyone needs.
At the same time, we should try to be our first users with JAMES Server
but we currently don't use it. If Robert design is a prerequisite to be
able to use mime4j in the IMAP implementation then this worth discussing
more.
I'd also like to know what Niklas Therning thinks about this issue. He
is one of the original authors and probably one of the few current users
of this library.
Stefano
---------------------------------------------------------------------
To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
For additional commands, e-mail: server-dev-help@james.apache.org
Re: [Mime4J] Support For Non-Streaming Inputs
Posted by Bernd Fondermann <be...@googlemail.com>.
On 9/23/07, Jochen Wiedmann <jo...@gmail.com> wrote:
>
>
>
> robert burrell donkin-2 wrote:
> >
> > this is the problem with lots of small patches: i don't understand
> > where you are taking the design
> >
>
> In general, non-committers are expected to split their work up into smaller
> patches, because it helps the reviewers.
Generally, yes. But if you'd like to present an alternative
experimental design as a whole, we could as well grant you a sandbox.
BTW, this would be the appropriate approach for such efforts even for
committers.
Bernd
---------------------------------------------------------------------
To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
For additional commands, e-mail: server-dev-help@james.apache.org
Re: [Mime4J] Support For Non-Streaming Inputs
Posted by Stefano Bagnara <ap...@bago.org>.
Noel J. Bergman ha scritto:
> Robert Burrell Donkin wrote:
>
>> the disadvantage with using a byte array rather than a bytebuffer is
>> that direct bytebuffers would have to copy their data out into a byte
>> array. using a byte buffer at the lowest level would solve this issue
>> without really an added overhead for the bio case (just create a byte
>> array backed buffer and then fill that buffer from the inputstream).
>
> Given my background in real-time, embedded, systems, I'd like to see us
> improving performance, and doing a lot less movement of data. So I'm in
> favor of changes that reduce data movement.
>
> Here's a question for the lot of you: is this similar to DOM vs SAX, and if
> so, can we come up with a StAX solution? Just go with the analogy, but the
> issue is a best of both worlds.
>
> --- Noel
I think that comparing this design issue to the existing XML
parsing/manipulating APIs may help (At least this helps me, and I see
you refer to it, too)
Manipulating a mime document is not so different from Manipulating an
xml document.
The key here is that, like for XML parser, there is not a best solution
for every use case. If you need backward navigation than you can't use
StAX or SAX, if you want to manipulate the XML at the source then your
only option is DOM. If you are mainly interested in transformation then
TrAX could be the best solution.
StAX is the best of both worlds only if you need SAX features but you
want a simpler interface, but using StAX you can't alter the original
"document".
As far as I understand Robert Cursor interface *is* the "StAX like"
solution.
After so many years I think no API let you do all you can do via the
DOM: the new APIs simply removed features to give us much better
performances.
Let's also take into consideration the main difference between "generic"
XML documents and MIME documents: MIME most time will have a smaller
structure tree, but bigger size for parts than most XML documents. E.g:
to manage a lazy loaded MIME skeleton in memory is feasible while
keeping small memory usage.
Another difference is that most time XML parsers do not need to know how
the exact source looked like, while we identified use cases where the
original text is very important (S/MIME).
Can you see other big differences between XML and MIME in this comparison?
The problem is, IMHO, that we already have a lot of very different use
cases for a MIME document "handling" library:
A. SMTPServer receiving message: receive data asynchronously via NIO
buffers and be able to create MIME events without blocking (push api).
To avoid big memory usage we need streaming here and a lot of processing
should be done "on the fly" *while* the message is temporarily saved in
the queue (anti-spam, content filtering, size limit, other).
B. IMAPServer sending data to a client: needs a fast way to know the
message structure and be able to retrieve each part without streaming
the whole message, but being able to stream (without blocking) a part
(or the full message) to the client (SEDA).
C. Mailets altering messages: most mailets will probably like to alter
the message in a DOM like way. getParts, addAttachment, removeAttachment
and similar things. Again we should find a way to avoid loading big
parts in memory when not needed. Few mailets may prefer a transformation
api based on an inputstream/outputstram solution.
The 3 above are 3 use cases for our "parsing" needs, but we also have to
take into consideration that there are 2 very different source
"transports" for this documents:
1. "The Internet": a BIO or NIO connection streaming the data in our
direction.
2. "The local storage": a file, a database record (an abstraction of the
file access), a JCR node (an abstraction of one of the previous).
The BIG difference between the 2 is that the former give us a "one-shot"
document (We can't go back in the document once we received it) while
the latter allow us to read it as many times as we like.
Another big aspect of "transports" is the BIO vs NIO and this applies
both to "the internet" and "the local storage"
The last aspect is the Synchronous processing vs Asynchronous processing
and the related patterns (thread pools, SEDA...). (e.g: the SAX-style
probably better fits the SEDA scenario than any other XML api).
My conclusion is that we can add as much complexity as we want to this
design discussion but we'll probably never find a single API for all of
the use cases we can think of, and probably we can't even find a single
API that will be the best one for our *current* use cases.
Maybe the wrong thing is to try to create an API to accomplish so
different tasks like removing/adding an attachment to a MIME document
stored in a local random access file, fast failing when an SMTP-incoming
MIME message has an attachment bigger than 1MB, converting an 8bit
message to 7bit and viceversa. Maybe it is better to create separate
APIs and simply reuse code under the hood.
Well, this was much more than my 2 cents,
Stefano
---------------------------------------------------------------------
To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
For additional commands, e-mail: server-dev-help@james.apache.org
RE: [Mime4J] Support For Non-Streaming Inputs
Posted by "Noel J. Bergman" <no...@devtech.com>.
Robert Burrell Donkin wrote:
> the disadvantage with using a byte array rather than a bytebuffer is
> that direct bytebuffers would have to copy their data out into a byte
> array. using a byte buffer at the lowest level would solve this issue
> without really an added overhead for the bio case (just create a byte
> array backed buffer and then fill that buffer from the inputstream).
Given my background in real-time, embedded, systems, I'd like to see us
improving performance, and doing a lot less movement of data. So I'm in
favor of changes that reduce data movement.
Here's a question for the lot of you: is this similar to DOM vs SAX, and if
so, can we come up with a StAX solution? Just go with the analogy, but the
issue is a best of both worlds.
--- Noel
---------------------------------------------------------------------
To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
For additional commands, e-mail: server-dev-help@james.apache.org
Re: [Mime4J] Support For Non-Streaming Inputs
Posted by Robert Burrell Donkin <ro...@gmail.com>.
On 9/23/07, Jochen Wiedmann <jo...@gmail.com> wrote:
<snip>
> robert burrell donkin-2 wrote:
> >
> > it's direct use of byte arrays which worried me. how do you propose to
> > handle memory mapped files without double buffering?
> >
>
> I have to admit that I do not know about memory mapped files and why they
> should be relevant. I'll be reading on that.
the disadvantage with using a byte array rather than a bytebuffer is
that direct bytebuffers would have to copy their data out into a byte
array. using a byte buffer at the lowest level would solve this issue
without really an added overhead for the bio case (just create a byte
array backed buffer and then fill that buffer from the inputstream).
- robert
---------------------------------------------------------------------
To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
For additional commands, e-mail: server-dev-help@james.apache.org
Re: [Mime4J] Support For Non-Streaming Inputs
Posted by Jochen Wiedmann <jo...@gmail.com>.
robert burrell donkin-2 wrote:
>
> this is the problem with lots of small patches: i don't understand
> where you are taking the design
>
In general, non-committers are expected to split their work up into smaller
patches, because it helps the reviewers.
Apart from that, this patch arises out of my work on the commons-fileupload
rewrite, into which you have complete insight, if you visit
http://people.apache.org/~jochen/commons-fileupload/
The MIME4J-30 patch is, absolutely unrelated to the current discussion. It
concerns the question, whether the MIME4J user should care about
transport-encoding or not. This question comes up regardless of the overall
design.
robert burrell donkin-2 wrote:
>
> it's direct use of byte arrays which worried me. how do you propose to
> handle memory mapped files without double buffering?
>
I have to admit that I do not know about memory mapped files and why they
should be relevant. I'll be reading on that.
--
View this message in context: http://www.nabble.com/-Mime4J--Support-For-Non-Streaming-Inputs-tf4490104.html#a12845823
Sent from the James - Dev mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
For additional commands, e-mail: server-dev-help@james.apache.org
Re: [Mime4J] Support For Non-Streaming Inputs
Posted by Robert Burrell Donkin <ro...@gmail.com>.
On 9/22/07, Jochen Wiedmann <jo...@gmail.com> wrote:
>
>
>
> robert burrell donkin-2 wrote:
> >
> > your patch would require double buffering when used with direct
> > buffers and would require convertion of structured data into bytes.
> > so, it's unsatisfactory for use cases 1 and 2. however, i'm not sure
> > whether this is something that is worthwhile arguing about.
> >
>
> You haven't even see the patch, so how do you know?
this is the problem with lots of small patches: i don't understand
where you are taking the design
> Double buffering: I don't see why. The patch was designed to work with the
> buffers provided by a thread that's doing select().
it's direct use of byte arrays which worried me. how do you propose to
handle memory mapped files without double buffering?
> Convertion of structured data into bytes: How's that different from what we
> have now? And, if you really want to provide structured data, then the
> proper way to do so is to refactor an interface out of the MimeTokenStream
> and provide an implementation that produces the same events. Or, do the same
> thing by firing events to a ContentHandler.
the design question is at what level this interface should be. the API
would be better if MimeTokenStream were an interface and i'm satisfied
that this design can satisfy the structured data use case.
can anyone else see any problems with this approach?
- robert
---------------------------------------------------------------------
To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
For additional commands, e-mail: server-dev-help@james.apache.org
Re: [Mime4J] Support For Non-Streaming Inputs
Posted by Jochen Wiedmann <jo...@gmail.com>.
robert burrell donkin-2 wrote:
>
> your patch would require double buffering when used with direct
> buffers and would require convertion of structured data into bytes.
> so, it's unsatisfactory for use cases 1 and 2. however, i'm not sure
> whether this is something that is worthwhile arguing about.
>
You haven't even see the patch, so how do you know?
Double buffering: I don't see why. The patch was designed to work with the
buffers provided by a thread that's doing select().
Convertion of structured data into bytes: How's that different from what we
have now? And, if you really want to provide structured data, then the
proper way to do so is to refactor an interface out of the MimeTokenStream
and provide an implementation that produces the same events. Or, do the same
thing by firing events to a ContentHandler.
Jochen
--
View this message in context: http://www.nabble.com/-Mime4J--Support-For-Non-Streaming-Inputs-tf4490104.html#a12838664
Sent from the James - Dev mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
For additional commands, e-mail: server-dev-help@james.apache.org
RE: [Mime4J] Support For Non-Streaming Inputs
Posted by "Noel J. Bergman" <no...@devtech.com>.
Robert Burrell Donkin wrote:
> your patch would require double buffering when used with direct
> buffers and would require convertion of structured data into bytes.
> so, it's unsatisfactory for use cases 1 and 2. however, i'm not sure
> whether this is something that is worthwhile arguing about.
Since it effects use cases 1 and 2, both of which I consider important, the
answer is YES! Emphatically and unequivocally.
I am speaking of whether or not it is worth discussing, not with respect to
the merits of a code change that hasn't been presented, since Jochen claims
that we have not seen what he intends to present.
--- Noel
---------------------------------------------------------------------
To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
For additional commands, e-mail: server-dev-help@james.apache.org
Re: [Mime4J] Support For Non-Streaming Inputs
Posted by Robert Burrell Donkin <ro...@gmail.com>.
On 9/22/07, Jochen Wiedmann <jo...@gmail.com> wrote:
>
>
>
> robert burrell donkin-2 wrote:
> >
> > i'm not sure that they are necessarily mutally exclusive but i do
> > believe that good support introduces complexity and leads to
> > unnecessary debates about design
> >
>
> And the proper place for that discussion seems to me to be when a proposed
> patch comes up. IMO, the solution that I have developed is comparable to the
> current in terms of complexity. It is also well suited for InputStream's (I
> developed it with the goal in mind to improve the current implementations
> performance with a servlet input stream in mind) as well as byte buffers.
your patch would require double buffering when used with direct
buffers and would require convertion of structured data into bytes.
so, it's unsatisfactory for use cases 1 and 2. however, i'm not sure
whether this is something that is worthwhile arguing about.
- robert
---------------------------------------------------------------------
To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
For additional commands, e-mail: server-dev-help@james.apache.org
Re: [Mime4J] Support For Non-Streaming Inputs
Posted by Jochen Wiedmann <jo...@gmail.com>.
robert burrell donkin-2 wrote:
>
> i'm not sure that they are necessarily mutally exclusive but i do
> believe that good support introduces complexity and leads to
> unnecessary debates about design
>
And the proper place for that discussion seems to me to be when a proposed
patch comes up. IMO, the solution that I have developed is comparable to the
current in terms of complexity. It is also well suited for InputStream's (I
developed it with the goal in mind to improve the current implementations
performance with a servlet input stream in mind) as well as byte buffers.
Whether it is actually faster, I cannot tell so far, because I am still
debugging. That can be seen when the benchmark is working.
Jochen
--
View this message in context: http://www.nabble.com/-Mime4J--Support-For-Non-Streaming-Inputs-tf4490104.html#a12835172
Sent from the James - Dev mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
For additional commands, e-mail: server-dev-help@james.apache.org
Re: [Mime4J] Support For Non-Streaming Inputs
Posted by Robert Burrell Donkin <ro...@gmail.com>.
On 9/21/07, Jochen Wiedmann <jo...@gmail.com> wrote:
>
>
>
> Serge Knystautas-2 wrote:
> >
> > Thanks for raising this project scope issue since IMHO these are the
> > bigger than technical decisions. If I can restate, currently we
> > support both non-streaming and streaming uses, and have a cursor API
> > out of that. The project scope question at hand is if we drop support
> > for non-streaming (and thus the cursor API) and just optimize for the
> > streaming use case.
> >
>
> Could anybody please be so kind and start with an explanation what
> "streaming", or "non-streaming" means? I have the definite impression, that
> we are talking quite different things here. In particular, I am far from
> understanding why they should be mutually exclusive.
i'm not sure that they are necessarily mutally exclusive but i do
believe that good support introduces complexity and leads to
unnecessary debates about design
some non-streaming use cases:
1 efficient parsing of nio data (primarily ByteBuffer)
1a (example) parse contents of single ByteBuffer (eg backed by byte
array) without double buffering
1b (example) parse contents of a Channel (eg memory mapped file)
without double buffering
2 efficient parsing of structured data
2a (example) parse MimeMessage without transfering raw data to an input stream
2b (example) parse mail message stored as fully or partially parsed
records in a data source (eg database) without transfer of raw data to
an input stream
- robert
---------------------------------------------------------------------
To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
For additional commands, e-mail: server-dev-help@james.apache.org
Re: [Mime4J] Support For Non-Streaming Inputs
Posted by Jochen Wiedmann <jo...@gmail.com>.
Serge Knystautas-2 wrote:
>
> Thanks for raising this project scope issue since IMHO these are the
> bigger than technical decisions. If I can restate, currently we
> support both non-streaming and streaming uses, and have a cursor API
> out of that. The project scope question at hand is if we drop support
> for non-streaming (and thus the cursor API) and just optimize for the
> streaming use case.
>
Could anybody please be so kind and start with an explanation what
"streaming", or "non-streaming" means? I have the definite impression, that
we are talking quite different things here. In particular, I am far from
understanding why they should be mutually exclusive.
Jochen
--
View this message in context: http://www.nabble.com/-Mime4J--Support-For-Non-Streaming-Inputs-tf4490104.html#a12829975
Sent from the James - Dev mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
For additional commands, e-mail: server-dev-help@james.apache.org
Re: [Mime4J] Support For Non-Streaming Inputs
Posted by Serge Knystautas <sk...@gmail.com>.
On 9/20/07, Robert Burrell Donkin <ro...@gmail.com> wrote:
> i'd like to gauge opinions about whether anyone else feels strongly
> about supporting non-streaming inputs for MimeTokenStream. i've been
> keeping this in mind whilst reviewing jochen's patches but this has
> lead to conflicts about the design
> (https://issues.apache.org/jira/browse/MIME4J-30). the price of
> support will be greater complexity.
Thanks for raising this project scope issue since IMHO these are the
bigger than technical decisions. If I can restate, currently we
support both non-streaming and streaming uses, and have a cursor API
out of that. The project scope question at hand is if we drop support
for non-streaming (and thus the cursor API) and just optimize for the
streaming use case.
Is that right? And if so, can you give me a use case or two for non-streaming?
--
Serge Knystautas
Lokitech >> software . strategy . design >> http://www.lokitech.com
p. 301.656.5501
e. sergek@lokitech.com
---------------------------------------------------------------------
To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
For additional commands, e-mail: server-dev-help@james.apache.org