You are viewing a plain text version of this content. The canonical link for it is here.
Posted to mime4j-dev@james.apache.org by Ioan Eugen Stan <st...@gmail.com> on 2012/02/09 11:23:11 UTC
NIO Iterator over messages in mbox file
Hello,
This is the second try to get the message on the mailing list.
I've started working on a small project called mbox-iterator that I wish
to integrate with mime4j later, when it's more usable. It's purpose is
to provide a iterator like interface over mbox files. Details about the
project are here [1].
I'm bringing it on because I think this would be a nice addition to
mime4j and I need some help.
One thing that I can't figure out is: does mime4j do charset decoding? I
haven't looked over the code but I know it expects a stream of bytes.
mbox-iterator does regexp matching to find From_ lines and that requires
a CharSequence. This means I have use/return CharBuffer's from the
iterator. The CharBuffers need to be re-encoded and streamed as bytes
(maybe with ByteArrayInputStream).
In the future I plan to use git-svn to experiment new stuff but I would
like to get it fast into mime4j. What do you think?
[1]
http://ieugen.blogspot.com/2012/02/nio-iterator-over-messages-in-mbox-file.html
Cheers,
--
Ioan Eugen Stan
http://ieugen.blogspot.com/
Re: NIO Iterator over messages in mbox file
Posted by Ioan Eugen Stan <st...@gmail.com>.
Hi Norman,
Pe 19.03.2012 12:45, Norman Maurer a scris:
> Could you tell us the url of the repos again?
https://github.com/ieugen/mbox-iterator
Uups :)
>
> Thx,
> Norman
>
> Sent from my iPhone. Excuse any typos....
>
> Am 19.03.2012 um 10:20 schrieb Ioan Eugen Stan<st...@gmail.com>:
>
>> 2012/2/11 Ioan Eugen STAN<st...@gmail.com>:
>>> On Sb 11 feb 2012 00:15:57 +0200, Markus Wiederkehr wrote:
>>>> Hi Ioan,
>>>>
>>>> Mime4j's BufferedLineReaderInputStream bridges the gap between byte and
>>>> character streams. It lets you read lines of text from a byte stream into a
>>>> ByteArrayBuffer. Then you can use class ContentUtil to decode the
>>>> ByteArrayBuffer into a String. You can also push back (unread) content.
>>>> Maybe that helps with your project.
>>>>
>>>> Cheers,
>>>> Markus
>>
>> Hi,
>>
>> I have a working version of the mbox iterator [1] with an example and
>> a test. I've tested it on mbox files created by gmane mailing list and
>> it works fine. There are more features to add and improvements but it
>> does it's job and I think it's ready for inclusion in mime4j.
>>
>> Please test it and tell me what do you think?
>>
>> Cheers,
>>
>> --
>> Ioan Eugen Stan
>> http://ieugen.blogspot.com/
--
Ioan Eugen Stan
http://ieugen.blogspot.com
Re: NIO Iterator over messages in mbox file
Posted by Norman Maurer <no...@googlemail.com>.
Could you tell us the url of the repos again?
Thx,
Norman
Sent from my iPhone. Excuse any typos....
Am 19.03.2012 um 10:20 schrieb Ioan Eugen Stan <st...@gmail.com>:
> 2012/2/11 Ioan Eugen STAN <st...@gmail.com>:
>> On Sb 11 feb 2012 00:15:57 +0200, Markus Wiederkehr wrote:
>>> Hi Ioan,
>>>
>>> Mime4j's BufferedLineReaderInputStream bridges the gap between byte and
>>> character streams. It lets you read lines of text from a byte stream into a
>>> ByteArrayBuffer. Then you can use class ContentUtil to decode the
>>> ByteArrayBuffer into a String. You can also push back (unread) content.
>>> Maybe that helps with your project.
>>>
>>> Cheers,
>>> Markus
>
> Hi,
>
> I have a working version of the mbox iterator [1] with an example and
> a test. I've tested it on mbox files created by gmane mailing list and
> it works fine. There are more features to add and improvements but it
> does it's job and I think it's ready for inclusion in mime4j.
>
> Please test it and tell me what do you think?
>
> Cheers,
>
> --
> Ioan Eugen Stan
> http://ieugen.blogspot.com/
Re: NIO Iterator over messages in mbox file
Posted by Ioan Eugen Stan <st...@gmail.com>.
2012/2/11 Ioan Eugen STAN <st...@gmail.com>:
> On Sb 11 feb 2012 00:15:57 +0200, Markus Wiederkehr wrote:
>> Hi Ioan,
>>
>> Mime4j's BufferedLineReaderInputStream bridges the gap between byte and
>> character streams. It lets you read lines of text from a byte stream into a
>> ByteArrayBuffer. Then you can use class ContentUtil to decode the
>> ByteArrayBuffer into a String. You can also push back (unread) content.
>> Maybe that helps with your project.
>>
>> Cheers,
>> Markus
Hi,
I have a working version of the mbox iterator [1] with an example and
a test. I've tested it on mbox files created by gmane mailing list and
it works fine. There are more features to add and improvements but it
does it's job and I think it's ready for inclusion in mime4j.
Please test it and tell me what do you think?
Cheers,
--
Ioan Eugen Stan
http://ieugen.blogspot.com/
Re: NIO Iterator over messages in mbox file
Posted by Ioan Eugen STAN <st...@gmail.com>.
On Sb 11 feb 2012 00:15:57 +0200, Markus Wiederkehr wrote:
> Hi Ioan,
>
> Mime4j's BufferedLineReaderInputStream bridges the gap between byte and
> character streams. It lets you read lines of text from a byte stream into a
> ByteArrayBuffer. Then you can use class ContentUtil to decode the
> ByteArrayBuffer into a String. You can also push back (unread) content.
> Maybe that helps with your project.
>
> Cheers,
> Markus
Thanks for clarifying Markus.
The only thing I'm not sure of right now is whether the mbox file has
one charset. It should be, because multi-charset text files are kind of
weird and would be very problematic (and I never heard of before). But I
am uncertain because messages can have an encoding specified with
Content-encoding header.
>From what you said, mime4j uses a charset per message because it doesn't
assume that all messages are part of a single file with one encoding.
I will update the code to provide for means of creating an iterator for
which you can specify:
- file charset
- From_ line regex
- sensible defaults otherwise.
After this, I'll find a place to plug it in mime4j.
Thanks,
--
Ioan Eugen Stan
Re: NIO Iterator over messages in mbox file
Posted by Markus Wiederkehr <ma...@gmail.com>.
Hi Ioan,
Mime4j's BufferedLineReaderInputStream bridges the gap between byte and
character streams. It lets you read lines of text from a byte stream into a
ByteArrayBuffer. Then you can use class ContentUtil to decode the
ByteArrayBuffer into a String. You can also push back (unread) content.
Maybe that helps with your project.
Cheers,
Markus
On Fri, Feb 10, 2012 at 9:46 PM, Ioan Eugen STAN <st...@gmail.com>wrote:
> On Jo 09 feb 2012 15:43:35 +0200, Oleg Kalnichevski wrote:
> > On Thu, 2012-02-09 at 12:23 +0200, Ioan Eugen Stan wrote:
> >> Hello,
> >>
> >> This is the second try to get the message on the mailing list.
> >>
> >> I've started working on a small project called mbox-iterator that I wish
> >> to integrate with mime4j later, when it's more usable. It's purpose is
> >> to provide a iterator like interface over mbox files. Details about the
> >> project are here [1].
> >>
> >> I'm bringing it on because I think this would be a nice addition to
> >> mime4j and I need some help.
> >>
> >> One thing that I can't figure out is: does mime4j do charset decoding? I
> >> haven't looked over the code but I know it expects a stream of bytes.
> >>
> >
> > The MIME parser takes raw data stream as input but can provide fully
> > decoded Reader for individual body parts.
> >
> > Hope this helps
> >
> > Oleg
>
> Hello Oleg,
>
> From what you said, messages are treated like byte streams or encoded
> using a 7/8 bit encoding. I will check this when I will do the actual
> merge with mime4j.
>
> Thanks,
>
> @Emilian Sorry for being late with the implementation. Nice to hear
> from you.
>
> >
> >> mbox-iterator does regexp matching to find From_ lines and that requires
> >> a CharSequence. This means I have use/return CharBuffer's from the
> >> iterator. The CharBuffers need to be re-encoded and streamed as bytes
> >> (maybe with ByteArrayInputStream).
> >>
> >> In the future I plan to use git-svn to experiment new stuff but I would
> >> like to get it fast into mime4j. What do you think?
> >>
> >>
> >> [1]
> >>
> http://ieugen.blogspot.com/2012/02/nio-iterator-over-messages-in-mbox-file.html
> >>
> >> Cheers,
> >
> >
>
>
>
Re: NIO Iterator over messages in mbox file
Posted by Ioan Eugen STAN <st...@gmail.com>.
On Jo 09 feb 2012 15:43:35 +0200, Oleg Kalnichevski wrote:
> On Thu, 2012-02-09 at 12:23 +0200, Ioan Eugen Stan wrote:
>> Hello,
>>
>> This is the second try to get the message on the mailing list.
>>
>> I've started working on a small project called mbox-iterator that I wish
>> to integrate with mime4j later, when it's more usable. It's purpose is
>> to provide a iterator like interface over mbox files. Details about the
>> project are here [1].
>>
>> I'm bringing it on because I think this would be a nice addition to
>> mime4j and I need some help.
>>
>> One thing that I can't figure out is: does mime4j do charset decoding? I
>> haven't looked over the code but I know it expects a stream of bytes.
>>
>
> The MIME parser takes raw data stream as input but can provide fully
> decoded Reader for individual body parts.
>
> Hope this helps
>
> Oleg
Hello Oleg,
From what you said, messages are treated like byte streams or encoded
using a 7/8 bit encoding. I will check this when I will do the actual
merge with mime4j.
Thanks,
@Emilian Sorry for being late with the implementation. Nice to hear
from you.
>
>> mbox-iterator does regexp matching to find From_ lines and that requires
>> a CharSequence. This means I have use/return CharBuffer's from the
>> iterator. The CharBuffers need to be re-encoded and streamed as bytes
>> (maybe with ByteArrayInputStream).
>>
>> In the future I plan to use git-svn to experiment new stuff but I would
>> like to get it fast into mime4j. What do you think?
>>
>>
>> [1]
>> http://ieugen.blogspot.com/2012/02/nio-iterator-over-messages-in-mbox-file.html
>>
>> Cheers,
>
>
Re: NIO Iterator over messages in mbox file
Posted by Oleg Kalnichevski <ol...@apache.org>.
On Thu, 2012-02-09 at 12:23 +0200, Ioan Eugen Stan wrote:
> Hello,
>
> This is the second try to get the message on the mailing list.
>
> I've started working on a small project called mbox-iterator that I wish
> to integrate with mime4j later, when it's more usable. It's purpose is
> to provide a iterator like interface over mbox files. Details about the
> project are here [1].
>
> I'm bringing it on because I think this would be a nice addition to
> mime4j and I need some help.
>
> One thing that I can't figure out is: does mime4j do charset decoding? I
> haven't looked over the code but I know it expects a stream of bytes.
>
The MIME parser takes raw data stream as input but can provide fully
decoded Reader for individual body parts.
Hope this helps
Oleg
> mbox-iterator does regexp matching to find From_ lines and that requires
> a CharSequence. This means I have use/return CharBuffer's from the
> iterator. The CharBuffers need to be re-encoded and streamed as bytes
> (maybe with ByteArrayInputStream).
>
> In the future I plan to use git-svn to experiment new stuff but I would
> like to get it fast into mime4j. What do you think?
>
>
> [1]
> http://ieugen.blogspot.com/2012/02/nio-iterator-over-messages-in-mbox-file.html
>
> Cheers,
Re: NIO Iterator over messages in mbox file
Posted by Emilian Bold <em...@gmail.com>.
Funny you are working on this. I needed such a library a month ago!
--emi
On Feb 9, 2012, at 12:23 PM, Ioan Eugen Stan <st...@gmail.com> wrote:
> Hello,
>
> This is the second try to get the message on the mailing list.
>
> I've started working on a small project called mbox-iterator that I wish
> to integrate with mime4j later, when it's more usable. It's purpose is
> to provide a iterator like interface over mbox files. Details about the
> project are here [1].
>
> I'm bringing it on because I think this would be a nice addition to
> mime4j and I need some help.
>
> One thing that I can't figure out is: does mime4j do charset decoding? I
> haven't looked over the code but I know it expects a stream of bytes.
>
> mbox-iterator does regexp matching to find From_ lines and that requires
> a CharSequence. This means I have use/return CharBuffer's from the
> iterator. The CharBuffers need to be re-encoded and streamed as bytes
> (maybe with ByteArrayInputStream).
>
> In the future I plan to use git-svn to experiment new stuff but I would
> like to get it fast into mime4j. What do you think?
>
>
> [1]
> http://ieugen.blogspot.com/2012/02/nio-iterator-over-messages-in-mbox-file.html
>
> Cheers,
> --
> Ioan Eugen Stan
> http://ieugen.blogspot.com/