You are viewing a plain text version of this content. The canonical link for it is here.
Posted to mime4j-dev@james.apache.org by Mariano Kamp <ma...@gmail.com> on 2014/01/04 23:33:29 UTC
org.apache.james.mime4j.io.MaxLineLimitException: Maximum line length
limit exceeded / NPE
Hey guys.
I want to learn Hadoop and use my gmail mbox file as a basis for that.
That brought me to mime4j and for the most part this is working out great.
Thank you, btw.
But it took me a while to get my hands on a version of it. So I am
wondering if this project is still active? The download link [1] is broken.
Is this the official download link?
After some time I found apache-mime4j-*-0.8.0-SNAPSHOT.jars. But it is a
higher version number as the [official|outdated] 0.7.2 and it also seems
different then what I can see in the trunk. Or at least it seemed to me as
the DefaultMessageBuilder is not part of the trunk, but in 0.8.0.
Or maybe I am getting it all wrong, no idea. Tried to look at the sources,
but it seems then I also need to learn Maven – as it needs to generate
source – and I would rather spent my time on Hadoop.
Anyway, I ran into two issues, see subject.
It seems that Mime4j doesn't like long lines in the input and so the
parsing fails.
Here is such a long line that looks legit to me. At least it is not a
corner case as it is created by Google Groups.
References: <e3...@googlegroups.com> <
afd247f5-47df-4367-acfa-b31e94b18dea@googlegroups.com> <
44152f22-3b3a-4137-aadb-d61151ceec13@googlegroups.com>
<BL...@phx.gbl> <
ae44f4eb-b3f4-4159-bed6-5dab479f0d54@googlegroups.com> <
5bbe5e89-3a1b-4e07-b339-9c719ed46aa5@googlegroups.com> <
181e5f61-9cee-4409-b6ca-cf62dc974e59@googlegroups.com> <
CALQcipbUU+Cp2memLx_7qS-hqmitQ9REM2kJhGDbp4YesU+xDA@mail.gmail.com> <
62124bf3-6830-4889-b982-505a047ef61d@googlegroups.com> <
6ae3238c-3b54-488d-bb80-51e35afdb960@hd10g2000pbc.googlegroups.com> <
8928045d-68fa-4bfc-98f1-7fddc24f7a3a@kt16g2000pbb.googlegroups.com> <
8ad18487-0d3d-4c6a-bba0-4286a2243dfd@googlegroups.com> <
fbd6f604-a703-401c-9546-a33f868c4b96@googlegroups.com>
<CA...@mail.gmail.com> <
11f38d9f-6805-47f6-881a-2e6a623dffa4@googlegroups.com> <
2baf7a3d-4ec0-4bfd-972b-d5d7db84289c@googlegroups.com> <
e7e3f172-85cf-4f75-a178-e9093ad28be1@googlegroups.com> <413a
1a0a-7fe0-41fd-a33e-3d880647ada0@googlegroups.com> <
2b700a14-0066-415c-b376-aca812aae3b3@googlegroups.com> <
82e4aa34-2c08-41bb-9970-34cf0bedde8a@googlegroups.com> <
405cc1fb-5eb2-4295-bd85-d7043866b182@googlegroups.com> <
4025bf45-ac66-4542-b116-d7eaaed33732@googlegroups.com>
Here is an excerpt from the stacktrace:
Caused by: org.apache.james.mime4j.io.MaxLineLimitException: Maximum line
length limit exceeded
at
org.apache.james.mime4j.io.BufferedLineReaderInputStream.readLine(BufferedLineReaderInputStream.java:218)
at
org.apache.james.mime4j.io.LineReaderInputStreamAdaptor.readLine(LineReaderInputStreamAdaptor.java:78)
at
org.apache.james.mime4j.stream.MimeEntity.readRawField(MimeEntity.java:215)
I attached the full offending message. At the end of the message you'll
find the stacktrace.
But to put things in perspective. I processed my mails from the last ten
years (100k+) and it only had issues with a few hundred. So it's not a
biggie, but wanted to give you this feedback. And I can provide you with
more examples if you need those.
Furthermore I ran into a NPE as well.
java.lang.NullPointerException
at
org.apache.james.mime4j.io.MimeBoundaryInputStream.<init>(MimeBoundaryInputStream.java:67)
at
org.apache.james.mime4j.stream.MimeEntity.createMimePartStream(MimeEntity.java:366)
at org.apache.james.mime4j.stream.MimeEntity.advance(MimeEntity.java:320)
at
org.apache.james.mime4j.stream.MimeTokenStream.next(MimeTokenStream.java:368)
at
org.apache.james.mime4j.parser.MimeStreamParser.parse(MimeStreamParser.java:176)
at
org.apache.james.mime4j.message.DefaultMessageBuilder.parseMessage(DefaultMessageBuilder.java:316)
at com.mboxanalytics.util.MboxUtil.parseMessage(MboxUtil.java:95)
...
UnsupportedEncodingException and IllegalCharsetNameException available as
well, but probably correct. Full list:
RecordLevelErrors
java.io.UnsupportedEncodingException/ISO-8859-8-I=1
java.io.UnsupportedEncodingException/ansi_x3.110-1983=1
java.io.UnsupportedEncodingException/csn_369103=1
java.io.UnsupportedEncodingException/en.UTF-8=3
java.io.UnsupportedEncodingException/iso-88592=4
java.io.UnsupportedEncodingException/unicode-1-1-utf-7=3
java.io.UnsupportedEncodingException/unknown-8bit=1
java.io.UnsupportedEncodingException/us-as=1
java.io.UnsupportedEncodingException/windows-1252http-equivCont=1
java.io.UnsupportedEncodingException/x-utf8utf8=1
java.lang.NullPointerException/null=1
java.nio.charset.IllegalCharsetNameException/\"us-ascii\"=2
java.nio.charset.IllegalCharsetNameException/iso-8859-1 Content=1
java.nio.charset.IllegalCharsetNameException/windows-1252 chars=1
org.apache.james.mime4j.MimeIOException/Maximum header length l=13
org.apache.james.mime4j.MimeIOException/org.apache.james.mime4j=228
Success
No of message successfully processed=129462
No of words processed=64267035
Hope this is helpful and thanks again for providing this library.
Best,
Mariano
[1] http://james.apache.org/download.cgi#Apache_Mime4J
Re: org.apache.james.mime4j.io.MaxLineLimitException: Maximum line
length limit exceeded / NPE
Posted by Mariano Kamp <ma...@gmail.com>.
Oleg.
> You should be using the stable branch for the time being
> https://svn.apache.org/repos/asf/james/mime4j/branches/apache-mime4j-0.7
A downloadable dist would be nice.
Re: NPE
Created the JIRA bug report:
https://issues.apache.org/jira/browse/MIME4J-235
Re: Max Line Length
Using MimeConfig does the trick.
messageBuilder.setMimeEntityConfig(new MimeConfig.Builder()
.setMaxLineLen(-1).setMaxHeaderLen(-1).setMaxHeaderCount(-1).build());
Thanks.
On Mon, Jan 6, 2014 at 3:56 PM, Oleg Kalnichevski <ol...@apache.org> wrote:
> On Sat, 2014-01-04 at 23:33 +0100, Mariano Kamp wrote:
> > Hey guys.
> >
> >
> > I want to learn Hadoop and use my gmail mbox file as a basis for that.
> > That brought me to mime4j and for the most part this is working out
> > great. Thank you, btw.
> >
> >
> > But it took me a while to get my hands on a version of it. So I am
> > wondering if this project is still active? The download link [1] is
> > broken. Is this the official download link?
> >
>
> Hi Mariano
>
> Unfortunately the project is has not been very active. On the positive
> side that presents potential contributors with more opportunities.
>
> >
> > After some time I found apache-mime4j-*-0.8.0-SNAPSHOT.jars. But it is
> > a higher version number as the [official|outdated] 0.7.2 and it also
> > seems different then what I can see in the trunk. Or at least it
> > seemed to me as the DefaultMessageBuilder is not part of the trunk,
> > but in 0.8.0.
> >
>
> You should be using the stable branch for the time being
>
> https://svn.apache.org/repos/asf/james/mime4j/branches/apache-mime4j-0.7
>
>
> > It seems that Mime4j doesn't like long lines in the input and so the
> > parsing fails.
> >
>
> No, it does not. For good reasons. You hardly want your mail server to
> enter an infinite loop while trying to parse a malformed message.
>
> You can disable max line length / max header count limits (or increase
> them) by using a custom MimeConfig object.
>
> ...
>
> >
> > But to put things in perspective. I processed my mails from the last
> > ten years (100k+) and it only had issues with a few hundred. So it's
> > not a biggie, but wanted to give you this feedback. And I can provide
> > you with more examples if you need those.
> >
> >
> > Furthermore I ran into a NPE as well.
> >
> >
> > java.lang.NullPointerException
> > at
> >
> org.apache.james.mime4j.io.MimeBoundaryInputStream.<init>(MimeBoundaryInputStream.java:67)
> > at
> >
> org.apache.james.mime4j.stream.MimeEntity.createMimePartStream(MimeEntity.java:366)
> > at
> > org.apache.james.mime4j.stream.MimeEntity.advance(MimeEntity.java:320)
> > at
> >
> org.apache.james.mime4j.stream.MimeTokenStream.next(MimeTokenStream.java:368)
> > at
> >
> org.apache.james.mime4j.parser.MimeStreamParser.parse(MimeStreamParser.java:176)
> > at
> >
> org.apache.james.mime4j.message.DefaultMessageBuilder.parseMessage(DefaultMessageBuilder.java:316)
> > at com.mboxanalytics.util.MboxUtil.parseMessage(MboxUtil.java:95)
> > ...
> >
>
> This one is a bug. Could you please raise a JIRA for this defect and
> attach the offending message to it?
>
> >
> >
> >
> > UnsupportedEncodingException and IllegalCharsetNameException available
> > as well, but probably correct. Full list:
> >
> >
>
> As far as UnsupportedEncodingException, IllegalCharsetNameException or
> similar are concerned one can easily avoid them, if so desired, by
> providing a custom BodyFactory implementation that applies some
> heuristics to map unsupported charsets to supported ones.
>
> Hope this helps
>
> Oleg
>
>
Re: org.apache.james.mime4j.io.MaxLineLimitException: Maximum line
length limit exceeded / NPE
Posted by Oleg Kalnichevski <ol...@apache.org>.
On Sat, 2014-01-04 at 23:33 +0100, Mariano Kamp wrote:
> Hey guys.
>
>
> I want to learn Hadoop and use my gmail mbox file as a basis for that.
> That brought me to mime4j and for the most part this is working out
> great. Thank you, btw.
>
>
> But it took me a while to get my hands on a version of it. So I am
> wondering if this project is still active? The download link [1] is
> broken. Is this the official download link?
>
Hi Mariano
Unfortunately the project is has not been very active. On the positive
side that presents potential contributors with more opportunities.
>
> After some time I found apache-mime4j-*-0.8.0-SNAPSHOT.jars. But it is
> a higher version number as the [official|outdated] 0.7.2 and it also
> seems different then what I can see in the trunk. Or at least it
> seemed to me as the DefaultMessageBuilder is not part of the trunk,
> but in 0.8.0.
>
You should be using the stable branch for the time being
https://svn.apache.org/repos/asf/james/mime4j/branches/apache-mime4j-0.7
> It seems that Mime4j doesn't like long lines in the input and so the
> parsing fails.
>
No, it does not. For good reasons. You hardly want your mail server to
enter an infinite loop while trying to parse a malformed message.
You can disable max line length / max header count limits (or increase
them) by using a custom MimeConfig object.
...
>
> But to put things in perspective. I processed my mails from the last
> ten years (100k+) and it only had issues with a few hundred. So it's
> not a biggie, but wanted to give you this feedback. And I can provide
> you with more examples if you need those.
>
>
> Furthermore I ran into a NPE as well.
>
>
> java.lang.NullPointerException
> at
> org.apache.james.mime4j.io.MimeBoundaryInputStream.<init>(MimeBoundaryInputStream.java:67)
> at
> org.apache.james.mime4j.stream.MimeEntity.createMimePartStream(MimeEntity.java:366)
> at
> org.apache.james.mime4j.stream.MimeEntity.advance(MimeEntity.java:320)
> at
> org.apache.james.mime4j.stream.MimeTokenStream.next(MimeTokenStream.java:368)
> at
> org.apache.james.mime4j.parser.MimeStreamParser.parse(MimeStreamParser.java:176)
> at
> org.apache.james.mime4j.message.DefaultMessageBuilder.parseMessage(DefaultMessageBuilder.java:316)
> at com.mboxanalytics.util.MboxUtil.parseMessage(MboxUtil.java:95)
> ...
>
This one is a bug. Could you please raise a JIRA for this defect and
attach the offending message to it?
>
>
>
> UnsupportedEncodingException and IllegalCharsetNameException available
> as well, but probably correct. Full list:
>
>
As far as UnsupportedEncodingException, IllegalCharsetNameException or
similar are concerned one can easily avoid them, if so desired, by
providing a custom BodyFactory implementation that applies some
heuristics to map unsupported charsets to supported ones.
Hope this helps
Oleg