You are viewing a plain text version of this content. The canonical link for it is here.
Posted to server-dev@james.apache.org by "Martin.Bergljung" <Ma...@opsera.com> on 2009/05/17 17:18:17 UTC
IMAP FETCH BODYSTRUCTURE problem
Hi,
I started to test the new IMAP Protocol with Lotus Notes
(wants MIME parsing on the server side compared to Outlook and
Thunderbird
that downloads the complete MIME and does it on the client side)
and got into some problems with FETCH BODYSTRUCTURE.
If I for example have a plain text message and execute a command like
"A9 FETCH 3 (RFC822.HEADER BODYSTRUCTURE)" then
the bodystructure response looks something like this:
* 1 FETCH (BODYSTRUCTURE ("TEXT" "PLAIN" ("charset" "utf-8") NIL NIL
"7BIT" 1257 37 NIL NIL ("en-us") NIL) RFC822.HEADER {868}
...
This does not work with Lotus Notes, it things this response is
incorrect.
The old IMAP implementation that I am using will respond as follows:
* 1 FETCH (BODYSTRUCTURE ("TEXT" "PLAIN" ("charset" "utf-8") NIL NIL
"7BIT" 1257 1) RFC822.HEADER {868}
...
And Lotus Notes is fine with this response.
Looks like there is some extra stuff in the response for the new
implementation: ... NIL NIL ("en-us") NIL ...
Note. It returns 1 for the number of lines in message but it works
anyway.
If I am using a multipart email with an attachment the response from the
new IMAP implementation would be something like:
* 2 FETCH (BODYSTRUCTURE
(
( ("TEXT" "PLAIN" ("charset" "us-ascii") NIL NIL "7BIT" 1120 30 NIL NIL
NIL NIL)
("TEXT" "HTML" ("charset" "us-ascii")
NIL NIL "QUOTED-PRINTABLE" 3874 124 NIL NIL NIL NIL)
"ALTERNATIVE" ("boundary"
"----=_NextPart_001_0009_01C9D707.766D0950") NIL NIL NIL)
("APPLICATION" "MSWORD" ("name" "XXXXXX Functional Specification.doc")
NIL NIL "BASE64" 737068 NIL
("attachment" ("filename" "XXXXXX Functional Specification.doc")) NIL
NIL)
"MIXED" ("boundary" "----=_NextPart_000_0008_01C9D707.766D0950") NIL
("en-us") NIL) RFC822.HEADER {636}
And from the old implementation:
* 2 FETCH (BODYSTRUCTURE
(
(("TEXT" "PLAIN" ("charset" "us-ascii") NIL NIL "7BIT" 1120 1)
("TEXT" "HTML" ("charset" "us-ascii") NIL NIL "QUOTED-PRINTABLE" 3874
1)
"ALTERNATIVE" ("boundary"
"----=_NextPart_001_0011_01C9D707.87F7B2B0") NIL NIL)
("APPLICATION" "MSWORD" ("name" "XXXXXX Functional Specification.doc")
NIL NIL "BASE64" 737068 NIL
("attachment" ("filename" "XXXXXX Functional Specification.doc")) NIL)
"MIXED" ("boundary"
"----=_NextPart_000_0010_01C9D707.87F7B2B0") NIL NIL) RFC822.HEADER
{636}
Looks like there is some extra stuff here too in the response for the
new implementation: ... NIL NIL NIL NIL... etc
Don't know which implementation is correct but the old one seems to
work...
Cheers,
Martin Bergljung
Re: IMAP FETCH BODYSTRUCTURE problem
Posted by Robert Burrell Donkin <ro...@gmail.com>.
On Mon, May 18, 2009 at 4:21 PM, Martin.Bergljung
<Ma...@opsera.com> wrote:
> Hi,
>
> It now works, I had a problem with the partial fetch, which is now
> fixed.
> I have played around with Lotus Notes for a while and it seems
> to be working now :-)
great :-)
- robert
---------------------------------------------------------------------
To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
For additional commands, e-mail: server-dev-help@james.apache.org
RE: IMAP FETCH BODYSTRUCTURE problem
Posted by "Martin.Bergljung" <Ma...@opsera.com>.
Hi,
It now works, I had a problem with the partial fetch, which is now
fixed.
I have played around with Lotus Notes for a while and it seems
to be working now :-)
Cheers,
Martin
-----Original Message-----
From: Robert Burrell Donkin [mailto:robertburrelldonkin@gmail.com]
Sent: 18 May 2009 11:57
To: James Developers List
Subject: Re: IMAP FETCH BODYSTRUCTURE problem
On Mon, May 18, 2009 at 10:35 AM, Martin.Bergljung
<Ma...@opsera.com> wrote:
> Hi,
>
> Turned off extensions in the FetchResponseEncoder.java and now it is
> starting to work normally in Lotus Notes and I can open messages.
i've committed a configuration parameter (see
https://issues.apache.org/jira/browse/IMAP-91). porting the functional
tests for this new configuration would be quite a lot of work so i
think it should work but if you could try the version in trunk and
report back, that'd be great.
- robert
---------------------------------------------------------------------
To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
For additional commands, e-mail: server-dev-help@james.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
For additional commands, e-mail: server-dev-help@james.apache.org
Re: IMAP FETCH BODYSTRUCTURE problem
Posted by ma...@opsera.com.
Hi,
I will try it. I think it might actually be the mailbox implementation of getMessages that is not returning data correctly. Have to check that all stuff in FetchGroup is returned correctly.
Martin
Sent from my BlackBerry® wireless device
-----Original Message-----
From: "Robert Burrell Donkin" <ro...@gmail.com>
Date: Mon, 18 May 2009 11:57:19
To: James Developers List<se...@james.apache.org>
Subject: Re: IMAP FETCH BODYSTRUCTURE problem
On Mon, May 18, 2009 at 10:35 AM, Martin.Bergljung
<Ma...@opsera.com> wrote:
> Hi,
>
> Turned off extensions in the FetchResponseEncoder.java and now it is
> starting to work normally in Lotus Notes and I can open messages.
i've committed a configuration parameter (see
https://issues.apache.org/jira/browse/IMAP-91). porting the functional
tests for this new configuration would be quite a lot of work so i
think it should work but if you could try the version in trunk and
report back, that'd be great.
- robert
---------------------------------------------------------------------
To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
For additional commands, e-mail: server-dev-help@james.apache.org
Re: IMAP FETCH BODYSTRUCTURE problem
Posted by Robert Burrell Donkin <ro...@gmail.com>.
On Mon, May 18, 2009 at 10:35 AM, Martin.Bergljung
<Ma...@opsera.com> wrote:
> Hi,
>
> Turned off extensions in the FetchResponseEncoder.java and now it is
> starting to work normally in Lotus Notes and I can open messages.
i've committed a configuration parameter (see
https://issues.apache.org/jira/browse/IMAP-91). porting the functional
tests for this new configuration would be quite a lot of work so i
think it should work but if you could try the version in trunk and
report back, that'd be great.
- robert
---------------------------------------------------------------------
To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
For additional commands, e-mail: server-dev-help@james.apache.org
RE: IMAP FETCH BODYSTRUCTURE problem
Posted by "Martin.Bergljung" <Ma...@opsera.com>.
Hi,
Turned off extensions in the FetchResponseEncoder.java and now it is
starting to work normally in Lotus Notes and I can open messages.
However, I get a NPE for some messages (multipart I think) as follows:
ERROR 10:06:07,367 | james.imapserver | Unexpected runtime exception:
null
java.lang.NullPointerException
at
org.apache.james.imap.processor.fetch.ContentBodyElement.size(ContentBod
yElement.java:53)
at
org.apache.james.imap.main.ChannelImapResponseWriter.literal(ChannelImap
ResponseWriter.java:207)
at
org.apache.james.imap.encode.base.ImapResponseComposerImpl.literal(ImapR
esponseComposerImpl.java:433)
at
org.apache.james.imap.encode.FetchResponseEncoder.encodeBodyElements(Fet
chResponseEncoder.java:259)
at
org.apache.james.imap.encode.FetchResponseEncoder.doEncode(FetchResponse
Encoder.java:64)
at
org.apache.james.imap.encode.base.AbstractChainedImapEncoder.encode(Abst
ractChainedImapEncoder.java:42)
at
org.apache.james.imap.encode.base.AbstractChainedImapEncoder.chainEncode
(AbstractChainedImapEncoder.java:50)
at
org.apache.james.imap.encode.base.AbstractChainedImapEncoder.encode(Abst
ractChainedImapEncoder.java:44)
at
org.apache.james.imap.encode.base.AbstractChainedImapEncoder.chainEncode
(AbstractChainedImapEncoder.java:50)
at
org.apache.james.imap.encode.base.AbstractChainedImapEncoder.encode(Abst
ractChainedImapEncoder.java:44)
at
org.apache.james.imap.encode.base.AbstractChainedImapEncoder.chainEncode
(AbstractChainedImapEncoder.java:50)
at
org.apache.james.imap.encode.base.AbstractChainedImapEncoder.encode(Abst
ractChainedImapEncoder.java:44)
at
org.apache.james.imap.encode.base.AbstractChainedImapEncoder.chainEncode
(AbstractChainedImapEncoder.java:50)
at
org.apache.james.imap.encode.base.AbstractChainedImapEncoder.encode(Abst
ractChainedImapEncoder.java:44)
at
org.apache.james.imap.encode.base.AbstractChainedImapEncoder.chainEncode
(AbstractChainedImapEncoder.java:50)
at
org.apache.james.imap.encode.base.AbstractChainedImapEncoder.encode(Abst
ractChainedImapEncoder.java:44)
at
org.apache.james.imap.encode.base.AbstractChainedImapEncoder.chainEncode
(AbstractChainedImapEncoder.java:50)
at
org.apache.james.imap.encode.base.AbstractChainedImapEncoder.encode(Abst
ractChainedImapEncoder.java:44)
at
org.apache.james.imap.encode.base.AbstractChainedImapEncoder.chainEncode
(AbstractChainedImapEncoder.java:50)
at
org.apache.james.imap.encode.base.AbstractChainedImapEncoder.encode(Abst
ractChainedImapEncoder.java:44)
at
org.apache.james.imap.encode.base.AbstractChainedImapEncoder.chainEncode
(AbstractChainedImapEncoder.java:50)
at
org.apache.james.imap.encode.base.AbstractChainedImapEncoder.encode(Abst
ractChainedImapEncoder.java:44)
at
org.apache.james.imap.main.ImapRequestHandler$ResponseEncoder.respond(Im
apRequestHandler.java:228)
at
org.apache.james.imap.processor.fetch.FetchProcessor.doProcess(FetchProc
essor.java:96)
at
org.apache.james.imap.processor.AbstractMailboxProcessor.doProcess(Abstr
actMailboxProcessor.java:123)
at
org.apache.james.imap.processor.AbstractMailboxProcessor.process(Abstrac
tMailboxProcessor.java:82)
at
org.apache.james.imap.processor.AbstractMailboxProcessor.doProcess(Abstr
actMailboxProcessor.java:75)
at
org.apache.james.imap.processor.base.AbstractChainedProcessor.process(Ab
stractChainedProcessor.java:44)
at
org.apache.james.imap.processor.base.AbstractChainedProcessor.process(Ab
stractChainedProcessor.java:46)
at
org.apache.james.imap.main.ImapRequestHandler.doProcessRequest(ImapReque
stHandler.java:184)
at
org.apache.james.imap.main.ImapRequestHandler.handleRequest(ImapRequestH
andler.java:108)
at
com.opsera.omm.james.imapserver.ImapHandler.handleRequest(ImapHandler.ja
va:119)
at
com.opsera.omm.james.imapserver.ImapHandler.handleProtocol(ImapHandler.j
ava:103)
at
org.apache.james.socket.DelegatingJamesHandler.handleProtocol(Delegating
JamesHandler.java:60)
at
org.apache.james.socket.AbstractJamesHandler.handleConnection(AbstractJa
mesHandler.java:262)
at
org.apache.james.util.connection.ServerConnection$ClientConnectionRunner
.run(ServerConnection.java:467)
at
org.apache.excalibur.thread.impl.ExecutableRunnable.execute(ExecutableRu
nnable.java:57)
at
org.apache.excalibur.thread.impl.WorkerThread.run(WorkerThread.java:118)
DEBUG 10:06:07,450 | james.imapserver | Calling disposeWatchdog()
default Worker #8
DEBUG 10:06:07,452 | james.imapserver | Watchdog default Worker #8 is
exiting run().
DEBUG 10:06:07,453 | james.imap-connections | Ending connection on
127.0.0.1:143 from 127.0.0.1:50154
The new code seems to be better structured and more efficient. It is
more stateless and commands are not created for every request for
example. I think it is quite important to try and keep it stateless and
not create loads of objects all the time. For a couple of reasons,
memory consumption and speed.
I think the code is more scalable now where each request is basically
independent, before it seemed to be more sticky sessions like, or I have
misunderstood it:-)
The APIs have also improved a lot in the new implementation.
Thanks
/Martin
-----Original Message-----
From: Robert Burrell Donkin [mailto:robertburrelldonkin@gmail.com]
Sent: 18 May 2009 09:52
To: James Developers List
Subject: Re: IMAP FETCH BODYSTRUCTURE problem
On Mon, May 18, 2009 at 9:30 AM, Martin.Bergljung
<Ma...@opsera.com> wrote:
> Hi,
>
> If we could add a configuration parameter to be able to turn on and
off
> the return of extension data that would be good.
> I have lots of trial users with Lotus Notes clients so I need to find
a
> solution.
i'll take a look at that now
(https://issues.apache.org/jira/browse/IMAP-91)
if you do figure out some selective way to do this, that'd be great.
maybe we could talk clueful clients into user a new USER_AGENT
extension command...
> BTW. I did loads of changes and updates to FetchCommand.java and
> SimpleMessageAttributes.java in the old implementation
> to get it to work.
how are you finding the new code?
- robert
---------------------------------------------------------------------
To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
For additional commands, e-mail: server-dev-help@james.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
For additional commands, e-mail: server-dev-help@james.apache.org
Re: IMAP FETCH BODYSTRUCTURE problem
Posted by Robert Burrell Donkin <ro...@gmail.com>.
On Mon, May 18, 2009 at 9:30 AM, Martin.Bergljung
<Ma...@opsera.com> wrote:
> Hi,
>
> If we could add a configuration parameter to be able to turn on and off
> the return of extension data that would be good.
> I have lots of trial users with Lotus Notes clients so I need to find a
> solution.
i'll take a look at that now (https://issues.apache.org/jira/browse/IMAP-91)
if you do figure out some selective way to do this, that'd be great.
maybe we could talk clueful clients into user a new USER_AGENT
extension command...
> BTW. I did loads of changes and updates to FetchCommand.java and
> SimpleMessageAttributes.java in the old implementation
> to get it to work.
how are you finding the new code?
- robert
---------------------------------------------------------------------
To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
For additional commands, e-mail: server-dev-help@james.apache.org
RE: IMAP FETCH BODYSTRUCTURE problem
Posted by "Martin.Bergljung" <Ma...@opsera.com>.
Hi,
If we could add a configuration parameter to be able to turn on and off
the return of extension data that would be good.
I have lots of trial users with Lotus Notes clients so I need to find a
solution.
BTW. I did loads of changes and updates to FetchCommand.java and
SimpleMessageAttributes.java in the old implementation
to get it to work.
Cheers,
Martin
-----Original Message-----
From: Robert Burrell Donkin [mailto:robertburrelldonkin@gmail.com]
Sent: 18 May 2009 08:24
To: James Developers List
Subject: Re: IMAP FETCH BODYSTRUCTURE problem
On Sun, May 17, 2009 at 4:18 PM, Martin.Bergljung
<Ma...@opsera.com> wrote:
> Hi,
>
>
>
> I started to test the new IMAP Protocol with Lotus Notes
>
> (wants MIME parsing on the server side compared to Outlook and
> Thunderbird
>
> that downloads the complete MIME and does it on the client side)
>
> and got into some problems with FETCH BODYSTRUCTURE.
>
>
>
> If I for example have a plain text message and execute a command like
> "A9 FETCH 3 (RFC822.HEADER BODYSTRUCTURE)" then
>
> the bodystructure response looks something like this:
>
>
>
> * 1 FETCH (BODYSTRUCTURE ("TEXT" "PLAIN" ("charset" "utf-8") NIL NIL
> "7BIT" 1257 37 NIL NIL ("en-us") NIL) RFC822.HEADER {868}
this looks right to me
> ...
>
>
>
> This does not work with Lotus Notes, it things this response is
> incorrect.
>
> The old IMAP implementation that I am using will respond as follows:
>
>
>
> * 1 FETCH (BODYSTRUCTURE ("TEXT" "PLAIN" ("charset" "utf-8") NIL NIL
> "7BIT" 1257 1) RFC822.HEADER {868}
this looks wrong to me
> ...
>
>
>
> And Lotus Notes is fine with this response.
the lotus client is broken:
<blockquote cite='http://www.faqs.org/rfcs/rfc3501.html'>
Client implementations that do a BODYSTRUCTURE fetch MUST be prepared
to accept such extension data.
</blockquote>
> Looks like there is some extra stuff in the response for the new
> implementation: ... NIL NIL ("en-us") NIL ...
this extension data is specified in RFC3501. the current IMAP
implementation does quite a full job, more than many servers. this
stuff is tricky to parse and isn't very useful so most servers don't
bother sending and clients don't bother parsing.
i don't know of any way for an IMAP server to work out what the client
application is. the server has to send a hello including this
information. it seems to be assumed by the specification that servers
will often be buggy whilst clients will always be correct. the
converse is usually the base :-/
AIUI the usual approach taken is to start adding configuration
parameters which tell a server to deviate from the specification. any
better ideas would be appreciated :-)
> Note. It returns 1 for the number of lines in message but it works
> anyway.
the old implementation is broken and crashs some client versions with
this behaviour
- robert
---------------------------------------------------------------------
To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
For additional commands, e-mail: server-dev-help@james.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
For additional commands, e-mail: server-dev-help@james.apache.org
Re: IMAP FETCH BODYSTRUCTURE problem
Posted by Robert Burrell Donkin <ro...@gmail.com>.
On Sun, May 17, 2009 at 4:18 PM, Martin.Bergljung
<Ma...@opsera.com> wrote:
> Hi,
>
>
>
> I started to test the new IMAP Protocol with Lotus Notes
>
> (wants MIME parsing on the server side compared to Outlook and
> Thunderbird
>
> that downloads the complete MIME and does it on the client side)
>
> and got into some problems with FETCH BODYSTRUCTURE.
>
>
>
> If I for example have a plain text message and execute a command like
> "A9 FETCH 3 (RFC822.HEADER BODYSTRUCTURE)" then
>
> the bodystructure response looks something like this:
>
>
>
> * 1 FETCH (BODYSTRUCTURE ("TEXT" "PLAIN" ("charset" "utf-8") NIL NIL
> "7BIT" 1257 37 NIL NIL ("en-us") NIL) RFC822.HEADER {868}
this looks right to me
> ...
>
>
>
> This does not work with Lotus Notes, it things this response is
> incorrect.
>
> The old IMAP implementation that I am using will respond as follows:
>
>
>
> * 1 FETCH (BODYSTRUCTURE ("TEXT" "PLAIN" ("charset" "utf-8") NIL NIL
> "7BIT" 1257 1) RFC822.HEADER {868}
this looks wrong to me
> ...
>
>
>
> And Lotus Notes is fine with this response.
the lotus client is broken:
<blockquote cite='http://www.faqs.org/rfcs/rfc3501.html'>
Client implementations that do a BODYSTRUCTURE fetch MUST be prepared
to accept such extension data.
</blockquote>
> Looks like there is some extra stuff in the response for the new
> implementation: ... NIL NIL ("en-us") NIL ...
this extension data is specified in RFC3501. the current IMAP
implementation does quite a full job, more than many servers. this
stuff is tricky to parse and isn't very useful so most servers don't
bother sending and clients don't bother parsing.
i don't know of any way for an IMAP server to work out what the client
application is. the server has to send a hello including this
information. it seems to be assumed by the specification that servers
will often be buggy whilst clients will always be correct. the
converse is usually the base :-/
AIUI the usual approach taken is to start adding configuration
parameters which tell a server to deviate from the specification. any
better ideas would be appreciated :-)
> Note. It returns 1 for the number of lines in message but it works
> anyway.
the old implementation is broken and crashs some client versions with
this behaviour
- robert
---------------------------------------------------------------------
To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
For additional commands, e-mail: server-dev-help@james.apache.org