You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@qpid.apache.org by Matthew Karlsen <MK...@idbs.com> on 2016/01/26 16:09:14 UTC
Sending/retrieving larger messages using Proton-J?
Hello all,
I am presently trying to send a 1.4 meg PDF in a message and am encountering substantial difficulties.
I have tried various combinations (though my central aim is to use Proton-J). Combinations as follows:
Qpid Proton-J 0.11.1 to HornetQ 2.4.0 Final to Qpid Proton-J 0.11.1
Qpid Proton-J 0.11.1 to Apache Artemis 1.2.0 to Qpid Proton-J 0.11.1
Qpid JMS 0.7 to HornetQ 2.4.0 Final to Qpid JMS 0.7
Qpid JMS 0.7 to Apache Artemis 1.2.0 to Qpid JMS 0.7
Have also tried MQLight and SwiftMQ clients (though they are not really a viable solution here due to their licenses).
The most bytes I have managed to send and receive is ~10,000.
I frequently get the two error messages below when sending (in the Artemis console window) -- relating to the Qpid Proton library.
(Note: I have tried increasing connection-ttl on the client side.)
My question is this: are there any important caveats when sending large messages via AMQP that I have somehow missed (or related caveats)?
I have read the Qpid documentation, HornetQ documentation and Apache Artemis documentation and there are some things that I think *may* be relevant, but I am not sure.
The question is clearly somewhat cross-technology but I felt that this was the best place to post the issue.
I am happy to supply Wireshark output if required, though I suspect the output would be quite voluminous.
Thank you for any help you are able to offer,
regards,
Matthew
-------------------------------------------------
java.nio.BufferUnderflowException
at java.nio.HeapByteBuffer.get(HeapByteBuffer.java:151)
at org.apache.qpid.proton.codec.DecoderImpl.readRaw(DecoderImpl.java:939)
at org.apache.qpid.proton.codec.BinaryType$LongBinaryEncoding.readValue(BinaryType.java:110)
at org.apache.qpid.proton.codec.BinaryType$LongBinaryEncoding.readValue(BinaryType.java:67)
at org.apache.qpid.proton.codec.DynamicTypeConstructor.readValue(DynamicTypeConstructor.java:39)
at org.apache.qpid.proton.codec.DecoderImpl.readObject(DecoderImpl.java:885)
at org.apache.qpid.proton.message.impl.MessageImpl.decode(MessageImpl.java:647)
at org.apache.qpid.proton.message.impl.MessageImpl.decode(MessageImpl.java:577)
at org.apache.qpid.proton.jms.EncodedMessage.decode(EncodedMessage.java:46)
at org.apache.qpid.proton.jms.JMSMappingInboundTransformer.transform(JMSMappingInboundTransformer.java:40)
at org.apache.activemq.artemis.core.protocol.proton.converter.ProtonMessageConverter.inboundJMSType(ProtonMessageConverter.java:57)
at org.apache.activemq.artemis.core.protocol.proton.converter.ProtonMessageConverter.inbound(ProtonMessageConverter.java:43)
at org.apache.activemq.artemis.core.protocol.proton.plug.ProtonSessionIntegrationCallback.serverSend(ProtonSessionIntegrationCallback.java:249)
at org.proton.plug.context.server.ProtonServerReceiverContext.onMessage(ProtonServerReceiverContext.java:116)
at org.proton.plug.context.AbstractConnectionContext$LocalListener.onDelivery(AbstractConnectionContext.java:250)
at org.proton.plug.handler.Events.dispatch(Events.java:100)
at org.proton.plug.handler.impl.ProtonHandlerImpl.dispatch(ProtonHandlerImpl.java:339)
at org.proton.plug.handler.impl.ProtonHandlerImpl.flush(ProtonHandlerImpl.java:259)
at org.proton.plug.handler.impl.ProtonHandlerImpl.inputBuffer(ProtonHandlerImpl.java:159)
at org.proton.plug.context.AbstractConnectionContext.inputBuffer(AbstractConnectionContext.java:82)
at org.apache.activemq.artemis.core.protocol.proton.ActiveMQProtonRemotingConnection.bufferReceived(ActiveMQProtonRemotingConnection.java:128)
at org.apache.activemq.artemis.core.remoting.server.impl.RemotingServiceImpl$DelegatingBufferHandler.bufferReceived(RemotingServiceImpl.java:620)
at org.apache.activemq.artemis.core.remoting.impl.netty.ActiveMQChannelHandler.channelRead(ActiveMQChannelHandler.java:68)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:308)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:294)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:846)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:131)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:511)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:468)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:382)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:354)
at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:112)
at java.lang.Thread.run(Thread.java:745)
14:45:05,373 WARN [org.apache.activemq.artemis.core.client] AMQ212037: Connection failure has been detected: AMQ119014: Did not receive data from null. It is likely the client has exited or crashed without closing its connection, or the network between the server and client has
failed. You also might have configured connection-ttl and client-failure-check-period incorrectly. Please check user manual for more information. The connection will now be closed. [code=CONNECTION_TIMEDOUT]
Re: Sending/retrieving larger messages using Proton-J?
Posted by Robbie Gemmell <ro...@gmail.com>.
On 26 January 2016 at 15:09, Matthew Karlsen <MK...@idbs.com> wrote:
> Hello all,
>
> I am presently trying to send a 1.4 meg PDF in a message and am encountering substantial difficulties.
>
> I have tried various combinations (though my central aim is to use Proton-J). Combinations as follows:
> Qpid Proton-J 0.11.1 to HornetQ 2.4.0 Final to Qpid Proton-J 0.11.1
> Qpid Proton-J 0.11.1 to Apache Artemis 1.2.0 to Qpid Proton-J 0.11.1
> Qpid JMS 0.7 to HornetQ 2.4.0 Final to Qpid JMS 0.7
> Qpid JMS 0.7 to Apache Artemis 1.2.0 to Qpid JMS 0.7
>
> Have also tried MQLight and SwiftMQ clients (though they are not really a viable solution here due to their licenses).
>
> The most bytes I have managed to send and receive is ~10,000.
>
> I frequently get the two error messages below when sending (in the Artemis console window) -- relating to the Qpid Proton library.
>
> (Note: I have tried increasing connection-ttl on the client side.)
>
> My question is this: are there any important caveats when sending large messages via AMQP that I have somehow missed (or related caveats)?
>
> I have read the Qpid documentation, HornetQ documentation and Apache Artemis documentation and there are some things that I think *may* be relevant, but I am not sure.
>
> The question is clearly somewhat cross-technology but I felt that this was the best place to post the issue.
>
> I am happy to supply Wireshark output if required, though I suspect the output would be quite voluminous.
>
> Thank you for any help you are able to offer,
> regards,
> Matthew
>
> -------------------------------------------------
>
> java.nio.BufferUnderflowException
> at java.nio.HeapByteBuffer.get(HeapByteBuffer.java:151)
> at org.apache.qpid.proton.codec.DecoderImpl.readRaw(DecoderImpl.java:939)
> at org.apache.qpid.proton.codec.BinaryType$LongBinaryEncoding.readValue(BinaryType.java:110)
> at org.apache.qpid.proton.codec.BinaryType$LongBinaryEncoding.readValue(BinaryType.java:67)
> at org.apache.qpid.proton.codec.DynamicTypeConstructor.readValue(DynamicTypeConstructor.java:39)
> at org.apache.qpid.proton.codec.DecoderImpl.readObject(DecoderImpl.java:885)
> at org.apache.qpid.proton.message.impl.MessageImpl.decode(MessageImpl.java:647)
> at org.apache.qpid.proton.message.impl.MessageImpl.decode(MessageImpl.java:577)
> at org.apache.qpid.proton.jms.EncodedMessage.decode(EncodedMessage.java:46)
> at org.apache.qpid.proton.jms.JMSMappingInboundTransformer.transform(JMSMappingInboundTransformer.java:40)
> at org.apache.activemq.artemis.core.protocol.proton.converter.ProtonMessageConverter.inboundJMSType(ProtonMessageConverter.java:57)
> at org.apache.activemq.artemis.core.protocol.proton.converter.ProtonMessageConverter.inbound(ProtonMessageConverter.java:43)
> at org.apache.activemq.artemis.core.protocol.proton.plug.ProtonSessionIntegrationCallback.serverSend(ProtonSessionIntegrationCallback.java:249)
> at org.proton.plug.context.server.ProtonServerReceiverContext.onMessage(ProtonServerReceiverContext.java:116)
> at org.proton.plug.context.AbstractConnectionContext$LocalListener.onDelivery(AbstractConnectionContext.java:250)
> at org.proton.plug.handler.Events.dispatch(Events.java:100)
> at org.proton.plug.handler.impl.ProtonHandlerImpl.dispatch(ProtonHandlerImpl.java:339)
> at org.proton.plug.handler.impl.ProtonHandlerImpl.flush(ProtonHandlerImpl.java:259)
> at org.proton.plug.handler.impl.ProtonHandlerImpl.inputBuffer(ProtonHandlerImpl.java:159)
> at org.proton.plug.context.AbstractConnectionContext.inputBuffer(AbstractConnectionContext.java:82)
> at org.apache.activemq.artemis.core.protocol.proton.ActiveMQProtonRemotingConnection.bufferReceived(ActiveMQProtonRemotingConnection.java:128)
> at org.apache.activemq.artemis.core.remoting.server.impl.RemotingServiceImpl$DelegatingBufferHandler.bufferReceived(RemotingServiceImpl.java:620)
> at org.apache.activemq.artemis.core.remoting.impl.netty.ActiveMQChannelHandler.channelRead(ActiveMQChannelHandler.java:68)
> at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:308)
> at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:294)
> at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:846)
> at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:131)
> at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:511)
> at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:468)
> at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:382)
> at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:354)
> at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:112)
> at java.lang.Thread.run(Thread.java:745)
>
> 14:45:05,373 WARN [org.apache.activemq.artemis.core.client] AMQ212037: Connection failure has been detected: AMQ119014: Did not receive data from null. It is likely the client has exited or crashed without closing its connection, or the network between the server and client has
> failed. You also might have configured connection-ttl and client-failure-check-period incorrectly. Please check user manual for more information. The connection will now be closed. [code=CONNECTION_TIMEDOUT]
>
Hi Mathew,
One thing would be good, in case it matters, is if you could clarify
what specifically you are using to send/recv, as 'Proton' encompasses
a few different areas/behaviours, so the name alone isn't enough to
nail down exactly what you are doing there.
I can't say there are any caveats I can think of right now that you'd
need to be aware of in relation to sending large messages,
particularly with regard to causing BufferUnderflowExceptions on the
broker. The Qpid JMS client test suite includes some interop testing
of sending and receiving larger messages against ActiveMQ 5 (which
also uses the proton-j engine for its AMQP 1.0 support) using various
message sizes up to 50MB.
You said you tried them without success, but did you see the exact
same behaviour while using the MQLight and SwiftMQ clients?
At first glance I'd have to say the Artemis/HornetQ brokers use of the
proton-j engine is likely to be the area to look at. I have run the
JMS clients examples against Artemis, but can't say that I've sent
larger messages against it.
The second log message appears to be from Artemis core client logging,
its not clear if its related or not.
Robbie
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
For additional commands, e-mail: users-help@qpid.apache.org