You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@uima.apache.org by Mihaela M <mm...@yahoo.com> on 2014/02/12 15:21:24 UTC
Re: uima-as 2.3.1 - java.io.IOException: Frame size of 147 MB larger than max allowed 100 MB
Hello,
I have upgraded uima-as to version 2.4.2 but I still encounter an issue with the wireFormat.maxFrameSize setting for the ActiveMQ broker.
1. I have updated the configuration for transport connector in activemq.xml file:
<transportConnectors>
<transportConnector name="openwire" uri="tcp://127.0.0.1:61616?wireFormat.maxInactivityDuration=0&wireFormat.maxFrameSize=209715200&jms.useCompression=true"/>
</transportConnectors>
2. I have set the brokerURL attribute in uima-as deployment descriptors to value: "tcp://127.0.0.1:61616?wireFormat.maxInactivityDuration=0&wireFormat.maxFrameSize=209715200&jms.useCompression=true"
3. I have set the TRACE level for logger org.apache.activemq.transport
After performing all the above settings I noticed that when I started the pipeline, for each remote delegate, multiple negotiations are performed by org.apache.activemq.transport.WireFormatNegotiator. All use the maxFrameSize of 200 MB that I specified, except one negotiation that is done using maxFrameSize of 100 MB.
I do not understand from where does come this limitation of 100 MB. Does exist in the UIMA client? By default I saw that ActiveMQ is using MAX_LONG for maxFrameSize so I really don't know from where does come this 100 MB setting for maxFrameSize.
Does anyone have an idea why is happening this? Could somebody tell me a starting point for looking in the uima code?
On the other hand does anybody know whether there are some limitations when using the "binary" serializer for remote delegates instead of "xmi" serializer? I found in one jira issue (https://issues.apache.org/jira/browse/UIMA-1196) that for the "binary" serializer is mandatory that all uima AS services use a common type system. Is this still an issue in uima-as 2.4.2?
Thank you!
Mihaela
On Monday, January 27, 2014 4:30 PM, Eddie Epstein <ea...@gmail.com> wrote:
On Thu, Jan 23, 2014 at 9:28 AM, Thomas Ginter <th...@utah.edu>wrote:
> It is likely then that your expansion is happening after the remote
> service is called or else is not yet big enough to be over the 100MB limit.
>
Also note that by default UIMA-AS [Java] services use a delta-CAS
interface. Only changes to the CAS
are returned from a service.
Besides deleting unnecessary FS from the final CAS to be returned, another
option to consider is to use compression on JMS messages:
jms.useCompression=true
This decoration can be added to the broker configuration file,
$UIMA_HOME/amq/conf/activemq-nojournal.xml
as
<transportConnector name="openwire" uri="tcp://
0.0.0.0:61616?jms.useCompression=true"/>
which will cause messages in all queues to be compressed.
Eddie
Re: uima-as 2.3.1 - java.io.IOException: Frame size of 147 MB larger than max allowed 100 MB
Posted by Mihaela M <mm...@yahoo.com>.
I don't know why it doesn't work because I have explicitly set the brokerUrl to the deployment descriptors, ran the UIMA_Service with argument -defaultBrokerURL and set also the system property -DdefaultBrokerURL. All were set with value: tcp://127.0.0.1:61616?wireFormat.maxInactivityDuration=0&wireFormat.maxFrameSize=209715200&jms.useCompression=true.
The pipeline has the following components: one main aggregate with a remote RDF Consumer and one remote aggregate. The remote aggregate has one remote primitive annotator and one local primitive annotator.
I don't know exactly how many connections are created for the services to communicate with each other but in the log file of each of the components I noticed that there is ONLY ONE connection which is built with default frame size of 100 MB instead of 200 MB as the others.
In the aggregators I see a line like the following:
02/13/2014 05:36:37.081 [DEBUG] [main] org.apache.activemq.transport.WireFormatNegotiator - Sending: WireFormatInfo { version=9, properties={MaxFrameSize=104857600, CacheSize=1024, CacheEnabled=true, SizePrefixDisabled=false, MaxInactivityDurationInitalDelay=10000, TcpNoDelayEnabled=true, MaxInactivityDuration=0, TightEncodingEnabled=true, StackTraceEnabled=true}, magic=[A,c,t,i,v,e,M,Q]}
02/13/2014 05:36:37.082 [TRACE] [ActiveMQ Transport: tcp://localhost/127.0.0.1:61616] org.apache.activemq.transport.tcp.TcpTransport - TCP consumer thread for tcp://localhost/127.0.0.1:61616 starting
02/13/2014 05:36:37.083 [DEBUG] [ActiveMQ Transport: tcp://localhost/127.0.0.1:61616] org.apache.activemq.transport.InactivityMonitor - Using min of local: WireFormatInfo { version=9, properties={MaxFrameSize=104857600, CacheSize=1024, CacheEnabled=true, SizePrefixDisabled=false, MaxInactivityDurationInitalDelay=10000, TcpNoDelayEnabled=true, MaxInactivityDuration=0, TightEncodingEnabled=true, StackTraceEnabled=true}, magic=[A,c,t,i,v,e,M,Q]} and remote: WireFormatInfo { version=9, properties={CacheSize=1024, MaxFrameSize=209715200, CacheEnabled=true, SizePrefixDisabled=false, TcpNoDelayEnabled=true, MaxInactivityDurationInitalDelay=10000, MaxInactivityDuration=0, TightEncodingEnabled=true, StackTraceEnabled=true}, magic=[A,c,t,i,v,e,M,Q]}
02/13/2014 05:36:37.083 [DEBUG] [ActiveMQ Transport: tcp://localhost/127.0.0.1:61616] org.apache.activemq.transport.WireFormatNegotiator - Received WireFormat: WireFormatInfo { version=9, properties={CacheSize=1024, MaxFrameSize=209715200, CacheEnabled=true, SizePrefixDisabled=false, TcpNoDelayEnabled=true, MaxInactivityDurationInitalDelay=10000, MaxInactivityDuration=0, TightEncodingEnabled=true, StackTraceEnabled=true}, magic=[A,c,t,i,v,e,M,Q]}
02/13/2014 05:36:37.083 [DEBUG] [ActiveMQ Transport: tcp://localhost/127.0.0.1:61616] org.apache.activemq.transport.WireFormatNegotiator - tcp://localhost/127.0.0.1:61616 before negotiation: OpenWireFormat{version=9, cacheEnabled=false, stackTraceEnabled=false, tightEncodingEnabled=false, sizePrefixDisabled=false, maxFrameSize=104857600}
02/13/2014 05:36:37.083 [DEBUG] [ActiveMQ Transport: tcp://localhost/127.0.0.1:61616] org.apache.activemq.transport.WireFormatNegotiator - tcp://localhost/127.0.0.1:61616 after negotiation: OpenWireFormat{version=9, cacheEnabled=true, stackTraceEnabled=true, tightEncodingEnabled=true, sizePrefixDisabled=false, maxFrameSize=104857600}
and in the primitive annotator I see the following:
02/13/2014 05:36:37.122 [DEBUG] [primitive_input_msgLsnrCntnr_TokenizerQueue_1_getMeta - Thread - 18] org.apache.uima.adapter.jms.activemq.UimaDefaultMessageListenerContainer - Received message of type [class org.apache.activemq.command.ActiveMQTextMessage] from consumer [ActiveMQMessageConsumer { value=ID:PCCS000630WS-56652-1392287793445-1:2:1:1, started=true }] of session [ActiveMQSession {id=ID:PCCS000630WS-56652-1392287793445-1:2:1,started=true}]
02/13/2014 05:36:37.143 [DEBUG][primitive_input_msgLsnrCntnr_TokenizerQueue_1_getMeta - Thread - 18] org.apache.activemq.transport.WireFormatNegotiator - Sending: WireFormatInfo { version=9, properties={MaxFrameSize=104857600, CacheSize=1024, CacheEnabled=true, SizePrefixDisabled=false, MaxInactivityDurationInitalDelay=10000, TcpNoDelayEnabled=true, MaxInactivityDuration=0, TightEncodingEnabled=true, StackTraceEnabled=true}, magic=[A,c,t,i,v,e,M,Q]}
02/13/2014 05:36:37.146 [TRACE] [ActiveMQ Transport: tcp://localhost/127.0.0.1:61616] org.apache.activemq.transport.tcp.TcpTransport - TCP consumer thread for tcp://localhost/127.0.0.1:61616 starting
02/13/2014 05:36:37.146 [DEBUG] [ActiveMQ Transport: tcp://localhost/127.0.0.1:61616] org.apache.activemq.transport.InactivityMonitor - Using min of local: WireFormatInfo { version=9, properties={MaxFrameSize=104857600, CacheSize=1024, CacheEnabled=true, SizePrefixDisabled=false, MaxInactivityDurationInitalDelay=10000, TcpNoDelayEnabled=true, MaxInactivityDuration=0, TightEncodingEnabled=true, StackTraceEnabled=true}, magic=[A,c,t,i,v,e,M,Q]} and remote: WireFormatInfo { version=9, properties={CacheSize=1024, MaxFrameSize=209715200, CacheEnabled=true, SizePrefixDisabled=false, TcpNoDelayEnabled=true, MaxInactivityDurationInitalDelay=10000, MaxInactivityDuration=0, TightEncodingEnabled=true, StackTraceEnabled=true}, magic=[A,c,t,i,v,e,M,Q]}
02/13/2014 05:36:37.147 [DEBUG] [ActiveMQ Transport: tcp://localhost/127.0.0.1:61616] org.apache.activemq.transport.WireFormatNegotiator - Received WireFormat: WireFormatInfo { version=9, properties={CacheSize=1024, MaxFrameSize=209715200, CacheEnabled=true, SizePrefixDisabled=false, TcpNoDelayEnabled=true, MaxInactivityDurationInitalDelay=10000, MaxInactivityDuration=0, TightEncodingEnabled=true, StackTraceEnabled=true}, magic=[A,c,t,i,v,e,M,Q]}
02/13/2014 05:36:37.147 [DEBUG] [ActiveMQ Transport: tcp://localhost/127.0.0.1:61616] org.apache.activemq.transport.WireFormatNegotiator - tcp://localhost/127.0.0.1:61616 before negotiation: OpenWireFormat{version=9, cacheEnabled=false, stackTraceEnabled=false, tightEncodingEnabled=false, sizePrefixDisabled=false, maxFrameSize=104857600}
02/13/2014 05:36:37.147 [DEBUG] [ActiveMQ Transport: tcp://localhost/127.0.0.1:61616] org.apache.activemq.transport.WireFormatNegotiator - tcp://localhost/127.0.0.1:61616 after negotiation: OpenWireFormat{version=9, cacheEnabled=true, stackTraceEnabled=true, tightEncodingEnabled=true, sizePrefixDisabled=false, maxFrameSize=104857600}
I have logged also the name of the thread and it seems that, at least for the remote primitive annotator, not the main thread starts negotiating using the default maxFrameSize of 100 MB. The thread name contains "meta". Could it be the thread used for checking the state of the component? Is it possible that this connection is not using the brokerUrl passed, but the default one?
Any feedback is appreciated.
Thanks,
Mihaela
On Wednesday, February 12, 2014 4:43 PM, Jaroslaw Cwiklik <ui...@gmail.com> wrote:
It seems like the ActimeMQ documentation (http://activemq.apache.org/configuring-wire-formats.html)
is wrong with respect to the default maxFrameSize being MAX_LONG. I checked ActiveMQ source code and the default is 100 MB:
publicfinalclassOpenWireFormatimplementsWireFormat{publicstaticfinalintDEFAULT_VERSION=CommandTypes.PROTOCOL_STORE_VERSION;publicstaticfinalintDEFAULT_WIRE_VERSION=CommandTypes.PROTOCOL_VERSION;publicstaticfinalintDEFAULT_MAX_FRAME_SIZE=100*1024*1024;//100 MB <-------------------------------------staticfinalbyteNULL_TYPE=CommandTypes.NULL;privatestaticfinalintMARSHAL_CACHE_SIZE=Short.MAX_VALUE/2;privatestaticfinalintMARSHAL_CACHE_FREE_SPACE=100;
The UIMA-AS doesnt set this value so the default is being used unless overriden. It seems to me that
either your service or a client is not overriding the default. Please check your deployment descriptors to make sure
that you changing the default in the brokerURL.
Jerry
On Wed, Feb 12, 2014 at 9:21 AM, Mihaela M <mm...@yahoo.com> wrote:
Hello,
>
>I have upgraded uima-as to version 2.4.2 but I still encounter an issue with the wireFormat.maxFrameSize setting for the ActiveMQ broker.
>1. I have updated the configuration for transport connector in activemq.xml file:
><transportConnectors>
> <transportConnector name="openwire" uri="tcp://127.0.0.1:61616?wireFormat.maxInactivityDuration=0&wireFormat.maxFrameSize=209715200&jms.useCompression=true"/>
></transportConnectors>
>2. I have set the brokerURL attribute in uima-as deployment descriptors to value: "tcp://127.0.0.1:61616?wireFormat.maxInactivityDuration=0&wireFormat.maxFrameSize=209715200&jms.useCompression=true"
>3. I have set the TRACE level for logger org.apache.activemq.transport
>
>After performing all the above settings I noticed that when I started the pipeline, for each remote delegate, multiple negotiations are performed by org.apache.activemq.transport.WireFormatNegotiator. All use the maxFrameSize of 200 MB that I specified, except one negotiation that is done using maxFrameSize of 100 MB.
>I do not understand from where does come this limitation of 100 MB. Does exist in the UIMA client? By default I saw that ActiveMQ is using MAX_LONG for maxFrameSize so I really don't know from where does come this 100 MB setting for maxFrameSize.
>
>Does anyone have an idea why is happening this? Could somebody tell me a starting point for looking in the uima code?
>
>
>On the other hand does anybody know whether there are some limitations when using the "binary" serializer for remote delegates instead of "xmi" serializer? I found in one jira issue (https://issues.apache.org/jira/browse/UIMA-1196) that for the "binary" serializer is mandatory that all uima AS services use a common type system. Is this still an issue in uima-as 2.4.2?
>
>Thank you!
>Mihaela
>
>
>
>
>
>On Monday, January 27, 2014 4:30 PM, Eddie Epstein <ea...@gmail.com> wrote:
>
>On Thu, Jan 23, 2014 at 9:28 AM, Thomas Ginter <th...@utah.edu>wrote:
>
>> It is likely then that your expansion is happening after the remote
>> service is called or else is not yet big enough to be over the 100MB limit.
>>
>
>Also note that by default UIMA-AS [Java] services use a delta-CAS
>interface. Only changes to the CAS
>are returned from a service.
>
>Besides deleting unnecessary FS from the final CAS to be returned, another
>option to consider is to use compression on JMS messages:
>jms.useCompression=true
>This decoration can be added to the broker configuration file,
> $UIMA_HOME/amq/conf/activemq-nojournal.xml
>
>as
> <transportConnector name="openwire" uri="tcp://
>0.0.0.0:61616?jms.useCompression=true"/>
>which will cause messages in all queues to be compressed.
>
>Eddie
Re: uima-as 2.3.1 - java.io.IOException: Frame size of 147 MB larger
than max allowed 100 MB
Posted by Jaroslaw Cwiklik <ui...@gmail.com>.
It seems like the ActimeMQ documentation (
http://activemq.apache.org/configuring-wire-formats.html)
is wrong with respect to the default maxFrameSize being MAX_LONG. I checked
ActiveMQ source code and the default is 100 MB:
public final class OpenWireFormat implements WireFormat { public
static final int DEFAULT_VERSION =
CommandTypes.PROTOCOL_STORE_VERSION; public static final int
DEFAULT_WIRE_VERSION = CommandTypes.PROTOCOL_VERSION; public static
final int *DEFAULT_MAX_FRAME_SIZE* = 100 * 1024 * 1024; //100 MB
<------------------------------------- static final byte NULL_TYPE
= CommandTypes.NULL; private static final int MARSHAL_CACHE_SIZE =
Short.MAX_VALUE / 2; private static final int
MARSHAL_CACHE_FREE_SPACE = 100;
The UIMA-AS doesnt set this value so the default is being used unless
overriden. It seems to me that
either your service or a client is not overriding the default. Please check
your deployment descriptors to make sure
that you changing the default in the brokerURL.
Jerry
On Wed, Feb 12, 2014 at 9:21 AM, Mihaela M <mm...@yahoo.com> wrote:
> Hello,
>
> I have upgraded uima-as to version 2.4.2 but I still encounter an issue
> with the wireFormat.maxFrameSize setting for the ActiveMQ broker.
> 1. I have updated the configuration for transport connector in
> activemq.xml file:
> <transportConnectors>
> <transportConnector name="openwire" uri="tcp://
> 127.0.0.1:61616?wireFormat.maxInactivityDuration=0&wireFormat.maxFrameSize=209715200&jms.useCompression=true
> "/>
> </transportConnectors>
> 2. I have set the brokerURL attribute in uima-as deployment descriptors to
> value: "tcp://
> 127.0.0.1:61616?wireFormat.maxInactivityDuration=0&wireFormat.maxFrameSize=209715200&jms.useCompression=true
> "
> 3. I have set the TRACE level for logger org.apache.activemq.transport
>
> After performing all the above settings I noticed that when I started the
> pipeline, for each remote delegate, multiple negotiations are performed
> by org.apache.activemq.transport.WireFormatNegotiator. All use the
> maxFrameSize of 200 MB that I specified, except one negotiation that is
> done using maxFrameSize of 100 MB.
> I do not understand from where does come this limitation of 100 MB. Does
> exist in the UIMA client? By default I saw that ActiveMQ is using MAX_LONG
> for maxFrameSize so I really don't know from where does come this 100 MB
> setting for maxFrameSize.
>
> Does anyone have an idea why is happening this? Could somebody tell me a
> starting point for looking in the uima code?
>
>
> On the other hand does anybody know whether there are some limitations
> when using the "binary" serializer for remote delegates instead of "xmi"
> serializer? I found in one jira issue (
> https://issues.apache.org/jira/browse/UIMA-1196) that for the "binary"
> serializer is mandatory that all uima AS services use a common type system.
> Is this still an issue in uima-as 2.4.2?
>
> Thank you!
> Mihaela
>
>
>
>
> On Monday, January 27, 2014 4:30 PM, Eddie Epstein <ea...@gmail.com>
> wrote:
>
> On Thu, Jan 23, 2014 at 9:28 AM, Thomas Ginter <thomas.ginter@utah.edu
> >wrote:
>
> > It is likely then that your expansion is happening after the remote
> > service is called or else is not yet big enough to be over the 100MB
> limit.
> >
>
> Also note that by default UIMA-AS [Java] services use a delta-CAS
> interface. Only changes to the CAS
> are returned from a service.
>
> Besides deleting unnecessary FS from the final CAS to be returned, another
> option to consider is to use compression on JMS messages:
> jms.useCompression=true
> This decoration can be added to the broker configuration file,
> $UIMA_HOME/amq/conf/activemq-nojournal.xml
>
> as
> <transportConnector name="openwire" uri="tcp://
> 0.0.0.0:61616?jms.useCompression=true"/>
> which will cause messages in all queues to be compressed.
>
> Eddie
>