You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@activemq.apache.org by Praveen Bysani <pr...@gmail.com> on 2013/07/10 11:27:36 UTC
Variable volume of messages across consumers
Hi,
I have the following setup in my project,
1. multiple producers (p1,p2,p3) in Hongkong region sending variable
size non-persistent messages to a topic t1 on the broker (b1) in
Phillippines region
2. p1,p2 and p3 send messages frequently (~1 second)
3. each message has a time to live of 30 seconds
4. a tcp network bridge from b1 to a new broker b2 in HK region
5. consumers c1 in hongkong and c2 in europe subscribed to b2 through
ssl and a consumer c3 subscribed to t1 on b1 through tcp
I don't see the same amount of messages received across c1, c2 and c3. The
number of messages received is in the following order c3 > c1 > c2. I use
the default settings for prefetch values and *vmCursor *for
*pendingSubscriberPolicy
*and *strictOrderDisptachpolicy*.
While i understand there could be delays in passing the message to these
consumers as c3 could be slow consumer and c1 is fast, shouldn't the
message eventually be received to all consumers of a topic. Why do i see
different volumes of messages across different consumers ? Someone please
help me understand what is going on !!
Another strange thing is after a while, there are no messages received by
c2 for a very long period. I suppose it is due to the prefetch size ? Since
i am using a non persistent topic, the default value is quite high
(according to activemq documentation), which again baffles me if that is
really the reason.
--
Regards,
Praveen Bysani
http://www.praveenbysani.com
Re: Variable volume of messages across consumers
Posted by Praveen Bysani <pr...@gmail.com>.
I did take a look at the activemq logs on both the brokers, i only have
information such as below in the primary broker. There is not much i can
infer from this.
2013-07-24 15:06:21,916 | DEBUG | bridging (localhost -> localhost)
5a2c1e2a-9df2-45a8-aebc-861c3ab271e0:1:1:161, consumer:
ID:global-desktop.com-40234-1374645969412-2:2:1:3, destination topic://DM,
brokerPath: [ID:global-desktop.com-40234-1374645969412-0:1], message:
ActiveMQBytesMessage {commandId = 165, responseRequired = false, messageId
= 5a2c1e2a-9df2-45a8-aebc-861c3ab271e0:1:1:161, originalDestination = null,
originalTransactionId = null, producerId =
ID:global-desktop.com-40234-1374645969412-2:1:1:1, destination =
topic://DM, transactionId = null, expiration = 1374649701750, timestamp =
1374649581750, arrival = 0, brokerInTime = 1374649581915, brokerOutTime =
1374649581916, correlationId = null, replyTo = null, persistent = false,
type = null, priority = 4, groupID = null, groupSequence = 0,
targetConsumerId = null, compressed = false, userID = null, content =
org.apache.activemq.util.ByteSequence@258185ce, marshalledProperties =
null, dataStructure = null, redeliveryCounter = 0, size = 16819, properties
= null, readOnlyProperties = false, readOnlyBody = false, droppable =
false} ActiveMQBytesMessage{ bytesOut = null, dataOut = null, dataIn = null
} | org.apache.activemq.network.DemandForwardingBridgeSupport | ActiveMQ
BrokerService[localhost] Task-10
On 24 July 2013 03:15, Christian Posta <ch...@gmail.com> wrote:
> Well, you have to make sure messages are getting bridged properly (take a
> close look at the bridge logging output) and are not bouncing back and
> forth for some reason.
>
>
> On Tue, Jul 23, 2013 at 12:27 AM, Praveen Bysani <praveen.iiith@gmail.com
> >wrote:
>
> > Hi Christian,
> >
> > Thank you for the clarification. What are the factors in activemq that
> > could effect this, other than time to live that is set in the publisher
> ? I
> > have a time to live of 2 minutes currently, how can i ensure that most
> > messages are delivered before expiry ?
> >
> > On 15 July 2013 20:04, Christian Posta <ch...@gmail.com>
> wrote:
> >
> > > Yes, that's what it means.
> > >
> > > On Sunday, July 14, 2013, Praveen Bysani wrote:
> > >
> > > > Hi,
> > > >
> > > > I observed a lot of following messages in my log,
> > > >
> > > > 2013-07-15 03:48:22,399 | DEBUG | Message expired
> ActiveMQBytesMessage
> > > > {commandId = 1498, responseRequired = false, messageId =
> > > > d7dfc864-e46a-44b0-a0ec-b5d9e3a07127:1:1:253, originalDestination =
> > null,
> > > > originalTransactionId = null, producerId = ID:global-desktop,
> > > destination =
> > > > topic://DM, transactionId = null, expiration = 1373831265580,
> > timestamp =
> > > > 1373831145580, arrival = 0, brokerInTime = 1373831302399,
> > brokerOutTime =
> > > > 1373831299130, correlationId = null, replyTo = null, persistent =
> > false,
> > > > type = null, priority = 4, groupID = null, groupSequence = 0,
> > > > targetConsumerId = null, compressed = false, userID = null, content =
> > > > org.apache.activemq.util.ByteSequence@7f94dadb,
> marshalledProperties =
> > > > null, dataStructure = null, redeliveryCounter = 0, size = 0,
> > properties =
> > > > null, readOnlyProperties = false, readOnlyBody = false, droppable =
> > > false}
> > > > ActiveMQBytesMessage{ bytesOut = null, dataOut = null, dataIn = null
> }
> > |
> > > > org.apache.activemq.broker.region.RegionBroker | ActiveMQ Transport:
> > > > tcp:///<ip-address>:40887@6122
> > > >
> > > > Does that mean it failed to deliver the messages before the time to
> > live
> > > ?
> > > >
> > > > On 12 July 2013 14:49, Praveen Bysani <pr...@gmail.com>
> wrote:
> > > >
> > > > > Hi,
> > > > > Thanks i added the log4j for all components under
> > org.apache.activemq.
> > > > > Hopefully i can see something useful.
> > > > >
> > > > >
> > > > > On 12 July 2013 00:21, Christian Posta <ch...@gmail.com>
> > > > wrote:
> > > > >
> > > > >> Add logging configuration to turn on logging for the bridge.
> > > > >>
> > > > >> e.g.,
> > > > >>
> > > > >> to your log4j.properties, add
> > > > >>
> > > > >>
> > > > >>
> > > >
> > >
> >
> log4j.logger.org.apache.activemq.network.DemandForwardingBridgeSupport=DEBUG
> > > > >>
> > > > >>
> > > > >> On Thu, Jul 11, 2013 at 3:38 AM, Praveen Bysani <
> > > > praveen.iiith@gmail.com
> > > > >> >wrote:
> > > > >>
> > > > >> > Hi,
> > > > >> >
> > > > >> > I found the following lines in the subscriber log
> > > > >> >
> > > > >> > | org.apache.activemq.transport.InactivityMonitor | DEBUG |
> > > 2013-07-11
> > > > >> > 07:34:44,384 | 30000 ms elapsed since last read check.
> > > > >> > | org.apache.activemq.transport.InactivityMonitor | DEBUG |
> > > 2013-07-11
> > > > >> > 07:34:44,384 |
> > > > >> > org.apache.activemq.transport.InactivityMonitor$2@61726a5c9999
> > > > >> > ms elapsed since last write check.
> > > > >> >
> > > > >> > I have disabled the inactivity period in the broker connection
> url
> > > by
> > > > >> > setting maxInactivityDuration to 0. Does it have anything to do
> > with
> > > > >> > messages not being received ?
> > > > >> >
> > > > >> > On 11 July 2013 12:18, Praveen Bysani <pr...@gmail.com>
> > > > wrote:
> > > > >> >
> > > > >> > > Hi Christian,
> > > > >> > >
> > > > >> > > Thanks for your input. But i have little idea of what those
> > terms
> > > > >> mean.
> > > > >> > > Could you elaborate please.
> > > > >> > >
> > > > >> > >
> > > > >> > > On 10 July 2013 21:06, Christian Posta <
> > christian.posta@gmail.com
> > > >
> > > > >> > wrote:
> > > > >> > >
> > > > >> > >> Your best bet would be to set logging to debug on the network
> > > > bridge
> > > > >> > >> (DemandForwardingBridgeSupport) and possibly the RegionBroker
> > to
> > > > see
> > > > >> > what
> > > > >> > >> is happening.
> > > > >> > >>
> > > > >> > >> On Wednesday, July 10, 2013, Praveen Bysani wrote:
> > > > >> > >>
> > > > >> > >> > Hi,
> > > > >> > >> > I have the following setup in my project,
> > > > >> > >> >
> > > > >> > >> > 1. multiple producers (p1,p2,p3) in Hongkong region
> > sending
> > > > >> > variable
> > > > >> > >> > size non-persistent messages to a topic t1 on the broker
> > > (b1)
> > > > in
> > > > >> > >> > Phillippines region
> > > > >> > >> > 2. p1,p2 and p3 send messages frequently (~1 second)
> > > > >> > >> > 3. each message has a time to live of 30 seconds
> > > > >> > >> > 4. a tcp network bridge from b1 to a new broker b2 in HK
> > > > region
> > > > >> > >> > 5. consumers c1 in hongkong and c2 in europe subscribed
> to
> > > b2
> > > > >> > through
> > > > >> > >> > ssl and a consumer c3 subscribed to t1 on b1 through tcp
> > > > >> > >> >
> > > > >> > >> > I don't see the same amount of messages received across c1,
> > c2
> > > > and
> > > > >> c3.
> > > > >> > >> The
> > > > >> > >> > number of messages received is in the following order c3 >
> > c1 >
> > > > >> c2. I
> > > > >> > >> use
> > > > >> > >> > the default settings for prefetch values and *vmCursor *for
> > > > >> > >> > *pendingSubscriberPolicy
> > > > >> > >> > *and *strictOrderDisptachpolicy*.
> > > > >> > >> >
> > > > >> > >> > While i understand there could be delays in passing the
> > message
> > > > to
> > > > >> > these
> > > > >> > >> > consumers as c3 could be slow consumer and c1 is fast,
> > > shouldn't
> > > > >> the
> > > > >> > >> > message eventually be received to all consumers of a topic.
> > Why
> > > > do
> > > > >> i
> > > > >> > see
> > > > >> > >> > different volumes of
> > >
> > >
> > >
> > > --
> > > *Christian Posta*
> > > http://www.christianposta.com/blog
> > > twitter: @christianposta
> > >
> >
> >
> >
> > --
> > Regards,
> > Praveen Bysani
> > http://www.praveenbysani.com
> >
>
>
>
> --
> *Christian Posta*
> http://www.christianposta.com/blog
> twitter: @christianposta
>
--
Regards,
Praveen Bysani
http://www.praveenbysani.com
Re: Variable volume of messages across consumers
Posted by Christian Posta <ch...@gmail.com>.
Well, you have to make sure messages are getting bridged properly (take a
close look at the bridge logging output) and are not bouncing back and
forth for some reason.
On Tue, Jul 23, 2013 at 12:27 AM, Praveen Bysani <pr...@gmail.com>wrote:
> Hi Christian,
>
> Thank you for the clarification. What are the factors in activemq that
> could effect this, other than time to live that is set in the publisher ? I
> have a time to live of 2 minutes currently, how can i ensure that most
> messages are delivered before expiry ?
>
> On 15 July 2013 20:04, Christian Posta <ch...@gmail.com> wrote:
>
> > Yes, that's what it means.
> >
> > On Sunday, July 14, 2013, Praveen Bysani wrote:
> >
> > > Hi,
> > >
> > > I observed a lot of following messages in my log,
> > >
> > > 2013-07-15 03:48:22,399 | DEBUG | Message expired ActiveMQBytesMessage
> > > {commandId = 1498, responseRequired = false, messageId =
> > > d7dfc864-e46a-44b0-a0ec-b5d9e3a07127:1:1:253, originalDestination =
> null,
> > > originalTransactionId = null, producerId = ID:global-desktop,
> > destination =
> > > topic://DM, transactionId = null, expiration = 1373831265580,
> timestamp =
> > > 1373831145580, arrival = 0, brokerInTime = 1373831302399,
> brokerOutTime =
> > > 1373831299130, correlationId = null, replyTo = null, persistent =
> false,
> > > type = null, priority = 4, groupID = null, groupSequence = 0,
> > > targetConsumerId = null, compressed = false, userID = null, content =
> > > org.apache.activemq.util.ByteSequence@7f94dadb, marshalledProperties =
> > > null, dataStructure = null, redeliveryCounter = 0, size = 0,
> properties =
> > > null, readOnlyProperties = false, readOnlyBody = false, droppable =
> > false}
> > > ActiveMQBytesMessage{ bytesOut = null, dataOut = null, dataIn = null }
> |
> > > org.apache.activemq.broker.region.RegionBroker | ActiveMQ Transport:
> > > tcp:///<ip-address>:40887@6122
> > >
> > > Does that mean it failed to deliver the messages before the time to
> live
> > ?
> > >
> > > On 12 July 2013 14:49, Praveen Bysani <pr...@gmail.com> wrote:
> > >
> > > > Hi,
> > > > Thanks i added the log4j for all components under
> org.apache.activemq.
> > > > Hopefully i can see something useful.
> > > >
> > > >
> > > > On 12 July 2013 00:21, Christian Posta <ch...@gmail.com>
> > > wrote:
> > > >
> > > >> Add logging configuration to turn on logging for the bridge.
> > > >>
> > > >> e.g.,
> > > >>
> > > >> to your log4j.properties, add
> > > >>
> > > >>
> > > >>
> > >
> >
> log4j.logger.org.apache.activemq.network.DemandForwardingBridgeSupport=DEBUG
> > > >>
> > > >>
> > > >> On Thu, Jul 11, 2013 at 3:38 AM, Praveen Bysani <
> > > praveen.iiith@gmail.com
> > > >> >wrote:
> > > >>
> > > >> > Hi,
> > > >> >
> > > >> > I found the following lines in the subscriber log
> > > >> >
> > > >> > | org.apache.activemq.transport.InactivityMonitor | DEBUG |
> > 2013-07-11
> > > >> > 07:34:44,384 | 30000 ms elapsed since last read check.
> > > >> > | org.apache.activemq.transport.InactivityMonitor | DEBUG |
> > 2013-07-11
> > > >> > 07:34:44,384 |
> > > >> > org.apache.activemq.transport.InactivityMonitor$2@61726a5c9999
> > > >> > ms elapsed since last write check.
> > > >> >
> > > >> > I have disabled the inactivity period in the broker connection url
> > by
> > > >> > setting maxInactivityDuration to 0. Does it have anything to do
> with
> > > >> > messages not being received ?
> > > >> >
> > > >> > On 11 July 2013 12:18, Praveen Bysani <pr...@gmail.com>
> > > wrote:
> > > >> >
> > > >> > > Hi Christian,
> > > >> > >
> > > >> > > Thanks for your input. But i have little idea of what those
> terms
> > > >> mean.
> > > >> > > Could you elaborate please.
> > > >> > >
> > > >> > >
> > > >> > > On 10 July 2013 21:06, Christian Posta <
> christian.posta@gmail.com
> > >
> > > >> > wrote:
> > > >> > >
> > > >> > >> Your best bet would be to set logging to debug on the network
> > > bridge
> > > >> > >> (DemandForwardingBridgeSupport) and possibly the RegionBroker
> to
> > > see
> > > >> > what
> > > >> > >> is happening.
> > > >> > >>
> > > >> > >> On Wednesday, July 10, 2013, Praveen Bysani wrote:
> > > >> > >>
> > > >> > >> > Hi,
> > > >> > >> > I have the following setup in my project,
> > > >> > >> >
> > > >> > >> > 1. multiple producers (p1,p2,p3) in Hongkong region
> sending
> > > >> > variable
> > > >> > >> > size non-persistent messages to a topic t1 on the broker
> > (b1)
> > > in
> > > >> > >> > Phillippines region
> > > >> > >> > 2. p1,p2 and p3 send messages frequently (~1 second)
> > > >> > >> > 3. each message has a time to live of 30 seconds
> > > >> > >> > 4. a tcp network bridge from b1 to a new broker b2 in HK
> > > region
> > > >> > >> > 5. consumers c1 in hongkong and c2 in europe subscribed to
> > b2
> > > >> > through
> > > >> > >> > ssl and a consumer c3 subscribed to t1 on b1 through tcp
> > > >> > >> >
> > > >> > >> > I don't see the same amount of messages received across c1,
> c2
> > > and
> > > >> c3.
> > > >> > >> The
> > > >> > >> > number of messages received is in the following order c3 >
> c1 >
> > > >> c2. I
> > > >> > >> use
> > > >> > >> > the default settings for prefetch values and *vmCursor *for
> > > >> > >> > *pendingSubscriberPolicy
> > > >> > >> > *and *strictOrderDisptachpolicy*.
> > > >> > >> >
> > > >> > >> > While i understand there could be delays in passing the
> message
> > > to
> > > >> > these
> > > >> > >> > consumers as c3 could be slow consumer and c1 is fast,
> > shouldn't
> > > >> the
> > > >> > >> > message eventually be received to all consumers of a topic.
> Why
> > > do
> > > >> i
> > > >> > see
> > > >> > >> > different volumes of
> >
> >
> >
> > --
> > *Christian Posta*
> > http://www.christianposta.com/blog
> > twitter: @christianposta
> >
>
>
>
> --
> Regards,
> Praveen Bysani
> http://www.praveenbysani.com
>
--
*Christian Posta*
http://www.christianposta.com/blog
twitter: @christianposta
Re: Variable volume of messages across consumers
Posted by Praveen Bysani <pr...@gmail.com>.
Hi Christian,
Thank you for the clarification. What are the factors in activemq that
could effect this, other than time to live that is set in the publisher ? I
have a time to live of 2 minutes currently, how can i ensure that most
messages are delivered before expiry ?
On 15 July 2013 20:04, Christian Posta <ch...@gmail.com> wrote:
> Yes, that's what it means.
>
> On Sunday, July 14, 2013, Praveen Bysani wrote:
>
> > Hi,
> >
> > I observed a lot of following messages in my log,
> >
> > 2013-07-15 03:48:22,399 | DEBUG | Message expired ActiveMQBytesMessage
> > {commandId = 1498, responseRequired = false, messageId =
> > d7dfc864-e46a-44b0-a0ec-b5d9e3a07127:1:1:253, originalDestination = null,
> > originalTransactionId = null, producerId = ID:global-desktop,
> destination =
> > topic://DM, transactionId = null, expiration = 1373831265580, timestamp =
> > 1373831145580, arrival = 0, brokerInTime = 1373831302399, brokerOutTime =
> > 1373831299130, correlationId = null, replyTo = null, persistent = false,
> > type = null, priority = 4, groupID = null, groupSequence = 0,
> > targetConsumerId = null, compressed = false, userID = null, content =
> > org.apache.activemq.util.ByteSequence@7f94dadb, marshalledProperties =
> > null, dataStructure = null, redeliveryCounter = 0, size = 0, properties =
> > null, readOnlyProperties = false, readOnlyBody = false, droppable =
> false}
> > ActiveMQBytesMessage{ bytesOut = null, dataOut = null, dataIn = null } |
> > org.apache.activemq.broker.region.RegionBroker | ActiveMQ Transport:
> > tcp:///<ip-address>:40887@6122
> >
> > Does that mean it failed to deliver the messages before the time to live
> ?
> >
> > On 12 July 2013 14:49, Praveen Bysani <pr...@gmail.com> wrote:
> >
> > > Hi,
> > > Thanks i added the log4j for all components under org.apache.activemq.
> > > Hopefully i can see something useful.
> > >
> > >
> > > On 12 July 2013 00:21, Christian Posta <ch...@gmail.com>
> > wrote:
> > >
> > >> Add logging configuration to turn on logging for the bridge.
> > >>
> > >> e.g.,
> > >>
> > >> to your log4j.properties, add
> > >>
> > >>
> > >>
> >
> log4j.logger.org.apache.activemq.network.DemandForwardingBridgeSupport=DEBUG
> > >>
> > >>
> > >> On Thu, Jul 11, 2013 at 3:38 AM, Praveen Bysani <
> > praveen.iiith@gmail.com
> > >> >wrote:
> > >>
> > >> > Hi,
> > >> >
> > >> > I found the following lines in the subscriber log
> > >> >
> > >> > | org.apache.activemq.transport.InactivityMonitor | DEBUG |
> 2013-07-11
> > >> > 07:34:44,384 | 30000 ms elapsed since last read check.
> > >> > | org.apache.activemq.transport.InactivityMonitor | DEBUG |
> 2013-07-11
> > >> > 07:34:44,384 |
> > >> > org.apache.activemq.transport.InactivityMonitor$2@61726a5c9999
> > >> > ms elapsed since last write check.
> > >> >
> > >> > I have disabled the inactivity period in the broker connection url
> by
> > >> > setting maxInactivityDuration to 0. Does it have anything to do with
> > >> > messages not being received ?
> > >> >
> > >> > On 11 July 2013 12:18, Praveen Bysani <pr...@gmail.com>
> > wrote:
> > >> >
> > >> > > Hi Christian,
> > >> > >
> > >> > > Thanks for your input. But i have little idea of what those terms
> > >> mean.
> > >> > > Could you elaborate please.
> > >> > >
> > >> > >
> > >> > > On 10 July 2013 21:06, Christian Posta <christian.posta@gmail.com
> >
> > >> > wrote:
> > >> > >
> > >> > >> Your best bet would be to set logging to debug on the network
> > bridge
> > >> > >> (DemandForwardingBridgeSupport) and possibly the RegionBroker to
> > see
> > >> > what
> > >> > >> is happening.
> > >> > >>
> > >> > >> On Wednesday, July 10, 2013, Praveen Bysani wrote:
> > >> > >>
> > >> > >> > Hi,
> > >> > >> > I have the following setup in my project,
> > >> > >> >
> > >> > >> > 1. multiple producers (p1,p2,p3) in Hongkong region sending
> > >> > variable
> > >> > >> > size non-persistent messages to a topic t1 on the broker
> (b1)
> > in
> > >> > >> > Phillippines region
> > >> > >> > 2. p1,p2 and p3 send messages frequently (~1 second)
> > >> > >> > 3. each message has a time to live of 30 seconds
> > >> > >> > 4. a tcp network bridge from b1 to a new broker b2 in HK
> > region
> > >> > >> > 5. consumers c1 in hongkong and c2 in europe subscribed to
> b2
> > >> > through
> > >> > >> > ssl and a consumer c3 subscribed to t1 on b1 through tcp
> > >> > >> >
> > >> > >> > I don't see the same amount of messages received across c1, c2
> > and
> > >> c3.
> > >> > >> The
> > >> > >> > number of messages received is in the following order c3 > c1 >
> > >> c2. I
> > >> > >> use
> > >> > >> > the default settings for prefetch values and *vmCursor *for
> > >> > >> > *pendingSubscriberPolicy
> > >> > >> > *and *strictOrderDisptachpolicy*.
> > >> > >> >
> > >> > >> > While i understand there could be delays in passing the message
> > to
> > >> > these
> > >> > >> > consumers as c3 could be slow consumer and c1 is fast,
> shouldn't
> > >> the
> > >> > >> > message eventually be received to all consumers of a topic. Why
> > do
> > >> i
> > >> > see
> > >> > >> > different volumes of
>
>
>
> --
> *Christian Posta*
> http://www.christianposta.com/blog
> twitter: @christianposta
>
--
Regards,
Praveen Bysani
http://www.praveenbysani.com
Re: Variable volume of messages across consumers
Posted by Christian Posta <ch...@gmail.com>.
Yes, that's what it means.
On Sunday, July 14, 2013, Praveen Bysani wrote:
> Hi,
>
> I observed a lot of following messages in my log,
>
> 2013-07-15 03:48:22,399 | DEBUG | Message expired ActiveMQBytesMessage
> {commandId = 1498, responseRequired = false, messageId =
> d7dfc864-e46a-44b0-a0ec-b5d9e3a07127:1:1:253, originalDestination = null,
> originalTransactionId = null, producerId = ID:global-desktop, destination =
> topic://DM, transactionId = null, expiration = 1373831265580, timestamp =
> 1373831145580, arrival = 0, brokerInTime = 1373831302399, brokerOutTime =
> 1373831299130, correlationId = null, replyTo = null, persistent = false,
> type = null, priority = 4, groupID = null, groupSequence = 0,
> targetConsumerId = null, compressed = false, userID = null, content =
> org.apache.activemq.util.ByteSequence@7f94dadb, marshalledProperties =
> null, dataStructure = null, redeliveryCounter = 0, size = 0, properties =
> null, readOnlyProperties = false, readOnlyBody = false, droppable = false}
> ActiveMQBytesMessage{ bytesOut = null, dataOut = null, dataIn = null } |
> org.apache.activemq.broker.region.RegionBroker | ActiveMQ Transport:
> tcp:///<ip-address>:40887@6122
>
> Does that mean it failed to deliver the messages before the time to live ?
>
> On 12 July 2013 14:49, Praveen Bysani <pr...@gmail.com> wrote:
>
> > Hi,
> > Thanks i added the log4j for all components under org.apache.activemq.
> > Hopefully i can see something useful.
> >
> >
> > On 12 July 2013 00:21, Christian Posta <ch...@gmail.com>
> wrote:
> >
> >> Add logging configuration to turn on logging for the bridge.
> >>
> >> e.g.,
> >>
> >> to your log4j.properties, add
> >>
> >>
> >>
> log4j.logger.org.apache.activemq.network.DemandForwardingBridgeSupport=DEBUG
> >>
> >>
> >> On Thu, Jul 11, 2013 at 3:38 AM, Praveen Bysani <
> praveen.iiith@gmail.com
> >> >wrote:
> >>
> >> > Hi,
> >> >
> >> > I found the following lines in the subscriber log
> >> >
> >> > | org.apache.activemq.transport.InactivityMonitor | DEBUG | 2013-07-11
> >> > 07:34:44,384 | 30000 ms elapsed since last read check.
> >> > | org.apache.activemq.transport.InactivityMonitor | DEBUG | 2013-07-11
> >> > 07:34:44,384 |
> >> > org.apache.activemq.transport.InactivityMonitor$2@61726a5c9999
> >> > ms elapsed since last write check.
> >> >
> >> > I have disabled the inactivity period in the broker connection url by
> >> > setting maxInactivityDuration to 0. Does it have anything to do with
> >> > messages not being received ?
> >> >
> >> > On 11 July 2013 12:18, Praveen Bysani <pr...@gmail.com>
> wrote:
> >> >
> >> > > Hi Christian,
> >> > >
> >> > > Thanks for your input. But i have little idea of what those terms
> >> mean.
> >> > > Could you elaborate please.
> >> > >
> >> > >
> >> > > On 10 July 2013 21:06, Christian Posta <ch...@gmail.com>
> >> > wrote:
> >> > >
> >> > >> Your best bet would be to set logging to debug on the network
> bridge
> >> > >> (DemandForwardingBridgeSupport) and possibly the RegionBroker to
> see
> >> > what
> >> > >> is happening.
> >> > >>
> >> > >> On Wednesday, July 10, 2013, Praveen Bysani wrote:
> >> > >>
> >> > >> > Hi,
> >> > >> > I have the following setup in my project,
> >> > >> >
> >> > >> > 1. multiple producers (p1,p2,p3) in Hongkong region sending
> >> > variable
> >> > >> > size non-persistent messages to a topic t1 on the broker (b1)
> in
> >> > >> > Phillippines region
> >> > >> > 2. p1,p2 and p3 send messages frequently (~1 second)
> >> > >> > 3. each message has a time to live of 30 seconds
> >> > >> > 4. a tcp network bridge from b1 to a new broker b2 in HK
> region
> >> > >> > 5. consumers c1 in hongkong and c2 in europe subscribed to b2
> >> > through
> >> > >> > ssl and a consumer c3 subscribed to t1 on b1 through tcp
> >> > >> >
> >> > >> > I don't see the same amount of messages received across c1, c2
> and
> >> c3.
> >> > >> The
> >> > >> > number of messages received is in the following order c3 > c1 >
> >> c2. I
> >> > >> use
> >> > >> > the default settings for prefetch values and *vmCursor *for
> >> > >> > *pendingSubscriberPolicy
> >> > >> > *and *strictOrderDisptachpolicy*.
> >> > >> >
> >> > >> > While i understand there could be delays in passing the message
> to
> >> > these
> >> > >> > consumers as c3 could be slow consumer and c1 is fast, shouldn't
> >> the
> >> > >> > message eventually be received to all consumers of a topic. Why
> do
> >> i
> >> > see
> >> > >> > different volumes of
--
*Christian Posta*
http://www.christianposta.com/blog
twitter: @christianposta
Re: Variable volume of messages across consumers
Posted by Praveen Bysani <pr...@gmail.com>.
Hi,
I observed a lot of following messages in my log,
2013-07-15 03:48:22,399 | DEBUG | Message expired ActiveMQBytesMessage
{commandId = 1498, responseRequired = false, messageId =
d7dfc864-e46a-44b0-a0ec-b5d9e3a07127:1:1:253, originalDestination = null,
originalTransactionId = null, producerId = ID:global-desktop, destination =
topic://DM, transactionId = null, expiration = 1373831265580, timestamp =
1373831145580, arrival = 0, brokerInTime = 1373831302399, brokerOutTime =
1373831299130, correlationId = null, replyTo = null, persistent = false,
type = null, priority = 4, groupID = null, groupSequence = 0,
targetConsumerId = null, compressed = false, userID = null, content =
org.apache.activemq.util.ByteSequence@7f94dadb, marshalledProperties =
null, dataStructure = null, redeliveryCounter = 0, size = 0, properties =
null, readOnlyProperties = false, readOnlyBody = false, droppable = false}
ActiveMQBytesMessage{ bytesOut = null, dataOut = null, dataIn = null } |
org.apache.activemq.broker.region.RegionBroker | ActiveMQ Transport:
tcp:///<ip-address>:40887@6122
Does that mean it failed to deliver the messages before the time to live ?
On 12 July 2013 14:49, Praveen Bysani <pr...@gmail.com> wrote:
> Hi,
> Thanks i added the log4j for all components under org.apache.activemq.
> Hopefully i can see something useful.
>
>
> On 12 July 2013 00:21, Christian Posta <ch...@gmail.com> wrote:
>
>> Add logging configuration to turn on logging for the bridge.
>>
>> e.g.,
>>
>> to your log4j.properties, add
>>
>>
>> log4j.logger.org.apache.activemq.network.DemandForwardingBridgeSupport=DEBUG
>>
>>
>> On Thu, Jul 11, 2013 at 3:38 AM, Praveen Bysani <praveen.iiith@gmail.com
>> >wrote:
>>
>> > Hi,
>> >
>> > I found the following lines in the subscriber log
>> >
>> > | org.apache.activemq.transport.InactivityMonitor | DEBUG | 2013-07-11
>> > 07:34:44,384 | 30000 ms elapsed since last read check.
>> > | org.apache.activemq.transport.InactivityMonitor | DEBUG | 2013-07-11
>> > 07:34:44,384 |
>> > org.apache.activemq.transport.InactivityMonitor$2@61726a5c9999
>> > ms elapsed since last write check.
>> >
>> > I have disabled the inactivity period in the broker connection url by
>> > setting maxInactivityDuration to 0. Does it have anything to do with
>> > messages not being received ?
>> >
>> > On 11 July 2013 12:18, Praveen Bysani <pr...@gmail.com> wrote:
>> >
>> > > Hi Christian,
>> > >
>> > > Thanks for your input. But i have little idea of what those terms
>> mean.
>> > > Could you elaborate please.
>> > >
>> > >
>> > > On 10 July 2013 21:06, Christian Posta <ch...@gmail.com>
>> > wrote:
>> > >
>> > >> Your best bet would be to set logging to debug on the network bridge
>> > >> (DemandForwardingBridgeSupport) and possibly the RegionBroker to see
>> > what
>> > >> is happening.
>> > >>
>> > >> On Wednesday, July 10, 2013, Praveen Bysani wrote:
>> > >>
>> > >> > Hi,
>> > >> > I have the following setup in my project,
>> > >> >
>> > >> > 1. multiple producers (p1,p2,p3) in Hongkong region sending
>> > variable
>> > >> > size non-persistent messages to a topic t1 on the broker (b1) in
>> > >> > Phillippines region
>> > >> > 2. p1,p2 and p3 send messages frequently (~1 second)
>> > >> > 3. each message has a time to live of 30 seconds
>> > >> > 4. a tcp network bridge from b1 to a new broker b2 in HK region
>> > >> > 5. consumers c1 in hongkong and c2 in europe subscribed to b2
>> > through
>> > >> > ssl and a consumer c3 subscribed to t1 on b1 through tcp
>> > >> >
>> > >> > I don't see the same amount of messages received across c1, c2 and
>> c3.
>> > >> The
>> > >> > number of messages received is in the following order c3 > c1 >
>> c2. I
>> > >> use
>> > >> > the default settings for prefetch values and *vmCursor *for
>> > >> > *pendingSubscriberPolicy
>> > >> > *and *strictOrderDisptachpolicy*.
>> > >> >
>> > >> > While i understand there could be delays in passing the message to
>> > these
>> > >> > consumers as c3 could be slow consumer and c1 is fast, shouldn't
>> the
>> > >> > message eventually be received to all consumers of a topic. Why do
>> i
>> > see
>> > >> > different volumes of messages across different consumers ? Someone
>> > >> please
>> > >> > help me understand what is going on !!
>> > >> >
>> > >> > Another strange thing is after a while, there are no messages
>> received
>> > >> by
>> > >> > c2 for a very long period. I suppose it is due to the prefetch
>> size ?
>> > >> Since
>> > >> > i am using a non persistent topic, the default value is quite high
>> > >> > (according to activemq documentation), which again baffles me if
>> that
>> > is
>> > >> > really the reason.
>> > >> > --
>> > >> > Regards,
>> > >> > Praveen Bysani
>> > >> > http://www.praveenbysani.com
>> > >> >
>> > >>
>> > >>
>> > >> --
>> > >> *Christian Posta*
>> > >> http://www.christianposta.com/blog
>> > >> twitter: @christianposta
>> > >>
>> > >
>> > >
>> > >
>> > > --
>> > > Regards,
>> > > Praveen Bysani
>> > > http://www.praveenbysani.com
>> > >
>> >
>> >
>> >
>> > --
>> > Regards,
>> > Praveen Bysani
>> > http://www.praveenbysani.com
>> >
>>
>>
>>
>> --
>> *Christian Posta*
>> http://www.christianposta.com/blog
>> twitter: @christianposta
>>
>
>
>
> --
> Regards,
> Praveen Bysani
> http://www.praveenbysani.com
>
--
Regards,
Praveen Bysani
http://www.praveenbysani.com
Re: Variable volume of messages across consumers
Posted by Praveen Bysani <pr...@gmail.com>.
Hi,
Thanks i added the log4j for all components under org.apache.activemq.
Hopefully i can see something useful.
On 12 July 2013 00:21, Christian Posta <ch...@gmail.com> wrote:
> Add logging configuration to turn on logging for the bridge.
>
> e.g.,
>
> to your log4j.properties, add
>
>
> log4j.logger.org.apache.activemq.network.DemandForwardingBridgeSupport=DEBUG
>
>
> On Thu, Jul 11, 2013 at 3:38 AM, Praveen Bysani <praveen.iiith@gmail.com
> >wrote:
>
> > Hi,
> >
> > I found the following lines in the subscriber log
> >
> > | org.apache.activemq.transport.InactivityMonitor | DEBUG | 2013-07-11
> > 07:34:44,384 | 30000 ms elapsed since last read check.
> > | org.apache.activemq.transport.InactivityMonitor | DEBUG | 2013-07-11
> > 07:34:44,384 |
> > org.apache.activemq.transport.InactivityMonitor$2@61726a5c9999
> > ms elapsed since last write check.
> >
> > I have disabled the inactivity period in the broker connection url by
> > setting maxInactivityDuration to 0. Does it have anything to do with
> > messages not being received ?
> >
> > On 11 July 2013 12:18, Praveen Bysani <pr...@gmail.com> wrote:
> >
> > > Hi Christian,
> > >
> > > Thanks for your input. But i have little idea of what those terms mean.
> > > Could you elaborate please.
> > >
> > >
> > > On 10 July 2013 21:06, Christian Posta <ch...@gmail.com>
> > wrote:
> > >
> > >> Your best bet would be to set logging to debug on the network bridge
> > >> (DemandForwardingBridgeSupport) and possibly the RegionBroker to see
> > what
> > >> is happening.
> > >>
> > >> On Wednesday, July 10, 2013, Praveen Bysani wrote:
> > >>
> > >> > Hi,
> > >> > I have the following setup in my project,
> > >> >
> > >> > 1. multiple producers (p1,p2,p3) in Hongkong region sending
> > variable
> > >> > size non-persistent messages to a topic t1 on the broker (b1) in
> > >> > Phillippines region
> > >> > 2. p1,p2 and p3 send messages frequently (~1 second)
> > >> > 3. each message has a time to live of 30 seconds
> > >> > 4. a tcp network bridge from b1 to a new broker b2 in HK region
> > >> > 5. consumers c1 in hongkong and c2 in europe subscribed to b2
> > through
> > >> > ssl and a consumer c3 subscribed to t1 on b1 through tcp
> > >> >
> > >> > I don't see the same amount of messages received across c1, c2 and
> c3.
> > >> The
> > >> > number of messages received is in the following order c3 > c1 > c2.
> I
> > >> use
> > >> > the default settings for prefetch values and *vmCursor *for
> > >> > *pendingSubscriberPolicy
> > >> > *and *strictOrderDisptachpolicy*.
> > >> >
> > >> > While i understand there could be delays in passing the message to
> > these
> > >> > consumers as c3 could be slow consumer and c1 is fast, shouldn't the
> > >> > message eventually be received to all consumers of a topic. Why do i
> > see
> > >> > different volumes of messages across different consumers ? Someone
> > >> please
> > >> > help me understand what is going on !!
> > >> >
> > >> > Another strange thing is after a while, there are no messages
> received
> > >> by
> > >> > c2 for a very long period. I suppose it is due to the prefetch size
> ?
> > >> Since
> > >> > i am using a non persistent topic, the default value is quite high
> > >> > (according to activemq documentation), which again baffles me if
> that
> > is
> > >> > really the reason.
> > >> > --
> > >> > Regards,
> > >> > Praveen Bysani
> > >> > http://www.praveenbysani.com
> > >> >
> > >>
> > >>
> > >> --
> > >> *Christian Posta*
> > >> http://www.christianposta.com/blog
> > >> twitter: @christianposta
> > >>
> > >
> > >
> > >
> > > --
> > > Regards,
> > > Praveen Bysani
> > > http://www.praveenbysani.com
> > >
> >
> >
> >
> > --
> > Regards,
> > Praveen Bysani
> > http://www.praveenbysani.com
> >
>
>
>
> --
> *Christian Posta*
> http://www.christianposta.com/blog
> twitter: @christianposta
>
--
Regards,
Praveen Bysani
http://www.praveenbysani.com
Re: Variable volume of messages across consumers
Posted by Christian Posta <ch...@gmail.com>.
Add logging configuration to turn on logging for the bridge.
e.g.,
to your log4j.properties, add
log4j.logger.org.apache.activemq.network.DemandForwardingBridgeSupport=DEBUG
On Thu, Jul 11, 2013 at 3:38 AM, Praveen Bysani <pr...@gmail.com>wrote:
> Hi,
>
> I found the following lines in the subscriber log
>
> | org.apache.activemq.transport.InactivityMonitor | DEBUG | 2013-07-11
> 07:34:44,384 | 30000 ms elapsed since last read check.
> | org.apache.activemq.transport.InactivityMonitor | DEBUG | 2013-07-11
> 07:34:44,384 |
> org.apache.activemq.transport.InactivityMonitor$2@61726a5c9999
> ms elapsed since last write check.
>
> I have disabled the inactivity period in the broker connection url by
> setting maxInactivityDuration to 0. Does it have anything to do with
> messages not being received ?
>
> On 11 July 2013 12:18, Praveen Bysani <pr...@gmail.com> wrote:
>
> > Hi Christian,
> >
> > Thanks for your input. But i have little idea of what those terms mean.
> > Could you elaborate please.
> >
> >
> > On 10 July 2013 21:06, Christian Posta <ch...@gmail.com>
> wrote:
> >
> >> Your best bet would be to set logging to debug on the network bridge
> >> (DemandForwardingBridgeSupport) and possibly the RegionBroker to see
> what
> >> is happening.
> >>
> >> On Wednesday, July 10, 2013, Praveen Bysani wrote:
> >>
> >> > Hi,
> >> > I have the following setup in my project,
> >> >
> >> > 1. multiple producers (p1,p2,p3) in Hongkong region sending
> variable
> >> > size non-persistent messages to a topic t1 on the broker (b1) in
> >> > Phillippines region
> >> > 2. p1,p2 and p3 send messages frequently (~1 second)
> >> > 3. each message has a time to live of 30 seconds
> >> > 4. a tcp network bridge from b1 to a new broker b2 in HK region
> >> > 5. consumers c1 in hongkong and c2 in europe subscribed to b2
> through
> >> > ssl and a consumer c3 subscribed to t1 on b1 through tcp
> >> >
> >> > I don't see the same amount of messages received across c1, c2 and c3.
> >> The
> >> > number of messages received is in the following order c3 > c1 > c2. I
> >> use
> >> > the default settings for prefetch values and *vmCursor *for
> >> > *pendingSubscriberPolicy
> >> > *and *strictOrderDisptachpolicy*.
> >> >
> >> > While i understand there could be delays in passing the message to
> these
> >> > consumers as c3 could be slow consumer and c1 is fast, shouldn't the
> >> > message eventually be received to all consumers of a topic. Why do i
> see
> >> > different volumes of messages across different consumers ? Someone
> >> please
> >> > help me understand what is going on !!
> >> >
> >> > Another strange thing is after a while, there are no messages received
> >> by
> >> > c2 for a very long period. I suppose it is due to the prefetch size ?
> >> Since
> >> > i am using a non persistent topic, the default value is quite high
> >> > (according to activemq documentation), which again baffles me if that
> is
> >> > really the reason.
> >> > --
> >> > Regards,
> >> > Praveen Bysani
> >> > http://www.praveenbysani.com
> >> >
> >>
> >>
> >> --
> >> *Christian Posta*
> >> http://www.christianposta.com/blog
> >> twitter: @christianposta
> >>
> >
> >
> >
> > --
> > Regards,
> > Praveen Bysani
> > http://www.praveenbysani.com
> >
>
>
>
> --
> Regards,
> Praveen Bysani
> http://www.praveenbysani.com
>
--
*Christian Posta*
http://www.christianposta.com/blog
twitter: @christianposta
Re: Variable volume of messages across consumers
Posted by Praveen Bysani <pr...@gmail.com>.
Hi,
I found the following lines in the subscriber log
| org.apache.activemq.transport.InactivityMonitor | DEBUG | 2013-07-11
07:34:44,384 | 30000 ms elapsed since last read check.
| org.apache.activemq.transport.InactivityMonitor | DEBUG | 2013-07-11
07:34:44,384 | org.apache.activemq.transport.InactivityMonitor$2@61726a5c9999
ms elapsed since last write check.
I have disabled the inactivity period in the broker connection url by
setting maxInactivityDuration to 0. Does it have anything to do with
messages not being received ?
On 11 July 2013 12:18, Praveen Bysani <pr...@gmail.com> wrote:
> Hi Christian,
>
> Thanks for your input. But i have little idea of what those terms mean.
> Could you elaborate please.
>
>
> On 10 July 2013 21:06, Christian Posta <ch...@gmail.com> wrote:
>
>> Your best bet would be to set logging to debug on the network bridge
>> (DemandForwardingBridgeSupport) and possibly the RegionBroker to see what
>> is happening.
>>
>> On Wednesday, July 10, 2013, Praveen Bysani wrote:
>>
>> > Hi,
>> > I have the following setup in my project,
>> >
>> > 1. multiple producers (p1,p2,p3) in Hongkong region sending variable
>> > size non-persistent messages to a topic t1 on the broker (b1) in
>> > Phillippines region
>> > 2. p1,p2 and p3 send messages frequently (~1 second)
>> > 3. each message has a time to live of 30 seconds
>> > 4. a tcp network bridge from b1 to a new broker b2 in HK region
>> > 5. consumers c1 in hongkong and c2 in europe subscribed to b2 through
>> > ssl and a consumer c3 subscribed to t1 on b1 through tcp
>> >
>> > I don't see the same amount of messages received across c1, c2 and c3.
>> The
>> > number of messages received is in the following order c3 > c1 > c2. I
>> use
>> > the default settings for prefetch values and *vmCursor *for
>> > *pendingSubscriberPolicy
>> > *and *strictOrderDisptachpolicy*.
>> >
>> > While i understand there could be delays in passing the message to these
>> > consumers as c3 could be slow consumer and c1 is fast, shouldn't the
>> > message eventually be received to all consumers of a topic. Why do i see
>> > different volumes of messages across different consumers ? Someone
>> please
>> > help me understand what is going on !!
>> >
>> > Another strange thing is after a while, there are no messages received
>> by
>> > c2 for a very long period. I suppose it is due to the prefetch size ?
>> Since
>> > i am using a non persistent topic, the default value is quite high
>> > (according to activemq documentation), which again baffles me if that is
>> > really the reason.
>> > --
>> > Regards,
>> > Praveen Bysani
>> > http://www.praveenbysani.com
>> >
>>
>>
>> --
>> *Christian Posta*
>> http://www.christianposta.com/blog
>> twitter: @christianposta
>>
>
>
>
> --
> Regards,
> Praveen Bysani
> http://www.praveenbysani.com
>
--
Regards,
Praveen Bysani
http://www.praveenbysani.com
Re: Variable volume of messages across consumers
Posted by Praveen Bysani <pr...@gmail.com>.
Hi Christian,
Thanks for your input. But i have little idea of what those terms mean.
Could you elaborate please.
On 10 July 2013 21:06, Christian Posta <ch...@gmail.com> wrote:
> Your best bet would be to set logging to debug on the network bridge
> (DemandForwardingBridgeSupport) and possibly the RegionBroker to see what
> is happening.
>
> On Wednesday, July 10, 2013, Praveen Bysani wrote:
>
> > Hi,
> > I have the following setup in my project,
> >
> > 1. multiple producers (p1,p2,p3) in Hongkong region sending variable
> > size non-persistent messages to a topic t1 on the broker (b1) in
> > Phillippines region
> > 2. p1,p2 and p3 send messages frequently (~1 second)
> > 3. each message has a time to live of 30 seconds
> > 4. a tcp network bridge from b1 to a new broker b2 in HK region
> > 5. consumers c1 in hongkong and c2 in europe subscribed to b2 through
> > ssl and a consumer c3 subscribed to t1 on b1 through tcp
> >
> > I don't see the same amount of messages received across c1, c2 and c3.
> The
> > number of messages received is in the following order c3 > c1 > c2. I use
> > the default settings for prefetch values and *vmCursor *for
> > *pendingSubscriberPolicy
> > *and *strictOrderDisptachpolicy*.
> >
> > While i understand there could be delays in passing the message to these
> > consumers as c3 could be slow consumer and c1 is fast, shouldn't the
> > message eventually be received to all consumers of a topic. Why do i see
> > different volumes of messages across different consumers ? Someone please
> > help me understand what is going on !!
> >
> > Another strange thing is after a while, there are no messages received
> by
> > c2 for a very long period. I suppose it is due to the prefetch size ?
> Since
> > i am using a non persistent topic, the default value is quite high
> > (according to activemq documentation), which again baffles me if that is
> > really the reason.
> > --
> > Regards,
> > Praveen Bysani
> > http://www.praveenbysani.com
> >
>
>
> --
> *Christian Posta*
> http://www.christianposta.com/blog
> twitter: @christianposta
>
--
Regards,
Praveen Bysani
http://www.praveenbysani.com
Re: Variable volume of messages across consumers
Posted by Christian Posta <ch...@gmail.com>.
Your best bet would be to set logging to debug on the network bridge
(DemandForwardingBridgeSupport) and possibly the RegionBroker to see what
is happening.
On Wednesday, July 10, 2013, Praveen Bysani wrote:
> Hi,
> I have the following setup in my project,
>
> 1. multiple producers (p1,p2,p3) in Hongkong region sending variable
> size non-persistent messages to a topic t1 on the broker (b1) in
> Phillippines region
> 2. p1,p2 and p3 send messages frequently (~1 second)
> 3. each message has a time to live of 30 seconds
> 4. a tcp network bridge from b1 to a new broker b2 in HK region
> 5. consumers c1 in hongkong and c2 in europe subscribed to b2 through
> ssl and a consumer c3 subscribed to t1 on b1 through tcp
>
> I don't see the same amount of messages received across c1, c2 and c3. The
> number of messages received is in the following order c3 > c1 > c2. I use
> the default settings for prefetch values and *vmCursor *for
> *pendingSubscriberPolicy
> *and *strictOrderDisptachpolicy*.
>
> While i understand there could be delays in passing the message to these
> consumers as c3 could be slow consumer and c1 is fast, shouldn't the
> message eventually be received to all consumers of a topic. Why do i see
> different volumes of messages across different consumers ? Someone please
> help me understand what is going on !!
>
> Another strange thing is after a while, there are no messages received by
> c2 for a very long period. I suppose it is due to the prefetch size ? Since
> i am using a non persistent topic, the default value is quite high
> (according to activemq documentation), which again baffles me if that is
> really the reason.
> --
> Regards,
> Praveen Bysani
> http://www.praveenbysani.com
>
--
*Christian Posta*
http://www.christianposta.com/blog
twitter: @christianposta