JMS (Tibco) Source - input messages status during transaction

[Roberto Coluccio <ro...@eng.it>]

Hi,

When leveraging a JMS (Tibco implementation) source to consume messages 
from a queue, are such messages immediately consumed , hence removed 
from the queue, even if the flume transaction hasn't been closed yet? 
What happens if the transaction is not committed (e.g. in case of errors)?

For instance, in a topology made by:

jms source -> memory or file channel -> hdfs sink

with a configured batch size of 100, when 99 of them are read from JMS 
and written to the channel, if an error occurs when trying to read the 
100th, to the best of my knowledge the whole batch of 100 is discarded, 
the transaction canceled, and the messages tried to be re-processed.

In such a scenario, are the first 99 messages already consumed from JMS 
to be considered as lost, supposing the ones in the channel are lost as 
well (because of other issues)?

thanks a lot in advance.

Best regards,
Roberto

Re: JMS (Tibco) Source - input messages status during transaction

[Attila Simon <sa...@cloudera.com>]

Indeed it can be stripped down to the question whether jms read
transactions are committed as a batch or as individual reads.

On Wednesday, 19 October 2016, Roberto Coluccio <ro...@eng.it>
wrote:

> Hi,
>
> When leveraging a JMS (Tibco implementation) source to consume messages
> from a queue, are such messages immediately consumed , hence removed from
> the queue, even if the flume transaction hasn't been closed yet? What
> happens if the transaction is not committed (e.g. in case of errors)?
>
> For instance, in a topology made by:
>
> jms source -> memory or file channel -> hdfs sink
>
> with a configured batch size of 100, when 99 of them are read from JMS and
> written to the channel, if an error occurs when trying to read the 100th,
> to the best of my knowledge the whole batch of 100 is discarded, the
> transaction canceled, and the messages tried to be re-processed.
>
> In such a scenario, are the first 99 messages already consumed from JMS to
> be considered as lost, supposing the ones in the channel are lost as well
> (because of other issues)?
>
> thanks a lot in advance.
>
> Best regards,
> Roberto
>


-- 

*Attila Simon*
Software Engineer
Email:   sati@cloudera.com

[image: Cloudera Inc.]