You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@storm.apache.org by jeremy p <at...@gmail.com> on 2015/03/31 20:41:15 UTC
Using Storm with IBM MQ Series
Hello all,
According to this webinar recap, Storm cannot work with IBM MQ Series (also
known as Websphere MQ) :
http://hortonworks.com/blog/discover-hdp-2-2-apache-kafka-apache-storm-stream-data-processing/
This is a real bummer for me, because my company uses IBM MQ Series, and we
have a new project that Storm would be perfect for. Is there currently an
effort underway to make IBM MQ Series interoperate with Storm? Do you
think it's even possible to make Storm and IBM MQ Series work together?
I'd really like to use Storm, if I can.
Re: Using Storm with IBM MQ Series
Posted by jeremy p <at...@gmail.com>.
All good. Thank you for your help! I really appreciate it. Please keep
us updated as to your findings. Likewise, I'll let you know if I uncover
anything.
--Jeremy
On Wed, Apr 1, 2015 at 11:37 AM, Parth Brahmbhatt <
pbrahmbhatt@hortonworks.com> wrote:
> Jeremy,
>
> Taylor and I work for the same company and the webinar contains the same
> info because it is coming from the same source. We will investigate further
> what caused the IBM-MQ integration failure but we can not promise a
> timeline. Once again apologies for providing inaccurate info in first place.
>
> Thanks
> Parth
>
> From: jeremy p <at...@gmail.com>
> Reply-To: user <us...@storm.apache.org>
> Date: Wednesday, April 1, 2015 at 8:19 AM
> To: user <us...@storm.apache.org>
>
> Subject: Re: Using Storm with IBM MQ Series
>
> Nathan : thanks for the response! It appears that the engineer who gave
> this webinar ran into the same problem as Parth :
>
> http://hortonworks.com/blog/discover-hdp-2-2-apache-kafka-apache-storm-stream-data-processing/
>
> He reports that he was unable to make MQ Series work with Storm because
> :
> "We ran into issues with IBM MQ-Series with respect to how messages are
> acknowledged by IBM MQ. IBM-MQ requires the thread that receives the
> message be the same thread that acks it. Storm’s framework cannot support
> this requirement, as the receiving and acking thread by design are
> different threads to achieve higher throughput."
>
> If threading wasn't the issue, how would you explain Storm's behavior in
> this situation?
>
> --Jeremy
>
> On Wed, Apr 1, 2015 at 12:11 AM, Parth Brahmbhatt <
> pbrahmbhatt@hortonworks.com> wrote:
>
>> Thanks for correcting me Nathan.
>>
>> Jeremy, in that case you can give storm-jms
>> <https://github.com/ptgoetz/storm-jms> a shot and see if it works for
>> you.
>>
>> Thanks
>> Parth
>>
>> From: Nathan Marz <na...@nathanmarz.com>
>> Reply-To: "user@storm.apache.org" <us...@storm.apache.org>
>> Date: Tuesday, March 31, 2015 at 6:11 PM
>> To: "user@storm.apache.org" <us...@storm.apache.org>
>> Subject: Re: Using Storm with IBM MQ Series
>>
>> That's not accurate. A spout task's nextTuple, ack, and fail methods
>> are all called by the exact same thread. You're confusing Storm acks with
>> acks that go back to the source message queue. Storm acks are about
>> detecting tuple DAG completion and have nothing to do with the ack to the
>> source message queue.
>>
>> So if you weren't able to get it to work then it was a different
>> problem.
>>
>> On Tue, Mar 31, 2015 at 2:48 PM, Parth Brahmbhatt <
>> pbrahmbhatt@hortonworks.com> wrote:
>>
>>> I tried this once and failed, following are my findings:
>>>
>>> IBM MQ has a strict limitation the thread that receives the message
>>> has to be the thread that acks it. This does not work with storm's
>>> threading model where one thread reads the message and sends it to the bolt
>>> another thread invokes spout's ack method when the bolt acks the message
>>> and only then the spout can ack the message.
>>>
>>> Based on
>>> http://www-01.ibm.com/support/knowledgecenter/SSFKSJ_8.0.0/com.ibm.mq.dev.doc/q031020_.htm?lang=en we
>>> tried looking into proprietary IBM MQ java client. The acking mechanism
>>> relies on following 3 APIS
>>>
>>>
>>> http://www-01.ibm.com/support/knowledgecenter/api/content/SSFKSJ_8.0.0/com.ibm.mq.javadoc.doc/WMQJavaClasses/com/ibm/mq/MQQueueManager.html#begin(
>>> )
>>>
>>> http://www-01.ibm.com/support/knowledgecenter/api/content/SSFKSJ_8.0.0/com.ibm.mq.javadoc.doc/WMQJavaClasses/com/ibm/mq/MQQueueManager.html#commit(
>>> )
>>>
>>> http://www-01.ibm.com/support/knowledgecenter/api/content/SSFKSJ_8.0.0/com.ibm.mq.javadoc.doc/WMQJavaClasses/com/ibm/mq/MQQueueManager.html#backout(
>>> )
>>>
>>> We could use these with one caveat. There is no way to cherry pick what
>>> messages, or range of messages like JMS where you can pick the message with
>>> lowest id, will get committed.
>>> This means in the spout we will keep an active list of pending messages,
>>> when ack method in spout is called for a message we remove the message from
>>> the pending list and call commit only if the pending list is empty. This
>>> works as long as the producer is slow or the consumers are slow but if both
>>> of them are really fast the pending list may never become empty causing lot
>>> of memory overhead on the box as well as server side overhead of keeping
>>> all the messages around.
>>> we could introduce a pseudo upper limit in spout where we stop handing
>>> out messages once pending message list reaches some upper bound but that
>>> will just slow down the topology.
>>>
>>> In summary, it seems to be impossible to implement a IBM MQ spout that
>>> does not run the risk of message loss and be performant.
>>>
>>>
>>> Thanks
>>>
>>> Parth
>>>
>>> From: jeremy p <at...@gmail.com>
>>> Reply-To: "user@storm.apache.org" <us...@storm.apache.org>
>>> Date: Tuesday, March 31, 2015 at 11:41 AM
>>> To: "user@storm.apache.org" <us...@storm.apache.org>
>>> Subject: Using Storm with IBM MQ Series
>>>
>>> Hello all,
>>>
>>> According to this webinar recap, Storm cannot work with IBM MQ Series
>>> (also known as Websphere MQ) :
>>>
>>> http://hortonworks.com/blog/discover-hdp-2-2-apache-kafka-apache-storm-stream-data-processing/
>>>
>>> This is a real bummer for me, because my company uses IBM MQ Series,
>>> and we have a new project that Storm would be perfect for. Is there
>>> currently an effort underway to make IBM MQ Series interoperate with
>>> Storm? Do you think it's even possible to make Storm and IBM MQ Series
>>> work together?
>>>
>>> I'd really like to use Storm, if I can.
>>>
>>
>>
>>
>> --
>> Twitter: @nathanmarz
>> http://nathanmarz.com
>>
>
>
Re: Using Storm with IBM MQ Series
Posted by jeremy p <at...@gmail.com>.
Hey Taylor : I would love to collaborate with you on this! However, I
don't have Storm up and running yet. Right now, I'm just investigating
whether or not we can use Storm for this project, or if I need to pick a
different technology. It sounds like we may be able to use Storm if we put
some development effort into it. Would it be alright if I reached out to
you once I have a Storm instance up and running?
On Wed, Apr 1, 2015 at 12:41 PM, P. Taylor Goetz <pt...@gmail.com> wrote:
> Memory slightly refreshed, but I don’t have access to an IBM-MQ instance
> at the moment, so I can’t verify…
>
> In the storm-jms spout[1], we receive a JMS message in the JMS
> `onMessage()` method (the thread that invokes that method is a JMS thread)
> and put it in a queue so we can perform the JMS message
> acknowledgement/session commit later when the spout `ack()` method is
> called (the thread that invokes that method is a Storm thread).
>
> I believe IBM-MQ’s JMS implementation was enforcing a policy such that the
> JMS message acknowledgement/commit must be called by the same thread that
> called the `onMessage()` method. Since the thread calling `ack()` (Storm
> thread) != the thread calling `onMessage()`, it violated that policy.
>
> Again, I could be wrong. I’d be happy to help if someone with access to an
> IBM-MQ instance would be interested in collaborating.
>
> -Taylor
>
> [1] https://github.com/ptgoetz/storm-jms
>
> On Apr 1, 2015, at 11:55 AM, P. Taylor Goetz <pt...@gmail.com> wrote:
>
> Hi Jeremy,
>
> I was the engineer who gave that webinar. And the statement you quoted is
> misleading (IIRC, that’s not the wording I used when the question was
> asked).
>
> Nathan is correct, the nextTuple(), ack() and fail() methods are all
> called from the same thread.
>
> At the moment I don’t have access to the specific errors that were
> encountered when using the storm-jms spout with IBM-MQ, but it was related
> that JMS implementation enforcing a strict policy around threads and JMS
> message acknowledgement (not Storm acks). I’ll try to dig up more details
> to refresh my memory and get back to you.
>
> -Taylor
>
>
> On Apr 1, 2015, at 11:37 AM, Parth Brahmbhatt <pb...@hortonworks.com>
> wrote:
>
> Jeremy,
>
> Taylor and I work for the same company and the webinar contains the same
> info because it is coming from the same source. We will investigate further
> what caused the IBM-MQ integration failure but we can not promise a
> timeline. Once again apologies for providing inaccurate info in first place.
>
> Thanks
> Parth
>
> From: jeremy p <at...@gmail.com>
> Reply-To: user <us...@storm.apache.org>
> Date: Wednesday, April 1, 2015 at 8:19 AM
> To: user <us...@storm.apache.org>
> Subject: Re: Using Storm with IBM MQ Series
>
> Nathan : thanks for the response! It appears that the engineer who gave
> this webinar ran into the same problem as Parth :
>
> http://hortonworks.com/blog/discover-hdp-2-2-apache-kafka-apache-storm-stream-data-processing/
>
> He reports that he was unable to make MQ Series work with Storm because
> :
> "We ran into issues with IBM MQ-Series with respect to how messages are
> acknowledged by IBM MQ. IBM-MQ requires the thread that receives the
> message be the same thread that acks it. Storm’s framework cannot support
> this requirement, as the receiving and acking thread by design are
> different threads to achieve higher throughput."
>
> If threading wasn't the issue, how would you explain Storm's behavior in
> this situation?
>
> --Jeremy
>
> On Wed, Apr 1, 2015 at 12:11 AM, Parth Brahmbhatt <
> pbrahmbhatt@hortonworks.com> wrote:
>
>> Thanks for correcting me Nathan.
>>
>> Jeremy, in that case you can give storm-jms
>> <https://github.com/ptgoetz/storm-jms> a shot and see if it works for
>> you.
>>
>> Thanks
>> Parth
>>
>> From: Nathan Marz <na...@nathanmarz.com>
>> Reply-To: "user@storm.apache.org" <us...@storm.apache.org>
>> Date: Tuesday, March 31, 2015 at 6:11 PM
>> To: "user@storm.apache.org" <us...@storm.apache.org>
>> Subject: Re: Using Storm with IBM MQ Series
>>
>> That's not accurate. A spout task's nextTuple, ack, and fail methods
>> are all called by the exact same thread. You're confusing Storm acks with
>> acks that go back to the source message queue. Storm acks are about
>> detecting tuple DAG completion and have nothing to do with the ack to the
>> source message queue.
>>
>> So if you weren't able to get it to work then it was a different
>> problem.
>>
>> On Tue, Mar 31, 2015 at 2:48 PM, Parth Brahmbhatt <
>> pbrahmbhatt@hortonworks.com> wrote:
>>
>>> I tried this once and failed, following are my findings:
>>>
>>> IBM MQ has a strict limitation the thread that receives the message
>>> has to be the thread that acks it. This does not work with storm's
>>> threading model where one thread reads the message and sends it to the bolt
>>> another thread invokes spout's ack method when the bolt acks the message
>>> and only then the spout can ack the message.
>>>
>>> Based on
>>> http://www-01.ibm.com/support/knowledgecenter/SSFKSJ_8.0.0/com.ibm.mq.dev.doc/q031020_.htm?lang=en we
>>> tried looking into proprietary IBM MQ java client. The acking mechanism
>>> relies on following 3 APIS
>>>
>>>
>>> http://www-01.ibm.com/support/knowledgecenter/api/content/SSFKSJ_8.0.0/com.ibm.mq.javadoc.doc/WMQJavaClasses/com/ibm/mq/MQQueueManager.html#begin(
>>> )
>>>
>>> http://www-01.ibm.com/support/knowledgecenter/api/content/SSFKSJ_8.0.0/com.ibm.mq.javadoc.doc/WMQJavaClasses/com/ibm/mq/MQQueueManager.html#commit(
>>> )
>>>
>>> http://www-01.ibm.com/support/knowledgecenter/api/content/SSFKSJ_8.0.0/com.ibm.mq.javadoc.doc/WMQJavaClasses/com/ibm/mq/MQQueueManager.html#backout(
>>> )
>>>
>>> We could use these with one caveat. There is no way to cherry pick what
>>> messages, or range of messages like JMS where you can pick the message with
>>> lowest id, will get committed.
>>> This means in the spout we will keep an active list of pending messages,
>>> when ack method in spout is called for a message we remove the message from
>>> the pending list and call commit only if the pending list is empty. This
>>> works as long as the producer is slow or the consumers are slow but if both
>>> of them are really fast the pending list may never become empty causing lot
>>> of memory overhead on the box as well as server side overhead of keeping
>>> all the messages around.
>>> we could introduce a pseudo upper limit in spout where we stop handing
>>> out messages once pending message list reaches some upper bound but that
>>> will just slow down the topology.
>>>
>>> In summary, it seems to be impossible to implement a IBM MQ spout that
>>> does not run the risk of message loss and be performant.
>>>
>>>
>>> Thanks
>>>
>>> Parth
>>>
>>> From: jeremy p <at...@gmail.com>
>>> Reply-To: "user@storm.apache.org" <us...@storm.apache.org>
>>> Date: Tuesday, March 31, 2015 at 11:41 AM
>>> To: "user@storm.apache.org" <us...@storm.apache.org>
>>> Subject: Using Storm with IBM MQ Series
>>>
>>> Hello all,
>>>
>>> According to this webinar recap, Storm cannot work with IBM MQ Series
>>> (also known as Websphere MQ) :
>>>
>>> http://hortonworks.com/blog/discover-hdp-2-2-apache-kafka-apache-storm-stream-data-processing/
>>>
>>> This is a real bummer for me, because my company uses IBM MQ Series,
>>> and we have a new project that Storm would be perfect for. Is there
>>> currently an effort underway to make IBM MQ Series interoperate with
>>> Storm? Do you think it's even possible to make Storm and IBM MQ Series
>>> work together?
>>>
>>> I'd really like to use Storm, if I can.
>>>
>>
>>
>>
>> --
>> Twitter: @nathanmarz
>> http://nathanmarz.com
>>
>
>
>
>
Re: Using Storm with IBM MQ Series
Posted by "P. Taylor Goetz" <pt...@gmail.com>.
Memory slightly refreshed, but I don’t have access to an IBM-MQ instance at the moment, so I can’t verify…
In the storm-jms spout[1], we receive a JMS message in the JMS `onMessage()` method (the thread that invokes that method is a JMS thread) and put it in a queue so we can perform the JMS message acknowledgement/session commit later when the spout `ack()` method is called (the thread that invokes that method is a Storm thread).
I believe IBM-MQ’s JMS implementation was enforcing a policy such that the JMS message acknowledgement/commit must be called by the same thread that called the `onMessage()` method. Since the thread calling `ack()` (Storm thread) != the thread calling `onMessage()`, it violated that policy.
Again, I could be wrong. I’d be happy to help if someone with access to an IBM-MQ instance would be interested in collaborating.
-Taylor
[1] https://github.com/ptgoetz/storm-jms
On Apr 1, 2015, at 11:55 AM, P. Taylor Goetz <pt...@gmail.com> wrote:
> Hi Jeremy,
>
> I was the engineer who gave that webinar. And the statement you quoted is misleading (IIRC, that’s not the wording I used when the question was asked).
>
> Nathan is correct, the nextTuple(), ack() and fail() methods are all called from the same thread.
>
> At the moment I don’t have access to the specific errors that were encountered when using the storm-jms spout with IBM-MQ, but it was related that JMS implementation enforcing a strict policy around threads and JMS message acknowledgement (not Storm acks). I’ll try to dig up more details to refresh my memory and get back to you.
>
> -Taylor
>
>
> On Apr 1, 2015, at 11:37 AM, Parth Brahmbhatt <pb...@hortonworks.com> wrote:
>
>> Jeremy,
>>
>> Taylor and I work for the same company and the webinar contains the same info because it is coming from the same source. We will investigate further what caused the IBM-MQ integration failure but we can not promise a timeline. Once again apologies for providing inaccurate info in first place.
>>
>> Thanks
>> Parth
>>
>> From: jeremy p <at...@gmail.com>
>> Reply-To: user <us...@storm.apache.org>
>> Date: Wednesday, April 1, 2015 at 8:19 AM
>> To: user <us...@storm.apache.org>
>> Subject: Re: Using Storm with IBM MQ Series
>>
>> Nathan : thanks for the response! It appears that the engineer who gave this webinar ran into the same problem as Parth :
>> http://hortonworks.com/blog/discover-hdp-2-2-apache-kafka-apache-storm-stream-data-processing/
>>
>> He reports that he was unable to make MQ Series work with Storm because :
>> "We ran into issues with IBM MQ-Series with respect to how messages are acknowledged by IBM MQ. IBM-MQ requires the thread that receives the message be the same thread that acks it. Storm’s framework cannot support this requirement, as the receiving and acking thread by design are different threads to achieve higher throughput."
>>
>> If threading wasn't the issue, how would you explain Storm's behavior in this situation?
>>
>> --Jeremy
>>
>> On Wed, Apr 1, 2015 at 12:11 AM, Parth Brahmbhatt <pb...@hortonworks.com> wrote:
>> Thanks for correcting me Nathan.
>>
>> Jeremy, in that case you can give storm-jms a shot and see if it works for you.
>>
>> Thanks
>> Parth
>>
>> From: Nathan Marz <na...@nathanmarz.com>
>> Reply-To: "user@storm.apache.org" <us...@storm.apache.org>
>> Date: Tuesday, March 31, 2015 at 6:11 PM
>> To: "user@storm.apache.org" <us...@storm.apache.org>
>> Subject: Re: Using Storm with IBM MQ Series
>>
>> That's not accurate. A spout task's nextTuple, ack, and fail methods are all called by the exact same thread. You're confusing Storm acks with acks that go back to the source message queue. Storm acks are about detecting tuple DAG completion and have nothing to do with the ack to the source message queue.
>>
>> So if you weren't able to get it to work then it was a different problem.
>>
>> On Tue, Mar 31, 2015 at 2:48 PM, Parth Brahmbhatt <pb...@hortonworks.com> wrote:
>> I tried this once and failed, following are my findings:
>>
>> IBM MQ has a strict limitation the thread that receives the message has to be the thread that acks it. This does not work with storm's threading model where one thread reads the message and sends it to the bolt another thread invokes spout's ack method when the bolt acks the message and only then the spout can ack the message.
>> Based on http://www-01.ibm.com/support/knowledgecenter/SSFKSJ_8.0.0/com.ibm.mq.dev.doc/q031020_.htm?lang=en we tried looking into proprietary IBM MQ java client. The acking mechanism relies on following 3 APIS
>> http://www-01.ibm.com/support/knowledgecenter/api/content/SSFKSJ_8.0.0/com.ibm.mq.javadoc.doc/WMQJavaClasses/com/ibm/mq/MQQueueManager.html#begin()
>> http://www-01.ibm.com/support/knowledgecenter/api/content/SSFKSJ_8.0.0/com.ibm.mq.javadoc.doc/WMQJavaClasses/com/ibm/mq/MQQueueManager.html#commit()
>> http://www-01.ibm.com/support/knowledgecenter/api/content/SSFKSJ_8.0.0/com.ibm.mq.javadoc.doc/WMQJavaClasses/com/ibm/mq/MQQueueManager.html#backout()
>> We could use these with one caveat. There is no way to cherry pick what messages, or range of messages like JMS where you can pick the message with lowest id, will get committed.
>> This means in the spout we will keep an active list of pending messages, when ack method in spout is called for a message we remove the message from the pending list and call commit only if the pending list is empty. This works as long as the producer is slow or the consumers are slow but if both of them are really fast the pending list may never become empty causing lot of memory overhead on the box as well as server side overhead of keeping all the messages around.
>> we could introduce a pseudo upper limit in spout where we stop handing out messages once pending message list reaches some upper bound but that will just slow down the topology.
>> In summary, it seems to be impossible to implement a IBM MQ spout that does not run the risk of message loss and be performant.
>>
>> Thanks
>> Parth
>>
>> From: jeremy p <at...@gmail.com>
>> Reply-To: "user@storm.apache.org" <us...@storm.apache.org>
>> Date: Tuesday, March 31, 2015 at 11:41 AM
>> To: "user@storm.apache.org" <us...@storm.apache.org>
>> Subject: Using Storm with IBM MQ Series
>>
>> Hello all,
>>
>> According to this webinar recap, Storm cannot work with IBM MQ Series (also known as Websphere MQ) :
>> http://hortonworks.com/blog/discover-hdp-2-2-apache-kafka-apache-storm-stream-data-processing/
>>
>> This is a real bummer for me, because my company uses IBM MQ Series, and we have a new project that Storm would be perfect for. Is there currently an effort underway to make IBM MQ Series interoperate with Storm? Do you think it's even possible to make Storm and IBM MQ Series work together?
>>
>> I'd really like to use Storm, if I can.
>>
>>
>>
>> --
>> Twitter: @nathanmarz
>> http://nathanmarz.com
>>
>
Re: Using Storm with IBM MQ Series
Posted by "P. Taylor Goetz" <pt...@gmail.com>.
Hi Jeremy,
I was the engineer who gave that webinar. And the statement you quoted is misleading (IIRC, that’s not the wording I used when the question was asked).
Nathan is correct, the nextTuple(), ack() and fail() methods are all called from the same thread.
At the moment I don’t have access to the specific errors that were encountered when using the storm-jms spout with IBM-MQ, but it was related that JMS implementation enforcing a strict policy around threads and JMS message acknowledgement (not Storm acks). I’ll try to dig up more details to refresh my memory and get back to you.
-Taylor
On Apr 1, 2015, at 11:37 AM, Parth Brahmbhatt <pb...@hortonworks.com> wrote:
> Jeremy,
>
> Taylor and I work for the same company and the webinar contains the same info because it is coming from the same source. We will investigate further what caused the IBM-MQ integration failure but we can not promise a timeline. Once again apologies for providing inaccurate info in first place.
>
> Thanks
> Parth
>
> From: jeremy p <at...@gmail.com>
> Reply-To: user <us...@storm.apache.org>
> Date: Wednesday, April 1, 2015 at 8:19 AM
> To: user <us...@storm.apache.org>
> Subject: Re: Using Storm with IBM MQ Series
>
> Nathan : thanks for the response! It appears that the engineer who gave this webinar ran into the same problem as Parth :
> http://hortonworks.com/blog/discover-hdp-2-2-apache-kafka-apache-storm-stream-data-processing/
>
> He reports that he was unable to make MQ Series work with Storm because :
> "We ran into issues with IBM MQ-Series with respect to how messages are acknowledged by IBM MQ. IBM-MQ requires the thread that receives the message be the same thread that acks it. Storm’s framework cannot support this requirement, as the receiving and acking thread by design are different threads to achieve higher throughput."
>
> If threading wasn't the issue, how would you explain Storm's behavior in this situation?
>
> --Jeremy
>
> On Wed, Apr 1, 2015 at 12:11 AM, Parth Brahmbhatt <pb...@hortonworks.com> wrote:
> Thanks for correcting me Nathan.
>
> Jeremy, in that case you can give storm-jms a shot and see if it works for you.
>
> Thanks
> Parth
>
> From: Nathan Marz <na...@nathanmarz.com>
> Reply-To: "user@storm.apache.org" <us...@storm.apache.org>
> Date: Tuesday, March 31, 2015 at 6:11 PM
> To: "user@storm.apache.org" <us...@storm.apache.org>
> Subject: Re: Using Storm with IBM MQ Series
>
> That's not accurate. A spout task's nextTuple, ack, and fail methods are all called by the exact same thread. You're confusing Storm acks with acks that go back to the source message queue. Storm acks are about detecting tuple DAG completion and have nothing to do with the ack to the source message queue.
>
> So if you weren't able to get it to work then it was a different problem.
>
> On Tue, Mar 31, 2015 at 2:48 PM, Parth Brahmbhatt <pb...@hortonworks.com> wrote:
> I tried this once and failed, following are my findings:
>
> IBM MQ has a strict limitation the thread that receives the message has to be the thread that acks it. This does not work with storm's threading model where one thread reads the message and sends it to the bolt another thread invokes spout's ack method when the bolt acks the message and only then the spout can ack the message.
> Based on http://www-01.ibm.com/support/knowledgecenter/SSFKSJ_8.0.0/com.ibm.mq.dev.doc/q031020_.htm?lang=en we tried looking into proprietary IBM MQ java client. The acking mechanism relies on following 3 APIS
> http://www-01.ibm.com/support/knowledgecenter/api/content/SSFKSJ_8.0.0/com.ibm.mq.javadoc.doc/WMQJavaClasses/com/ibm/mq/MQQueueManager.html#begin()
> http://www-01.ibm.com/support/knowledgecenter/api/content/SSFKSJ_8.0.0/com.ibm.mq.javadoc.doc/WMQJavaClasses/com/ibm/mq/MQQueueManager.html#commit()
> http://www-01.ibm.com/support/knowledgecenter/api/content/SSFKSJ_8.0.0/com.ibm.mq.javadoc.doc/WMQJavaClasses/com/ibm/mq/MQQueueManager.html#backout()
> We could use these with one caveat. There is no way to cherry pick what messages, or range of messages like JMS where you can pick the message with lowest id, will get committed.
> This means in the spout we will keep an active list of pending messages, when ack method in spout is called for a message we remove the message from the pending list and call commit only if the pending list is empty. This works as long as the producer is slow or the consumers are slow but if both of them are really fast the pending list may never become empty causing lot of memory overhead on the box as well as server side overhead of keeping all the messages around.
> we could introduce a pseudo upper limit in spout where we stop handing out messages once pending message list reaches some upper bound but that will just slow down the topology.
> In summary, it seems to be impossible to implement a IBM MQ spout that does not run the risk of message loss and be performant.
>
> Thanks
> Parth
>
> From: jeremy p <at...@gmail.com>
> Reply-To: "user@storm.apache.org" <us...@storm.apache.org>
> Date: Tuesday, March 31, 2015 at 11:41 AM
> To: "user@storm.apache.org" <us...@storm.apache.org>
> Subject: Using Storm with IBM MQ Series
>
> Hello all,
>
> According to this webinar recap, Storm cannot work with IBM MQ Series (also known as Websphere MQ) :
> http://hortonworks.com/blog/discover-hdp-2-2-apache-kafka-apache-storm-stream-data-processing/
>
> This is a real bummer for me, because my company uses IBM MQ Series, and we have a new project that Storm would be perfect for. Is there currently an effort underway to make IBM MQ Series interoperate with Storm? Do you think it's even possible to make Storm and IBM MQ Series work together?
>
> I'd really like to use Storm, if I can.
>
>
>
> --
> Twitter: @nathanmarz
> http://nathanmarz.com
>
Re: Using Storm with IBM MQ Series
Posted by Parth Brahmbhatt <pb...@hortonworks.com>.
Jeremy,
Taylor and I work for the same company and the webinar contains the same info because it is coming from the same source. We will investigate further what caused the IBM-MQ integration failure but we can not promise a timeline. Once again apologies for providing inaccurate info in first place.
Thanks
Parth
From: jeremy p <at...@gmail.com>>
Reply-To: user <us...@storm.apache.org>>
Date: Wednesday, April 1, 2015 at 8:19 AM
To: user <us...@storm.apache.org>>
Subject: Re: Using Storm with IBM MQ Series
Nathan : thanks for the response! It appears that the engineer who gave this webinar ran into the same problem as Parth :
http://hortonworks.com/blog/discover-hdp-2-2-apache-kafka-apache-storm-stream-data-processing/
He reports that he was unable to make MQ Series work with Storm because :
"We ran into issues with IBM MQ-Series with respect to how messages are acknowledged by IBM MQ. IBM-MQ requires the thread that receives the message be the same thread that acks it. Storm’s framework cannot support this requirement, as the receiving and acking thread by design are different threads to achieve higher throughput."
If threading wasn't the issue, how would you explain Storm's behavior in this situation?
--Jeremy
On Wed, Apr 1, 2015 at 12:11 AM, Parth Brahmbhatt <pb...@hortonworks.com>> wrote:
Thanks for correcting me Nathan.
Jeremy, in that case you can give storm-jms<https://github.com/ptgoetz/storm-jms> a shot and see if it works for you.
Thanks
Parth
From: Nathan Marz <na...@nathanmarz.com>>
Reply-To: "user@storm.apache.org<ma...@storm.apache.org>" <us...@storm.apache.org>>
Date: Tuesday, March 31, 2015 at 6:11 PM
To: "user@storm.apache.org<ma...@storm.apache.org>" <us...@storm.apache.org>>
Subject: Re: Using Storm with IBM MQ Series
That's not accurate. A spout task's nextTuple, ack, and fail methods are all called by the exact same thread. You're confusing Storm acks with acks that go back to the source message queue. Storm acks are about detecting tuple DAG completion and have nothing to do with the ack to the source message queue.
So if you weren't able to get it to work then it was a different problem.
On Tue, Mar 31, 2015 at 2:48 PM, Parth Brahmbhatt <pb...@hortonworks.com>> wrote:
I tried this once and failed, following are my findings:
IBM MQ has a strict limitation the thread that receives the message has to be the thread that acks it. This does not work with storm's threading model where one thread reads the message and sends it to the bolt another thread invokes spout's ack method when the bolt acks the message and only then the spout can ack the message.
Based on http://www-01.ibm.com/support/knowledgecenter/SSFKSJ_8.0.0/com.ibm.mq.dev.doc/q031020_.htm?lang=en we tried looking into proprietary IBM MQ java client. The acking mechanism relies on following 3 APIS
http://www-01.ibm.com/support/knowledgecenter/api/content/SSFKSJ_8.0.0/com.ibm.mq.javadoc.doc/WMQJavaClasses/com/ibm/mq/MQQueueManager.html#begin()
http://www-01.ibm.com/support/knowledgecenter/api/content/SSFKSJ_8.0.0/com.ibm.mq.javadoc.doc/WMQJavaClasses/com/ibm/mq/MQQueueManager.html#commit()
http://www-01.ibm.com/support/knowledgecenter/api/content/SSFKSJ_8.0.0/com.ibm.mq.javadoc.doc/WMQJavaClasses/com/ibm/mq/MQQueueManager.html#backout()
We could use these with one caveat. There is no way to cherry pick what messages, or range of messages like JMS where you can pick the message with lowest id, will get committed.
This means in the spout we will keep an active list of pending messages, when ack method in spout is called for a message we remove the message from the pending list and call commit only if the pending list is empty. This works as long as the producer is slow or the consumers are slow but if both of them are really fast the pending list may never become empty causing lot of memory overhead on the box as well as server side overhead of keeping all the messages around.
we could introduce a pseudo upper limit in spout where we stop handing out messages once pending message list reaches some upper bound but that will just slow down the topology.
In summary, it seems to be impossible to implement a IBM MQ spout that does not run the risk of message loss and be performant.
Thanks
Parth
From: jeremy p <at...@gmail.com>>
Reply-To: "user@storm.apache.org<ma...@storm.apache.org>" <us...@storm.apache.org>>
Date: Tuesday, March 31, 2015 at 11:41 AM
To: "user@storm.apache.org<ma...@storm.apache.org>" <us...@storm.apache.org>>
Subject: Using Storm with IBM MQ Series
Hello all,
According to this webinar recap, Storm cannot work with IBM MQ Series (also known as Websphere MQ) :
http://hortonworks.com/blog/discover-hdp-2-2-apache-kafka-apache-storm-stream-data-processing/
This is a real bummer for me, because my company uses IBM MQ Series, and we have a new project that Storm would be perfect for. Is there currently an effort underway to make IBM MQ Series interoperate with Storm? Do you think it's even possible to make Storm and IBM MQ Series work together?
I'd really like to use Storm, if I can.
--
Twitter: @nathanmarz
http://nathanmarz.com<http://nathanmarz.com/>
Re: Using Storm with IBM MQ Series
Posted by jeremy p <at...@gmail.com>.
Thank you, Alesky. A couple of questions :
1) How well do MQ Series and Storm work together? Have you run into any
performance issues? You mentioned that you are not happy with the current
solution. What do you dislike about it?
2) You mention that you need to scale topology throughput adding more
spouts instead of using TOPOLOGY_MAX_SPOUT_PENDING. Could you please
explain what this means? Does it make a difference to performance?
Apologies for my lack of knowledge -- I am still new to Storm.
3) It sounds like you wrote some code to adapt MQ Series to Storm. Could
you please share your code?
Thanks again for the help.
--Jeremy
On Wed, Apr 1, 2015 at 12:01 PM, <al...@ubs.com> wrote:
> Hello Parth,
>
>
>
> We are using JMS API, means in spout I use Connection, Session,
> Destination, MessageConsumer, etc. Implementation is MQ provided, and, as I
> know, it's not open source.
>
>
>
> Thanks,
>
> Aleksey.
>
>
>
> *From:* Parth Brahmbhatt [mailto:pbrahmbhatt@hortonworks.com]
> *Sent:* Wednesday, April 01, 2015 11:53 AM
> *To:* user@storm.apache.org
>
> *Subject:* Re: Using Storm with IBM MQ Series
>
>
>
> Hi Aleksey,
>
>
>
> I am guessing you are using proprietary IBM-MQ Apis instead of using the
> JMS API. Is that right? Is this code open sourced?
>
>
>
> Thanks
>
> Parth
>
>
>
> *From: *"aleksey.didik@ubs.com" <al...@ubs.com>
> *Reply-To: *"user@storm.apache.org" <us...@storm.apache.org>
> *Date: *Wednesday, April 1, 2015 at 8:48 AM
> *To: *"user@storm.apache.org" <us...@storm.apache.org>
> *Subject: *RE: Using Storm with IBM MQ Series
>
>
>
> Hello Jeremy,
>
>
>
> I'm using Storm with IBM MQ for my project a half of a year.
>
>
>
> In MQ you have no per message acknowledgment – that the problem. You
> could ack all messages received in current session only altogether. That
> makes Parth to make his trick with pending list committing only when list
> is empty.
>
> We did we do – made one spout process one message only at the moment:
>
> 1. Create TRANSACTED session in spout
>
> 2. Create one consumer for this session in spout.
>
> 3. Receive message.
>
> 4. Emit to bolts.
>
> 5. When get ack – commit session.
>
> 6. When get fail – rollback session.
>
> 7. Start once again from point 2.
>
>
>
> One session has one message in that case, so we have per message ack.
>
> It makes us to scale topology throughput adding more spouts instead of
> using TOPOLOGY_MAX_SPOUT_PENDING. Personally myself don't like this
> solution, but I have nothing better now.
>
>
>
> I have never met limitation of different threads in current solution.
>
>
>
> HTH,
>
> Aleksey.
>
>
>
> *From:* jeremy p [mailto:athomewithagroovebox@gmail.com
> <at...@gmail.com>]
> *Sent:* Wednesday, April 01, 2015 11:19 AM
> *To:* user
> *Subject:* Re: Using Storm with IBM MQ Series
>
>
>
> Nathan : thanks for the response! It appears that the engineer who gave
> this webinar ran into the same problem as Parth :
>
>
> http://hortonworks.com/blog/discover-hdp-2-2-apache-kafka-apache-storm-stream-data-processing/
>
>
>
> He reports that he was unable to make MQ Series work with Storm because :
>
> "We ran into issues with IBM MQ-Series with respect to how messages are
> acknowledged by IBM MQ. IBM-MQ requires the thread that receives the
> message be the same thread that acks it. Storm’s framework cannot support
> this requirement, as the receiving and acking thread by design are
> different threads to achieve higher throughput."
>
>
>
> If threading wasn't the issue, how would you explain Storm's behavior in
> this situation?
>
>
>
> --Jeremy
>
>
>
> On Wed, Apr 1, 2015 at 12:11 AM, Parth Brahmbhatt <
> pbrahmbhatt@hortonworks.com> wrote:
>
> Thanks for correcting me Nathan.
>
>
>
> Jeremy, in that case you can give storm-jms
> <https://github.com/ptgoetz/storm-jms> a shot and see if it works for
> you.
>
>
>
> Thanks
>
> Parth
>
>
>
> *From: *Nathan Marz <na...@nathanmarz.com>
> *Reply-To: *"user@storm.apache.org" <us...@storm.apache.org>
> *Date: *Tuesday, March 31, 2015 at 6:11 PM
> *To: *"user@storm.apache.org" <us...@storm.apache.org>
> *Subject: *Re: Using Storm with IBM MQ Series
>
>
>
> That's not accurate. A spout task's nextTuple, ack, and fail methods are
> all called by the exact same thread. You're confusing Storm acks with acks
> that go back to the source message queue. Storm acks are about detecting
> tuple DAG completion and have nothing to do with the ack to the source
> message queue.
>
>
>
> So if you weren't able to get it to work then it was a different problem.
>
>
>
> On Tue, Mar 31, 2015 at 2:48 PM, Parth Brahmbhatt <
> pbrahmbhatt@hortonworks.com> wrote:
>
> I tried this once and failed, following are my findings:
>
>
>
> IBM MQ has a strict limitation the thread that receives the message has to
> be the thread that acks it. This does not work with storm's threading model
> where one thread reads the message and sends it to the bolt another thread
> invokes spout's ack method when the bolt acks the message and only then the
> spout can ack the message.
>
> Based on
> http://www-01.ibm.com/support/knowledgecenter/SSFKSJ_8.0.0/com.ibm.mq.dev.doc/q031020_.htm?lang=en we
> tried looking into proprietary IBM MQ java client. The acking mechanism
> relies on following 3 APIS
>
>
> http://www-01.ibm.com/support/knowledgecenter/api/content/SSFKSJ_8.0.0/com.ibm.mq.javadoc.doc/WMQJavaClasses/com/ibm/mq/MQQueueManager.html#begin(
> )
>
> http://www-01.ibm.com/support/knowledgecenter/api/content/SSFKSJ_8.0.0/com.ibm.mq.javadoc.doc/WMQJavaClasses/com/ibm/mq/MQQueueManager.html#commit(
> )
>
> http://www-01.ibm.com/support/knowledgecenter/api/content/SSFKSJ_8.0.0/com.ibm.mq.javadoc.doc/WMQJavaClasses/com/ibm/mq/MQQueueManager.html#backout(
> )
>
> We could use these with one caveat. There is no way to cherry pick what
> messages, or range of messages like JMS where you can pick the message with
> lowest id, will get committed.
> This means in the spout we will keep an active list of pending messages,
> when ack method in spout is called for a message we remove the message from
> the pending list and call commit only if the pending list is empty. This
> works as long as the producer is slow or the consumers are slow but if both
> of them are really fast the pending list may never become empty causing lot
> of memory overhead on the box as well as server side overhead of keeping
> all the messages around.
> we could introduce a pseudo upper limit in spout where we stop handing out
> messages once pending message list reaches some upper bound but that will
> just slow down the topology.
>
> In summary, it seems to be impossible to implement a IBM MQ spout that
> does not run the risk of message loss and be performant.
>
>
>
> Thanks
>
> Parth
>
>
>
> *From: *jeremy p <at...@gmail.com>
> *Reply-To: *"user@storm.apache.org" <us...@storm.apache.org>
> *Date: *Tuesday, March 31, 2015 at 11:41 AM
> *To: *"user@storm.apache.org" <us...@storm.apache.org>
> *Subject: *Using Storm with IBM MQ Series
>
>
>
> Hello all,
>
>
>
> According to this webinar recap, Storm cannot work with IBM MQ Series
> (also known as Websphere MQ) :
>
>
> http://hortonworks.com/blog/discover-hdp-2-2-apache-kafka-apache-storm-stream-data-processing/
>
>
>
> This is a real bummer for me, because my company uses IBM MQ Series, and
> we have a new project that Storm would be perfect for. Is there currently
> an effort underway to make IBM MQ Series interoperate with Storm? Do you
> think it's even possible to make Storm and IBM MQ Series work together?
>
>
>
> I'd really like to use Storm, if I can.
>
>
>
>
>
> --
>
> Twitter: @nathanmarz
>
> http://nathanmarz.com
>
>
>
> Please visit our website at
> http://financialservicesinc.ubs.com/wealth/E-maildisclaimer.html
> for important disclosures and information about our e-mail
> policies. For your protection, please do not transmit orders
> or instructions by e-mail or include account numbers, Social
> Security numbers, credit card numbers, passwords, or other
> personal information.
>
RE: Using Storm with IBM MQ Series
Posted by al...@ubs.com.
Hello Parth,
We are using JMS API, means in spout I use Connection, Session,
Destination, MessageConsumer, etc. Implementation is MQ provided, and,
as I know, it's not open source.
Thanks,
Aleksey.
From: Parth Brahmbhatt [mailto:pbrahmbhatt@hortonworks.com]
Sent: Wednesday, April 01, 2015 11:53 AM
To: user@storm.apache.org
Subject: Re: Using Storm with IBM MQ Series
Hi Aleksey,
I am guessing you are using proprietary IBM-MQ Apis instead of using the
JMS API. Is that right? Is this code open sourced?
Thanks
Parth
From: "aleksey.didik@ubs.com" <al...@ubs.com>
Reply-To: "user@storm.apache.org" <us...@storm.apache.org>
Date: Wednesday, April 1, 2015 at 8:48 AM
To: "user@storm.apache.org" <us...@storm.apache.org>
Subject: RE: Using Storm with IBM MQ Series
Hello Jeremy,
I'm using Storm with IBM MQ for my project a half of a year.
In MQ you have no per message acknowledgment - that the problem. You
could ack all messages received in current session only altogether.
That makes Parth to make his trick with pending list committing only
when list is empty.
We did we do - made one spout process one message only at the moment:
1. Create TRANSACTED session in spout
2. Create one consumer for this session in spout.
3. Receive message.
4. Emit to bolts.
5. When get ack - commit session.
6. When get fail - rollback session.
7. Start once again from point 2.
One session has one message in that case, so we have per message ack.
It makes us to scale topology throughput adding more spouts instead of
using TOPOLOGY_MAX_SPOUT_PENDING. Personally myself don't like this
solution, but I have nothing better now.
I have never met limitation of different threads in current solution.
HTH,
Aleksey.
From: jeremy p [mailto:athomewithagroovebox@gmail.com]
Sent: Wednesday, April 01, 2015 11:19 AM
To: user
Subject: Re: Using Storm with IBM MQ Series
Nathan : thanks for the response! It appears that the engineer who gave
this webinar ran into the same problem as Parth :
http://hortonworks.com/blog/discover-hdp-2-2-apache-kafka-apache-storm-s
tream-data-processing/
He reports that he was unable to make MQ Series work with Storm because
:
"We ran into issues with IBM MQ-Series with respect to how messages are
acknowledged by IBM MQ. IBM-MQ requires the thread that receives the
message be the same thread that acks it. Storm's framework cannot
support this requirement, as the receiving and acking thread by design
are different threads to achieve higher throughput."
If threading wasn't the issue, how would you explain Storm's behavior in
this situation?
--Jeremy
On Wed, Apr 1, 2015 at 12:11 AM, Parth Brahmbhatt
<pb...@hortonworks.com> wrote:
Thanks for correcting me Nathan.
Jeremy, in that case you can give storm-jms
<https://github.com/ptgoetz/storm-jms> a shot and see if it works for
you.
Thanks
Parth
From: Nathan Marz <na...@nathanmarz.com>
Reply-To: "user@storm.apache.org" <us...@storm.apache.org>
Date: Tuesday, March 31, 2015 at 6:11 PM
To: "user@storm.apache.org" <us...@storm.apache.org>
Subject: Re: Using Storm with IBM MQ Series
That's not accurate. A spout task's nextTuple, ack, and fail methods are
all called by the exact same thread. You're confusing Storm acks with
acks that go back to the source message queue. Storm acks are about
detecting tuple DAG completion and have nothing to do with the ack to
the source message queue.
So if you weren't able to get it to work then it was a different
problem.
On Tue, Mar 31, 2015 at 2:48 PM, Parth Brahmbhatt
<pb...@hortonworks.com> wrote:
I tried this once and failed, following are my findings:
IBM MQ has a strict limitation the thread that receives the message has
to be the thread that acks it. This does not work with storm's threading
model where one thread reads the message and sends it to the bolt
another thread invokes spout's ack method when the bolt acks the message
and only then the spout can ack the message.
Based on
http://www-01.ibm.com/support/knowledgecenter/SSFKSJ_8.0.0/com.ibm.mq.de
v.doc/q031020_.htm?lang=en
<http://www-01.ibm.com/support/knowledgecenter/SSFKSJ_8.0.0/com.ibm.mq.d
ev.doc/q031020_.htm?lang=en> we tried looking into proprietary IBM MQ
java client. The acking mechanism relies on following 3 APIS
http://www-01.ibm.com/support/knowledgecenter/api/content/SSFKSJ_8.0.0/c
om.ibm.mq.javadoc.doc/WMQJavaClasses/com/ibm/mq/MQQueueManager.html#begi
n(
<http://www-01.ibm.com/support/knowledgecenter/api/content/SSFKSJ_8.0.0/
com.ibm.mq.javadoc.doc/WMQJavaClasses/com/ibm/mq/MQQueueManager.html#beg
in(> )
http://www-01.ibm.com/support/knowledgecenter/api/content/SSFKSJ_8.0.0/c
om.ibm.mq.javadoc.doc/WMQJavaClasses/com/ibm/mq/MQQueueManager.html#comm
it(
<http://www-01.ibm.com/support/knowledgecenter/api/content/SSFKSJ_8.0.0/
com.ibm.mq.javadoc.doc/WMQJavaClasses/com/ibm/mq/MQQueueManager.html#com
mit(> )
http://www-01.ibm.com/support/knowledgecenter/api/content/SSFKSJ_8.0.0/c
om.ibm.mq.javadoc.doc/WMQJavaClasses/com/ibm/mq/MQQueueManager.html#back
out(
<http://www-01.ibm.com/support/knowledgecenter/api/content/SSFKSJ_8.0.0/
com.ibm.mq.javadoc.doc/WMQJavaClasses/com/ibm/mq/MQQueueManager.html#bac
kout(> )
We could use these with one caveat. There is no way to cherry pick what
messages, or range of messages like JMS where you can pick the message
with lowest id, will get committed.
This means in the spout we will keep an active list of pending messages,
when ack method in spout is called for a message we remove the message
from the pending list and call commit only if the pending list is empty.
This works as long as the producer is slow or the consumers are slow but
if both of them are really fast the pending list may never become empty
causing lot of memory overhead on the box as well as server side
overhead of keeping all the messages around.
we could introduce a pseudo upper limit in spout where we stop handing
out messages once pending message list reaches some upper bound but that
will just slow down the topology.
In summary, it seems to be impossible to implement a IBM MQ spout that
does not run the risk of message loss and be performant.
Thanks
Parth
From: jeremy p <at...@gmail.com>
Reply-To: "user@storm.apache.org" <us...@storm.apache.org>
Date: Tuesday, March 31, 2015 at 11:41 AM
To: "user@storm.apache.org" <us...@storm.apache.org>
Subject: Using Storm with IBM MQ Series
Hello all,
According to this webinar recap, Storm cannot work with IBM MQ Series
(also known as Websphere MQ) :
http://hortonworks.com/blog/discover-hdp-2-2-apache-kafka-apache-storm-s
tream-data-processing/
This is a real bummer for me, because my company uses IBM MQ Series, and
we have a new project that Storm would be perfect for. Is there
currently an effort underway to make IBM MQ Series interoperate with
Storm? Do you think it's even possible to make Storm and IBM MQ Series
work together?
I'd really like to use Storm, if I can.
--
Twitter: @nathanmarz
http://nathanmarz.com <http://nathanmarz.com/>
Re: Using Storm with IBM MQ Series
Posted by Parth Brahmbhatt <pb...@hortonworks.com>.
Hi Aleksey,
I am guessing you are using proprietary IBM-MQ Apis instead of using the JMS API. Is that right? Is this code open sourced?
Thanks
Parth
From: "aleksey.didik@ubs.com<ma...@ubs.com>" <al...@ubs.com>>
Reply-To: "user@storm.apache.org<ma...@storm.apache.org>" <us...@storm.apache.org>>
Date: Wednesday, April 1, 2015 at 8:48 AM
To: "user@storm.apache.org<ma...@storm.apache.org>" <us...@storm.apache.org>>
Subject: RE: Using Storm with IBM MQ Series
Hello Jeremy,
I'm using Storm with IBM MQ for my project a half of a year.
In MQ you have no per message acknowledgment – that the problem. You could ack all messages received in current session only altogether. That makes Parth to make his trick with pending list committing only when list is empty.
We did we do – made one spout process one message only at the moment:
1. Create TRANSACTED session in spout
2. Create one consumer for this session in spout.
3. Receive message.
4. Emit to bolts.
5. When get ack – commit session.
6. When get fail – rollback session.
7. Start once again from point 2.
One session has one message in that case, so we have per message ack.
It makes us to scale topology throughput adding more spouts instead of using TOPOLOGY_MAX_SPOUT_PENDING. Personally myself don't like this solution, but I have nothing better now.
I have never met limitation of different threads in current solution.
HTH,
Aleksey.
From: jeremy p [mailto:athomewithagroovebox@gmail.com]
Sent: Wednesday, April 01, 2015 11:19 AM
To: user
Subject: Re: Using Storm with IBM MQ Series
Nathan : thanks for the response! It appears that the engineer who gave this webinar ran into the same problem as Parth :
http://hortonworks.com/blog/discover-hdp-2-2-apache-kafka-apache-storm-stream-data-processing/
He reports that he was unable to make MQ Series work with Storm because :
"We ran into issues with IBM MQ-Series with respect to how messages are acknowledged by IBM MQ. IBM-MQ requires the thread that receives the message be the same thread that acks it. Storm’s framework cannot support this requirement, as the receiving and acking thread by design are different threads to achieve higher throughput."
If threading wasn't the issue, how would you explain Storm's behavior in this situation?
--Jeremy
On Wed, Apr 1, 2015 at 12:11 AM, Parth Brahmbhatt <pb...@hortonworks.com>> wrote:
Thanks for correcting me Nathan.
Jeremy, in that case you can give storm-jms<https://github.com/ptgoetz/storm-jms> a shot and see if it works for you.
Thanks
Parth
From: Nathan Marz <na...@nathanmarz.com>>
Reply-To: "user@storm.apache.org<ma...@storm.apache.org>" <us...@storm.apache.org>>
Date: Tuesday, March 31, 2015 at 6:11 PM
To: "user@storm.apache.org<ma...@storm.apache.org>" <us...@storm.apache.org>>
Subject: Re: Using Storm with IBM MQ Series
That's not accurate. A spout task's nextTuple, ack, and fail methods are all called by the exact same thread. You're confusing Storm acks with acks that go back to the source message queue. Storm acks are about detecting tuple DAG completion and have nothing to do with the ack to the source message queue.
So if you weren't able to get it to work then it was a different problem.
On Tue, Mar 31, 2015 at 2:48 PM, Parth Brahmbhatt <pb...@hortonworks.com>> wrote:
I tried this once and failed, following are my findings:
IBM MQ has a strict limitation the thread that receives the message has to be the thread that acks it. This does not work with storm's threading model where one thread reads the message and sends it to the bolt another thread invokes spout's ack method when the bolt acks the message and only then the spout can ack the message.
Based on http://www-01.ibm.com/support/knowledgecenter/SSFKSJ_8.0.0/com.ibm.mq.dev.doc/q031020_.htm?lang=en we tried looking into proprietary IBM MQ java client. The acking mechanism relies on following 3 APIS
http://www-01.ibm.com/support/knowledgecenter/api/content/SSFKSJ_8.0.0/com.ibm.mq.javadoc.doc/WMQJavaClasses/com/ibm/mq/MQQueueManager.html#begin()
http://www-01.ibm.com/support/knowledgecenter/api/content/SSFKSJ_8.0.0/com.ibm.mq.javadoc.doc/WMQJavaClasses/com/ibm/mq/MQQueueManager.html#commit()
http://www-01.ibm.com/support/knowledgecenter/api/content/SSFKSJ_8.0.0/com.ibm.mq.javadoc.doc/WMQJavaClasses/com/ibm/mq/MQQueueManager.html#backout()
We could use these with one caveat. There is no way to cherry pick what messages, or range of messages like JMS where you can pick the message with lowest id, will get committed.
This means in the spout we will keep an active list of pending messages, when ack method in spout is called for a message we remove the message from the pending list and call commit only if the pending list is empty. This works as long as the producer is slow or the consumers are slow but if both of them are really fast the pending list may never become empty causing lot of memory overhead on the box as well as server side overhead of keeping all the messages around.
we could introduce a pseudo upper limit in spout where we stop handing out messages once pending message list reaches some upper bound but that will just slow down the topology.
In summary, it seems to be impossible to implement a IBM MQ spout that does not run the risk of message loss and be performant.
Thanks
Parth
From: jeremy p <at...@gmail.com>>
Reply-To: "user@storm.apache.org<ma...@storm.apache.org>" <us...@storm.apache.org>>
Date: Tuesday, March 31, 2015 at 11:41 AM
To: "user@storm.apache.org<ma...@storm.apache.org>" <us...@storm.apache.org>>
Subject: Using Storm with IBM MQ Series
Hello all,
According to this webinar recap, Storm cannot work with IBM MQ Series (also known as Websphere MQ) :
http://hortonworks.com/blog/discover-hdp-2-2-apache-kafka-apache-storm-stream-data-processing/
This is a real bummer for me, because my company uses IBM MQ Series, and we have a new project that Storm would be perfect for. Is there currently an effort underway to make IBM MQ Series interoperate with Storm? Do you think it's even possible to make Storm and IBM MQ Series work together?
I'd really like to use Storm, if I can.
--
Twitter: @nathanmarz
http://nathanmarz.com<http://nathanmarz.com/>
RE: Using Storm with IBM MQ Series
Posted by al...@ubs.com.
Hello Jeremy,
I'm using Storm with IBM MQ for my project a half of a year.
In MQ you have no per message acknowledgment – that the problem. You could ack all messages received in current session only altogether. That makes Parth to make his trick with pending list committing only when list is empty.
We did we do – made one spout process one message only at the moment:
1. Create TRANSACTED session in spout
2. Create one consumer for this session in spout.
3. Receive message.
4. Emit to bolts.
5. When get ack – commit session.
6. When get fail – rollback session.
7. Start once again from point 2.
One session has one message in that case, so we have per message ack.
It makes us to scale topology throughput adding more spouts instead of using TOPOLOGY_MAX_SPOUT_PENDING. Personally myself don't like this solution, but I have nothing better now.
I have never met limitation of different threads in current solution.
HTH,
Aleksey.
From: jeremy p [mailto:athomewithagroovebox@gmail.com]
Sent: Wednesday, April 01, 2015 11:19 AM
To: user
Subject: Re: Using Storm with IBM MQ Series
Nathan : thanks for the response! It appears that the engineer who gave this webinar ran into the same problem as Parth :
http://hortonworks.com/blog/discover-hdp-2-2-apache-kafka-apache-storm-stream-data-processing/
He reports that he was unable to make MQ Series work with Storm because :
"We ran into issues with IBM MQ-Series with respect to how messages are acknowledged by IBM MQ. IBM-MQ requires the thread that receives the message be the same thread that acks it. Storm’s framework cannot support this requirement, as the receiving and acking thread by design are different threads to achieve higher throughput."
If threading wasn't the issue, how would you explain Storm's behavior in this situation?
--Jeremy
On Wed, Apr 1, 2015 at 12:11 AM, Parth Brahmbhatt <pb...@hortonworks.com> wrote:
Thanks for correcting me Nathan.
Jeremy, in that case you can give storm-jms <https://github.com/ptgoetz/storm-jms> a shot and see if it works for you.
Thanks
Parth
From: Nathan Marz <na...@nathanmarz.com>
Reply-To: "user@storm.apache.org" <us...@storm.apache.org>
Date: Tuesday, March 31, 2015 at 6:11 PM
To: "user@storm.apache.org" <us...@storm.apache.org>
Subject: Re: Using Storm with IBM MQ Series
That's not accurate. A spout task's nextTuple, ack, and fail methods are all called by the exact same thread. You're confusing Storm acks with acks that go back to the source message queue. Storm acks are about detecting tuple DAG completion and have nothing to do with the ack to the source message queue.
So if you weren't able to get it to work then it was a different problem.
On Tue, Mar 31, 2015 at 2:48 PM, Parth Brahmbhatt <pb...@hortonworks.com> wrote:
I tried this once and failed, following are my findings:
IBM MQ has a strict limitation the thread that receives the message has to be the thread that acks it. This does not work with storm's threading model where one thread reads the message and sends it to the bolt another thread invokes spout's ack method when the bolt acks the message and only then the spout can ack the message.
Based on http://www-01.ibm.com/support/knowledgecenter/SSFKSJ_8.0.0/com.ibm.mq.dev.doc/q031020_.htm?lang=en <http://www-01.ibm.com/support/knowledgecenter/SSFKSJ_8.0.0/com.ibm.mq.dev.doc/q031020_.htm?lang=en> we tried looking into proprietary IBM MQ java client. The acking mechanism relies on following 3 APIS
http://www-01.ibm.com/support/knowledgecenter/api/content/SSFKSJ_8.0.0/com.ibm.mq.javadoc.doc/WMQJavaClasses/com/ibm/mq/MQQueueManager.html#begin( <http://www-01.ibm.com/support/knowledgecenter/api/content/SSFKSJ_8.0.0/com.ibm.mq.javadoc.doc/WMQJavaClasses/com/ibm/mq/MQQueueManager.html#begin(> )
http://www-01.ibm.com/support/knowledgecenter/api/content/SSFKSJ_8.0.0/com.ibm.mq.javadoc.doc/WMQJavaClasses/com/ibm/mq/MQQueueManager.html#commit( <http://www-01.ibm.com/support/knowledgecenter/api/content/SSFKSJ_8.0.0/com.ibm.mq.javadoc.doc/WMQJavaClasses/com/ibm/mq/MQQueueManager.html#commit(> )
http://www-01.ibm.com/support/knowledgecenter/api/content/SSFKSJ_8.0.0/com.ibm.mq.javadoc.doc/WMQJavaClasses/com/ibm/mq/MQQueueManager.html#backout( <http://www-01.ibm.com/support/knowledgecenter/api/content/SSFKSJ_8.0.0/com.ibm.mq.javadoc.doc/WMQJavaClasses/com/ibm/mq/MQQueueManager.html#backout(> )
We could use these with one caveat. There is no way to cherry pick what messages, or range of messages like JMS where you can pick the message with lowest id, will get committed.
This means in the spout we will keep an active list of pending messages, when ack method in spout is called for a message we remove the message from the pending list and call commit only if the pending list is empty. This works as long as the producer is slow or the consumers are slow but if both of them are really fast the pending list may never become empty causing lot of memory overhead on the box as well as server side overhead of keeping all the messages around.
we could introduce a pseudo upper limit in spout where we stop handing out messages once pending message list reaches some upper bound but that will just slow down the topology.
In summary, it seems to be impossible to implement a IBM MQ spout that does not run the risk of message loss and be performant.
Thanks
Parth
From: jeremy p <at...@gmail.com>
Reply-To: "user@storm.apache.org" <us...@storm.apache.org>
Date: Tuesday, March 31, 2015 at 11:41 AM
To: "user@storm.apache.org" <us...@storm.apache.org>
Subject: Using Storm with IBM MQ Series
Hello all,
According to this webinar recap, Storm cannot work with IBM MQ Series (also known as Websphere MQ) :
http://hortonworks.com/blog/discover-hdp-2-2-apache-kafka-apache-storm-stream-data-processing/
This is a real bummer for me, because my company uses IBM MQ Series, and we have a new project that Storm would be perfect for. Is there currently an effort underway to make IBM MQ Series interoperate with Storm? Do you think it's even possible to make Storm and IBM MQ Series work together?
I'd really like to use Storm, if I can.
--
Twitter: @nathanmarz
http://nathanmarz.com <http://nathanmarz.com/>
Re: Using Storm with IBM MQ Series
Posted by jeremy p <at...@gmail.com>.
Nathan : thanks for the response! It appears that the engineer who gave
this webinar ran into the same problem as Parth :
http://hortonworks.com/blog/discover-hdp-2-2-apache-kafka-apache-storm-stream-data-processing/
He reports that he was unable to make MQ Series work with Storm because :
"We ran into issues with IBM MQ-Series with respect to how messages are
acknowledged by IBM MQ. IBM-MQ requires the thread that receives the
message be the same thread that acks it. Storm’s framework cannot support
this requirement, as the receiving and acking thread by design are
different threads to achieve higher throughput."
If threading wasn't the issue, how would you explain Storm's behavior in
this situation?
--Jeremy
On Wed, Apr 1, 2015 at 12:11 AM, Parth Brahmbhatt <
pbrahmbhatt@hortonworks.com> wrote:
> Thanks for correcting me Nathan.
>
> Jeremy, in that case you can give storm-jms
> <https://github.com/ptgoetz/storm-jms> a shot and see if it works for
> you.
>
> Thanks
> Parth
>
> From: Nathan Marz <na...@nathanmarz.com>
> Reply-To: "user@storm.apache.org" <us...@storm.apache.org>
> Date: Tuesday, March 31, 2015 at 6:11 PM
> To: "user@storm.apache.org" <us...@storm.apache.org>
> Subject: Re: Using Storm with IBM MQ Series
>
> That's not accurate. A spout task's nextTuple, ack, and fail methods
> are all called by the exact same thread. You're confusing Storm acks with
> acks that go back to the source message queue. Storm acks are about
> detecting tuple DAG completion and have nothing to do with the ack to the
> source message queue.
>
> So if you weren't able to get it to work then it was a different problem.
>
> On Tue, Mar 31, 2015 at 2:48 PM, Parth Brahmbhatt <
> pbrahmbhatt@hortonworks.com> wrote:
>
>> I tried this once and failed, following are my findings:
>>
>> IBM MQ has a strict limitation the thread that receives the message
>> has to be the thread that acks it. This does not work with storm's
>> threading model where one thread reads the message and sends it to the bolt
>> another thread invokes spout's ack method when the bolt acks the message
>> and only then the spout can ack the message.
>>
>> Based on
>> http://www-01.ibm.com/support/knowledgecenter/SSFKSJ_8.0.0/com.ibm.mq.dev.doc/q031020_.htm?lang=en we
>> tried looking into proprietary IBM MQ java client. The acking mechanism
>> relies on following 3 APIS
>>
>>
>> http://www-01.ibm.com/support/knowledgecenter/api/content/SSFKSJ_8.0.0/com.ibm.mq.javadoc.doc/WMQJavaClasses/com/ibm/mq/MQQueueManager.html#begin(
>> )
>>
>> http://www-01.ibm.com/support/knowledgecenter/api/content/SSFKSJ_8.0.0/com.ibm.mq.javadoc.doc/WMQJavaClasses/com/ibm/mq/MQQueueManager.html#commit(
>> )
>>
>> http://www-01.ibm.com/support/knowledgecenter/api/content/SSFKSJ_8.0.0/com.ibm.mq.javadoc.doc/WMQJavaClasses/com/ibm/mq/MQQueueManager.html#backout(
>> )
>>
>> We could use these with one caveat. There is no way to cherry pick what
>> messages, or range of messages like JMS where you can pick the message with
>> lowest id, will get committed.
>> This means in the spout we will keep an active list of pending messages,
>> when ack method in spout is called for a message we remove the message from
>> the pending list and call commit only if the pending list is empty. This
>> works as long as the producer is slow or the consumers are slow but if both
>> of them are really fast the pending list may never become empty causing lot
>> of memory overhead on the box as well as server side overhead of keeping
>> all the messages around.
>> we could introduce a pseudo upper limit in spout where we stop handing
>> out messages once pending message list reaches some upper bound but that
>> will just slow down the topology.
>>
>> In summary, it seems to be impossible to implement a IBM MQ spout that
>> does not run the risk of message loss and be performant.
>>
>>
>> Thanks
>>
>> Parth
>>
>> From: jeremy p <at...@gmail.com>
>> Reply-To: "user@storm.apache.org" <us...@storm.apache.org>
>> Date: Tuesday, March 31, 2015 at 11:41 AM
>> To: "user@storm.apache.org" <us...@storm.apache.org>
>> Subject: Using Storm with IBM MQ Series
>>
>> Hello all,
>>
>> According to this webinar recap, Storm cannot work with IBM MQ Series
>> (also known as Websphere MQ) :
>>
>> http://hortonworks.com/blog/discover-hdp-2-2-apache-kafka-apache-storm-stream-data-processing/
>>
>> This is a real bummer for me, because my company uses IBM MQ Series,
>> and we have a new project that Storm would be perfect for. Is there
>> currently an effort underway to make IBM MQ Series interoperate with
>> Storm? Do you think it's even possible to make Storm and IBM MQ Series
>> work together?
>>
>> I'd really like to use Storm, if I can.
>>
>
>
>
> --
> Twitter: @nathanmarz
> http://nathanmarz.com
>
Re: Using Storm with IBM MQ Series
Posted by Parth Brahmbhatt <pb...@hortonworks.com>.
Thanks for correcting me Nathan.
Jeremy, in that case you can give storm-jms<https://github.com/ptgoetz/storm-jms> a shot and see if it works for you.
Thanks
Parth
From: Nathan Marz <na...@nathanmarz.com>>
Reply-To: "user@storm.apache.org<ma...@storm.apache.org>" <us...@storm.apache.org>>
Date: Tuesday, March 31, 2015 at 6:11 PM
To: "user@storm.apache.org<ma...@storm.apache.org>" <us...@storm.apache.org>>
Subject: Re: Using Storm with IBM MQ Series
That's not accurate. A spout task's nextTuple, ack, and fail methods are all called by the exact same thread. You're confusing Storm acks with acks that go back to the source message queue. Storm acks are about detecting tuple DAG completion and have nothing to do with the ack to the source message queue.
So if you weren't able to get it to work then it was a different problem.
On Tue, Mar 31, 2015 at 2:48 PM, Parth Brahmbhatt <pb...@hortonworks.com>> wrote:
I tried this once and failed, following are my findings:
IBM MQ has a strict limitation the thread that receives the message has to be the thread that acks it. This does not work with storm's threading model where one thread reads the message and sends it to the bolt another thread invokes spout's ack method when the bolt acks the message and only then the spout can ack the message.
Based on http://www-01.ibm.com/support/knowledgecenter/SSFKSJ_8.0.0/com.ibm.mq.dev.doc/q031020_.htm?lang=en we tried looking into proprietary IBM MQ java client. The acking mechanism relies on following 3 APIS
http://www-01.ibm.com/support/knowledgecenter/api/content/SSFKSJ_8.0.0/com.ibm.mq.javadoc.doc/WMQJavaClasses/com/ibm/mq/MQQueueManager.html#begin()
http://www-01.ibm.com/support/knowledgecenter/api/content/SSFKSJ_8.0.0/com.ibm.mq.javadoc.doc/WMQJavaClasses/com/ibm/mq/MQQueueManager.html#commit()
http://www-01.ibm.com/support/knowledgecenter/api/content/SSFKSJ_8.0.0/com.ibm.mq.javadoc.doc/WMQJavaClasses/com/ibm/mq/MQQueueManager.html#backout()
We could use these with one caveat. There is no way to cherry pick what messages, or range of messages like JMS where you can pick the message with lowest id, will get committed.
This means in the spout we will keep an active list of pending messages, when ack method in spout is called for a message we remove the message from the pending list and call commit only if the pending list is empty. This works as long as the producer is slow or the consumers are slow but if both of them are really fast the pending list may never become empty causing lot of memory overhead on the box as well as server side overhead of keeping all the messages around.
we could introduce a pseudo upper limit in spout where we stop handing out messages once pending message list reaches some upper bound but that will just slow down the topology.
In summary, it seems to be impossible to implement a IBM MQ spout that does not run the risk of message loss and be performant.
Thanks
Parth
From: jeremy p <at...@gmail.com>>
Reply-To: "user@storm.apache.org<ma...@storm.apache.org>" <us...@storm.apache.org>>
Date: Tuesday, March 31, 2015 at 11:41 AM
To: "user@storm.apache.org<ma...@storm.apache.org>" <us...@storm.apache.org>>
Subject: Using Storm with IBM MQ Series
Hello all,
According to this webinar recap, Storm cannot work with IBM MQ Series (also known as Websphere MQ) :
http://hortonworks.com/blog/discover-hdp-2-2-apache-kafka-apache-storm-stream-data-processing/
This is a real bummer for me, because my company uses IBM MQ Series, and we have a new project that Storm would be perfect for. Is there currently an effort underway to make IBM MQ Series interoperate with Storm? Do you think it's even possible to make Storm and IBM MQ Series work together?
I'd really like to use Storm, if I can.
--
Twitter: @nathanmarz
http://nathanmarz.com<http://nathanmarz.com/>
Re: Using Storm with IBM MQ Series
Posted by Nathan Marz <na...@nathanmarz.com>.
That's not accurate. A spout task's nextTuple, ack, and fail methods are
all called by the exact same thread. You're confusing Storm acks with acks
that go back to the source message queue. Storm acks are about detecting
tuple DAG completion and have nothing to do with the ack to the source
message queue.
So if you weren't able to get it to work then it was a different problem.
On Tue, Mar 31, 2015 at 2:48 PM, Parth Brahmbhatt <
pbrahmbhatt@hortonworks.com> wrote:
> I tried this once and failed, following are my findings:
>
> IBM MQ has a strict limitation the thread that receives the message has
> to be the thread that acks it. This does not work with storm's threading
> model where one thread reads the message and sends it to the bolt another
> thread invokes spout's ack method when the bolt acks the message and only
> then the spout can ack the message.
>
> Based on
> http://www-01.ibm.com/support/knowledgecenter/SSFKSJ_8.0.0/com.ibm.mq.dev.doc/q031020_.htm?lang=en we
> tried looking into proprietary IBM MQ java client. The acking mechanism
> relies on following 3 APIS
>
>
> http://www-01.ibm.com/support/knowledgecenter/api/content/SSFKSJ_8.0.0/com.ibm.mq.javadoc.doc/WMQJavaClasses/com/ibm/mq/MQQueueManager.html#begin(
> )
>
> http://www-01.ibm.com/support/knowledgecenter/api/content/SSFKSJ_8.0.0/com.ibm.mq.javadoc.doc/WMQJavaClasses/com/ibm/mq/MQQueueManager.html#commit(
> )
>
> http://www-01.ibm.com/support/knowledgecenter/api/content/SSFKSJ_8.0.0/com.ibm.mq.javadoc.doc/WMQJavaClasses/com/ibm/mq/MQQueueManager.html#backout(
> )
>
> We could use these with one caveat. There is no way to cherry pick what
> messages, or range of messages like JMS where you can pick the message with
> lowest id, will get committed.
> This means in the spout we will keep an active list of pending messages,
> when ack method in spout is called for a message we remove the message from
> the pending list and call commit only if the pending list is empty. This
> works as long as the producer is slow or the consumers are slow but if both
> of them are really fast the pending list may never become empty causing lot
> of memory overhead on the box as well as server side overhead of keeping
> all the messages around.
> we could introduce a pseudo upper limit in spout where we stop handing out
> messages once pending message list reaches some upper bound but that will
> just slow down the topology.
>
> In summary, it seems to be impossible to implement a IBM MQ spout that
> does not run the risk of message loss and be performant.
>
>
> Thanks
>
> Parth
>
> From: jeremy p <at...@gmail.com>
> Reply-To: "user@storm.apache.org" <us...@storm.apache.org>
> Date: Tuesday, March 31, 2015 at 11:41 AM
> To: "user@storm.apache.org" <us...@storm.apache.org>
> Subject: Using Storm with IBM MQ Series
>
> Hello all,
>
> According to this webinar recap, Storm cannot work with IBM MQ Series
> (also known as Websphere MQ) :
>
> http://hortonworks.com/blog/discover-hdp-2-2-apache-kafka-apache-storm-stream-data-processing/
>
> This is a real bummer for me, because my company uses IBM MQ Series, and
> we have a new project that Storm would be perfect for. Is there currently
> an effort underway to make IBM MQ Series interoperate with Storm? Do you
> think it's even possible to make Storm and IBM MQ Series work together?
>
> I'd really like to use Storm, if I can.
>
--
Twitter: @nathanmarz
http://nathanmarz.com
Re: Using Storm with IBM MQ Series
Posted by Michael Rose <mi...@fullcontact.com>.
Why couldn't a thread inside the spout be responsible for communications
with IBM MQ?
That thread would read messages from the MQ, push them into a queue (most
spouts do that anyways), the spout can deliver messages from the queue.
ack/fail could put events in another queue. That same thread that's
filling the queue can also read the ack/fail queue and communicate with IBM
MQ.
*Michael Rose*
Senior Software Engineer, Data Science
*Full*Contact | fullcontact.com
<https://www.fullcontact.com/?utm_source=FullContact%20-%20Email%20Signatures&utm_medium=email&utm_content=Signature%20Link&utm_campaign=FullContact%20-%20Email%20Signatures>
m: +1.720.837.1357 | t: @xorlev
All Your Contacts, Updated and In One Place.
Try FullContact for Free
<https://www.fullcontact.com/?utm_source=FullContact%20-%20Email%20Signatures&utm_medium=email&utm_content=Signature%20Link&utm_campaign=FullContact%20-%20Email%20Signatures>
On Tue, Mar 31, 2015 at 1:00 PM, Parth Brahmbhatt <
pbrahmbhatt@hortonworks.com> wrote:
> Sure, as long as you can figure out a performant way to push messages
> from IBM MQ to kafka it should work.
>
> Thanks
> Parth
> From: jeremy p <at...@gmail.com>
> Reply-To: user <us...@storm.apache.org>
> Date: Tuesday, March 31, 2015 at 11:56 AM
> To: user <us...@storm.apache.org>
> Subject: Re: Using Storm with IBM MQ Series
>
> Thank you for the detailed answer! Question : what if we used
> something like Kafka as an intermediary between MQ Series and Storm? Like,
> if we just used Kafka as a way to pass messages from MQ Series to Storm?
> Do you think something like that would work?
>
> --Jeremy
>
> On Tue, Mar 31, 2015 at 2:48 PM, Parth Brahmbhatt <
> pbrahmbhatt@hortonworks.com> wrote:
>
>> I tried this once and failed, following are my findings:
>>
>> IBM MQ has a strict limitation the thread that receives the message
>> has to be the thread that acks it. This does not work with storm's
>> threading model where one thread reads the message and sends it to the bolt
>> another thread invokes spout's ack method when the bolt acks the message
>> and only then the spout can ack the message.
>>
>> Based on
>> http://www-01.ibm.com/support/knowledgecenter/SSFKSJ_8.0.0/com.ibm.mq.dev.doc/q031020_.htm?lang=en we
>> tried looking into proprietary IBM MQ java client. The acking mechanism
>> relies on following 3 APIS
>>
>>
>> http://www-01.ibm.com/support/knowledgecenter/api/content/SSFKSJ_8.0.0/com.ibm.mq.javadoc.doc/WMQJavaClasses/com/ibm/mq/MQQueueManager.html#begin(
>> )
>>
>> http://www-01.ibm.com/support/knowledgecenter/api/content/SSFKSJ_8.0.0/com.ibm.mq.javadoc.doc/WMQJavaClasses/com/ibm/mq/MQQueueManager.html#commit(
>> )
>>
>> http://www-01.ibm.com/support/knowledgecenter/api/content/SSFKSJ_8.0.0/com.ibm.mq.javadoc.doc/WMQJavaClasses/com/ibm/mq/MQQueueManager.html#backout(
>> )
>>
>> We could use these with one caveat. There is no way to cherry pick what
>> messages, or range of messages like JMS where you can pick the message with
>> lowest id, will get committed.
>> This means in the spout we will keep an active list of pending messages,
>> when ack method in spout is called for a message we remove the message from
>> the pending list and call commit only if the pending list is empty. This
>> works as long as the producer is slow or the consumers are slow but if both
>> of them are really fast the pending list may never become empty causing lot
>> of memory overhead on the box as well as server side overhead of keeping
>> all the messages around.
>> we could introduce a pseudo upper limit in spout where we stop handing
>> out messages once pending message list reaches some upper bound but that
>> will just slow down the topology.
>>
>> In summary, it seems to be impossible to implement a IBM MQ spout that
>> does not run the risk of message loss and be performant.
>>
>>
>> Thanks
>>
>> Parth
>>
>> From: jeremy p <at...@gmail.com>
>> Reply-To: "user@storm.apache.org" <us...@storm.apache.org>
>> Date: Tuesday, March 31, 2015 at 11:41 AM
>> To: "user@storm.apache.org" <us...@storm.apache.org>
>> Subject: Using Storm with IBM MQ Series
>>
>> Hello all,
>>
>> According to this webinar recap, Storm cannot work with IBM MQ Series
>> (also known as Websphere MQ) :
>>
>> http://hortonworks.com/blog/discover-hdp-2-2-apache-kafka-apache-storm-stream-data-processing/
>>
>> This is a real bummer for me, because my company uses IBM MQ Series,
>> and we have a new project that Storm would be perfect for. Is there
>> currently an effort underway to make IBM MQ Series interoperate with
>> Storm? Do you think it's even possible to make Storm and IBM MQ Series
>> work together?
>>
>> I'd really like to use Storm, if I can.
>>
>
>
Re: Using Storm with IBM MQ Series
Posted by Parth Brahmbhatt <pb...@hortonworks.com>.
Sure, as long as you can figure out a performant way to push messages from IBM MQ to kafka it should work.
Thanks
Parth
From: jeremy p <at...@gmail.com>>
Reply-To: user <us...@storm.apache.org>>
Date: Tuesday, March 31, 2015 at 11:56 AM
To: user <us...@storm.apache.org>>
Subject: Re: Using Storm with IBM MQ Series
Thank you for the detailed answer! Question : what if we used something like Kafka as an intermediary between MQ Series and Storm? Like, if we just used Kafka as a way to pass messages from MQ Series to Storm? Do you think something like that would work?
--Jeremy
On Tue, Mar 31, 2015 at 2:48 PM, Parth Brahmbhatt <pb...@hortonworks.com>> wrote:
I tried this once and failed, following are my findings:
IBM MQ has a strict limitation the thread that receives the message has to be the thread that acks it. This does not work with storm's threading model where one thread reads the message and sends it to the bolt another thread invokes spout's ack method when the bolt acks the message and only then the spout can ack the message.
Based on http://www-01.ibm.com/support/knowledgecenter/SSFKSJ_8.0.0/com.ibm.mq.dev.doc/q031020_.htm?lang=en we tried looking into proprietary IBM MQ java client. The acking mechanism relies on following 3 APIS
http://www-01.ibm.com/support/knowledgecenter/api/content/SSFKSJ_8.0.0/com.ibm.mq.javadoc.doc/WMQJavaClasses/com/ibm/mq/MQQueueManager.html#begin()
http://www-01.ibm.com/support/knowledgecenter/api/content/SSFKSJ_8.0.0/com.ibm.mq.javadoc.doc/WMQJavaClasses/com/ibm/mq/MQQueueManager.html#commit()
http://www-01.ibm.com/support/knowledgecenter/api/content/SSFKSJ_8.0.0/com.ibm.mq.javadoc.doc/WMQJavaClasses/com/ibm/mq/MQQueueManager.html#backout()
We could use these with one caveat. There is no way to cherry pick what messages, or range of messages like JMS where you can pick the message with lowest id, will get committed.
This means in the spout we will keep an active list of pending messages, when ack method in spout is called for a message we remove the message from the pending list and call commit only if the pending list is empty. This works as long as the producer is slow or the consumers are slow but if both of them are really fast the pending list may never become empty causing lot of memory overhead on the box as well as server side overhead of keeping all the messages around.
we could introduce a pseudo upper limit in spout where we stop handing out messages once pending message list reaches some upper bound but that will just slow down the topology.
In summary, it seems to be impossible to implement a IBM MQ spout that does not run the risk of message loss and be performant.
Thanks
Parth
From: jeremy p <at...@gmail.com>>
Reply-To: "user@storm.apache.org<ma...@storm.apache.org>" <us...@storm.apache.org>>
Date: Tuesday, March 31, 2015 at 11:41 AM
To: "user@storm.apache.org<ma...@storm.apache.org>" <us...@storm.apache.org>>
Subject: Using Storm with IBM MQ Series
Hello all,
According to this webinar recap, Storm cannot work with IBM MQ Series (also known as Websphere MQ) :
http://hortonworks.com/blog/discover-hdp-2-2-apache-kafka-apache-storm-stream-data-processing/
This is a real bummer for me, because my company uses IBM MQ Series, and we have a new project that Storm would be perfect for. Is there currently an effort underway to make IBM MQ Series interoperate with Storm? Do you think it's even possible to make Storm and IBM MQ Series work together?
I'd really like to use Storm, if I can.
Re: Using Storm with IBM MQ Series
Posted by jeremy p <at...@gmail.com>.
Thank you for the detailed answer! Question : what if we used something
like Kafka as an intermediary between MQ Series and Storm? Like, if we
just used Kafka as a way to pass messages from MQ Series to Storm? Do you
think something like that would work?
--Jeremy
On Tue, Mar 31, 2015 at 2:48 PM, Parth Brahmbhatt <
pbrahmbhatt@hortonworks.com> wrote:
> I tried this once and failed, following are my findings:
>
> IBM MQ has a strict limitation the thread that receives the message has
> to be the thread that acks it. This does not work with storm's threading
> model where one thread reads the message and sends it to the bolt another
> thread invokes spout's ack method when the bolt acks the message and only
> then the spout can ack the message.
>
> Based on
> http://www-01.ibm.com/support/knowledgecenter/SSFKSJ_8.0.0/com.ibm.mq.dev.doc/q031020_.htm?lang=en we
> tried looking into proprietary IBM MQ java client. The acking mechanism
> relies on following 3 APIS
>
>
> http://www-01.ibm.com/support/knowledgecenter/api/content/SSFKSJ_8.0.0/com.ibm.mq.javadoc.doc/WMQJavaClasses/com/ibm/mq/MQQueueManager.html#begin(
> )
>
> http://www-01.ibm.com/support/knowledgecenter/api/content/SSFKSJ_8.0.0/com.ibm.mq.javadoc.doc/WMQJavaClasses/com/ibm/mq/MQQueueManager.html#commit(
> )
>
> http://www-01.ibm.com/support/knowledgecenter/api/content/SSFKSJ_8.0.0/com.ibm.mq.javadoc.doc/WMQJavaClasses/com/ibm/mq/MQQueueManager.html#backout(
> )
>
> We could use these with one caveat. There is no way to cherry pick what
> messages, or range of messages like JMS where you can pick the message with
> lowest id, will get committed.
> This means in the spout we will keep an active list of pending messages,
> when ack method in spout is called for a message we remove the message from
> the pending list and call commit only if the pending list is empty. This
> works as long as the producer is slow or the consumers are slow but if both
> of them are really fast the pending list may never become empty causing lot
> of memory overhead on the box as well as server side overhead of keeping
> all the messages around.
> we could introduce a pseudo upper limit in spout where we stop handing out
> messages once pending message list reaches some upper bound but that will
> just slow down the topology.
>
> In summary, it seems to be impossible to implement a IBM MQ spout that
> does not run the risk of message loss and be performant.
>
>
> Thanks
>
> Parth
>
> From: jeremy p <at...@gmail.com>
> Reply-To: "user@storm.apache.org" <us...@storm.apache.org>
> Date: Tuesday, March 31, 2015 at 11:41 AM
> To: "user@storm.apache.org" <us...@storm.apache.org>
> Subject: Using Storm with IBM MQ Series
>
> Hello all,
>
> According to this webinar recap, Storm cannot work with IBM MQ Series
> (also known as Websphere MQ) :
>
> http://hortonworks.com/blog/discover-hdp-2-2-apache-kafka-apache-storm-stream-data-processing/
>
> This is a real bummer for me, because my company uses IBM MQ Series, and
> we have a new project that Storm would be perfect for. Is there currently
> an effort underway to make IBM MQ Series interoperate with Storm? Do you
> think it's even possible to make Storm and IBM MQ Series work together?
>
> I'd really like to use Storm, if I can.
>
Re: Using Storm with IBM MQ Series
Posted by Parth Brahmbhatt <pb...@hortonworks.com>.
I tried this once and failed, following are my findings:
IBM MQ has a strict limitation the thread that receives the message has to be the thread that acks it. This does not work with storm's threading model where one thread reads the message and sends it to the bolt another thread invokes spout's ack method when the bolt acks the message and only then the spout can ack the message.
Based on http://www-01.ibm.com/support/knowledgecenter/SSFKSJ_8.0.0/com.ibm.mq.dev.doc/q031020_.htm?lang=en we tried looking into proprietary IBM MQ java client. The acking mechanism relies on following 3 APIS
http://www-01.ibm.com/support/knowledgecenter/api/content/SSFKSJ_8.0.0/com.ibm.mq.javadoc.doc/WMQJavaClasses/com/ibm/mq/MQQueueManager.html#begin()
http://www-01.ibm.com/support/knowledgecenter/api/content/SSFKSJ_8.0.0/com.ibm.mq.javadoc.doc/WMQJavaClasses/com/ibm/mq/MQQueueManager.html#commit()
http://www-01.ibm.com/support/knowledgecenter/api/content/SSFKSJ_8.0.0/com.ibm.mq.javadoc.doc/WMQJavaClasses/com/ibm/mq/MQQueueManager.html#backout()
We could use these with one caveat. There is no way to cherry pick what messages, or range of messages like JMS where you can pick the message with lowest id, will get committed.
This means in the spout we will keep an active list of pending messages, when ack method in spout is called for a message we remove the message from the pending list and call commit only if the pending list is empty. This works as long as the producer is slow or the consumers are slow but if both of them are really fast the pending list may never become empty causing lot of memory overhead on the box as well as server side overhead of keeping all the messages around.
we could introduce a pseudo upper limit in spout where we stop handing out messages once pending message list reaches some upper bound but that will just slow down the topology.
In summary, it seems to be impossible to implement a IBM MQ spout that does not run the risk of message loss and be performant.
Thanks
Parth
From: jeremy p <at...@gmail.com>>
Reply-To: "user@storm.apache.org<ma...@storm.apache.org>" <us...@storm.apache.org>>
Date: Tuesday, March 31, 2015 at 11:41 AM
To: "user@storm.apache.org<ma...@storm.apache.org>" <us...@storm.apache.org>>
Subject: Using Storm with IBM MQ Series
Hello all,
According to this webinar recap, Storm cannot work with IBM MQ Series (also known as Websphere MQ) :
http://hortonworks.com/blog/discover-hdp-2-2-apache-kafka-apache-storm-stream-data-processing/
This is a real bummer for me, because my company uses IBM MQ Series, and we have a new project that Storm would be perfect for. Is there currently an effort underway to make IBM MQ Series interoperate with Storm? Do you think it's even possible to make Storm and IBM MQ Series work together?
I'd really like to use Storm, if I can.