You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@apex.apache.org by Sanjay Pujare <sa...@datatorrent.com> on 2016/07/14 18:16:47 UTC
JMS Input Operator enhancements
Hi all,
I am proposing the following enhancements to the Malhar JMS input operator.
Let me know if you have any comments.
Enhancements are proposed to the malhar JMS Input Operator (in the package
com.datatorrent.lib.io.jms) to make it usable with any JMS compatible
message broker for basic input functionality.
For now we will make this input operator work with ActiveMQ and Amazon SQS
through the JMS interface. The current code contains implementation and
tests for ActiveMQ, and we will add support and tests for SQS without
breaking the ActiveMQ support.
The motivation for this enhancement is as follows. JMS is NOT a wire
protocol but just an API specification for Java programs, so JMS is just a
“wrapper” around an actual implementation library that talks to the
respective message broker. Moreover each message broker has its own
semantics for more advanced features like partitioning or dynamic scaling
that are impossible or difficult to capture via the JMS abstraction. With
this enhancement, we expect the JMS input operator to be usable as a
generic “JMS input operator” with no support for advanced features like
partionability or dynamic scalability.
Full featured input operators for SQS and ActiveMQ will be developed
separately (without necessarily using the JMS interface but the most
appropriate interface) and the design and implementation of those operators
are out of scope here.
The enhancement includes:
-
adding amazon-sqs-java-messaging-lib and aws-android-sdk-sqs libraries
to the pom
-
adding test cases for SQS, matching those for ActiveMQ.
-
making both sets of test cases work
-
cleaning up the existing code (e.g. removing hard coded values like
“TEST.FOO” in JMSBase)
SQS ConsiderationsNo Support for Topics
SQS only supports queues, so topics will not be supported for the SQS
variant.
Idempotency
We will implement a simple idempotent design that uses the
WindowDataManager to store the whole message and deletes the message from
SQS so we are not impacted by SQS’s visibility timeout feature.
Sanjay
Re: JMS Input Operator enhancements
Posted by Sanjay Pujare <sa...@datatorrent.com>.
Yes. ActiveMQ is already there and uses Apache so nothing to do there. The libs for AWS SQS also use the Apache license.
On 7/14/16, 2:42 PM, "Pramod Immaneni" <pr...@datatorrent.com> wrote:
What is the license covering the libraries, Apache?
On Thu, Jul 14, 2016 at 2:37 PM, Sanjay Pujare <sa...@datatorrent.com>
wrote:
> Sounds good. Will go with test scope.
>
> On 7/14/16, 12:57 PM, "Ashwin Chandra Putta" <as...@gmail.com>
> wrote:
>
> They can be made test scoped in malhar for testing. The user of this
> operator can add the implementation specific dependency in their
> application pom where they use this operator and supply the
> corresponding
> connection factory class name.
>
> Regards,
> Ashwin.
>
> On Thu, Jul 14, 2016 at 11:55 AM, Sanjay Pujare <
> sanjay@datatorrent.com>
> wrote:
>
> > Pramod,
> >
> > The JMS “wrapper” library needs the actual client library as you can
> see
> > in com.datatorrent.lib.io.jms.JMSBase.getConnectionFactory()
> >
> > Class<ConnectionFactory> clazz =
> > (Class<ConnectionFactory>)Class.forName(connectionFactoryClass);
> >
> > Where it loads the client library’s connectionFactoryClass by name.
> > Instead of putting the onus on the caller, I thought we can include
> these
> > libraries, so the current class loader can load the required class.
> >
> > Sanjay
> >
> >
> > On 7/14/16, 11:36 AM, "Pramod Immaneni" <pr...@datatorrent.com>
> wrote:
> >
> > Hi Sanjay,
> >
> > If there are going to separate operators developed for SQS and
> ActiveMQ
> > using the native api, why do we need to add the client libraries
> for
> > those
> > to JMS? Are these only needed for testing?
> >
> > Thanks
> >
> > On Thu, Jul 14, 2016 at 11:16 AM, Sanjay Pujare <
> > sanjay@datatorrent.com>
> > wrote:
> >
> > > Hi all,
> > >
> > > I am proposing the following enhancements to the Malhar JMS
> input
> > operator.
> > > Let me know if you have any comments.
> > >
> > > Enhancements are proposed to the malhar JMS Input Operator (in
> the
> > package
> > > com.datatorrent.lib.io.jms) to make it usable with any JMS
> compatible
> > > message broker for basic input functionality.
> > >
> > > For now we will make this input operator work with ActiveMQ and
> > Amazon SQS
> > > through the JMS interface. The current code contains
> implementation
> > and
> > > tests for ActiveMQ, and we will add support and tests for SQS
> without
> > > breaking the ActiveMQ support.
> > >
> > > The motivation for this enhancement is as follows. JMS is NOT
> a wire
> > > protocol but just an API specification for Java programs, so
> JMS is
> > just a
> > > “wrapper” around an actual implementation library that talks
> to the
> > > respective message broker. Moreover each message broker has
> its own
> > > semantics for more advanced features like partitioning or
> dynamic
> > scaling
> > > that are impossible or difficult to capture via the JMS
> abstraction.
> > With
> > > this enhancement, we expect the JMS input operator to be
> usable as a
> > > generic “JMS input operator” with no support for advanced
> features
> > like
> > > partionability or dynamic scalability.
> > >
> > > Full featured input operators for SQS and ActiveMQ will be
> developed
> > > separately (without necessarily using the JMS interface but
> the most
> > > appropriate interface) and the design and implementation of
> those
> > operators
> > > are out of scope here.
> > >
> > > The enhancement includes:
> > >
> > > -
> > >
> > > adding amazon-sqs-java-messaging-lib and aws-android-sdk-sqs
> > libraries
> > > to the pom
> > > -
> > >
> > > adding test cases for SQS, matching those for ActiveMQ.
> > > -
> > >
> > > making both sets of test cases work
> > > -
> > >
> > > cleaning up the existing code (e.g. removing hard coded
> values
> > like
> > > “TEST.FOO” in JMSBase)
> > >
> > > SQS ConsiderationsNo Support for Topics
> > >
> > > SQS only supports queues, so topics will not be supported for
> the SQS
> > > variant.
> > > Idempotency
> > >
> > > We will implement a simple idempotent design that uses the
> > > WindowDataManager to store the whole message and deletes the
> message
> > from
> > > SQS so we are not impacted by SQS’s visibility timeout feature.
> > >
> > >
> > >
> > > Sanjay
> > >
> >
> >
> >
> >
>
>
> --
>
> Regards,
> Ashwin.
>
>
>
>
Re: JMS Input Operator enhancements
Posted by Pramod Immaneni <pr...@datatorrent.com>.
What is the license covering the libraries, Apache?
On Thu, Jul 14, 2016 at 2:37 PM, Sanjay Pujare <sa...@datatorrent.com>
wrote:
> Sounds good. Will go with test scope.
>
> On 7/14/16, 12:57 PM, "Ashwin Chandra Putta" <as...@gmail.com>
> wrote:
>
> They can be made test scoped in malhar for testing. The user of this
> operator can add the implementation specific dependency in their
> application pom where they use this operator and supply the
> corresponding
> connection factory class name.
>
> Regards,
> Ashwin.
>
> On Thu, Jul 14, 2016 at 11:55 AM, Sanjay Pujare <
> sanjay@datatorrent.com>
> wrote:
>
> > Pramod,
> >
> > The JMS “wrapper” library needs the actual client library as you can
> see
> > in com.datatorrent.lib.io.jms.JMSBase.getConnectionFactory()
> >
> > Class<ConnectionFactory> clazz =
> > (Class<ConnectionFactory>)Class.forName(connectionFactoryClass);
> >
> > Where it loads the client library’s connectionFactoryClass by name.
> > Instead of putting the onus on the caller, I thought we can include
> these
> > libraries, so the current class loader can load the required class.
> >
> > Sanjay
> >
> >
> > On 7/14/16, 11:36 AM, "Pramod Immaneni" <pr...@datatorrent.com>
> wrote:
> >
> > Hi Sanjay,
> >
> > If there are going to separate operators developed for SQS and
> ActiveMQ
> > using the native api, why do we need to add the client libraries
> for
> > those
> > to JMS? Are these only needed for testing?
> >
> > Thanks
> >
> > On Thu, Jul 14, 2016 at 11:16 AM, Sanjay Pujare <
> > sanjay@datatorrent.com>
> > wrote:
> >
> > > Hi all,
> > >
> > > I am proposing the following enhancements to the Malhar JMS
> input
> > operator.
> > > Let me know if you have any comments.
> > >
> > > Enhancements are proposed to the malhar JMS Input Operator (in
> the
> > package
> > > com.datatorrent.lib.io.jms) to make it usable with any JMS
> compatible
> > > message broker for basic input functionality.
> > >
> > > For now we will make this input operator work with ActiveMQ and
> > Amazon SQS
> > > through the JMS interface. The current code contains
> implementation
> > and
> > > tests for ActiveMQ, and we will add support and tests for SQS
> without
> > > breaking the ActiveMQ support.
> > >
> > > The motivation for this enhancement is as follows. JMS is NOT
> a wire
> > > protocol but just an API specification for Java programs, so
> JMS is
> > just a
> > > “wrapper” around an actual implementation library that talks
> to the
> > > respective message broker. Moreover each message broker has
> its own
> > > semantics for more advanced features like partitioning or
> dynamic
> > scaling
> > > that are impossible or difficult to capture via the JMS
> abstraction.
> > With
> > > this enhancement, we expect the JMS input operator to be
> usable as a
> > > generic “JMS input operator” with no support for advanced
> features
> > like
> > > partionability or dynamic scalability.
> > >
> > > Full featured input operators for SQS and ActiveMQ will be
> developed
> > > separately (without necessarily using the JMS interface but
> the most
> > > appropriate interface) and the design and implementation of
> those
> > operators
> > > are out of scope here.
> > >
> > > The enhancement includes:
> > >
> > > -
> > >
> > > adding amazon-sqs-java-messaging-lib and aws-android-sdk-sqs
> > libraries
> > > to the pom
> > > -
> > >
> > > adding test cases for SQS, matching those for ActiveMQ.
> > > -
> > >
> > > making both sets of test cases work
> > > -
> > >
> > > cleaning up the existing code (e.g. removing hard coded
> values
> > like
> > > “TEST.FOO” in JMSBase)
> > >
> > > SQS ConsiderationsNo Support for Topics
> > >
> > > SQS only supports queues, so topics will not be supported for
> the SQS
> > > variant.
> > > Idempotency
> > >
> > > We will implement a simple idempotent design that uses the
> > > WindowDataManager to store the whole message and deletes the
> message
> > from
> > > SQS so we are not impacted by SQS’s visibility timeout feature.
> > >
> > >
> > >
> > > Sanjay
> > >
> >
> >
> >
> >
>
>
> --
>
> Regards,
> Ashwin.
>
>
>
>
Re: JMS Input Operator enhancements
Posted by Sanjay Pujare <sa...@datatorrent.com>.
Sounds good. Will go with test scope.
On 7/14/16, 12:57 PM, "Ashwin Chandra Putta" <as...@gmail.com> wrote:
They can be made test scoped in malhar for testing. The user of this
operator can add the implementation specific dependency in their
application pom where they use this operator and supply the corresponding
connection factory class name.
Regards,
Ashwin.
On Thu, Jul 14, 2016 at 11:55 AM, Sanjay Pujare <sa...@datatorrent.com>
wrote:
> Pramod,
>
> The JMS “wrapper” library needs the actual client library as you can see
> in com.datatorrent.lib.io.jms.JMSBase.getConnectionFactory()
>
> Class<ConnectionFactory> clazz =
> (Class<ConnectionFactory>)Class.forName(connectionFactoryClass);
>
> Where it loads the client library’s connectionFactoryClass by name.
> Instead of putting the onus on the caller, I thought we can include these
> libraries, so the current class loader can load the required class.
>
> Sanjay
>
>
> On 7/14/16, 11:36 AM, "Pramod Immaneni" <pr...@datatorrent.com> wrote:
>
> Hi Sanjay,
>
> If there are going to separate operators developed for SQS and ActiveMQ
> using the native api, why do we need to add the client libraries for
> those
> to JMS? Are these only needed for testing?
>
> Thanks
>
> On Thu, Jul 14, 2016 at 11:16 AM, Sanjay Pujare <
> sanjay@datatorrent.com>
> wrote:
>
> > Hi all,
> >
> > I am proposing the following enhancements to the Malhar JMS input
> operator.
> > Let me know if you have any comments.
> >
> > Enhancements are proposed to the malhar JMS Input Operator (in the
> package
> > com.datatorrent.lib.io.jms) to make it usable with any JMS compatible
> > message broker for basic input functionality.
> >
> > For now we will make this input operator work with ActiveMQ and
> Amazon SQS
> > through the JMS interface. The current code contains implementation
> and
> > tests for ActiveMQ, and we will add support and tests for SQS without
> > breaking the ActiveMQ support.
> >
> > The motivation for this enhancement is as follows. JMS is NOT a wire
> > protocol but just an API specification for Java programs, so JMS is
> just a
> > “wrapper” around an actual implementation library that talks to the
> > respective message broker. Moreover each message broker has its own
> > semantics for more advanced features like partitioning or dynamic
> scaling
> > that are impossible or difficult to capture via the JMS abstraction.
> With
> > this enhancement, we expect the JMS input operator to be usable as a
> > generic “JMS input operator” with no support for advanced features
> like
> > partionability or dynamic scalability.
> >
> > Full featured input operators for SQS and ActiveMQ will be developed
> > separately (without necessarily using the JMS interface but the most
> > appropriate interface) and the design and implementation of those
> operators
> > are out of scope here.
> >
> > The enhancement includes:
> >
> > -
> >
> > adding amazon-sqs-java-messaging-lib and aws-android-sdk-sqs
> libraries
> > to the pom
> > -
> >
> > adding test cases for SQS, matching those for ActiveMQ.
> > -
> >
> > making both sets of test cases work
> > -
> >
> > cleaning up the existing code (e.g. removing hard coded values
> like
> > “TEST.FOO” in JMSBase)
> >
> > SQS ConsiderationsNo Support for Topics
> >
> > SQS only supports queues, so topics will not be supported for the SQS
> > variant.
> > Idempotency
> >
> > We will implement a simple idempotent design that uses the
> > WindowDataManager to store the whole message and deletes the message
> from
> > SQS so we are not impacted by SQS’s visibility timeout feature.
> >
> >
> >
> > Sanjay
> >
>
>
>
>
--
Regards,
Ashwin.
Re: JMS Input Operator enhancements
Posted by Ashwin Chandra Putta <as...@gmail.com>.
They can be made test scoped in malhar for testing. The user of this
operator can add the implementation specific dependency in their
application pom where they use this operator and supply the corresponding
connection factory class name.
Regards,
Ashwin.
On Thu, Jul 14, 2016 at 11:55 AM, Sanjay Pujare <sa...@datatorrent.com>
wrote:
> Pramod,
>
> The JMS “wrapper” library needs the actual client library as you can see
> in com.datatorrent.lib.io.jms.JMSBase.getConnectionFactory()
>
> Class<ConnectionFactory> clazz =
> (Class<ConnectionFactory>)Class.forName(connectionFactoryClass);
>
> Where it loads the client library’s connectionFactoryClass by name.
> Instead of putting the onus on the caller, I thought we can include these
> libraries, so the current class loader can load the required class.
>
> Sanjay
>
>
> On 7/14/16, 11:36 AM, "Pramod Immaneni" <pr...@datatorrent.com> wrote:
>
> Hi Sanjay,
>
> If there are going to separate operators developed for SQS and ActiveMQ
> using the native api, why do we need to add the client libraries for
> those
> to JMS? Are these only needed for testing?
>
> Thanks
>
> On Thu, Jul 14, 2016 at 11:16 AM, Sanjay Pujare <
> sanjay@datatorrent.com>
> wrote:
>
> > Hi all,
> >
> > I am proposing the following enhancements to the Malhar JMS input
> operator.
> > Let me know if you have any comments.
> >
> > Enhancements are proposed to the malhar JMS Input Operator (in the
> package
> > com.datatorrent.lib.io.jms) to make it usable with any JMS compatible
> > message broker for basic input functionality.
> >
> > For now we will make this input operator work with ActiveMQ and
> Amazon SQS
> > through the JMS interface. The current code contains implementation
> and
> > tests for ActiveMQ, and we will add support and tests for SQS without
> > breaking the ActiveMQ support.
> >
> > The motivation for this enhancement is as follows. JMS is NOT a wire
> > protocol but just an API specification for Java programs, so JMS is
> just a
> > “wrapper” around an actual implementation library that talks to the
> > respective message broker. Moreover each message broker has its own
> > semantics for more advanced features like partitioning or dynamic
> scaling
> > that are impossible or difficult to capture via the JMS abstraction.
> With
> > this enhancement, we expect the JMS input operator to be usable as a
> > generic “JMS input operator” with no support for advanced features
> like
> > partionability or dynamic scalability.
> >
> > Full featured input operators for SQS and ActiveMQ will be developed
> > separately (without necessarily using the JMS interface but the most
> > appropriate interface) and the design and implementation of those
> operators
> > are out of scope here.
> >
> > The enhancement includes:
> >
> > -
> >
> > adding amazon-sqs-java-messaging-lib and aws-android-sdk-sqs
> libraries
> > to the pom
> > -
> >
> > adding test cases for SQS, matching those for ActiveMQ.
> > -
> >
> > making both sets of test cases work
> > -
> >
> > cleaning up the existing code (e.g. removing hard coded values
> like
> > “TEST.FOO” in JMSBase)
> >
> > SQS ConsiderationsNo Support for Topics
> >
> > SQS only supports queues, so topics will not be supported for the SQS
> > variant.
> > Idempotency
> >
> > We will implement a simple idempotent design that uses the
> > WindowDataManager to store the whole message and deletes the message
> from
> > SQS so we are not impacted by SQS’s visibility timeout feature.
> >
> >
> >
> > Sanjay
> >
>
>
>
>
--
Regards,
Ashwin.
Re: JMS Input Operator enhancements
Posted by Sanjay Pujare <sa...@datatorrent.com>.
Pramod,
The JMS “wrapper” library needs the actual client library as you can see in com.datatorrent.lib.io.jms.JMSBase.getConnectionFactory()
Class<ConnectionFactory> clazz = (Class<ConnectionFactory>)Class.forName(connectionFactoryClass);
Where it loads the client library’s connectionFactoryClass by name. Instead of putting the onus on the caller, I thought we can include these libraries, so the current class loader can load the required class.
Sanjay
On 7/14/16, 11:36 AM, "Pramod Immaneni" <pr...@datatorrent.com> wrote:
Hi Sanjay,
If there are going to separate operators developed for SQS and ActiveMQ
using the native api, why do we need to add the client libraries for those
to JMS? Are these only needed for testing?
Thanks
On Thu, Jul 14, 2016 at 11:16 AM, Sanjay Pujare <sa...@datatorrent.com>
wrote:
> Hi all,
>
> I am proposing the following enhancements to the Malhar JMS input operator.
> Let me know if you have any comments.
>
> Enhancements are proposed to the malhar JMS Input Operator (in the package
> com.datatorrent.lib.io.jms) to make it usable with any JMS compatible
> message broker for basic input functionality.
>
> For now we will make this input operator work with ActiveMQ and Amazon SQS
> through the JMS interface. The current code contains implementation and
> tests for ActiveMQ, and we will add support and tests for SQS without
> breaking the ActiveMQ support.
>
> The motivation for this enhancement is as follows. JMS is NOT a wire
> protocol but just an API specification for Java programs, so JMS is just a
> “wrapper” around an actual implementation library that talks to the
> respective message broker. Moreover each message broker has its own
> semantics for more advanced features like partitioning or dynamic scaling
> that are impossible or difficult to capture via the JMS abstraction. With
> this enhancement, we expect the JMS input operator to be usable as a
> generic “JMS input operator” with no support for advanced features like
> partionability or dynamic scalability.
>
> Full featured input operators for SQS and ActiveMQ will be developed
> separately (without necessarily using the JMS interface but the most
> appropriate interface) and the design and implementation of those operators
> are out of scope here.
>
> The enhancement includes:
>
> -
>
> adding amazon-sqs-java-messaging-lib and aws-android-sdk-sqs libraries
> to the pom
> -
>
> adding test cases for SQS, matching those for ActiveMQ.
> -
>
> making both sets of test cases work
> -
>
> cleaning up the existing code (e.g. removing hard coded values like
> “TEST.FOO” in JMSBase)
>
> SQS ConsiderationsNo Support for Topics
>
> SQS only supports queues, so topics will not be supported for the SQS
> variant.
> Idempotency
>
> We will implement a simple idempotent design that uses the
> WindowDataManager to store the whole message and deletes the message from
> SQS so we are not impacted by SQS’s visibility timeout feature.
>
>
>
> Sanjay
>
Re: JMS Input Operator enhancements
Posted by Pramod Immaneni <pr...@datatorrent.com>.
Hi Sanjay,
If there are going to separate operators developed for SQS and ActiveMQ
using the native api, why do we need to add the client libraries for those
to JMS? Are these only needed for testing?
Thanks
On Thu, Jul 14, 2016 at 11:16 AM, Sanjay Pujare <sa...@datatorrent.com>
wrote:
> Hi all,
>
> I am proposing the following enhancements to the Malhar JMS input operator.
> Let me know if you have any comments.
>
> Enhancements are proposed to the malhar JMS Input Operator (in the package
> com.datatorrent.lib.io.jms) to make it usable with any JMS compatible
> message broker for basic input functionality.
>
> For now we will make this input operator work with ActiveMQ and Amazon SQS
> through the JMS interface. The current code contains implementation and
> tests for ActiveMQ, and we will add support and tests for SQS without
> breaking the ActiveMQ support.
>
> The motivation for this enhancement is as follows. JMS is NOT a wire
> protocol but just an API specification for Java programs, so JMS is just a
> “wrapper” around an actual implementation library that talks to the
> respective message broker. Moreover each message broker has its own
> semantics for more advanced features like partitioning or dynamic scaling
> that are impossible or difficult to capture via the JMS abstraction. With
> this enhancement, we expect the JMS input operator to be usable as a
> generic “JMS input operator” with no support for advanced features like
> partionability or dynamic scalability.
>
> Full featured input operators for SQS and ActiveMQ will be developed
> separately (without necessarily using the JMS interface but the most
> appropriate interface) and the design and implementation of those operators
> are out of scope here.
>
> The enhancement includes:
>
> -
>
> adding amazon-sqs-java-messaging-lib and aws-android-sdk-sqs libraries
> to the pom
> -
>
> adding test cases for SQS, matching those for ActiveMQ.
> -
>
> making both sets of test cases work
> -
>
> cleaning up the existing code (e.g. removing hard coded values like
> “TEST.FOO” in JMSBase)
>
> SQS ConsiderationsNo Support for Topics
>
> SQS only supports queues, so topics will not be supported for the SQS
> variant.
> Idempotency
>
> We will implement a simple idempotent design that uses the
> WindowDataManager to store the whole message and deletes the message from
> SQS so we are not impacted by SQS’s visibility timeout feature.
>
>
>
> Sanjay
>