You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@apex.apache.org by Ananth Gundabattula <ag...@gmail.com> on 2016/04/17 00:30:55 UTC
Apex engine and malhar maven versions for Kafka 0.9 version operator
Hello all,
I would like to run a unit test code using the new Kafka Operator that
supports the 0.9 version protocol.
In this process, I included the malhar-kafka library version (
3.3.1-incubating ) and am using the apex-engine ( version 3.3.0 ) with
<scope> as test/provided.
The compilation works fine but my unit tests fail to run properly with "
java.lang.ClassNotFoundException: com.datatorrent.lib.util.KryoCloneUtils"
exception.
What is the recommended way to run a unit test which uses Kafka 0.9
operator integrated with the Apex engine ? I am assuming that the
malhar-contrib library kafka operator is not 0.9 compliant ..
The unit test code is like this :
The class CassandraEventDetailsStreamingApp extends the
AbstractKafkaInputOperator in the below snippet of code.
The exception arises in the method lma.getController();
@Test
public void testApplication() throws IOException, Exception {
try {
LocalMode lma = LocalMode.newInstance();
Configuration conf = new Configuration(false);
conf.addResource(this.getClass().getResourceAsStream("/dag-test-props.xml"));
lma.prepareDAG(new CassandraEventDetailsStreamingApp(), conf);
LocalMode.Controller lc = lma.getController();
lc.run();
} catch (ConstraintViolationException e) {
Assert.fail("constraint violations: " + e.getConstraintViolations());
}
}
Regards,
Ananth
Re: Apex engine and malhar maven versions for Kafka 0.9 version operator
Posted by Thomas Weise <th...@gmail.com>.
Ram added an example app for the new Kafka input operator:
https://github.com/DataTorrent/examples/tree/master/tutorials/kafka
Thanks,
Thomas
On Sat, Apr 16, 2016 at 10:47 PM, Ananth Gundabattula <
agundabattula@gmail.com> wrote:
> Thanks for the clarification Siyuan.
>
> On Sun, Apr 17, 2016 at 3:34 PM, hsy541@gmail.com <hs...@gmail.com>
> wrote:
>
>> Hey Ananth,
>>
>> We have semantic version control. For public library/Api, they should
>> stay no change in major release, so most operator in 3.3.x should be able
>> to work with 3.3.0 engine. The only exception is Api that marked
>> @evolving(nonstable). Kafka 0.9 is one of them. But I don't see any feature
>> of that operator couldn't work with 3.3.0 engine. And we will try best to
>> not change the public interface for Kafka 0.9 operator, unless there is a
>> big demand for some new feature from the community
>>
>> Regards,
>> Siyuan
>>
>>
>> On Saturday, April 16, 2016, Ananth Gundabattula <ag...@gmail.com>
>> wrote:
>>
>>> Hello Siyuan/Ram,
>>>
>>> Thanks for the inputs.
>>>
>>> I was able to resolve the problem by excluding the dependencies of
>>> malhar-library and malhar-contrib from the dependency sections of the
>>> apex-engine , apex-api .
>>>
>>> This made the 3.3.1-incubating version of the malhar kick into the
>>> classpath and subsequently the malhar-kafka library with 3.3.1-incubating
>>> version) .
>>>
>>> Thanks for your time.
>>>
>>> This brings in one more question : Is it an acceptable practice that I
>>> package malhar-operator library in the lib path of my "apa" package ? In my
>>> case I had version 3.3.1 in my packaging structure which I want in my
>>> runtime as well ( due to 0.9 kafka operator ) but the apex engine itself is
>>> a version behind ( 3.3.0 ) .. I believe malhar-versions and apex-version
>>> will be on different pace with respect to evolution and versioning ?
>>>
>>>
>>>
>>> Regards,
>>> Ananth
>>>
>>> On Sun, Apr 17, 2016 at 1:43 PM, hsy541@gmail.com <hs...@gmail.com>
>>> wrote:
>>>
>>>> Hi Ananth,
>>>>
>>>> Could you please try to add malhar-library in test scope? Thanks!
>>>>
>>>> Regards,
>>>> Siyuan
>>>>
>>>>
>>>> On Saturday, April 16, 2016, Ananth Gundabattula <
>>>> agundabattula@gmail.com> wrote:
>>>>
>>>>> Hello all,
>>>>>
>>>>> I would like to run a unit test code using the new Kafka Operator that
>>>>> supports the 0.9 version protocol.
>>>>>
>>>>> In this process, I included the malhar-kafka library version (
>>>>> 3.3.1-incubating ) and am using the apex-engine ( version 3.3.0 ) with
>>>>> <scope> as test/provided.
>>>>>
>>>>> The compilation works fine but my unit tests fail to run properly with
>>>>> " java.lang.ClassNotFoundException:
>>>>> com.datatorrent.lib.util.KryoCloneUtils" exception.
>>>>>
>>>>>
>>>>> What is the recommended way to run a unit test which uses Kafka 0.9
>>>>> operator integrated with the Apex engine ? I am assuming that the
>>>>> malhar-contrib library kafka operator is not 0.9 compliant ..
>>>>>
>>>>> The unit test code is like this :
>>>>>
>>>>> The class CassandraEventDetailsStreamingApp extends the
>>>>> AbstractKafkaInputOperator in the below snippet of code.
>>>>>
>>>>> The exception arises in the method lma.getController();
>>>>>
>>>>> @Test
>>>>> public void testApplication() throws IOException, Exception {
>>>>> try {
>>>>> LocalMode lma = LocalMode.newInstance();
>>>>> Configuration conf = new Configuration(false);
>>>>> conf.addResource(this.getClass().getResourceAsStream("/dag-test-props.xml"));
>>>>> lma.prepareDAG(new CassandraEventDetailsStreamingApp(), conf);
>>>>> LocalMode.Controller lc = lma.getController();
>>>>> lc.run();
>>>>> } catch (ConstraintViolationException e) {
>>>>> Assert.fail("constraint violations: " + e.getConstraintViolations());
>>>>> }
>>>>> }
>>>>>
>>>>>
>>>>>
>>>>> Regards,
>>>>> Ananth
>>>>>
>>>>
>>>
>
Re: Apex engine and malhar maven versions for Kafka 0.9 version operator
Posted by Ananth Gundabattula <ag...@gmail.com>.
Thanks for the clarification Siyuan.
On Sun, Apr 17, 2016 at 3:34 PM, hsy541@gmail.com <hs...@gmail.com> wrote:
> Hey Ananth,
>
> We have semantic version control. For public library/Api, they should stay
> no change in major release, so most operator in 3.3.x should be able to
> work with 3.3.0 engine. The only exception is Api that marked
> @evolving(nonstable). Kafka 0.9 is one of them. But I don't see any feature
> of that operator couldn't work with 3.3.0 engine. And we will try best to
> not change the public interface for Kafka 0.9 operator, unless there is a
> big demand for some new feature from the community
>
> Regards,
> Siyuan
>
>
> On Saturday, April 16, 2016, Ananth Gundabattula <ag...@gmail.com>
> wrote:
>
>> Hello Siyuan/Ram,
>>
>> Thanks for the inputs.
>>
>> I was able to resolve the problem by excluding the dependencies of
>> malhar-library and malhar-contrib from the dependency sections of the
>> apex-engine , apex-api .
>>
>> This made the 3.3.1-incubating version of the malhar kick into the
>> classpath and subsequently the malhar-kafka library with 3.3.1-incubating
>> version) .
>>
>> Thanks for your time.
>>
>> This brings in one more question : Is it an acceptable practice that I
>> package malhar-operator library in the lib path of my "apa" package ? In my
>> case I had version 3.3.1 in my packaging structure which I want in my
>> runtime as well ( due to 0.9 kafka operator ) but the apex engine itself is
>> a version behind ( 3.3.0 ) .. I believe malhar-versions and apex-version
>> will be on different pace with respect to evolution and versioning ?
>>
>>
>>
>> Regards,
>> Ananth
>>
>> On Sun, Apr 17, 2016 at 1:43 PM, hsy541@gmail.com <hs...@gmail.com>
>> wrote:
>>
>>> Hi Ananth,
>>>
>>> Could you please try to add malhar-library in test scope? Thanks!
>>>
>>> Regards,
>>> Siyuan
>>>
>>>
>>> On Saturday, April 16, 2016, Ananth Gundabattula <
>>> agundabattula@gmail.com> wrote:
>>>
>>>> Hello all,
>>>>
>>>> I would like to run a unit test code using the new Kafka Operator that
>>>> supports the 0.9 version protocol.
>>>>
>>>> In this process, I included the malhar-kafka library version (
>>>> 3.3.1-incubating ) and am using the apex-engine ( version 3.3.0 ) with
>>>> <scope> as test/provided.
>>>>
>>>> The compilation works fine but my unit tests fail to run properly with
>>>> " java.lang.ClassNotFoundException:
>>>> com.datatorrent.lib.util.KryoCloneUtils" exception.
>>>>
>>>>
>>>> What is the recommended way to run a unit test which uses Kafka 0.9
>>>> operator integrated with the Apex engine ? I am assuming that the
>>>> malhar-contrib library kafka operator is not 0.9 compliant ..
>>>>
>>>> The unit test code is like this :
>>>>
>>>> The class CassandraEventDetailsStreamingApp extends the
>>>> AbstractKafkaInputOperator in the below snippet of code.
>>>>
>>>> The exception arises in the method lma.getController();
>>>>
>>>> @Test
>>>> public void testApplication() throws IOException, Exception {
>>>> try {
>>>> LocalMode lma = LocalMode.newInstance();
>>>> Configuration conf = new Configuration(false);
>>>> conf.addResource(this.getClass().getResourceAsStream("/dag-test-props.xml"));
>>>> lma.prepareDAG(new CassandraEventDetailsStreamingApp(), conf);
>>>> LocalMode.Controller lc = lma.getController();
>>>> lc.run();
>>>> } catch (ConstraintViolationException e) {
>>>> Assert.fail("constraint violations: " + e.getConstraintViolations());
>>>> }
>>>> }
>>>>
>>>>
>>>>
>>>> Regards,
>>>> Ananth
>>>>
>>>
>>
Re: Apex engine and malhar maven versions for Kafka 0.9 version operator
Posted by "hsy541@gmail.com" <hs...@gmail.com>.
Hey Ananth,
We have semantic version control. For public library/Api, they should stay
no change in major release, so most operator in 3.3.x should be able to
work with 3.3.0 engine. The only exception is Api that marked
@evolving(nonstable). Kafka 0.9 is one of them. But I don't see any feature
of that operator couldn't work with 3.3.0 engine. And we will try best to
not change the public interface for Kafka 0.9 operator, unless there is a
big demand for some new feature from the community
Regards,
Siyuan
On Saturday, April 16, 2016, Ananth Gundabattula <ag...@gmail.com>
wrote:
> Hello Siyuan/Ram,
>
> Thanks for the inputs.
>
> I was able to resolve the problem by excluding the dependencies of
> malhar-library and malhar-contrib from the dependency sections of the
> apex-engine , apex-api .
>
> This made the 3.3.1-incubating version of the malhar kick into the
> classpath and subsequently the malhar-kafka library with 3.3.1-incubating
> version) .
>
> Thanks for your time.
>
> This brings in one more question : Is it an acceptable practice that I
> package malhar-operator library in the lib path of my "apa" package ? In my
> case I had version 3.3.1 in my packaging structure which I want in my
> runtime as well ( due to 0.9 kafka operator ) but the apex engine itself is
> a version behind ( 3.3.0 ) .. I believe malhar-versions and apex-version
> will be on different pace with respect to evolution and versioning ?
>
>
>
> Regards,
> Ananth
>
> On Sun, Apr 17, 2016 at 1:43 PM, hsy541@gmail.com
> <javascript:_e(%7B%7D,'cvml','hsy541@gmail.com');> <hsy541@gmail.com
> <javascript:_e(%7B%7D,'cvml','hsy541@gmail.com');>> wrote:
>
>> Hi Ananth,
>>
>> Could you please try to add malhar-library in test scope? Thanks!
>>
>> Regards,
>> Siyuan
>>
>>
>> On Saturday, April 16, 2016, Ananth Gundabattula <agundabattula@gmail.com
>> <javascript:_e(%7B%7D,'cvml','agundabattula@gmail.com');>> wrote:
>>
>>> Hello all,
>>>
>>> I would like to run a unit test code using the new Kafka Operator that
>>> supports the 0.9 version protocol.
>>>
>>> In this process, I included the malhar-kafka library version (
>>> 3.3.1-incubating ) and am using the apex-engine ( version 3.3.0 ) with
>>> <scope> as test/provided.
>>>
>>> The compilation works fine but my unit tests fail to run properly with "
>>> java.lang.ClassNotFoundException: com.datatorrent.lib.util.KryoCloneUtils"
>>> exception.
>>>
>>>
>>> What is the recommended way to run a unit test which uses Kafka 0.9
>>> operator integrated with the Apex engine ? I am assuming that the
>>> malhar-contrib library kafka operator is not 0.9 compliant ..
>>>
>>> The unit test code is like this :
>>>
>>> The class CassandraEventDetailsStreamingApp extends the
>>> AbstractKafkaInputOperator in the below snippet of code.
>>>
>>> The exception arises in the method lma.getController();
>>>
>>> @Test
>>> public void testApplication() throws IOException, Exception {
>>> try {
>>> LocalMode lma = LocalMode.newInstance();
>>> Configuration conf = new Configuration(false);
>>> conf.addResource(this.getClass().getResourceAsStream("/dag-test-props.xml"));
>>> lma.prepareDAG(new CassandraEventDetailsStreamingApp(), conf);
>>> LocalMode.Controller lc = lma.getController();
>>> lc.run();
>>> } catch (ConstraintViolationException e) {
>>> Assert.fail("constraint violations: " + e.getConstraintViolations());
>>> }
>>> }
>>>
>>>
>>>
>>> Regards,
>>> Ananth
>>>
>>
>
Re: Apex engine and malhar maven versions for Kafka 0.9 version operator
Posted by Ananth Gundabattula <ag...@gmail.com>.
Hello Siyuan/Ram,
Thanks for the inputs.
I was able to resolve the problem by excluding the dependencies of
malhar-library and malhar-contrib from the dependency sections of the
apex-engine , apex-api .
This made the 3.3.1-incubating version of the malhar kick into the
classpath and subsequently the malhar-kafka library with 3.3.1-incubating
version) .
Thanks for your time.
This brings in one more question : Is it an acceptable practice that I
package malhar-operator library in the lib path of my "apa" package ? In my
case I had version 3.3.1 in my packaging structure which I want in my
runtime as well ( due to 0.9 kafka operator ) but the apex engine itself is
a version behind ( 3.3.0 ) .. I believe malhar-versions and apex-version
will be on different pace with respect to evolution and versioning ?
Regards,
Ananth
On Sun, Apr 17, 2016 at 1:43 PM, hsy541@gmail.com <hs...@gmail.com> wrote:
> Hi Ananth,
>
> Could you please try to add malhar-library in test scope? Thanks!
>
> Regards,
> Siyuan
>
>
> On Saturday, April 16, 2016, Ananth Gundabattula <ag...@gmail.com>
> wrote:
>
>> Hello all,
>>
>> I would like to run a unit test code using the new Kafka Operator that
>> supports the 0.9 version protocol.
>>
>> In this process, I included the malhar-kafka library version (
>> 3.3.1-incubating ) and am using the apex-engine ( version 3.3.0 ) with
>> <scope> as test/provided.
>>
>> The compilation works fine but my unit tests fail to run properly with "
>> java.lang.ClassNotFoundException: com.datatorrent.lib.util.KryoCloneUtils"
>> exception.
>>
>>
>> What is the recommended way to run a unit test which uses Kafka 0.9
>> operator integrated with the Apex engine ? I am assuming that the
>> malhar-contrib library kafka operator is not 0.9 compliant ..
>>
>> The unit test code is like this :
>>
>> The class CassandraEventDetailsStreamingApp extends the
>> AbstractKafkaInputOperator in the below snippet of code.
>>
>> The exception arises in the method lma.getController();
>>
>> @Test
>> public void testApplication() throws IOException, Exception {
>> try {
>> LocalMode lma = LocalMode.newInstance();
>> Configuration conf = new Configuration(false);
>> conf.addResource(this.getClass().getResourceAsStream("/dag-test-props.xml"));
>> lma.prepareDAG(new CassandraEventDetailsStreamingApp(), conf);
>> LocalMode.Controller lc = lma.getController();
>> lc.run();
>> } catch (ConstraintViolationException e) {
>> Assert.fail("constraint violations: " + e.getConstraintViolations());
>> }
>> }
>>
>>
>>
>> Regards,
>> Ananth
>>
>
Re: Apex engine and malhar maven versions for Kafka 0.9 version operator
Posted by "hsy541@gmail.com" <hs...@gmail.com>.
Hi Ananth,
Could you please try to add malhar-library in test scope? Thanks!
Regards,
Siyuan
On Saturday, April 16, 2016, Ananth Gundabattula <ag...@gmail.com>
wrote:
> Hello all,
>
> I would like to run a unit test code using the new Kafka Operator that
> supports the 0.9 version protocol.
>
> In this process, I included the malhar-kafka library version (
> 3.3.1-incubating ) and am using the apex-engine ( version 3.3.0 ) with
> <scope> as test/provided.
>
> The compilation works fine but my unit tests fail to run properly with "
> java.lang.ClassNotFoundException: com.datatorrent.lib.util.KryoCloneUtils"
> exception.
>
>
> What is the recommended way to run a unit test which uses Kafka 0.9
> operator integrated with the Apex engine ? I am assuming that the
> malhar-contrib library kafka operator is not 0.9 compliant ..
>
> The unit test code is like this :
>
> The class CassandraEventDetailsStreamingApp extends the
> AbstractKafkaInputOperator in the below snippet of code.
>
> The exception arises in the method lma.getController();
>
> @Test
> public void testApplication() throws IOException, Exception {
> try {
> LocalMode lma = LocalMode.newInstance();
> Configuration conf = new Configuration(false);
> conf.addResource(this.getClass().getResourceAsStream("/dag-test-props.xml"));
> lma.prepareDAG(new CassandraEventDetailsStreamingApp(), conf);
> LocalMode.Controller lc = lma.getController();
> lc.run();
> } catch (ConstraintViolationException e) {
> Assert.fail("constraint violations: " + e.getConstraintViolations());
> }
> }
>
>
>
> Regards,
> Ananth
>
Re: Apex engine and malhar maven versions for Kafka 0.9 version operator
Posted by Munagala Ramanath <ra...@datatorrent.com>.
Ananth,
Could you make your pom.xml accessible somewhere ?
Ram
On Sat, Apr 16, 2016 at 3:30 PM, Ananth Gundabattula <
agundabattula@gmail.com> wrote:
> Hello all,
>
> I would like to run a unit test code using the new Kafka Operator that
> supports the 0.9 version protocol.
>
> In this process, I included the malhar-kafka library version (
> 3.3.1-incubating ) and am using the apex-engine ( version 3.3.0 ) with
> <scope> as test/provided.
>
> The compilation works fine but my unit tests fail to run properly with "
> java.lang.ClassNotFoundException: com.datatorrent.lib.util.KryoCloneUtils"
> exception.
>
>
> What is the recommended way to run a unit test which uses Kafka 0.9
> operator integrated with the Apex engine ? I am assuming that the
> malhar-contrib library kafka operator is not 0.9 compliant ..
>
> The unit test code is like this :
>
> The class CassandraEventDetailsStreamingApp extends the
> AbstractKafkaInputOperator in the below snippet of code.
>
> The exception arises in the method lma.getController();
>
> @Test
> public void testApplication() throws IOException, Exception {
> try {
> LocalMode lma = LocalMode.newInstance();
> Configuration conf = new Configuration(false);
> conf.addResource(this.getClass().getResourceAsStream("/dag-test-props.xml"));
> lma.prepareDAG(new CassandraEventDetailsStreamingApp(), conf);
> LocalMode.Controller lc = lma.getController();
> lc.run();
> } catch (ConstraintViolationException e) {
> Assert.fail("constraint violations: " + e.getConstraintViolations());
> }
> }
>
>
>
> Regards,
> Ananth
>