You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@apex.apache.org by Akshay S Harale <ak...@synerzip.com> on 2016/06/29 09:33:29 UTC
Running application using apex CLI
In our application we are fetching data from kafka, doing some
modifications in data and then persist that data into elastic search.
As we are using elasticsearch verion 2.2.3, first we did upgrade of elastic
search client API in malhar contrib from 1.1.2 to 2.3.3 [ here is pull
request : https://github.com/apache/apex-malhar/pull/325 ]
With new elastic search api we able to run the application in local mode
using junite test case which worked very well and able to see the data in
elastic search.
Now we are trying to run our application using APEX cli. First we get the
exception about the googles guava library:
*java.lang.NoSuchMethodError:
com.google.common.util.concurrent.MoreExecutors.directExecutor()Ljava/util/concurrent/Executor;
at
org.elasticsearch.threadpool.ThreadPool.<clinit>(ThreadPool.java:190)
at
org.elasticsearch.client.transport.TransportClient$Builder.build(TransportClient.java:131)
at
com.datatorrent.contrib.elasticsearch.ElasticSearchConnectable.connect(ElasticSearchConnectable.java:123)*
The reason behind above exception is : elastic search java client requires
guava 18.0 and the Hadoop 2.2 comes with guava 11.0.2.
In our application package (apa file) we have guava 18.0 jar. But i think
while launching application apex gives preference to the guava 11.0.2.
So we updated the apex-engine.jar and changed guava version to 18.0
manually which is not good practice :)
After this we are able to see that application is running in hadoop cluster
but in log there is an exception :
*2016-06-28 15:54:28,721 ERROR com.datatorrent.netlet.AbstractClient:
Exception in event loop {id=ProcessWideEventLoop, head=1, tail=1,
capacity=1024} java.net.ConnectException: Connection refused at
sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) at
sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
at
com.datatorrent.netlet.DefaultEventLoop.handleSelectedKey(DefaultEventLoop.java:371)
at
com.datatorrent.netlet.OptimizedEventLoop$SelectedSelectionKeySet.forEach(OptimizedEventLoop.java:59)
at
com.datatorrent.netlet.OptimizedEventLoop.runEventLoop(OptimizedEventLoop.java:192)*
*We want to know about how to override the default apex dependencies with
dependency that we are using in our application.*
Regards,
Akshay S. Harale
Software Developer @ Synerzip
Skype – akshayharale
--
This e-mail, including any attached files, may contain confidential and
privileged information for the sole use of the intended recipient. Any
review, use, distribution, or disclosure by others is strictly prohibited.
If you are not the intended recipient (or authorized to receive information
for the intended recipient), please contact the sender by reply e-mail and
delete all copies of this message.
Re: Running application using apex CLI
Posted by Chinmay Kolhatkar <ch...@datatorrent.com>.
Hi Akshay,
Thanks for contribution to apex-malhar about update to elastic search
operators.
We have a contributing guidelines on apex website (
http://apex.apache.org/contributing.html).
One of the important steps in contributing to apex is to communicate on
dev@apex.apache.org and share your intent and approach with the community.
This is critical for your changes as there are dependency changes involved
in your PR and development community should be able to discuss and share
your feedback before changes can be merged.
About guava dependency, as such apex does not provide a way to override
default dependency.
I would not recommend to change guava dependency in apex. The guava
dependency is essentially coming from hadoop and not apex.
But to make your case work, there is a way to shade dependency of guava and
elastic search using maven. Please have a look at maven shade plugin:
https://maven.apache.org/plugins/maven-shade-plugin/
Thanks,
Chinmay.
On Wed, Jun 29, 2016 at 2:33 AM, Akshay S Harale <akshay.harale@synerzip.com
> wrote:
> In our application we are fetching data from kafka, doing some
> modifications in data and then persist that data into elastic search.
>
> As we are using elasticsearch verion 2.2.3, first we did upgrade of
> elastic search client API in malhar contrib from 1.1.2 to 2.3.3 [ here is
> pull request : https://github.com/apache/apex-malhar/pull/325 ]
>
> With new elastic search api we able to run the application in local mode
> using junite test case which worked very well and able to see the data in
> elastic search.
>
> Now we are trying to run our application using APEX cli. First we get the
> exception about the googles guava library:
>
>
>
>
> *java.lang.NoSuchMethodError:
> com.google.common.util.concurrent.MoreExecutors.directExecutor()Ljava/util/concurrent/Executor;
> at
> org.elasticsearch.threadpool.ThreadPool.<clinit>(ThreadPool.java:190)
> at
> org.elasticsearch.client.transport.TransportClient$Builder.build(TransportClient.java:131)
> at
> com.datatorrent.contrib.elasticsearch.ElasticSearchConnectable.connect(ElasticSearchConnectable.java:123)*
>
> The reason behind above exception is : elastic search java client
> requires guava 18.0 and the Hadoop 2.2 comes with guava 11.0.2.
> In our application package (apa file) we have guava 18.0 jar. But i think
> while launching application apex gives preference to the guava 11.0.2.
>
> So we updated the apex-engine.jar and changed guava version to 18.0
> manually which is not good practice :)
> After this we are able to see that application is running in hadoop
> cluster but in log there is an exception :
>
>
>
>
>
>
>
> *2016-06-28 15:54:28,721 ERROR com.datatorrent.netlet.AbstractClient:
> Exception in event loop {id=ProcessWideEventLoop, head=1, tail=1,
> capacity=1024} java.net.ConnectException: Connection refused at
> sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) at
> sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
> at
> com.datatorrent.netlet.DefaultEventLoop.handleSelectedKey(DefaultEventLoop.java:371)
> at
> com.datatorrent.netlet.OptimizedEventLoop$SelectedSelectionKeySet.forEach(OptimizedEventLoop.java:59)
> at
> com.datatorrent.netlet.OptimizedEventLoop.runEventLoop(OptimizedEventLoop.java:192)*
>
> *We want to know about how to override the default apex dependencies with
> dependency that we are using in our application.*
>
> Regards,
> Akshay S. Harale
> Software Developer @ Synerzip
> Skype – akshayharale
>
> This e-mail, including any attached files, may contain confidential and
> privileged information for the sole use of the intended recipient. Any
> review, use, distribution, or disclosure by others is strictly prohibited.
> If you are not the intended recipient (or authorized to receive information
> for the intended recipient), please contact the sender by reply e-mail and
> delete all copies of this message.
>
>