Regarding issue APEXMALHAR-2122

[Akshay S Harale <ak...@synerzip.com>]

Hi,

I have created the pull request for issue APEXMALHAR-2122.

The reason behind this is to support the latest version of elastic search.
Also updated the AbstractElasticSearchOutputOperator.processBatch() api to
process the batch only if its not empty.
I have created the pull request for the same : apex-malhar 325
<https://github.com/apache/apex-malhar/pull/325>

I created the pull request after running the example in local mode(junit
test case).

Now I am trying to run application using APEX cli. First got the exception
about the googles guava library:




*java.lang.NoSuchMethodError:
com.google.common.util.concurrent.MoreExecutors.directExecutor()Ljava/util/concurrent/Executor;
      at
org.elasticsearch.threadpool.ThreadPool.<clinit>(ThreadPool.java:190)
  at
org.elasticsearch.client.transport.TransportClient$Builder.build(TransportClient.java:131)
      at
com.datatorrent.contrib.elasticsearch.ElasticSearchConnectable.connect(ElasticSearchConnectable.java:123)*

The reason behind above exception is : elastic search java client requires
guava 18.0 and the Hadoop 2.2 comes with guava 11.0.2.
In our application package (apa file) we have guava 18.0 jar. But i think
while launching application apex gives preference to the guava 11.0.2.

So we updated the apex-engine.jar and changed guava version to 18.0
manually which is not good practice :)
After this we are able to see that application is running in hadoop cluster
but in log there is an exception :







*2016-06-28 15:54:28,721 ERROR com.datatorrent.netlet.AbstractClient:
Exception in event loop {id=ProcessWideEventLoop, head=1, tail=1,
capacity=1024} java.net.ConnectException: Connection refused         at
sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)         at
sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
    at
com.datatorrent.netlet.DefaultEventLoop.handleSelectedKey(DefaultEventLoop.java:371)
      at
com.datatorrent.netlet.OptimizedEventLoop$SelectedSelectionKeySet.forEach(OptimizedEventLoop.java:59)
      at
com.datatorrent.netlet.OptimizedEventLoop.runEventLoop(OptimizedEventLoop.java:192)*

Also tried using maven shade plugin to include guava classes in output jar
but that also not working.

Regards,
Akshay

-- 
This e-mail, including any attached files, may contain confidential and 
privileged information for the sole use of the intended recipient. Any 
review, use, distribution, or disclosure by others is strictly prohibited. 
If you are not the intended recipient (or authorized to receive information 
for the intended recipient), please contact the sender by reply e-mail and 
delete all copies of this message.

Re: Regarding issue APEXMALHAR-2122

[Priyanka Gugale <pr...@datatorrent.com>]

I am not very sure, but I would suggest to explore option of using multiple
class loaders. It will be tricky to make elastic search code use your
custom class loader.
Loading different versions of same class in one class loader seems to be
impossible.

-Priyanka

On Tue, Jul 5, 2016 at 4:40 PM, Akshay S Harale <ak...@synerzip.com>
wrote:

> Hi,
>
> I have created the pull request for issue APEXMALHAR-2122.
>
> The reason behind this is to support the latest version of elastic search.
> Also updated the AbstractElasticSearchOutputOperator.processBatch() api to
> process the batch only if its not empty.
> I have created the pull request for the same : apex-malhar 325
> <https://github.com/apache/apex-malhar/pull/325>
>
> I created the pull request after running the example in local mode(junit
> test case).
>
> Now I am trying to run application using APEX cli. First got the exception
> about the googles guava library:
>
>
>
>
> *java.lang.NoSuchMethodError:
>
> com.google.common.util.concurrent.MoreExecutors.directExecutor()Ljava/util/concurrent/Executor;
>       at
> org.elasticsearch.threadpool.ThreadPool.<clinit>(ThreadPool.java:190)
>   at
>
> org.elasticsearch.client.transport.TransportClient$Builder.build(TransportClient.java:131)
>       at
>
> com.datatorrent.contrib.elasticsearch.ElasticSearchConnectable.connect(ElasticSearchConnectable.java:123)*
>
> The reason behind above exception is : elastic search java client requires
> guava 18.0 and the Hadoop 2.2 comes with guava 11.0.2.
> In our application package (apa file) we have guava 18.0 jar. But i think
> while launching application apex gives preference to the guava 11.0.2.
>
> So we updated the apex-engine.jar and changed guava version to 18.0
> manually which is not good practice :)
> After this we are able to see that application is running in hadoop cluster
> but in log there is an exception :
>
>
>
>
>
>
>
> *2016-06-28 15:54:28,721 ERROR com.datatorrent.netlet.AbstractClient:
> Exception in event loop {id=ProcessWideEventLoop, head=1, tail=1,
> capacity=1024} java.net.ConnectException: Connection refused         at
> sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)         at
> sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
>     at
>
> com.datatorrent.netlet.DefaultEventLoop.handleSelectedKey(DefaultEventLoop.java:371)
>       at
>
> com.datatorrent.netlet.OptimizedEventLoop$SelectedSelectionKeySet.forEach(OptimizedEventLoop.java:59)
>       at
>
> com.datatorrent.netlet.OptimizedEventLoop.runEventLoop(OptimizedEventLoop.java:192)*
>
> Also tried using maven shade plugin to include guava classes in output jar
> but that also not working.
>
> Regards,
> Akshay
>
> --
> This e-mail, including any attached files, may contain confidential and
> privileged information for the sole use of the intended recipient. Any
> review, use, distribution, or disclosure by others is strictly prohibited.
> If you are not the intended recipient (or authorized to receive information
> for the intended recipient), please contact the sender by reply e-mail and
> delete all copies of this message.
>
>