You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@cassandra.apache.org by Patrick McFadin <pm...@gmail.com> on 2015/11/24 01:31:21 UTC
Re: Hadoop + Yarn / DataStax driver issues
That's a pretty basic error from the driver. I would look at the target
node and make sure that the IP/port combination is actually bound and free
of any firewall shenanigans.
Patrick
On Mon, Nov 23, 2015 at 4:00 PM, Cyril Scetbon <cy...@free.fr>
wrote:
> Let me also add that I use Cassandra 2.1.11 with Hadoop 2.7.1 (HDP 2.3.2).
>
> > Trying to execute an Hadoop job on Yarn, I got errors from Cassandra's
> internal code. It seems that connections are closed but I can't understand
> why ...
> >
> > Here is a subtract of the errors :
> >
> > 15/11/22 20:05:54 [main]: DEBUG core.RequestHandler: Error querying
> node006.internal.net/192.168.12.22:9042, trying next host (error is:
> com.datastax.driver.core.ConnectionException: [
> node006.internal.net/192.168.12.22:9042] Pool is shutdown)
> > Failed with exception java.io.IOException:java.io.IOException:
> com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s)
> tried
> > for
> > query failed (tried: node006.internal.net/192.168.12.22:9042
> (com.datastax.driver.core.ConnectionException: [
> node006.internal.net/192.168.12.22:9042] Pool is shutdown))
> > 15/11/22 20:05:54 [main]: ERROR CliDriver: Failed with exception
> java.io.IOException:java.io.IOException:
> com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s)
> tried
> > for
> > query failed (tried: node006.internal.net/192.168.12.22:9042
> (com.datastax.driver.core.ConnectionException: [
> node006.internal.net/192.168.12.22:9042] Pool is shutdown))
> > java.io.IOException: java.io.IOException:
> com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s)
> tried
> > for
> > query failed (tried: node006.internal.net/192.168.12.22:9042
> (com.datastax.driver.core.ConnectionException: [
> node006.internal.net/192.168.12.22:9042] Pool is shutdown))
> > at
> org.apache.hadoop.hive.ql.exec.FetchOperator.getNextRow(FetchOperator.java:508)
> > at
> org.apache.hadoop.hive.ql.exec.FetchOperator.pushRow(FetchOperator.java:415)
> > at
> org.apache.hadoop.hive.ql.exec.FetchTask.fetch(FetchTask.java:140)
> > at org.apache.hadoop.hive.ql.Driver.getResults(Driver.java:1672)
> > at
> org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:233)
> > at
> org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:165)
> > at
> org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:376)
> > at
> org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:736)
> > at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:681)
> > at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:621)
> > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> > at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> > at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> > at java.lang.reflect.Method.invoke(Method.java:497)
> > at org.apache.hadoop.util.RunJar.run(RunJar.java:221)
> > at org.apache.hadoop.util.RunJar.main(RunJar.java:136)
> > Caused by: java.io.IOException:
> com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s)
> tried
> > for
> > query failed (tried: node006.internal.net/192.168.12.22:9042
> (com.datastax.driver.core.ConnectionException: [
> node006.internal.net/192.168.12.22:9042] Pool is shutdown))
> > at
> org.apache.hadoop.hive.cassandra.input.cql.HiveCqlInputFormat.getRecordReader(HiveCqlInputFormat.java:132)
> > at
> org.apache.hadoop.hive.ql.exec.FetchOperator$FetchInputFormatSplit.getRecordReader(FetchOperator.java:674)
> > at
> org.apache.hadoop.hive.ql.exec.FetchOperator.getRecordReader(FetchOperator.java:324)
> > at
> org.apache.hadoop.hive.ql.exec.FetchOperator.getNextRow(FetchOperator.java:446)
> > ... 15 more
> > Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException:
> All host(s) tried
> > for
> > query failed (tried: node006.internal.net/192.168.12.22:9042
> (com.datastax.driver.core.ConnectionException: [
> node006.internal.net/192.168.12.22:9042] Pool is shutdown))
> > at
> com.datastax.driver.core.exceptions.NoHostAvailableException.copy(NoHostAvailableException.java:84)
> > at
> com.datastax.driver.core.DriverThrowables.propagateCause(DriverThrowables.java:37)
> > at
> com.datastax.driver.core.DefaultResultSetFuture.getUninterruptibly(DefaultResultSetFuture.java:214)
> > at
> com.datastax.driver.core.AbstractSession.execute(AbstractSession.java:52)
> > at
> com.datastax.driver.core.AbstractSession.execute(AbstractSession.java:36)
> > at
> org.apache.cassandra.hadoop.cql3.CqlRecordReader.fetchKeys(CqlRecordReader.java:578)
> > at
> org.apache.cassandra.hadoop.cql3.CqlRecordReader.buildQuery(CqlRecordReader.java:526)
> > at
> org.apache.cassandra.hadoop.cql3.CqlRecordReader.initialize(CqlRecordReader.java:148)
> I also add a file with the complete debug logs.
> >
> > You can also find the complete log at
> https://www.dropbox.com/s/thwugpw8amv5x9q/output.log?dl=0
>
>
Re: Hadoop + Yarn / DataStax driver issues
Posted by Cyril Scetbon <cy...@free.fr>.
Similar tests with Spark and cassandra connector (which uses the native driver too) work well ... I was wondering if Yarn was restricting the network access but even executing spark (using spark-shell) with --master yarn-client or --master yarn-server worked without any issue. You could also see in the logs that it succeeded to connect to node006.internal.net
[cluster1-nio-worker-0]: DEBUG core.Connection: Connection[node006.internal.net/192.168.12.22:9042-1, inFlight=0, closed=false] Connection opened successfully
[cluster1-nio-worker-1]: DEBUG Host.STATES: [node006.internal.net/192.168.12.22:9042] new connection created, total = 2
then later got
[node006.internal.net/192.168.12.22:9042] Pool is shutdown
However I've seen that before that, there is an error :
[cluster1-blocking-task-worker-0]: DEBUG Host.STATES: Defuncting Connection[node006.internal.net/192.168.12.22:9042-3, inFlight=0, closed=true] because: [node006.internal.net/192.168.12.22:9042] Error while setting keyspace
and after that error the node is marked as down :
DEBUG Host.STATES: [node006.internal.net/192.168.12.22:9042] marking host DOWN
still investigating ...
> On Nov 23, 2015, at 19:31, Patrick McFadin <pm...@gmail.com> wrote:
>
> That's a pretty basic error from the driver. I would look at the target node and make sure that the IP/port combination is actually bound and free of any firewall shenanigans.
>
> Patrick
>
> On Mon, Nov 23, 2015 at 4:00 PM, Cyril Scetbon <cyril.scetbon@free.fr <ma...@free.fr>> wrote:
> Let me also add that I use Cassandra 2.1.11 with Hadoop 2.7.1 (HDP 2.3.2).
>
> > Trying to execute an Hadoop job on Yarn, I got errors from Cassandra's internal code. It seems that connections are closed but I can't understand why ...
> >
> > Here is a subtract of the errors :
> >
> > 15/11/22 20:05:54 [main]: DEBUG core.RequestHandler: Error querying node006.internal.net/192.168.12.22:9042 <http://node006.internal.net/192.168.12.22:9042>, trying next host (error is: com.datastax.driver.core.ConnectionException: [node006.internal.net/192.168.12.22:9042 <http://node006.internal.net/192.168.12.22:9042>] Pool is shutdown)
> > Failed with exception java.io.IOException:java.io.IOException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried
> > for
> > query failed (tried: node006.internal.net/192.168.12.22:9042 <http://node006.internal.net/192.168.12.22:9042> (com.datastax.driver.core.ConnectionException: [node006.internal.net/192.168.12.22:9042 <http://node006.internal.net/192.168.12.22:9042>] Pool is shutdown))
> > 15/11/22 20:05:54 [main]: ERROR CliDriver: Failed with exception java.io.IOException:java.io.IOException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried
> > for
> > query failed (tried: node006.internal.net/192.168.12.22:9042 <http://node006.internal.net/192.168.12.22:9042> (com.datastax.driver.core.ConnectionException: [node006.internal.net/192.168.12.22:9042 <http://node006.internal.net/192.168.12.22:9042>] Pool is shutdown))
> > java.io.IOException: java.io.IOException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried
> > for
> > query failed (tried: node006.internal.net/192.168.12.22:9042 <http://node006.internal.net/192.168.12.22:9042> (com.datastax.driver.core.ConnectionException: [node006.internal.net/192.168.12.22:9042 <http://node006.internal.net/192.168.12.22:9042>] Pool is shutdown))
> > at org.apache.hadoop.hive.ql.exec.FetchOperator.getNextRow(FetchOperator.java:508)
> > at org.apache.hadoop.hive.ql.exec.FetchOperator.pushRow(FetchOperator.java:415)
> > at org.apache.hadoop.hive.ql.exec.FetchTask.fetch(FetchTask.java:140)
> > at org.apache.hadoop.hive.ql.Driver.getResults(Driver.java:1672)
> > at org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:233)
> > at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:165)
> > at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:376)
> > at org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:736)
> > at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:681)
> > at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:621)
> > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> > at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> > at java.lang.reflect.Method.invoke(Method.java:497)
> > at org.apache.hadoop.util.RunJar.run(RunJar.java:221)
> > at org.apache.hadoop.util.RunJar.main(RunJar.java:136)
> > Caused by: java.io.IOException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried
> > for
> > query failed (tried: node006.internal.net/192.168.12.22:9042 <http://node006.internal.net/192.168.12.22:9042> (com.datastax.driver.core.ConnectionException: [node006.internal.net/192.168.12.22:9042 <http://node006.internal.net/192.168.12.22:9042>] Pool is shutdown))
> > at org.apache.hadoop.hive.cassandra.input.cql.HiveCqlInputFormat.getRecordReader(HiveCqlInputFormat.java:132)
> > at org.apache.hadoop.hive.ql.exec.FetchOperator$FetchInputFormatSplit.getRecordReader(FetchOperator.java:674)
> > at org.apache.hadoop.hive.ql.exec.FetchOperator.getRecordReader(FetchOperator.java:324)
> > at org.apache.hadoop.hive.ql.exec.FetchOperator.getNextRow(FetchOperator.java:446)
> > ... 15 more
> > Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried
> > for
> > query failed (tried: node006.internal.net/192.168.12.22:9042 <http://node006.internal.net/192.168.12.22:9042> (com.datastax.driver.core.ConnectionException: [node006.internal.net/192.168.12.22:9042 <http://node006.internal.net/192.168.12.22:9042>] Pool is shutdown))
> > at com.datastax.driver.core.exceptions.NoHostAvailableException.copy(NoHostAvailableException.java:84)
> > at com.datastax.driver.core.DriverThrowables.propagateCause(DriverThrowables.java:37)
> > at com.datastax.driver.core.DefaultResultSetFuture.getUninterruptibly(DefaultResultSetFuture.java:214)
> > at com.datastax.driver.core.AbstractSession.execute(AbstractSession.java:52)
> > at com.datastax.driver.core.AbstractSession.execute(AbstractSession.java:36)
> > at org.apache.cassandra.hadoop.cql3.CqlRecordReader.fetchKeys(CqlRecordReader.java:578)
> > at org.apache.cassandra.hadoop.cql3.CqlRecordReader.buildQuery(CqlRecordReader.java:526)
> > at org.apache.cassandra.hadoop.cql3.CqlRecordReader.initialize(CqlRecordReader.java:148) I also add a file with the complete debug logs.
> >
> > You can also find the complete log at https://www.dropbox.com/s/thwugpw8amv5x9q/output.log?dl=0 <https://www.dropbox.com/s/thwugpw8amv5x9q/output.log?dl=0>
>
>