You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@cassandra.apache.org by "mck (JIRA)" <ji...@apache.org> on 2018/05/01 11:45:00 UTC
[jira] [Comment Edited] (CASSANDRA-10751) "Pool is shutdown" error
when running Hadoop jobs on Yarn
[ https://issues.apache.org/jira/browse/CASSANDRA-10751?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16459637#comment-16459637 ]
mck edited comment on CASSANDRA-10751 at 5/1/18 11:44 AM:
----------------------------------------------------------
Sorry [~cscetbon] that this got completely forgotten. The patch makes sense.
I've put your patch into relevant branches, and will commit once they go green.
|| Branch || uTest || dTest ||
|[cassandra-2.2_10751|https://github.com/thelastpickle/cassandra/tree/mck/cassandra-2.2_10751]|[!https://circleci.com/gh/thelastpickle/cassandra/tree/mck%2Fcassandra-2.2_10751.svg?style=svg!|https://circleci.com/gh/thelastpickle/cassandra/tree/mck%2Fcassandra-2.2_10751]| https://builds.apache.org/view/A-D/view/Cassandra/job/Cassandra-devbranch-dtest/534/ |
|[cassandra-3.0_10751|https://github.com/thelastpickle/cassandra/tree/mck/cassandra-3.0_10751]|[!https://circleci.com/gh/thelastpickle/cassandra/tree/mck%2Fcassandra-3.0_10751.svg?style=svg!|https://circleci.com/gh/thelastpickle/cassandra/tree/mck%2Fcassandra-3.0_10751]| https://builds.apache.org/view/A-D/view/Cassandra/job/Cassandra-devbranch-dtest/535/ |
|[cassandra-3.11_10751|https://github.com/thelastpickle/cassandra/tree/mck/cassandra-3.11_10751]|[!https://circleci.com/gh/thelastpickle/cassandra/tree/mck%2Fcassandra-3.11_10751.svg?style=svg!|https://circleci.com/gh/thelastpickle/cassandra/tree/mck%2Fcassandra-3.11_10751]| https://builds.apache.org/view/A-D/view/Cassandra/job/Cassandra-devbranch-dtest/536/ |
|[trunk_10751|https://github.com/thelastpickle/cassandra/tree/mck/trunk_10751]|[!https://circleci.com/gh/thelastpickle/cassandra/tree/mck%2Ftrunk_10751.svg?style=svg!|https://circleci.com/gh/thelastpickle/cassandra/tree/mck%2Ftrunk_10751]| https://builds.apache.org/view/A-D/view/Cassandra/job/Cassandra-devbranch-dtest/537/ |
was (Author: michaelsembwever):
Sorry [~cscetbon] that this got completely forgotten. The patch makes sense.
I've put your patch into relevant branches, and will commit once they go green.
Here's the patch for trunk, 3.0 and 3.11:
|| Branch || uTest || dTest ||
|[cassandra-2.2_10751|https://github.com/thelastpickle/cassandra/tree/mck/cassandra-2.2_10751]|[!https://circleci.com/gh/thelastpickle/cassandra/tree/mck%2Fcassandra-2.2_10751.svg?style=svg!|https://circleci.com/gh/thelastpickle/cassandra/tree/mck%2Fcassandra-2.2_10751]| https://builds.apache.org/view/A-D/view/Cassandra/job/Cassandra-devbranch-dtest/534/ |
|[cassandra-3.0_10751|https://github.com/thelastpickle/cassandra/tree/mck/cassandra-3.0_10751]|[!https://circleci.com/gh/thelastpickle/cassandra/tree/mck%2Fcassandra-3.0_10751.svg?style=svg!|https://circleci.com/gh/thelastpickle/cassandra/tree/mck%2Fcassandra-3.0_10751]| https://builds.apache.org/view/A-D/view/Cassandra/job/Cassandra-devbranch-dtest/535/ |
|[cassandra-3.11_10751|https://github.com/thelastpickle/cassandra/tree/mck/cassandra-3.11_10751]|[!https://circleci.com/gh/thelastpickle/cassandra/tree/mck%2Fcassandra-3.11_10751.svg?style=svg!|https://circleci.com/gh/thelastpickle/cassandra/tree/mck%2Fcassandra-3.11_10751]| https://builds.apache.org/view/A-D/view/Cassandra/job/Cassandra-devbranch-dtest/536/ |
|[trunk_10751|https://github.com/thelastpickle/cassandra/tree/mck/trunk_10751]|[!https://circleci.com/gh/thelastpickle/cassandra/tree/mck%2Ftrunk_10751.svg?style=svg!|https://circleci.com/gh/thelastpickle/cassandra/tree/mck%2Ftrunk_10751]| https://builds.apache.org/view/A-D/view/Cassandra/job/Cassandra-devbranch-dtest/537/ |
> "Pool is shutdown" error when running Hadoop jobs on Yarn
> ---------------------------------------------------------
>
> Key: CASSANDRA-10751
> URL: https://issues.apache.org/jira/browse/CASSANDRA-10751
> Project: Cassandra
> Issue Type: Bug
> Environment: Hadoop 2.7.1 (HDP 2.3.2)
> Cassandra 2.1.11
> Reporter: Cyril Scetbon
> Assignee: Cyril Scetbon
> Priority: Major
> Attachments: CASSANDRA-10751-2.2.patch, CASSANDRA-10751-3.0.patch, output.log
>
>
> Trying to execute an Hadoop job on Yarn, I get errors from Cassandra's internal code. It seems that connections are shutdown but we can't understand why ...
> Here is a subtract of the errors. I also add a file with the complete debug logs.
> {code}
> 15/11/22 20:05:54 [main]: DEBUG core.RequestHandler: Error querying node006.internal.net/192.168.12.22:9042, trying next host (error is: com.datastax.driver.core.ConnectionException: [node006.internal.net/192.168.12.22:9042] Pool is shutdown)
> Failed with exception java.io.IOException:java.io.IOException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: node006.internal.net/192.168.12.22:9042 (com.datastax.driver.core.ConnectionException: [node006.internal.net/192.168.12.22:9042] Pool is shutdown))
> 15/11/22 20:05:54 [main]: ERROR CliDriver: Failed with exception java.io.IOException:java.io.IOException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: node006.internal.net/192.168.12.22:9042 (com.datastax.driver.core.ConnectionException: [node006.internal.net/192.168.12.22:9042] Pool is shutdown))
> java.io.IOException: java.io.IOException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: node006.internal.net/192.168.12.22:9042 (com.datastax.driver.core.ConnectionException: [node006.internal.net/192.168.12.22:9042] Pool is shutdown))
> at org.apache.hadoop.hive.ql.exec.FetchOperator.getNextRow(FetchOperator.java:508)
> at org.apache.hadoop.hive.ql.exec.FetchOperator.pushRow(FetchOperator.java:415)
> at org.apache.hadoop.hive.ql.exec.FetchTask.fetch(FetchTask.java:140)
> at org.apache.hadoop.hive.ql.Driver.getResults(Driver.java:1672)
> at org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:233)
> at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:165)
> at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:376)
> at org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:736)
> at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:681)
> at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:621)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:497)
> at org.apache.hadoop.util.RunJar.run(RunJar.java:221)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:136)
> Caused by: java.io.IOException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: node006.internal.net/192.168.12.22:9042 (com.datastax.driver.core.ConnectionException: [node006.internal.net/192.168.12.22:9042] Pool is shutdown))
> at org.apache.hadoop.hive.cassandra.input.cql.HiveCqlInputFormat.getRecordReader(HiveCqlInputFormat.java:132)
> at org.apache.hadoop.hive.ql.exec.FetchOperator$FetchInputFormatSplit.getRecordReader(FetchOperator.java:674)
> at org.apache.hadoop.hive.ql.exec.FetchOperator.getRecordReader(FetchOperator.java:324)
> at org.apache.hadoop.hive.ql.exec.FetchOperator.getNextRow(FetchOperator.java:446)
> ... 15 more
> Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: node006.internal.net/192.168.12.22:9042 (com.datastax.driver.core.ConnectionException: [node006.internal.net/192.168.12.22:9042] Pool is shutdown))
> at com.datastax.driver.core.exceptions.NoHostAvailableException.copy(NoHostAvailableException.java:84)
> at com.datastax.driver.core.DriverThrowables.propagateCause(DriverThrowables.java:37)
> at com.datastax.driver.core.DefaultResultSetFuture.getUninterruptibly(DefaultResultSetFuture.java:214)
> at com.datastax.driver.core.AbstractSession.execute(AbstractSession.java:52)
> at com.datastax.driver.core.AbstractSession.execute(AbstractSession.java:36)
> at org.apache.cassandra.hadoop.cql3.CqlRecordReader.fetchKeys(CqlRecordReader.java:578)
> at org.apache.cassandra.hadoop.cql3.CqlRecordReader.buildQuery(CqlRecordReader.java:526)
> at org.apache.cassandra.hadoop.cql3.CqlRecordReader.initialize(CqlRecordReader.java:148)
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@cassandra.apache.org
For additional commands, e-mail: commits-help@cassandra.apache.org