All Datanodes are Bad

[Joseph Obernberger <jo...@gmail.com>]

Anyone ever seen an error like this?  We are running using HDFS for the
index.  At the time of the error, we are doing a lot of indexing.

Two errors:
java.io.IOException: All datanodes DatanodeInfoWithStorage[
172.16.100.220:50010,DS-4b806395-0661-4a70-a32b-deef82a85359,DISK] are bad.
Aborting...
at
org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.setupPipelineForAppendOrRecovery(DFSOutputStream.java:1357)
at
org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.processDatanodeError(DFSOutputStream.java:1119)
at
org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:622)

and

auto commit error...:org.apache.solr.common.SolrException:
java.io.IOException: All datanodes DatanodeInfoWithStorage[
172.16.100.220:50010,DS-4b806395-0661-4a70-a32b-deef82a85359,DISK] are bad.
Aborting...
at
org.apache.solr.update.HdfsTransactionLog.close(HdfsTransactionLog.java:321)
at org.apache.solr.update.TransactionLog.decref(TransactionLog.java:510)
at org.apache.solr.update.UpdateLog.addOldLog(UpdateLog.java:372)
at org.apache.solr.update.UpdateLog.postCommit(UpdateLog.java:668)
at
org.apache.solr.update.DirectUpdateHandler2.commit(DirectUpdateHandler2.java:658)
at org.apache.solr.update.CommitTracker.run(CommitTracker.java:217)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:178)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:292)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.io.IOException: All datanodes DatanodeInfoWithStorage[
172.16.100.220:50010,DS-4b806395-0661-4a70-a32b-deef82a85359,DISK] are bad.
Aborting...
at
org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.setupPipelineForAppendOrRecovery(DFSOutputStream.java:1357)
at
org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.processDatanodeError(DFSOutputStream.java:1119)
at
org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:622)

On the client side doing the indexing, we see:

org.apache.solr.client.solrj.impl.CloudSolrClient$RouteException: Error
from server at http://deimos:9100/solr/UNCLASS: java.io.IOException: All
datanodes DatanodeInfoWithStorage[172.16.100.220:50010,DS-f0e14105-9557-4a59-8918-43724aaa8346,DISK]
are bad. Aborting...
        at
org.apache.solr.client.solrj.impl.CloudSolrClient.directUpdate(CloudSolrClient.java:632)
        at
org.apache.solr.client.solrj.impl.CloudSolrClient.sendRequest(CloudSolrClient.java:981)
        at
org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:870)
        at
org.apache.solr.client.solrj.impl.CloudSolrClient.request(CloudSolrClient.java:806)
        at
org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:149)
        at org.apache.solr.client.solrj.SolrClient.add(SolrClient.java:106)
        at org.apache.solr.client.solrj.SolrClient.add(SolrClient.java:71)
        at org.apache.solr.client.solrj.SolrClient.add(SolrClient.java:85)
.
.
.
Caused by:
org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException: Error
from server at http://deimos:9100/solr/UNCLASS: java.io.IOException: All
datanodes DatanodeInfoWithStorage[172.16.100.220:50010,DS-f0e14105-9557-4a59-8918-43724aaa8346,DISK]
are bad. Aborting...
        at
org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:576)
        at
org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:240)
        at
org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:229)
        at
org.apache.solr.client.solrj.impl.LBHttpSolrClient.doRequest(LBHttpSolrClient.java:372)
        at
org.apache.solr.client.solrj.impl.LBHttpSolrClient.request(LBHttpSolrClient.java:325)
        at
org.apache.solr.client.solrj.impl.CloudSolrClient$2.call(CloudSolrClient.java:607)
        at
org.apache.solr.client.solrj.impl.CloudSolrClient$2.call(CloudSolrClient.java:604)
        at java.util.concurrent.FutureTask.run(FutureTask.java:262)
        at
org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$1.run(ExecutorUtil.java:231)

Thanks!

-Joe