[GitHub] adamresson opened a new issue #428: GCS Hadoop Connector can't recover from Tablet Server failure.

[GitBox <gi...@apache.org>]

adamresson opened a new issue #428: GCS Hadoop Connector can't recover from Tablet Server failure.
URL: https://github.com/apache/accumulo/issues/428
 
 
   I was attempting to use the GCS Connector (https://github.com/GoogleCloudPlatform/bigdata-interop/tree/master/gcs) to back Accumulo on GCP.
   
   All pretty straight forward (just pointed instance.volumes to my bucket `gs://<bucketname>/accumulo`
   
   One of my Tablet Servers OOMed, and when I try to recover it, I end up getting the following error:
   ```
   Failed to initiate log sort gs://<bucketname>/accumulo/wal/accumulo-gcs-w-1+9997/aa166493-637e-48e8-a9a6-3655dfb59a6c
   	java.lang.IllegalStateException: Don't know how to recover a lease for com.google.cloud.hadoop.fs.gcs.GoogleHadoopFileSystem
   		at org.apache.accumulo.server.master.recovery.HadoopLogCloser.close(HadoopLogCloser.java:70)
   		at org.apache.accumulo.master.recovery.RecoveryManager$LogSortTask.run(RecoveryManager.java:96)
   		at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
   		at java.util.concurrent.FutureTask.run(FutureTask.java:266)
   		at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
   		at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
   		at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
   		at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
   		at org.apache.accumulo.fate.util.LoggingRunnable.run(LoggingRunnable.java:35)
   		at java.lang.Thread.run(Thread.java:748)
   ``` 
   
   I traced that back to HadoopLogCloser, and since the GoogleHadoopFileSystem is a FileSystem, not a `DistributedFileSystem`, `LocalFileSystem`, `RawLocalFileSystem`, it bottoms out here: https://github.com/apache/accumulo/blob/5d216d414cc9ce9e79b161f12bf4c92a430c803d/server/base/src/main/java/org/apache/accumulo/server/master/recovery/HadoopLogCloser.java#L71
   
   I'm not sure what to do from here. I was also told that Azure should work with accumulo no problem, but it looks like the NativeAzureFileSystem similarly doesn't implement any of these interfaces, so I would presume it would hit the same issue.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services