[GitHub] a2l007 opened a new issue #6104: Webhdfs support for orc-extension

[GitBox <gi...@apache.org>]

a2l007 opened a new issue #6104: Webhdfs support for orc-extension
URL: https://github.com/apache/incubator-druid/issues/6104
 
 
   It doesn't look like the `druid-orc-extensions` extension works for WebHDFS. Got it working for HDFS after removing some conflicting jars,  but for WebHDFS the ingestion task fails with the following error:
   ```
   
   ERROR [ORC_GET_SPLITS #0] org.apache.hadoop.hive.ql.io.AcidUtils - Failed to get files with ID; using regular API
   java.lang.UnsupportedOperationException: Only supported for DFS; got class org.apache.hadoop.hdfs.web.WebHdfsFileSystem
   	at org.apache.hadoop.hive.shims.Hadoop23Shims.ensureDfs(Hadoop23Shims.java:813) ~[hive-exec-2.0.0.jar:2.0.0]
   	at org.apache.hadoop.hive.shims.Hadoop23Shims.listLocatedHdfsStatus(Hadoop23Shims.java:784) ~[hive-exec-2.0.0.jar:2.0.0]
   	at org.apache.hadoop.hive.ql.io.AcidUtils.getAcidState(AcidUtils.java:477) [hive-exec-2.0.0.jar:2.0.0]
   	at org.apache.hadoop.hive.ql.io.orc.OrcInputFormat$FileGenerator.call(OrcInputFormat.java:890) [hive-exec-2.0.0.jar:2.0.0]
   	at org.apache.hadoop.hive.ql.io.orc.OrcInputFormat$FileGenerator.call(OrcInputFormat.java:875) [hive-exec-2.0.0.jar:2.0.0]
   	at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_131]
   	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_131]
   	at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_131]
   	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [?:1.8.0_131]
   	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:1.8.0_131]
   	at java.lang.Thread.run(Thread.java:748) [?:1.8.0_131]
   ```
   
   From the trace it looks like it is attempting to perform an operation that is not supported for WebHDFS. I'm raising this issue to check if any of the extension users here has run into this problem before. 
   @gianm  @sirpkt , any input here?

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org