You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by GitBox <gi...@apache.org> on 2021/07/23 06:16:33 UTC

[GitHub] [incubator-doris] huozhanfeng opened a new issue #6307: [BUG]Broker load failed when loading OSS/S3A files

huozhanfeng opened a new issue #6307:
URL: https://github.com/apache/incubator-doris/issues/6307


   **Describe the bug**
   Currently, Doris supports loading OSS/S3A files by using params like `fs.s3a.access.key`, but there is a bug when using it to load such type files.
   
   Here is the error log when loading OSS files
   <pre>
   java.lang.UnsupportedOperationException: Byte-buffer read unsupported by input stream
       at org.apache.hadoop.fs.FSDataInputStream.read(FSDataInputStream.java:146)
       at org.apache.doris.broker.hdfs.FileSystemManager.readByteBuffer(FileSystemManager.java:678)
       at org.apache.doris.broker.hdfs.FileSystemManager.readByteBufferFully(FileSystemManager.java:689)
       at org.apache.doris.broker.hdfs.FileSystemManager.pread(FileSystemManager.java:570)
       at org.apache.doris.broker.hdfs.HDFSBrokerServiceImpl.pread(HDFSBrokerServiceImpl.java:157)
       at org.apache.doris.thrift.TPaloBrokerService$Processor$pread.getResult(TPaloBrokerService.java:937)
       at org.apache.doris.thrift.TPaloBrokerService$Processor$pread.getResult(TPaloBrokerService.java:922)
       at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
       at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
       at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:286)
       at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
       at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
       at java.lang.Thread.run(Thread.java:748)
   </pre>
   **To Reproduce**
   We can simply use such load configuration to reproduce the bug
   <pre>
   WITH BROKER hdfs_broker (
       "fs.s3a.access.key"="...",
       "fs.s3a.secret.key"="...",
       "fs.s3a.endpoint"="..."
     )
   </pre>
   
   
   **Additional context**
   It's a bug related to this issue.
   
   S3A input stream to support ByteBufferReadable
   https://issues.apache.org/jira/browse/HADOOP-14603
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org

[GitHub] [incubator-doris] huozhanfeng closed issue #6307: [BUG]Broker can't load OSS/S3A files

Posted by GitBox <gi...@apache.org>.
huozhanfeng closed issue #6307:
URL: https://github.com/apache/incubator-doris/issues/6307


   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org