You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-issues@hadoop.apache.org by "Steve Loughran (Jira)" <ji...@apache.org> on 2022/03/16 17:37:00 UTC
[jira] [Commented] (HADOOP-18161) [WASB] Retry not getting implemented when using wasb scheme in hadoop-azure 2.7.4
[ https://issues.apache.org/jira/browse/HADOOP-18161?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17507765#comment-17507765 ]
Steve Loughran commented on HADOOP-18161:
-----------------------------------------
hadoop 2.7 is six years old
please upgrade to hadoop 3.3, retest and, if the problem still exists, we will take patches on the branch-3.3 line.
downgrading to minor
> [WASB] Retry not getting implemented when using wasb scheme in hadoop-azure 2.7.4
> ----------------------------------------------------------------------------------
>
> Key: HADOOP-18161
> URL: https://issues.apache.org/jira/browse/HADOOP-18161
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs/azure, hadoop-thirdparty
> Affects Versions: 2.7.4
> Reporter: Aryan
> Priority: Major
>
> I am using prestodb to read data from blob.
> Presto is using hadoop-azure-2.7.4 jar.
> I'm using *wasb* scheme to query the data on blob. I'm afraid for some reason the hadoop-azure library is not retrying when getting IO exception.
> Attaching the stack trace below,
> {code:java}
> com.facebook.presto.spi.PrestoException: Error reading from wasb://oemdpv3prd-v1@oemdpv3prd.blob.core.windows.net/data/pipelines/hudi/kafka/telemetrics_v2/dp.hmi.quectel.bms.data.packet.v2/dt=2022-01-15/e576abc3-942a-434d-be02-6899798258eb-0_5-13327-290407_20220115211203.parquet at position 65924529
> at com.facebook.presto.hive.parquet.HdfsParquetDataSource.readInternal(HdfsParquetDataSource.java:66)
> at com.facebook.presto.parquet.AbstractParquetDataSource.readFully(AbstractParquetDataSource.java:60)
> at com.facebook.presto.parquet.AbstractParquetDataSource.readFully(AbstractParquetDataSource.java:51)
> at com.facebook.presto.parquet.reader.ParquetReader.readPrimitive(ParquetReader.java:247)
> at com.facebook.presto.parquet.reader.ParquetReader.readColumnChunk(ParquetReader.java:330)
> at com.facebook.presto.parquet.reader.ParquetReader.readBlock(ParquetReader.java:313)
> at com.facebook.presto.hive.parquet.ParquetPageSource$ParquetBlockLoader.load(ParquetPageSource.java:182)
> at com.facebook.presto.hive.parquet.ParquetPageSource$ParquetBlockLoader.load(ParquetPageSource.java:160)
> at com.facebook.presto.common.block.LazyBlock.assureLoaded(LazyBlock.java:291)
> at com.facebook.presto.common.block.LazyBlock.getLoadedBlock(LazyBlock.java:282)
> at com.facebook.presto.operator.ScanFilterAndProjectOperator$RecordingLazyBlockLoader.load(ScanFilterAndProjectOperator.java:314)
> at com.facebook.presto.operator.ScanFilterAndProjectOperator$RecordingLazyBlockLoader.load(ScanFilterAndProjectOperator.java:300)
> at com.facebook.presto.common.block.LazyBlock.assureLoaded(LazyBlock.java:291)
> at com.facebook.presto.common.block.LazyBlock.getLoadedBlock(LazyBlock.java:282)
> at com.facebook.presto.operator.project.InputPageProjection.project(InputPageProjection.java:69)
> at com.facebook.presto.operator.project.PageProjectionWithOutputs.project(PageProjectionWithOutputs.java:56)
> at com.facebook.presto.operator.project.PageProcessor$ProjectSelectedPositions.processBatch(PageProcessor.java:323)
> at com.facebook.presto.operator.project.PageProcessor$ProjectSelectedPositions.process(PageProcessor.java:197)
> at com.facebook.presto.operator.WorkProcessorUtils$ProcessWorkProcessor.process(WorkProcessorUtils.java:315)
> at com.facebook.presto.operator.WorkProcessorUtils$YieldingIterator.computeNext(WorkProcessorUtils.java:79)
> at com.facebook.presto.operator.WorkProcessorUtils$YieldingIterator.computeNext(WorkProcessorUtils.java:65)
> at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:141)
> at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:136)
> at com.facebook.presto.operator.project.MergingPageOutput.getOutput(MergingPageOutput.java:113)
> at com.facebook.presto.operator.ScanFilterAndProjectOperator.processPageSource(ScanFilterAndProjectOperator.java:295)
> at com.facebook.presto.operator.ScanFilterAndProjectOperator.getOutput(ScanFilterAndProjectOperator.java:242)
> at com.facebook.presto.operator.Driver.processInternal(Driver.java:418)
> at com.facebook.presto.operator.Driver.lambda$processFor$9(Driver.java:301)
> at com.facebook.presto.operator.Driver.tryWithLock(Driver.java:722)
> at com.facebook.presto.operator.Driver.processFor(Driver.java:294)
> at com.facebook.presto.execution.SqlTaskExecution$DriverSplitRunner.processFor(SqlTaskExecution.java:1077)
> at com.facebook.presto.execution.executor.PrioritizedSplitRunner.process(PrioritizedSplitRunner.java:162)
> at com.facebook.presto.execution.executor.TaskExecutor$TaskRunner.run(TaskExecutor.java:599)
> at com.facebook.presto.$gen.Presto_0_256_2ed7f73____20220112_133056_1.run(Unknown Source)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> Caused by: java.io.IOException: undefined
> at com.facebook.presto.hadoop.$internal.com.microsoft.azure.storage.core.Utility.initIOException(Utility.java:598)
> at com.facebook.presto.hadoop.$internal.com.microsoft.azure.storage.blob.BlobInputStream.dispatchRead(BlobInputStream.java:264)
> at com.facebook.presto.hadoop.$internal.com.microsoft.azure.storage.blob.BlobInputStream.readInternal(BlobInputStream.java:448)
> at com.facebook.presto.hadoop.$internal.com.microsoft.azure.storage.blob.BlobInputStream.read(BlobInputStream.java:420)
> at java.io.BufferedInputStream.read1(BufferedInputStream.java:284)
> at java.io.BufferedInputStream.read(BufferedInputStream.java:345)
> at java.io.DataInputStream.read(DataInputStream.java:149)
> at org.apache.hadoop.fs.azure.NativeAzureFileSystem$NativeAzureFsInputStream.read(NativeAzureFileSystem.java:735)
> at org.apache.hadoop.fs.FSInputStream.read(FSInputStream.java:65)
> at org.apache.hadoop.fs.FSInputStream.readFully(FSInputStream.java:78)
> at org.apache.hadoop.fs.BufferedFSInputStream.readFully(BufferedFSInputStream.java:113)
> at org.apache.hadoop.fs.FSDataInputStream.readFully(FSDataInputStream.java:107)
> at com.facebook.presto.hive.parquet.HdfsParquetDataSource.readInternal(HdfsParquetDataSource.java:58)
> ... 36 more
> Caused by: com.facebook.presto.hadoop.$internal.com.microsoft.azure.storage.StorageException: Blob hash mismatch (integrity check failed), Expected value is gRfj04hQvdOqOaAdqo+lGw==, retrieved 4V4j3H2KabAsssDyGnJN9w==.
> at com.facebook.presto.hadoop.$internal.com.microsoft.azure.storage.blob.CloudBlob$10.postProcessResponse(CloudBlob.java:1714)
> at com.facebook.presto.hadoop.$internal.com.microsoft.azure.storage.blob.CloudBlob$10.postProcessResponse(CloudBlob.java:1613)
> at com.facebook.presto.hadoop.$internal.com.microsoft.azure.storage.core.ExecutionEngine.executeWithRetry(ExecutionEngine.java:148)
> at com.facebook.presto.hadoop.$internal.com.microsoft.azure.storage.blob.CloudBlob.downloadRangeInternal(CloudBlob.java:1468)
> at com.facebook.presto.hadoop.$internal.com.microsoft.azure.storage.blob.BlobInputStream.dispatchRead(BlobInputStream.java:255)
> ... 47 more{code}
> Please suggest a way to get the retry implemented.
> I want the retry to be implemented if there is an IO exception while calling *read* method below,
> {code:java}
> org.apache.hadoop.fs.azure.NativeAzureFileSystem$NativeAzureFsInputStream.read(NativeAzureFileSystem.java:735){code}
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org