You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "hbgstc123 (via GitHub)" <gi...@apache.org> on 2023/03/30 08:21:04 UTC
[GitHub] [hudi] hbgstc123 opened a new issue, #8325: [SUPPORT] spark read hudi error
hbgstc123 opened a new issue, #8325:
URL: https://github.com/apache/hudi/issues/8325
**Describe the problem you faced**
spark read hudi meet this error, happen every time, only happen in new hudi version 0.12.2, works fine when switch to hudi version 0.10.1
**To Reproduce**
Steps to reproduce the behavior:
1.write hudi table with spark
2.read hudi table with spark
**Expected behavior**
A clear and concise description of what you expected to happen.
**Environment Description**
* Hudi version : 0.12.2 (have error), 0.10.1 (work fine)
* Spark version : 3.1
* Storage (HDFS/S3/GCS..) :HDFS
* Running on Docker? (yes/no) :no
**Additional context**
**Stacktrace**
```
at org.apache.spark.sql.catalyst.errors.package$.attachTree(package.scala:56)
at org.apache.spark.sql.execution.exchange.ShuffleExchangeExec.doExecute(ShuffleExchangeExec.scala:169)
at org.apache.spark.sql.execution.SparkPlan.$anonfun$execute$1(SparkPlan.scala:181)
at org.apache.spark.sql.execution.SparkPlan.$anonfun$executeQuery$1(SparkPlan.scala:219)
at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151)
at org.apache.spark.sql.execution.SparkPlan.executeQuery(SparkPlan.scala:216)
at org.apache.spark.sql.execution.SparkPlan.execute(SparkPlan.scala:177)
at org.apache.spark.sql.execution.InputAdapter.inputRDD(WholeStageCodegenExec.scala:525)
at org.apache.spark.sql.execution.InputRDDCodegen.inputRDDs(WholeStageCodegenExec.scala:453)
at org.apache.spark.sql.execution.InputRDDCodegen.inputRDDs$(WholeStageCodegenExec.scala:452)
at org.apache.spark.sql.execution.InputAdapter.inputRDDs(WholeStageCodegenExec.scala:496)
at org.apache.spark.sql.execution.SortExec.inputRDDs(SortExec.scala:132)
at org.apache.spark.sql.execution.WholeStageCodegenExec.doExecute(WholeStageCodegenExec.scala:746)
at org.apache.spark.sql.execution.SparkPlan.$anonfun$execute$1(SparkPlan.scala:181)
at org.apache.spark.sql.execution.SparkPlan.$anonfun$executeQuery$1(SparkPlan.scala:219)
at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151)
at org.apache.spark.sql.execution.SparkPlan.executeQuery(SparkPlan.scala:216)
at org.apache.spark.sql.execution.SparkPlan.execute(SparkPlan.scala:177)
at org.apache.spark.sql.execution.window.WindowExec.doExecute(WindowExec.scala:121)
at org.apache.spark.sql.execution.SparkPlan.$anonfun$execute$1(SparkPlan.scala:181)
at org.apache.spark.sql.execution.SparkPlan.$anonfun$executeQuery$1(SparkPlan.scala:219)
at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151)
at org.apache.spark.sql.execution.SparkPlan.executeQuery(SparkPlan.scala:216)
at org.apache.spark.sql.execution.SparkPlan.execute(SparkPlan.scala:177)
at org.apache.spark.sql.execution.InputAdapter.inputRDD(WholeStageCodegenExec.scala:525)
at org.apache.spark.sql.execution.InputRDDCodegen.inputRDDs(WholeStageCodegenExec.scala:453)
at org.apache.spark.sql.execution.InputRDDCodegen.inputRDDs$(WholeStageCodegenExec.scala:452)
at org.apache.spark.sql.execution.InputAdapter.inputRDDs(WholeStageCodegenExec.scala:496)
at org.apache.spark.sql.execution.FilterExec.inputRDDs(basicPhysicalOperators.scala:149)
at org.apache.spark.sql.execution.ProjectExec.inputRDDs(basicPhysicalOperators.scala:50)
at org.apache.spark.sql.execution.WholeStageCodegenExec.doExecute(WholeStageCodegenExec.scala:746)
at org.apache.spark.sql.execution.SparkPlan.$anonfun$execute$1(SparkPlan.scala:181)
at org.apache.spark.sql.execution.SparkPlan.$anonfun$executeQuery$1(SparkPlan.scala:219)
at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151)
at org.apache.spark.sql.execution.SparkPlan.executeQuery(SparkPlan.scala:216)
at org.apache.spark.sql.execution.SparkPlan.execute(SparkPlan.scala:177)
at org.apache.spark.sql.execution.exchange.ShuffleExchangeExec.inputRDD$lzycompute(ShuffleExchangeExec.scala:123)
at org.apache.spark.sql.execution.exchange.ShuffleExchangeExec.inputRDD(ShuffleExchangeExec.scala:123)
at org.apache.spark.sql.execution.exchange.ShuffleExchangeExec.shuffleDependency$lzycompute(ShuffleExchangeExec.scala:157)
at org.apache.spark.sql.execution.exchange.ShuffleExchangeExec.shuffleDependency(ShuffleExchangeExec.scala:155)
at org.apache.spark.sql.execution.exchange.ShuffleExchangeExec.$anonfun$doExecute$1(ShuffleExchangeExec.scala:172)
at org.apache.spark.sql.catalyst.errors.package$.attachTree(package.scala:52)
... 170 more
Caused by: org.apache.hudi.exception.HoodieException: Unable to instantiate class org.apache.hudi.common.bootstrap.index.HFileBootstrapIndex
at org.apache.hudi.common.util.ReflectionUtils.loadClass(ReflectionUtils.java:91)
at org.apache.hudi.common.bootstrap.index.BootstrapIndex.getBootstrapIndex(BootstrapIndex.java:163)
at org.apache.hudi.common.table.view.AbstractTableFileSystemView.init(AbstractTableFileSystemView.java:109)
at org.apache.hudi.common.table.view.HoodieTableFileSystemView.init(HoodieTableFileSystemView.java:108)
at org.apache.hudi.common.table.view.HoodieTableFileSystemView.<init>(HoodieTableFileSystemView.java:102)
at org.apache.hudi.common.table.view.HoodieTableFileSystemView.<init>(HoodieTableFileSystemView.java:93)
at org.apache.hudi.common.table.view.HoodieTableFileSystemView.<init>(HoodieTableFileSystemView.java:166)
at org.apache.hudi.BaseHoodieTableFileIndex.loadFileSlicesForPartitions(BaseHoodieTableFileIndex.java:246)
at org.apache.hudi.BaseHoodieTableFileIndex.ensurePreloadedPartitions(BaseHoodieTableFileIndex.java:234)
at org.apache.hudi.BaseHoodieTableFileIndex.getInputFileSlices(BaseHoodieTableFileIndex.java:220)
at org.apache.hudi.HoodieFileIndex.listFiles(HoodieFileIndex.scala:137)
at org.apache.spark.sql.execution.FileSourceScanExec.selectedPartitions$lzycompute(DataSourceScanExec.scala:215)
at org.apache.spark.sql.execution.FileSourceScanExec.selectedPartitions(DataSourceScanExec.scala:210)
at org.apache.spark.sql.execution.FileSourceScanExec.dynamicallySelectedPartitions$lzycompute(DataSourceScanExec.scala:245)
at org.apache.spark.sql.execution.FileSourceScanExec.dynamicallySelectedPartitions(DataSourceScanExec.scala:226)
at org.apache.spark.sql.execution.FileSourceScanExec.inputRDD$lzycompute(DataSourceScanExec.scala:413)
at org.apache.spark.sql.execution.FileSourceScanExec.inputRDD(DataSourceScanExec.scala:398)
at org.apache.spark.sql.execution.FileSourceScanExec.doExecuteColumnar(DataSourceScanExec.scala:497)
at org.apache.spark.sql.execution.SparkPlan.$anonfun$executeColumnar$1(SparkPlan.scala:208)
at org.apache.spark.sql.execution.SparkPlan.$anonfun$executeQuery$1(SparkPlan.scala:219)
at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151)
at org.apache.spark.sql.execution.SparkPlan.executeQuery(SparkPlan.scala:216)
at org.apache.spark.sql.execution.SparkPlan.executeColumnar(SparkPlan.scala:204)
at org.apache.spark.sql.execution.InputAdapter.doExecuteColumnar(WholeStageCodegenExec.scala:519)
at org.apache.spark.sql.execution.SparkPlan.$anonfun$executeColumnar$1(SparkPlan.scala:208)
at org.apache.spark.sql.execution.SparkPlan.$anonfun$executeQuery$1(SparkPlan.scala:219)
at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151)
at org.apache.spark.sql.execution.SparkPlan.executeQuery(SparkPlan.scala:216)
at org.apache.spark.sql.execution.SparkPlan.executeColumnar(SparkPlan.scala:204)
at org.apache.spark.sql.execution.ColumnarToRowExec.inputRDDs(Columnar.scala:202)
at org.apache.spark.sql.execution.WholeStageCodegenExec.doExecute(WholeStageCodegenExec.scala:746)
at org.apache.spark.sql.execution.SparkPlan.$anonfun$execute$1(SparkPlan.scala:181)
at org.apache.spark.sql.execution.SparkPlan.$anonfun$executeQuery$1(SparkPlan.scala:219)
at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151)
at org.apache.spark.sql.execution.SparkPlan.executeQuery(SparkPlan.scala:216)
at org.apache.spark.sql.execution.SparkPlan.execute(SparkPlan.scala:177)
at org.apache.spark.sql.execution.FilterExec.doExecute(basicPhysicalOperators.scala:245)
at org.apache.spark.sql.execution.SparkPlan.$anonfun$execute$1(SparkPlan.scala:181)
at org.apache.spark.sql.execution.SparkPlan.$anonfun$executeQuery$1(SparkPlan.scala:219)
at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151)
at org.apache.spark.sql.execution.SparkPlan.executeQuery(SparkPlan.scala:216)
at org.apache.spark.sql.execution.SparkPlan.execute(SparkPlan.scala:177)
at org.apache.spark.sql.execution.ProjectExec.doExecute(basicPhysicalOperators.scala:92)
at org.apache.spark.sql.execution.SparkPlan.$anonfun$execute$1(SparkPlan.scala:181)
at org.apache.spark.sql.execution.SparkPlan.$anonfun$executeQuery$1(SparkPlan.scala:219)
at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151)
at org.apache.spark.sql.execution.SparkPlan.executeQuery(SparkPlan.scala:216)
at org.apache.spark.sql.execution.SparkPlan.execute(SparkPlan.scala:177)
at org.apache.spark.sql.execution.exchange.ShuffleExchangeExec.inputRDD$lzycompute(ShuffleExchangeExec.scala:123)
at org.apache.spark.sql.execution.exchange.ShuffleExchangeExec.inputRDD(ShuffleExchangeExec.scala:123)
at org.apache.spark.sql.execution.exchange.ShuffleExchangeExec.shuffleDependency$lzycompute(ShuffleExchangeExec.scala:157)
at org.apache.spark.sql.execution.exchange.ShuffleExchangeExec.shuffleDependency(ShuffleExchangeExec.scala:155)
at org.apache.spark.sql.execution.exchange.ShuffleExchangeExec.$anonfun$doExecute$1(ShuffleExchangeExec.scala:172)
at org.apache.spark.sql.catalyst.errors.package$.attachTree(package.scala:52)
... 211 more
Caused by: java.lang.reflect.InvocationTargetException
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at org.apache.hudi.common.util.ReflectionUtils.loadClass(ReflectionUtils.java:89)
... 264 more
Caused by: org.apache.hudi.exception.HoodieIOException: Filesystem closed
at org.apache.hudi.common.bootstrap.index.HFileBootstrapIndex.<init>(HFileBootstrapIndex.java:106)
... 269 more
Caused by: java.io.IOException: Filesystem closed
at org.apache.hadoop.hdfs.DFSClient.checkOpen(DFSClient.java:475)
at org.apache.hadoop.hdfs.DFSClient.getFileInfo(DFSClient.java:1685)
at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1591)
at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1588)
at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1603)
at org.apache.hadoop.hdfs.ForwardDistributedFileSystem.getFileStatus(ForwardDistributedFileSystem.java:1197)
at org.apache.hadoop.fs.FileSystem.exists(FileSystem.java:1683)
at org.apache.hudi.common.fs.HoodieWrapperFileSystem.exists(HoodieWrapperFileSystem.java:558)
at org.apache.hudi.common.bootstrap.index.HFileBootstrapIndex.<init>(HFileBootstrapIndex.java:104)
... 269 more
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] jfrylings-twilio commented on issue #8325: [SUPPORT] spark read hudi error: Unable to instantiate HFileBootstrapIndex
Posted by "jfrylings-twilio (via GitHub)" <gi...@apache.org>.
jfrylings-twilio commented on issue #8325:
URL: https://github.com/apache/hudi/issues/8325#issuecomment-1537044024
Setting `fs.s3a.connection.maximum=200` fixed the issue for me. The issue looks like it was caused by recent input touching more partitions than usual.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] danny0405 commented on issue #8325: [SUPPORT] spark read hudi error: Unable to instantiate HFileBootstrapIndex
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on issue #8325:
URL: https://github.com/apache/hudi/issues/8325#issuecomment-1489922846
@yihua Do you have any idea whether it is fixed in master now?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] jfrylings-twilio commented on issue #8325: [SUPPORT] spark read hudi error: Unable to instantiate HFileBootstrapIndex
Posted by "jfrylings-twilio (via GitHub)" <gi...@apache.org>.
jfrylings-twilio commented on issue #8325:
URL: https://github.com/apache/hudi/issues/8325#issuecomment-1535478999
I'm seeing the same issue with hudi 0.8 and Spark 3.2.1 but not using Flink or metadata table.
(I know this combination of Hudi/Spark versions is not officially supported)
This job continues to fail over and over with the same error even when manually restarting it. It has been working for months with no issue and no recent code changes. Other jobs using the same code but different input data source and different data output continue to work fine. Those other jobs that continue to work are running in the same cluster and are not having any connectivity issues to s3.
The file it is erroring on does not exist. There is nothing in that directory. I haven't redacted the file name it really is filled with mostly 0s
`s3a://<bucket>/<path>/.hoodie/.aux/.bootstrap/.partitions/00000000-0000-0000-0000-000000000000-0_1-0-1_00000000000001.hfile`
```
23/05/04 21:50:31 WARN TaskSetManager: Lost task 120.0 in stage 26.0 (TID 10224) (10.221.232.192 executor 1): org.apache.hudi.exception.HoodieUpsertException: Error upserting bucketType UPDATE for partition :120
at org.apache.hudi.table.action.commit.BaseSparkCommitActionExecutor.handleUpsertPartition(BaseSparkCommitActionExecutor.java:288)
at org.apache.hudi.table.action.commit.BaseSparkCommitActionExecutor.lambda$execute$ecf5068c$1(BaseSparkCommitActionExecutor.java:139)
at org.apache.spark.api.java.JavaRDDLike.$anonfun$mapPartitionsWithIndex$1(JavaRDDLike.scala:102)
at org.apache.spark.api.java.JavaRDDLike.$anonfun$mapPartitionsWithIndex$1$adapted(JavaRDDLike.scala:102)
at org.apache.spark.rdd.RDD.$anonfun$mapPartitionsWithIndex$2(RDD.scala:915)
at org.apache.spark.rdd.RDD.$anonfun$mapPartitionsWithIndex$2$adapted(RDD.scala:915)
at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:52)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:373)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:337)
at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:52)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:373)
at org.apache.spark.rdd.RDD.$anonfun$getOrCompute$1(RDD.scala:386)
at org.apache.spark.storage.BlockManager.$anonfun$doPutIterator$1(BlockManager.scala:1498)
at org.apache.spark.storage.BlockManager.org$apache$spark$storage$BlockManager$$doPut(BlockManager.scala:1408)
at org.apache.spark.storage.BlockManager.doPutIterator(BlockManager.scala:1472)
at org.apache.spark.storage.BlockManager.getOrElseUpdate(BlockManager.scala:1295)
at org.apache.spark.rdd.RDD.getOrCompute(RDD.scala:384)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:335)
at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:52)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:373)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:337)
at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:90)
at org.apache.spark.scheduler.Task.run(Task.scala:131)
at org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$3(Executor.scala:506)
at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1462)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:509)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.base/java.lang.Thread.run(Unknown Source)
Caused by: org.apache.hudi.exception.HoodieUpsertException: Failed to close UpdateHandle
at org.apache.hudi.io.HoodieMergeHandle.close(HoodieMergeHandle.java:359)
at org.apache.hudi.table.action.commit.SparkMergeHelper.runMerge(SparkMergeHelper.java:107)
at org.apache.hudi.table.action.commit.BaseSparkCommitActionExecutor.handleUpdateInternal(BaseSparkCommitActionExecutor.java:317)
at org.apache.hudi.table.action.commit.BaseSparkCommitActionExecutor.handleUpdate(BaseSparkCommitActionExecutor.java:308)
at org.apache.hudi.table.action.commit.BaseSparkCommitActionExecutor.handleUpsertPartition(BaseSparkCommitActionExecutor.java:281)
... 28 more
Caused by: java.io.InterruptedIOException: Writing Object on <s3_path>/<a_new_file_that_doesn't_exist>.parquet: com.amazonaws.SdkClientException: Unable to execute HTTP request: Timeout waiting for connection from pool
at org.apache.hadoop.fs.s3a.S3AUtils.translateInterruptedException(S3AUtils.java:389)
at org.apache.hadoop.fs.s3a.S3AUtils.translateException(S3AUtils.java:196)
at org.apache.hadoop.fs.s3a.Invoker.once(Invoker.java:117)
at org.apache.hadoop.fs.s3a.Invoker.lambda$retry$4(Invoker.java:320)
at org.apache.hadoop.fs.s3a.Invoker.retryUntranslated(Invoker.java:412)
at org.apache.hadoop.fs.s3a.Invoker.retry(Invoker.java:316)
at org.apache.hadoop.fs.s3a.Invoker.retry(Invoker.java:291)
at org.apache.hadoop.fs.s3a.WriteOperationHelper.retry(WriteOperationHelper.java:168)
at org.apache.hadoop.fs.s3a.WriteOperationHelper.putObject(WriteOperationHelper.java:515)
at org.apache.hadoop.fs.s3a.S3ABlockOutputStream.lambda$putObject$0(S3ABlockOutputStream.java:548)
at org.apache.hadoop.thirdparty.com.google.common.util.concurrent.TrustedListenableFutureTask$TrustedFutureInterruptibleTask.runInterruptibly(TrustedListenableFutureTask.java:125)
at org.apache.hadoop.thirdparty.com.google.common.util.concurrent.InterruptibleTask.run(InterruptibleTask.java:69)
at org.apache.hadoop.thirdparty.com.google.common.util.concurrent.TrustedListenableFutureTask.run(TrustedListenableFutureTask.java:78)
at org.apache.hadoop.util.SemaphoredDelegatingExecutor$RunnableWithPermitRelease.run(SemaphoredDelegatingExecutor.java:196)
at org.apache.hadoop.util.SemaphoredDelegatingExecutor$RunnableWithPermitRelease.run(SemaphoredDelegatingExecutor.java:196)
... 3 more
Caused by: com.amazonaws.SdkClientException: Unable to execute HTTP request: Timeout waiting for connection from pool
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleRetryableException(AmazonHttpClient.java:1207)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1153)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:802)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:770)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:744)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:704)
at com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:686)
at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:550)
at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:530)
at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:5227)
at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:5173)
at com.amazonaws.services.s3.AmazonS3Client.access$300(AmazonS3Client.java:415)
at com.amazonaws.services.s3.AmazonS3Client$PutObjectStrategy.invokeServiceCall(AmazonS3Client.java:6289)
at com.amazonaws.services.s3.AmazonS3Client.uploadObject(AmazonS3Client.java:1834)
at com.amazonaws.services.s3.AmazonS3Client.putObject(AmazonS3Client.java:1794)
at org.apache.hadoop.fs.s3a.S3AFileSystem.putObjectDirect(S3AFileSystem.java:2432)
at org.apache.hadoop.fs.s3a.WriteOperationHelper.lambda$putObject$6(WriteOperationHelper.java:517)
at org.apache.hadoop.fs.s3a.Invoker.once(Invoker.java:115)
... 15 more
Caused by: com.amazonaws.thirdparty.apache.http.conn.ConnectionPoolTimeoutException: Timeout waiting for connection from pool
at com.amazonaws.thirdparty.apache.http.impl.conn.PoolingHttpClientConnectionManager.leaseConnection(PoolingHttpClientConnectionManager.java:316)
at com.amazonaws.thirdparty.apache.http.impl.conn.PoolingHttpClientConnectionManager$1.get(PoolingHttpClientConnectionManager.java:282)
at jdk.internal.reflect.GeneratedMethodAccessor30.invoke(Unknown Source)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
at java.base/java.lang.reflect.Method.invoke(Unknown Source)
at com.amazonaws.http.conn.ClientConnectionRequestFactory$Handler.invoke(ClientConnectionRequestFactory.java:70)
at com.amazonaws.http.conn.$Proxy20.get(Unknown Source)
at com.amazonaws.thirdparty.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:190)
at com.amazonaws.thirdparty.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186)
at com.amazonaws.thirdparty.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185)
at com.amazonaws.thirdparty.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83)
at com.amazonaws.thirdparty.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:56)
at com.amazonaws.http.apache.client.impl.SdkHttpClient.execute(SdkHttpClient.java:72)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1333)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1145)
... 31 more
at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:52)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:373)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:337)
at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:52)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:373)
at org.apache.spark.rdd.RDD.$anonfun$getOrCompute$1(RDD.scala:386)
at org.apache.spark.storage.BlockManager.$anonfun$doPutIterator$1(BlockManager.scala:1498)
at org.apache.spark.storage.BlockManager.org$apache$spark$storage$BlockManager$$doPut(BlockManager.scala:1408)
at org.apache.spark.storage.BlockManager.doPutIterator(BlockManager.scala:1472)
at org.apache.spark.storage.BlockManager.getOrElseUpdate(BlockManager.scala:1295)
at org.apache.spark.rdd.RDD.getOrCompute(RDD.scala:384)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:335)
at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:52)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:373)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:337)
at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:90)
at org.apache.spark.scheduler.Task.run(Task.scala:131)
at org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$3(Executor.scala:506)
at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1462)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:509)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.base/java.lang.Thread.run(Unknown Source)
Caused by: org.apache.hudi.exception.HoodieException: Unable to instantiate class
at org.apache.hudi.common.util.ReflectionUtils.loadClass(ReflectionUtils.java:89)
at org.apache.hudi.common.util.ReflectionUtils.loadClass(ReflectionUtils.java:98)
at org.apache.hudi.common.bootstrap.index.BootstrapIndex.getBootstrapIndex(BootstrapIndex.java:159)
at org.apache.hudi.common.table.view.AbstractTableFileSystemView.init(AbstractTableFileSystemView.java:107)
at org.apache.hudi.common.table.view.HoodieTableFileSystemView.init(HoodieTableFileSystemView.java:106)
at org.apache.hudi.common.table.view.HoodieTableFileSystemView.<init>(HoodieTableFileSystemView.java:100)
at org.apache.hudi.common.table.view.FileSystemViewManager.createInMemoryFileSystemView(FileSystemViewManager.java:167)
at org.apache.hudi.common.table.view.FileSystemViewManager.lambda$createViewManager$8894a6ab$1(FileSystemViewManager.java:255)
at org.apache.hudi.common.table.view.FileSystemViewManager.lambda$getFileSystemView$1(FileSystemViewManager.java:110)
at java.base/java.util.concurrent.ConcurrentHashMap.computeIfAbsent(Unknown Source)
at org.apache.hudi.common.table.view.FileSystemViewManager.getFileSystemView(FileSystemViewManager.java:109)
at org.apache.hudi.table.HoodieTable.getBaseFileOnlyView(HoodieTable.java:264)
at org.apache.hudi.io.HoodieMergeHandle.<init>(HoodieMergeHandle.java:111)
at org.apache.hudi.table.action.commit.BaseSparkCommitActionExecutor.getUpdateHandle(BaseSparkCommitActionExecutor.java:335)
at org.apache.hudi.table.action.commit.BaseSparkCommitActionExecutor.handleUpdate(BaseSparkCommitActionExecutor.java:307)
at org.apache.hudi.table.action.commit.BaseSparkCommitActionExecutor.handleUpsertPartition(BaseSparkCommitActionExecutor.java:281)
... 28 more
Caused by: java.lang.reflect.InvocationTargetException
at jdk.internal.reflect.GeneratedConstructorAccessor87.newInstance(Unknown Source)
at java.base/jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(Unknown Source)
at java.base/java.lang.reflect.Constructor.newInstance(Unknown Source)
at org.apache.hudi.common.util.ReflectionUtils.loadClass(ReflectionUtils.java:87)
... 43 more
Caused by: org.apache.hudi.exception.HoodieIOException: getFileStatus on s3a://<bucket>/<path>/.hoodie/.aux/.bootstrap/.partitions/00000000-0000-0000-0000-000000000000-0_1-0-1_00000000000001.hfile: com.amazonaws.SdkClientException: Unable to execute HTTP request: Timeout waiting for connection from pool
at org.apache.hudi.common.bootstrap.index.HFileBootstrapIndex.<init>(HFileBootstrapIndex.java:104)
... 47 more
Caused by: java.io.InterruptedIOException: getFileStatus on s3a://<bucket>/<path>/.hoodie/.aux/.bootstrap/.partitions/00000000-0000-0000-0000-000000000000-0_1-0-1_00000000000001.hfile: com.amazonaws.SdkClientException: Unable to execute HTTP request: Timeout waiting for connection from pool
at org.apache.hadoop.fs.s3a.S3AUtils.translateInterruptedException(S3AUtils.java:389)
at org.apache.hadoop.fs.s3a.S3AUtils.translateException(S3AUtils.java:196)
at org.apache.hadoop.fs.s3a.S3AUtils.translateException(S3AUtils.java:170)
at org.apache.hadoop.fs.s3a.S3AFileSystem.s3GetFileStatus(S3AFileSystem.java:3289)
at org.apache.hadoop.fs.s3a.S3AFileSystem.innerGetFileStatus(S3AFileSystem.java:3185)
at org.apache.hadoop.fs.s3a.S3AFileSystem.getFileStatus(S3AFileSystem.java:3053)
at org.apache.hadoop.fs.FileSystem.exists(FileSystem.java:1760)
at org.apache.hadoop.fs.s3a.S3AFileSystem.exists(S3AFileSystem.java:4263)
at org.apache.hudi.common.fs.HoodieWrapperFileSystem.exists(HoodieWrapperFileSystem.java:549)
at org.apache.hudi.common.bootstrap.index.HFileBootstrapIndex.<init>(HFileBootstrapIndex.java:102)
... 47 more
Caused by: com.amazonaws.SdkClientException: Unable to execute HTTP request: Timeout waiting for connection from pool
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleRetryableException(AmazonHttpClient.java:1207)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1153)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:802)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:770)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:744)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:704)
at com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:686)
at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:550)
at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:530)
at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:5227)
at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:5173)
at com.amazonaws.services.s3.AmazonS3Client.getObjectMetadata(AmazonS3Client.java:1360)
at org.apache.hadoop.fs.s3a.S3AFileSystem.lambda$getObjectMetadata$6(S3AFileSystem.java:2066)
at org.apache.hadoop.fs.s3a.Invoker.retryUntranslated(Invoker.java:412)
at org.apache.hadoop.fs.s3a.Invoker.retryUntranslated(Invoker.java:375)
at org.apache.hadoop.fs.s3a.S3AFileSystem.getObjectMetadata(S3AFileSystem.java:2056)
at org.apache.hadoop.fs.s3a.S3AFileSystem.getObjectMetadata(S3AFileSystem.java:2032)
at org.apache.hadoop.fs.s3a.S3AFileSystem.s3GetFileStatus(S3AFileSystem.java:3273)
... 53 more
Caused by: com.amazonaws.thirdparty.apache.http.conn.ConnectionPoolTimeoutException: Timeout waiting for connection from pool
at com.amazonaws.thirdparty.apache.http.impl.conn.PoolingHttpClientConnectionManager.leaseConnection(PoolingHttpClientConnectionManager.java:316)
at com.amazonaws.thirdparty.apache.http.impl.conn.PoolingHttpClientConnectionManager$1.get(PoolingHttpClientConnectionManager.java:282)
at jdk.internal.reflect.GeneratedMethodAccessor30.invoke(Unknown Source)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
at java.base/java.lang.reflect.Method.invoke(Unknown Source)
at com.amazonaws.http.conn.ClientConnectionRequestFactory$Handler.invoke(ClientConnectionRequestFactory.java:70)
at com.amazonaws.http.conn.$Proxy20.get(Unknown Source)
at com.amazonaws.thirdparty.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:190)
at com.amazonaws.thirdparty.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186)
at com.amazonaws.thirdparty.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185)
at com.amazonaws.thirdparty.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83)
at com.amazonaws.thirdparty.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:56)
at com.amazonaws.http.apache.client.impl.SdkHttpClient.execute(SdkHttpClient.java:72)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1333)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1145)
... 69 more
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] danny0405 commented on issue #8325: [SUPPORT] spark read hudi error: Unable to instantiate HFileBootstrapIndex
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on issue #8325:
URL: https://github.com/apache/hudi/issues/8325#issuecomment-1506439431
> Maybe we could move some logic out of the constructor
Is there any possibiity we can have some code refactoring ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] yihua commented on issue #8325: [SUPPORT] spark read hudi error: Unable to instantiate HFileBootstrapIndex
Posted by "yihua (via GitHub)" <gi...@apache.org>.
yihua commented on issue #8325:
URL: https://github.com/apache/hudi/issues/8325#issuecomment-1539190582
It looks like the issues are due to transient connection issues. Also, recently we have fixed the metadata table read in a few code paths so that the numbers FS calls do not rely on the number of partitions. @jfrylings-twilio Thanks for confirming it works after setting the maximum connections.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] ad1happy2go commented on issue #8325: [SUPPORT] spark read hudi error: Unable to instantiate HFileBootstrapIndex
Posted by "ad1happy2go (via GitHub)" <gi...@apache.org>.
ad1happy2go commented on issue #8325:
URL: https://github.com/apache/hudi/issues/8325#issuecomment-1536063152
@hbgstc123 Can you try setting property, spark.hadoop.fs.s3.maxConnections -> 1000
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] xccui commented on issue #8325: [SUPPORT] spark read hudi error: Unable to instantiate HFileBootstrapIndex
Posted by "xccui (via GitHub)" <gi...@apache.org>.
xccui commented on issue #8325:
URL: https://github.com/apache/hudi/issues/8325#issuecomment-1504472901
Got some time today to take a closer look at the errors. `HFileBootstrapIndex` needs to access some remote data during initialization. There should be some connection issues (e.g. file system closed or connection interrupted due to some reason) causing the initialization to fail. Shouldn't be a compatibility problem.
Maybe we could move some logic out of the constructor.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] xccui commented on issue #8325: [SUPPORT] spark read hudi error: Unable to instantiate HFileBootstrapIndex
Posted by "xccui (via GitHub)" <gi...@apache.org>.
xccui commented on issue #8325:
URL: https://github.com/apache/hudi/issues/8325#issuecomment-1496179790
> > I hit the same exception in a Flink writer job. It happened when the job was trying to recover from a failure.
> > Hudi version: 0.13.0 Flink version: 1.16.1
>
> have you try to reproduce the error? does it happen when you restart the job manually? If it does reproduce, can you share your configs and table fields type so we can try to reproduce locally and do some debugging.
It happened when a job tried to recover from a failure. Here is the full stack trace.
```
2023-04-03 18:25:00 [stream_write: response_insertion_ids (5/5)#4] ERROR org.apache.hudi.io.HoodieAppendHandle [] - Error writing record HoodieRecord{key=HoodieKey { recordKey=platformId:120,requestId:cfuleqeijb4qrau0tl4g partitionPath=dt=2023-02-28/hr=01}, currentLocation='HoodieRecordLocation {instantTime=U, fileId=null}', newLocation='null'}
org.apache.hudi.exception.HoodieException: Unable to instantiate class org.apache.hudi.common.bootstrap.index.HFileBootstrapIndex
at org.apache.hudi.common.util.ReflectionUtils.loadClass(ReflectionUtils.java:79) ~[blob_p-17a9236a425d801a5f243ade3abda3e034663e62-8dfab769092e9e11eec10f6efa74d93b:?]
at org.apache.hudi.common.bootstrap.index.BootstrapIndex.getBootstrapIndex(BootstrapIndex.java:163) ~[blob_p-17a9236a425d801a5f243ade3abda3e034663e62-8dfab769092e9e11eec10f6efa74d93b:?]
at org.apache.hudi.common.table.view.AbstractTableFileSystemView.init(AbstractTableFileSystemView.java:114) ~[blob_p-17a9236a425d801a5f243ade3abda3e034663e62-8dfab769092e9e11eec10f6efa74d93b:?]
at org.apache.hudi.common.table.view.HoodieTableFileSystemView.init(HoodieTableFileSystemView.java:113) ~[blob_p-17a9236a425d801a5f243ade3abda3e034663e62-8dfab769092e9e11eec10f6efa74d93b:?]
at org.apache.hudi.common.table.view.HoodieTableFileSystemView.<init>(HoodieTableFileSystemView.java:107) ~[blob_p-17a9236a425d801a5f243ade3abda3e034663e62-8dfab769092e9e11eec10f6efa74d93b:?]
at org.apache.hudi.common.table.view.FileSystemViewManager.createInMemoryFileSystemView(FileSystemViewManager.java:177) ~[blob_p-17a9236a425d801a5f243ade3abda3e034663e62-8dfab769092e9e11eec10f6efa74d93b:?]
at org.apache.hudi.common.table.view.FileSystemViewManager.lambda$createViewManager$5fcdabfe$1(FileSystemViewManager.java:272) ~[blob_p-17a9236a425d801a5f243ade3abda3e034663e62-8dfab769092e9e11eec10f6efa74d93b:?]
at org.apache.hudi.common.table.view.FileSystemViewManager.lambda$getFileSystemView$1(FileSystemViewManager.java:115) ~[blob_p-17a9236a425d801a5f243ade3abda3e034663e62-8dfab769092e9e11eec10f6efa74d93b:?]
at java.util.concurrent.ConcurrentHashMap.computeIfAbsent(Unknown Source) ~[?:?]
at org.apache.hudi.common.table.view.FileSystemViewManager.getFileSystemView(FileSystemViewManager.java:114) ~[blob_p-17a9236a425d801a5f243ade3abda3e034663e62-8dfab769092e9e11eec10f6efa74d93b:?]
at org.apache.hudi.table.HoodieTable.getSliceView(HoodieTable.java:316) ~[blob_p-17a9236a425d801a5f243ade3abda3e034663e62-8dfab769092e9e11eec10f6efa74d93b:?]
at org.apache.hudi.io.HoodieAppendHandle.init(HoodieAppendHandle.java:162) ~[blob_p-17a9236a425d801a5f243ade3abda3e034663e62-8dfab769092e9e11eec10f6efa74d93b:?]
at org.apache.hudi.io.HoodieAppendHandle.doWrite(HoodieAppendHandle.java:486) ~[blob_p-17a9236a425d801a5f243ade3abda3e034663e62-8dfab769092e9e11eec10f6efa74d93b:?]
at org.apache.hudi.io.HoodieWriteHandle.write(HoodieWriteHandle.java:175) ~[blob_p-17a9236a425d801a5f243ade3abda3e034663e62-8dfab769092e9e11eec10f6efa74d93b:?]
at org.apache.hudi.execution.ExplicitWriteHandler.consume(ExplicitWriteHandler.java:49) ~[blob_p-17a9236a425d801a5f243ade3abda3e034663e62-8dfab769092e9e11eec10f6efa74d93b:?]
at org.apache.hudi.execution.ExplicitWriteHandler.consume(ExplicitWriteHandler.java:35) ~[blob_p-17a9236a425d801a5f243ade3abda3e034663e62-8dfab769092e9e11eec10f6efa74d93b:?]
at org.apache.hudi.common.util.queue.SimpleExecutor.execute(SimpleExecutor.java:67) ~[blob_p-17a9236a425d801a5f243ade3abda3e034663e62-8dfab769092e9e11eec10f6efa74d93b:?]
at org.apache.hudi.execution.FlinkLazyInsertIterable.computeNext(FlinkLazyInsertIterable.java:64) ~[blob_p-17a9236a425d801a5f243ade3abda3e034663e62-8dfab769092e9e11eec10f6efa74d93b:?]
at org.apache.hudi.execution.FlinkLazyInsertIterable.computeNext(FlinkLazyInsertIterable.java:43) ~[blob_p-17a9236a425d801a5f243ade3abda3e034663e62-8dfab769092e9e11eec10f6efa74d93b:?]
at org.apache.hudi.client.utils.LazyIterableIterator.next(LazyIterableIterator.java:119) ~[blob_p-17a9236a425d801a5f243ade3abda3e034663e62-8dfab769092e9e11eec10f6efa74d93b:?]
at java.util.Iterator.forEachRemaining(Unknown Source) ~[?:?]
at org.apache.hudi.table.action.commit.BaseFlinkCommitActionExecutor.execute(BaseFlinkCommitActionExecutor.java:113) ~[blob_p-17a9236a425d801a5f243ade3abda3e034663e62-8dfab769092e9e11eec10f6efa74d93b:?]
at org.apache.hudi.table.action.commit.BaseFlinkCommitActionExecutor.execute(BaseFlinkCommitActionExecutor.java:69) ~[blob_p-17a9236a425d801a5f243ade3abda3e034663e62-8dfab769092e9e11eec10f6efa74d93b:?]
at org.apache.hudi.table.action.commit.FlinkWriteHelper.write(FlinkWriteHelper.java:77) ~[blob_p-17a9236a425d801a5f243ade3abda3e034663e62-8dfab769092e9e11eec10f6efa74d93b:?]
at org.apache.hudi.table.action.commit.delta.FlinkUpsertDeltaCommitActionExecutor.execute(FlinkUpsertDeltaCommitActionExecutor.java:51) ~[blob_p-17a9236a425d801a5f243ade3abda3e034663e62-8dfab769092e9e11eec10f6efa74d93b:?]
at org.apache.hudi.table.HoodieFlinkMergeOnReadTable.upsert(HoodieFlinkMergeOnReadTable.java:69) ~[blob_p-17a9236a425d801a5f243ade3abda3e034663e62-8dfab769092e9e11eec10f6efa74d93b:?]
at org.apache.hudi.client.HoodieFlinkWriteClient.upsert(HoodieFlinkWriteClient.java:147) ~[blob_p-17a9236a425d801a5f243ade3abda3e034663e62-8dfab769092e9e11eec10f6efa74d93b:?]
at org.apache.hudi.sink.StreamWriteFunction.lambda$initWriteFunction$1(StreamWriteFunction.java:191) ~[blob_p-17a9236a425d801a5f243ade3abda3e034663e62-8dfab769092e9e11eec10f6efa74d93b:?]
at org.apache.hudi.sink.StreamWriteFunction.lambda$flushRemaining$7(StreamWriteFunction.java:478) ~[blob_p-17a9236a425d801a5f243ade3abda3e034663e62-8dfab769092e9e11eec10f6efa74d93b:?]
at java.util.LinkedHashMap$LinkedValues.forEach(Unknown Source) ~[?:?]
at org.apache.hudi.sink.StreamWriteFunction.flushRemaining(StreamWriteFunction.java:468) ~[blob_p-17a9236a425d801a5f243ade3abda3e034663e62-8dfab769092e9e11eec10f6efa74d93b:?]
at org.apache.hudi.sink.StreamWriteFunction.snapshotState(StreamWriteFunction.java:138) ~[blob_p-17a9236a425d801a5f243ade3abda3e034663e62-8dfab769092e9e11eec10f6efa74d93b:?]
at org.apache.hudi.sink.common.AbstractStreamWriteFunction.snapshotState(AbstractStreamWriteFunction.java:167) ~[blob_p-17a9236a425d801a5f243ade3abda3e034663e62-8dfab769092e9e11eec10f6efa74d93b:?]
at org.apache.flink.streaming.util.functions.StreamingFunctionUtils.trySnapshotFunctionState(StreamingFunctionUtils.java:118) ~[flink-dist-1.16.1.jar:1.16.1]
at org.apache.flink.streaming.util.functions.StreamingFunctionUtils.snapshotFunctionState(StreamingFunctionUtils.java:99) ~[flink-dist-1.16.1.jar:1.16.1]
at org.apache.flink.streaming.api.operators.AbstractUdfStreamOperator.snapshotState(AbstractUdfStreamOperator.java:87) ~[flink-dist-1.16.1.jar:1.16.1]
at org.apache.flink.streaming.api.operators.StreamOperatorStateHandler.snapshotState(StreamOperatorStateHandler.java:222) ~[flink-dist-1.16.1.jar:1.16.1]
at org.apache.flink.streaming.api.operators.StreamOperatorStateHandler.snapshotState(StreamOperatorStateHandler.java:173) ~[flink-dist-1.16.1.jar:1.16.1]
at org.apache.flink.streaming.api.operators.AbstractStreamOperator.snapshotState(AbstractStreamOperator.java:345) ~[flink-dist-1.16.1.jar:1.16.1]
at org.apache.flink.streaming.runtime.tasks.RegularOperatorChain.checkpointStreamOperator(RegularOperatorChain.java:228) ~[flink-dist-1.16.1.jar:1.16.1]
at org.apache.flink.streaming.runtime.tasks.RegularOperatorChain.buildOperatorSnapshotFutures(RegularOperatorChain.java:213) ~[flink-dist-1.16.1.jar:1.16.1]
at org.apache.flink.streaming.runtime.tasks.RegularOperatorChain.snapshotState(RegularOperatorChain.java:192) ~[flink-dist-1.16.1.jar:1.16.1]
at org.apache.flink.streaming.runtime.tasks.SubtaskCheckpointCoordinatorImpl.takeSnapshotSync(SubtaskCheckpointCoordinatorImpl.java:726) ~[flink-dist-1.16.1.jar:1.16.1]
at org.apache.flink.streaming.runtime.tasks.SubtaskCheckpointCoordinatorImpl.checkpointState(SubtaskCheckpointCoordinatorImpl.java:363) ~[flink-dist-1.16.1.jar:1.16.1]
at org.apache.flink.streaming.runtime.tasks.StreamTask.lambda$performCheckpoint$13(StreamTask.java:1281) ~[flink-dist-1.16.1.jar:1.16.1]
at org.apache.flink.streaming.runtime.tasks.StreamTaskActionExecutor$1.runThrowing(StreamTaskActionExecutor.java:50) ~[flink-dist-1.16.1.jar:1.16.1]
at org.apache.flink.streaming.runtime.tasks.StreamTask.performCheckpoint(StreamTask.java:1269) ~[flink-dist-1.16.1.jar:1.16.1]
at org.apache.flink.streaming.runtime.tasks.StreamTask.triggerCheckpointOnBarrier(StreamTask.java:1226) ~[flink-dist-1.16.1.jar:1.16.1]
at org.apache.flink.streaming.runtime.io.checkpointing.CheckpointBarrierHandler.notifyCheckpoint(CheckpointBarrierHandler.java:147) ~[flink-dist-1.16.1.jar:1.16.1]
at org.apache.flink.streaming.runtime.io.checkpointing.SingleCheckpointBarrierHandler.triggerCheckpoint(SingleCheckpointBarrierHandler.java:287) ~[flink-dist-1.16.1.jar:1.16.1]
at org.apache.flink.streaming.runtime.io.checkpointing.SingleCheckpointBarrierHandler.access$100(SingleCheckpointBarrierHandler.java:64) ~[flink-dist-1.16.1.jar:1.16.1]
at org.apache.flink.streaming.runtime.io.checkpointing.SingleCheckpointBarrierHandler$ControllerImpl.triggerGlobalCheckpoint(SingleCheckpointBarrierHandler.java:488) ~[flink-dist-1.16.1.jar:1.16.1]
at org.apache.flink.streaming.runtime.io.checkpointing.AlternatingWaitingForFirstBarrierUnaligned.barrierReceived(AlternatingWaitingForFirstBarrierUnaligned.java:78) ~[flink-dist-1.16.1.jar:1.16.1]
at org.apache.flink.streaming.runtime.io.checkpointing.AbstractAlternatingAlignedBarrierHandlerState.barrierReceived(AbstractAlternatingAlignedBarrierHandlerState.java:55) ~[flink-dist-1.16.1.jar:1.16.1]
at org.apache.flink.streaming.runtime.io.checkpointing.SingleCheckpointBarrierHandler.lambda$processBarrier$2(SingleCheckpointBarrierHandler.java:234) ~[flink-dist-1.16.1.jar:1.16.1]
at org.apache.flink.streaming.runtime.io.checkpointing.SingleCheckpointBarrierHandler.markCheckpointAlignedAndTransformState(SingleCheckpointBarrierHandler.java:262) ~[flink-dist-1.16.1.jar:1.16.1]
at org.apache.flink.streaming.runtime.io.checkpointing.SingleCheckpointBarrierHandler.processBarrier(SingleCheckpointBarrierHandler.java:231) ~[flink-dist-1.16.1.jar:1.16.1]
at org.apache.flink.streaming.runtime.io.checkpointing.CheckpointedInputGate.handleEvent(CheckpointedInputGate.java:181) ~[flink-dist-1.16.1.jar:1.16.1]
at org.apache.flink.streaming.runtime.io.checkpointing.CheckpointedInputGate.pollNext(CheckpointedInputGate.java:159) ~[flink-dist-1.16.1.jar:1.16.1]
at org.apache.flink.streaming.runtime.io.AbstractStreamTaskNetworkInput.emitNext(AbstractStreamTaskNetworkInput.java:110) ~[flink-dist-1.16.1.jar:1.16.1]
at org.apache.flink.streaming.runtime.io.StreamOneInputProcessor.processInput(StreamOneInputProcessor.java:65) ~[flink-dist-1.16.1.jar:1.16.1]
at org.apache.flink.streaming.runtime.tasks.StreamTask.processInput(StreamTask.java:542) ~[flink-dist-1.16.1.jar:1.16.1]
at org.apache.flink.streaming.runtime.tasks.mailbox.MailboxProcessor.runMailboxLoop(MailboxProcessor.java:231) ~[flink-dist-1.16.1.jar:1.16.1]
at org.apache.flink.streaming.runtime.tasks.StreamTask.runMailboxLoop(StreamTask.java:831) ~[flink-dist-1.16.1.jar:1.16.1]
at org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:780) ~[flink-dist-1.16.1.jar:1.16.1]
at org.apache.flink.runtime.taskmanager.Task.runWithSystemExitMonitoring(Task.java:935) [flink-dist-1.16.1.jar:1.16.1]
at org.apache.flink.runtime.taskmanager.Task.restoreAndInvoke(Task.java:914) [flink-dist-1.16.1.jar:1.16.1]
at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:728) [flink-dist-1.16.1.jar:1.16.1]
at org.apache.flink.runtime.taskmanager.Task.run(Task.java:550) [flink-dist-1.16.1.jar:1.16.1]
at java.lang.Thread.run(Unknown Source) [?:?]
Caused by: java.lang.reflect.InvocationTargetException
at jdk.internal.reflect.GeneratedConstructorAccessor265.newInstance(Unknown Source) ~[?:?]
at jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(Unknown Source) ~[?:?]
at java.lang.reflect.Constructor.newInstance(Unknown Source) ~[?:?]
at org.apache.hudi.common.util.ReflectionUtils.loadClass(ReflectionUtils.java:77) ~[blob_p-17a9236a425d801a5f243ade3abda3e034663e62-8dfab769092e9e11eec10f6efa74d93b:?]
... 69 more
Caused by: org.apache.hudi.exception.HoodieIOException: getFileStatus on s3a://path-to-table/.hoodie/.aux/.bootstrap/.partitions/00000000-0000-0000-0000-000000000000-0_1-0-1_00000000000001.hfile: com.amazonaws.AbortedException:
at org.apache.hudi.common.bootstrap.index.HFileBootstrapIndex.<init>(HFileBootstrapIndex.java:110) ~[blob_p-17a9236a425d801a5f243ade3abda3e034663e62-8dfab769092e9e11eec10f6efa74d93b:?]
at jdk.internal.reflect.GeneratedConstructorAccessor265.newInstance(Unknown Source) ~[?:?]
at jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(Unknown Source) ~[?:?]
at java.lang.reflect.Constructor.newInstance(Unknown Source) ~[?:?]
at org.apache.hudi.common.util.ReflectionUtils.loadClass(ReflectionUtils.java:77) ~[blob_p-17a9236a425d801a5f243ade3abda3e034663e62-8dfab769092e9e11eec10f6efa74d93b:?]
... 69 more
Caused by: java.io.InterruptedIOException: getFileStatus on s3a://path-to-table/.hoodie/.aux/.bootstrap/.partitions/00000000-0000-0000-0000-000000000000-0_1-0-1_00000000000001.hfile: com.amazonaws.AbortedException:
at org.apache.hadoop.fs.s3a.S3AUtils.translateInterruptedException(S3AUtils.java:395) ~[blob_p-17a9236a425d801a5f243ade3abda3e034663e62-8dfab769092e9e11eec10f6efa74d93b:?]
at org.apache.hadoop.fs.s3a.S3AUtils.translateException(S3AUtils.java:201) ~[blob_p-17a9236a425d801a5f243ade3abda3e034663e62-8dfab769092e9e11eec10f6efa74d93b:?]
at org.apache.hadoop.fs.s3a.S3AUtils.translateException(S3AUtils.java:175) ~[blob_p-17a9236a425d801a5f243ade3abda3e034663e62-8dfab769092e9e11eec10f6efa74d93b:?]
at org.apache.hadoop.fs.s3a.S3AFileSystem.s3GetFileStatus(S3AFileSystem.java:3799) ~[blob_p-17a9236a425d801a5f243ade3abda3e034663e62-8dfab769092e9e11eec10f6efa74d93b:?]
at org.apache.hadoop.fs.s3a.S3AFileSystem.innerGetFileStatus(S3AFileSystem.java:3688) ~[blob_p-17a9236a425d801a5f243ade3abda3e034663e62-8dfab769092e9e11eec10f6efa74d93b:?]
at org.apache.hadoop.fs.s3a.S3AFileSystem.lambda$exists$34(S3AFileSystem.java:4703) ~[blob_p-17a9236a425d801a5f243ade3abda3e034663e62-8dfab769092e9e11eec10f6efa74d93b:?]
at org.apache.hadoop.fs.statistics.impl.IOStatisticsBinding.lambda$trackDurationOfOperation$5(IOStatisticsBinding.java:499) ~[blob_p-17a9236a425d801a5f243ade3abda3e034663e62-8dfab769092e9e11eec10f6efa74d93b:?]
at org.apache.hadoop.fs.statistics.impl.IOStatisticsBinding.trackDuration(IOStatisticsBinding.java:444) ~[blob_p-17a9236a425d801a5f243ade3abda3e034663e62-8dfab769092e9e11eec10f6efa74d93b:?]
at org.apache.hadoop.fs.s3a.S3AFileSystem.trackDurationAndSpan(S3AFileSystem.java:2337) ~[blob_p-17a9236a425d801a5f243ade3abda3e034663e62-8dfab769092e9e11eec10f6efa74d93b:?]
at org.apache.hadoop.fs.s3a.S3AFileSystem.trackDurationAndSpan(S3AFileSystem.java:2356) ~[blob_p-17a9236a425d801a5f243ade3abda3e034663e62-8dfab769092e9e11eec10f6efa74d93b:?]
at org.apache.hadoop.fs.s3a.S3AFileSystem.exists(S3AFileSystem.java:4701) ~[blob_p-17a9236a425d801a5f243ade3abda3e034663e62-8dfab769092e9e11eec10f6efa74d93b:?]
at org.apache.hudi.common.fs.HoodieWrapperFileSystem.exists(HoodieWrapperFileSystem.java:566) ~[blob_p-17a9236a425d801a5f243ade3abda3e034663e62-8dfab769092e9e11eec10f6efa74d93b:?]
at org.apache.hudi.common.bootstrap.index.HFileBootstrapIndex.<init>(HFileBootstrapIndex.java:108) ~[blob_p-17a9236a425d801a5f243ade3abda3e034663e62-8dfab769092e9e11eec10f6efa74d93b:?]
at jdk.internal.reflect.GeneratedConstructorAccessor265.newInstance(Unknown Source) ~[?:?]
at jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(Unknown Source) ~[?:?]
at java.lang.reflect.Constructor.newInstance(Unknown Source) ~[?:?]
at org.apache.hudi.common.util.ReflectionUtils.loadClass(ReflectionUtils.java:77) ~[blob_p-17a9236a425d801a5f243ade3abda3e034663e62-8dfab769092e9e11eec10f6efa74d93b:?]
... 69 more
Caused by: com.amazonaws.AbortedException:
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleInterruptedException(AmazonHttpClient.java:880) ~[blob_p-17a9236a425d801a5f243ade3abda3e034663e62-8dfab769092e9e11eec10f6efa74d93b:?]
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:757) ~[blob_p-17a9236a425d801a5f243ade3abda3e034663e62-8dfab769092e9e11eec10f6efa74d93b:?]
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:715) ~[blob_p-17a9236a425d801a5f243ade3abda3e034663e62-8dfab769092e9e11eec10f6efa74d93b:?]
at com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:697) ~[blob_p-17a9236a425d801a5f243ade3abda3e034663e62-8dfab769092e9e11eec10f6efa74d93b:?]
at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:561) ~[blob_p-17a9236a425d801a5f243ade3abda3e034663e62-8dfab769092e9e11eec10f6efa74d93b:?]
at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:541) ~[blob_p-17a9236a425d801a5f243ade3abda3e034663e62-8dfab769092e9e11eec10f6efa74d93b:?]
at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:5456) ~[blob_p-17a9236a425d801a5f243ade3abda3e034663e62-8dfab769092e9e11eec10f6efa74d93b:?]
at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:5403) ~[blob_p-17a9236a425d801a5f243ade3abda3e034663e62-8dfab769092e9e11eec10f6efa74d93b:?]
at com.amazonaws.services.s3.AmazonS3Client.getObjectMetadata(AmazonS3Client.java:1372) ~[blob_p-17a9236a425d801a5f243ade3abda3e034663e62-8dfab769092e9e11eec10f6efa74d93b:?]
at org.apache.hadoop.fs.s3a.S3AFileSystem.lambda$getObjectMetadata$10(S3AFileSystem.java:2545) ~[blob_p-17a9236a425d801a5f243ade3abda3e034663e62-8dfab769092e9e11eec10f6efa74d93b:?]
at org.apache.hadoop.fs.s3a.Invoker.retryUntranslated(Invoker.java:414) ~[blob_p-17a9236a425d801a5f243ade3abda3e034663e62-8dfab769092e9e11eec10f6efa74d93b:?]
at org.apache.hadoop.fs.s3a.Invoker.retryUntranslated(Invoker.java:377) ~[blob_p-17a9236a425d801a5f243ade3abda3e034663e62-8dfab769092e9e11eec10f6efa74d93b:?]
at org.apache.hadoop.fs.s3a.S3AFileSystem.getObjectMetadata(S3AFileSystem.java:2533) ~[blob_p-17a9236a425d801a5f243ade3abda3e034663e62-8dfab769092e9e11eec10f6efa74d93b:?]
at org.apache.hadoop.fs.s3a.S3AFileSystem.getObjectMetadata(S3AFileSystem.java:2513) ~[blob_p-17a9236a425d801a5f243ade3abda3e034663e62-8dfab769092e9e11eec10f6efa74d93b:?]
at org.apache.hadoop.fs.s3a.S3AFileSystem.s3GetFileStatus(S3AFileSystem.java:3776) ~[blob_p-17a9236a425d801a5f243ade3abda3e034663e62-8dfab769092e9e11eec10f6efa74d93b:?]
at org.apache.hadoop.fs.s3a.S3AFileSystem.innerGetFileStatus(S3AFileSystem.java:3688) ~[blob_p-17a9236a425d801a5f243ade3abda3e034663e62-8dfab769092e9e11eec10f6efa74d93b:?]
at org.apache.hadoop.fs.s3a.S3AFileSystem.lambda$exists$34(S3AFileSystem.java:4703) ~[blob_p-17a9236a425d801a5f243ade3abda3e034663e62-8dfab769092e9e11eec10f6efa74d93b:?]
at org.apache.hadoop.fs.statistics.impl.IOStatisticsBinding.lambda$trackDurationOfOperation$5(IOStatisticsBinding.java:499) ~[blob_p-17a9236a425d801a5f243ade3abda3e034663e62-8dfab769092e9e11eec10f6efa74d93b:?]
at org.apache.hadoop.fs.statistics.impl.IOStatisticsBinding.trackDuration(IOStatisticsBinding.java:444) ~[blob_p-17a9236a425d801a5f243ade3abda3e034663e62-8dfab769092e9e11eec10f6efa74d93b:?]
at org.apache.hadoop.fs.s3a.S3AFileSystem.trackDurationAndSpan(S3AFileSystem.java:2337) ~[blob_p-17a9236a425d801a5f243ade3abda3e034663e62-8dfab769092e9e11eec10f6efa74d93b:?]
at org.apache.hadoop.fs.s3a.S3AFileSystem.trackDurationAndSpan(S3AFileSystem.java:2356) ~[blob_p-17a9236a425d801a5f243ade3abda3e034663e62-8dfab769092e9e11eec10f6efa74d93b:?]
at org.apache.hadoop.fs.s3a.S3AFileSystem.exists(S3AFileSystem.java:4701) ~[blob_p-17a9236a425d801a5f243ade3abda3e034663e62-8dfab769092e9e11eec10f6efa74d93b:?]
at org.apache.hudi.common.fs.HoodieWrapperFileSystem.exists(HoodieWrapperFileSystem.java:566) ~[blob_p-17a9236a425d801a5f243ade3abda3e034663e62-8dfab769092e9e11eec10f6efa74d93b:?]
at org.apache.hudi.common.bootstrap.index.HFileBootstrapIndex.<init>(HFileBootstrapIndex.java:108) ~[blob_p-17a9236a425d801a5f243ade3abda3e034663e62-8dfab769092e9e11eec10f6efa74d93b:?]
at jdk.internal.reflect.GeneratedConstructorAccessor265.newInstance(Unknown Source) ~[?:?]
at jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(Unknown Source) ~[?:?]
at java.lang.reflect.Constructor.newInstance(Unknown Source) ~[?:?]
at org.apache.hudi.common.util.ReflectionUtils.loadClass(ReflectionUtils.java:77) ~[blob_p-17a9236a425d801a5f243ade3abda3e034663e62-8dfab769092e9e11eec10f6efa74d93b:?]
... 69 more
Caused by: com.amazonaws.http.timers.client.SdkInterruptedException
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.checkInterrupted(AmazonHttpClient.java:935) ~[blob_p-17a9236a425d801a5f243ade3abda3e034663e62-8dfab769092e9e11eec10f6efa74d93b:?]
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.checkInterrupted(AmazonHttpClient.java:921) ~[blob_p-17a9236a425d801a5f243ade3abda3e034663e62-8dfab769092e9e11eec10f6efa74d93b:?]
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1115) ~[blob_p-17a9236a425d801a5f243ade3abda3e034663e62-8dfab769092e9e11eec10f6efa74d93b:?]
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:814) ~[blob_p-17a9236a425d801a5f243ade3abda3e034663e62-8dfab769092e9e11eec10f6efa74d93b:?]
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:781) ~[blob_p-17a9236a425d801a5f243ade3abda3e034663e62-8dfab769092e9e11eec10f6efa74d93b:?]
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:755) ~[blob_p-17a9236a425d801a5f243ade3abda3e034663e62-8dfab769092e9e11eec10f6efa74d93b:?]
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:715) ~[blob_p-17a9236a425d801a5f243ade3abda3e034663e62-8dfab769092e9e11eec10f6efa74d93b:?]
at com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:697) ~[blob_p-17a9236a425d801a5f243ade3abda3e034663e62-8dfab769092e9e11eec10f6efa74d93b:?]
at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:561) ~[blob_p-17a9236a425d801a5f243ade3abda3e034663e62-8dfab769092e9e11eec10f6efa74d93b:?]
at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:541) ~[blob_p-17a9236a425d801a5f243ade3abda3e034663e62-8dfab769092e9e11eec10f6efa74d93b:?]
at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:5456) ~[blob_p-17a9236a425d801a5f243ade3abda3e034663e62-8dfab769092e9e11eec10f6efa74d93b:?]
at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:5403) ~[blob_p-17a9236a425d801a5f243ade3abda3e034663e62-8dfab769092e9e11eec10f6efa74d93b:?]
at com.amazonaws.services.s3.AmazonS3Client.getObjectMetadata(AmazonS3Client.java:1372) ~[blob_p-17a9236a425d801a5f243ade3abda3e034663e62-8dfab769092e9e11eec10f6efa74d93b:?]
at org.apache.hadoop.fs.s3a.S3AFileSystem.lambda$getObjectMetadata$10(S3AFileSystem.java:2545) ~[blob_p-17a9236a425d801a5f243ade3abda3e034663e62-8dfab769092e9e11eec10f6efa74d93b:?]
at org.apache.hadoop.fs.s3a.Invoker.retryUntranslated(Invoker.java:414) ~[blob_p-17a9236a425d801a5f243ade3abda3e034663e62-8dfab769092e9e11eec10f6efa74d93b:?]
at org.apache.hadoop.fs.s3a.Invoker.retryUntranslated(Invoker.java:377) ~[blob_p-17a9236a425d801a5f243ade3abda3e034663e62-8dfab769092e9e11eec10f6efa74d93b:?]
at org.apache.hadoop.fs.s3a.S3AFileSystem.getObjectMetadata(S3AFileSystem.java:2533) ~[blob_p-17a9236a425d801a5f243ade3abda3e034663e62-8dfab769092e9e11eec10f6efa74d93b:?]
at org.apache.hadoop.fs.s3a.S3AFileSystem.getObjectMetadata(S3AFileSystem.java:2513) ~[blob_p-17a9236a425d801a5f243ade3abda3e034663e62-8dfab769092e9e11eec10f6efa74d93b:?]
at org.apache.hadoop.fs.s3a.S3AFileSystem.s3GetFileStatus(S3AFileSystem.java:3776) ~[blob_p-17a9236a425d801a5f243ade3abda3e034663e62-8dfab769092e9e11eec10f6efa74d93b:?]
at org.apache.hadoop.fs.s3a.S3AFileSystem.innerGetFileStatus(S3AFileSystem.java:3688) ~[blob_p-17a9236a425d801a5f243ade3abda3e034663e62-8dfab769092e9e11eec10f6efa74d93b:?]
at org.apache.hadoop.fs.s3a.S3AFileSystem.lambda$exists$34(S3AFileSystem.java:4703) ~[blob_p-17a9236a425d801a5f243ade3abda3e034663e62-8dfab769092e9e11eec10f6efa74d93b:?]
at org.apache.hadoop.fs.statistics.impl.IOStatisticsBinding.lambda$trackDurationOfOperation$5(IOStatisticsBinding.java:499) ~[blob_p-17a9236a425d801a5f243ade3abda3e034663e62-8dfab769092e9e11eec10f6efa74d93b:?]
at org.apache.hadoop.fs.statistics.impl.IOStatisticsBinding.trackDuration(IOStatisticsBinding.java:444) ~[blob_p-17a9236a425d801a5f243ade3abda3e034663e62-8dfab769092e9e11eec10f6efa74d93b:?]
at org.apache.hadoop.fs.s3a.S3AFileSystem.trackDurationAndSpan(S3AFileSystem.java:2337) ~[blob_p-17a9236a425d801a5f243ade3abda3e034663e62-8dfab769092e9e11eec10f6efa74d93b:?]
at org.apache.hadoop.fs.s3a.S3AFileSystem.trackDurationAndSpan(S3AFileSystem.java:2356) ~[blob_p-17a9236a425d801a5f243ade3abda3e034663e62-8dfab769092e9e11eec10f6efa74d93b:?]
at org.apache.hadoop.fs.s3a.S3AFileSystem.exists(S3AFileSystem.java:4701) ~[blob_p-17a9236a425d801a5f243ade3abda3e034663e62-8dfab769092e9e11eec10f6efa74d93b:?]
at org.apache.hudi.common.fs.HoodieWrapperFileSystem.exists(HoodieWrapperFileSystem.java:566) ~[blob_p-17a9236a425d801a5f243ade3abda3e034663e62-8dfab769092e9e11eec10f6efa74d93b:?]
at org.apache.hudi.common.bootstrap.index.HFileBootstrapIndex.<init>(HFileBootstrapIndex.java:108) ~[blob_p-17a9236a425d801a5f243ade3abda3e034663e62-8dfab769092e9e11eec10f6efa74d93b:?]
at jdk.internal.reflect.GeneratedConstructorAccessor265.newInstance(Unknown Source) ~[?:?]
at jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(Unknown Source) ~[?:?]
at java.lang.reflect.Constructor.newInstance(Unknown Source) ~[?:?]
at org.apache.hudi.common.util.ReflectionUtils.loadClass(ReflectionUtils.java:77) ~[blob_p-17a9236a425d801a5f243ade3abda3e034663e62-8dfab769092e9e11eec10f6efa74d93b:?]
... 69 more
```
It happened more often when I enabled the metadata table in the past.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hbgstc123 commented on issue #8325: [SUPPORT] spark read hudi error: Unable to instantiate HFileBootstrapIndex
Posted by "hbgstc123 (via GitHub)" <gi...@apache.org>.
hbgstc123 commented on issue #8325:
URL: https://github.com/apache/hudi/issues/8325#issuecomment-1495690105
> I hit the same exception in a Flink writer job. It happened when the job was trying to recover from a failure.
>
> Hudi version: 0.13.0 Flink version: 1.16.1
have you try to reproduce the error?
does it happen when you restart the job manually?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] ad1happy2go commented on issue #8325: [SUPPORT] spark read hudi error: Unable to instantiate HFileBootstrapIndex
Posted by "ad1happy2go (via GitHub)" <gi...@apache.org>.
ad1happy2go commented on issue #8325:
URL: https://github.com/apache/hudi/issues/8325#issuecomment-1549054916
@jfrylings-twilio Closing this issue then, Please reopen in case of any more issues.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] codope closed issue #8325: [SUPPORT] spark read hudi error: Unable to instantiate HFileBootstrapIndex
Posted by "codope (via GitHub)" <gi...@apache.org>.
codope closed issue #8325: [SUPPORT] spark read hudi error: Unable to instantiate HFileBootstrapIndex
URL: https://github.com/apache/hudi/issues/8325
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] xccui commented on issue #8325: [SUPPORT] spark read hudi error: Unable to instantiate HFileBootstrapIndex
Posted by "xccui (via GitHub)" <gi...@apache.org>.
xccui commented on issue #8325:
URL: https://github.com/apache/hudi/issues/8325#issuecomment-1494840392
I hit the same exception in a Flink writer job. It happened when the job was trying to recover from a failure.
Hudi version: 0.13.0
Flink version: 1.16.1
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org