You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by GitBox <gi...@apache.org> on 2022/07/05 12:54:06 UTC
[GitHub] [hudi] noahtaite opened a new issue, #6048: [SUPPORT] S3 throttling while loading a table written with "hoodie.metadata.enable" = true
noahtaite opened a new issue, #6048:
URL: https://github.com/apache/hudi/issues/6048
**Describe the problem you faced**
Our Hudi data lake is heavily partitioned by datasource, year, and month.
We have 1000 datasources currently loaded into the lake, and are looking to load 1000 more over 2 bulk_insert batches. Our Hudi data lake is a Java application that has custom schema validation logic. It is achieved by calling `spark.read.format("hudi").load(existingTable)` to retrieve the schema for comparison with the incoming data.
When trying to create our Hudi data lake on 0.7 with no metadata table, we observed S3 slow down errors during the stage `Listing leaf files and directories for xxx paths:`. We upgraded to Hudi 0.9 and enabled metadata table, restarting our experiment. However, we still observe intermittent S3 throttling issues when calling load() on the existing table.
Hardware:
Master: 1 x m6g.8xlarge
Core: 48 x r5.8xlarge
**To Reproduce**
Steps to reproduce the behavior:
1. Create a large, heavily partitioned, COW Hudi table with metadata enabled
2. Our size currently is 7k+ paths, 216k objects, 3.5TB compressed parquet
3. Load the table with hardware above
4. Observe S3 throttling issue
**Expected behavior**
I expected the S3 throttling issue to not occur, as it is documented that the purpose of the metadata table is to prevent these issues on reads.
**Environment Description**
* Hudi version : 0.9.0-amzn-0 (EMR 5.34.0)
* Spark version : 2.4.8
* Hive version : 2.3.8
* Hadoop version : 2.10.1
* Storage (HDFS/S3/GCS..) : S3
* Running on Docker? (yes/no) : no
**Additional context**
Write operation type for all records so far: bulk_insert
Metadata table enabled: hoodie.metadata.enable = true
**Stacktrace**
```
Job aborted due to stage failure: Task 1272 in stage 63.0 failed 4 times, most recent failure: Lost task 1272.3 in stage 63.0 (TID 185468, ip-xx-xx-5-238.ec2.internal, executor 11): java.io.IOException: com.amazon.ws.emr.hadoop.fs.shaded.com.amazonaws.services.s3.model.AmazonS3Exception: Please reduce your request rate. (Service: Amazon S3; Status Code: 503; Error Code: SlowDown; Request ID: xx; S3 Extended Request ID: xx; Proxy: null), S3 Extended Request ID: xx at com.amazon.ws.emr.hadoop.fs.s3n.Jets3tNativeFileSystemStore.list(Jets3tNativeFileSystemStore.java:421)
at com.amazon.ws.emr.hadoop.fs.s3n.S3NativeFileSystem.listStatus(S3NativeFileSystem.java:654)
at com.amazon.ws.emr.hadoop.fs.s3n.S3NativeFileSystem.listStatus(S3NativeFileSystem.java:625)
at com.amazon.ws.emr.hadoop.fs.EmrFileSystem.listStatus(EmrFileSystem.java:473)
at org.apache.spark.sql.execution.datasources.InMemoryFileIndex$.org$apache$spark$sql$execution$datasources$InMemoryFileIndex$$listLeafFiles(InMemoryFileIndex.scala:320)
at org.apache.spark.sql.execution.datasources.InMemoryFileIndex$$anonfun$3$$anonfun$apply$2.apply(InMemoryFileIndex.scala:218)
at org.apache.spark.sql.execution.datasources.InMemoryFileIndex$$anonfun$3$$anonfun$apply$2.apply(InMemoryFileIndex.scala:217)
at scala.collection.immutable.Stream.map(Stream.scala:418)
at org.apache.spark.sql.execution.datasources.InMemoryFileIndex$$anonfun$3.apply(InMemoryFileIndex.scala:217)
at org.apache.spark.sql.execution.datasources.InMemoryFileIndex$$anonfun$3.apply(InMemoryFileIndex.scala:215)
at org.apache.spark.rdd.RDD$$anonfun$mapPartitions$1$$anonfun$apply$23.apply(RDD.scala:823)
at org.apache.spark.rdd.RDD$$anonfun$mapPartitions$1$$anonfun$apply$23.apply(RDD.scala:823)
at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:52)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:346)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:310)
at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:52)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:346)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:310)
at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:90)
at org.apache.spark.scheduler.Task.run(Task.scala:123)
at org.apache.spark.executor.Executor$TaskRunner$$anonfun$10.apply(Executor.scala:411)
at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1405)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:417)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:750)
Caused by: com.amazon.ws.emr.hadoop.fs.shaded.com.amazonaws.services.s3.model.AmazonS3Exception: Please reduce your request rate. (Service: Amazon S3; Status Code: 503; Error Code: SlowDown; Request ID: xx; S3 Extended Request ID: xx; Proxy: null), S3 Extended Request ID: xx
at com.amazon.ws.emr.hadoop.fs.shaded.com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleErrorResponse(AmazonHttpClient.java:1862)
at com.amazon.ws.emr.hadoop.fs.shaded.com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleServiceErrorResponse(AmazonHttpClient.java:1415)
at com.amazon.ws.emr.hadoop.fs.shaded.com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1384)
at com.amazon.ws.emr.hadoop.fs.shaded.com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1154)
at com.amazon.ws.emr.hadoop.fs.shaded.com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:811)
at com.amazon.ws.emr.hadoop.fs.shaded.com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:779)
at com.amazon.ws.emr.hadoop.fs.shaded.com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:753)
at com.amazon.ws.emr.hadoop.fs.shaded.com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:713)
at com.amazon.ws.emr.hadoop.fs.shaded.com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:695)
at com.amazon.ws.emr.hadoop.fs.shaded.com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:559)
at com.amazon.ws.emr.hadoop.fs.shaded.com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:539)
at com.amazon.ws.emr.hadoop.fs.shaded.com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:5445)
at com.amazon.ws.emr.hadoop.fs.shaded.com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:5392)
at com.amazon.ws.emr.hadoop.fs.shaded.com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:5386)
at com.amazon.ws.emr.hadoop.fs.shaded.com.amazonaws.services.s3.AmazonS3Client.listObjectsV2(AmazonS3Client.java:971)
at com.amazon.ws.emr.hadoop.fs.s3.lite.call.ListObjectsV2Call.perform(ListObjectsV2Call.java:26)
at com.amazon.ws.emr.hadoop.fs.s3.lite.call.ListObjectsV2Call.perform(ListObjectsV2Call.java:12)
at com.amazon.ws.emr.hadoop.fs.s3.lite.executor.GlobalS3Executor$CallPerformer.call(GlobalS3Executor.java:108)
at com.amazon.ws.emr.hadoop.fs.s3.lite.executor.GlobalS3Executor.execute(GlobalS3Executor.java:135)
at com.amazon.ws.emr.hadoop.fs.s3.lite.AmazonS3LiteClient.invoke(AmazonS3LiteClient.java:191)
at com.amazon.ws.emr.hadoop.fs.s3.lite.AmazonS3LiteClient.invoke(AmazonS3LiteClient.java:186)
at com.amazon.ws.emr.hadoop.fs.s3.lite.AmazonS3LiteClient.listObjectsV2(AmazonS3LiteClient.java:75)
at com.amazon.ws.emr.hadoop.fs.s3n.Jets3tNativeFileSystemStore.list(Jets3tNativeFileSystemStore.java:412)
... 25 more
```
Note: This does not happen every time, and it appears to occur more frequently with more executors provided. However, if I lower the # of executors the queries take far too long for data ingestion.
I'm willing to provide additional information and configuration as required. We are trying to understand the root cause of this S3 throttling issue as it breaks the pipeline intermittently
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan closed issue #6048: [SUPPORT] S3 throttling while loading a table written with "hoodie.metadata.enable" = true
Posted by GitBox <gi...@apache.org>.
nsivabalan closed issue #6048: [SUPPORT] S3 throttling while loading a table written with "hoodie.metadata.enable" = true
URL: https://github.com/apache/hudi/issues/6048
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] xushiyan commented on issue #6048: [SUPPORT] S3 throttling while loading a table written with "hoodie.metadata.enable" = true
Posted by GitBox <gi...@apache.org>.
xushiyan commented on issue #6048:
URL: https://github.com/apache/hudi/issues/6048#issuecomment-1296300272
@noahtaite assume things are going well with 0.11 on EMR, good to close this?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan commented on issue #6048: [SUPPORT] S3 throttling while loading a table written with "hoodie.metadata.enable" = true
Posted by GitBox <gi...@apache.org>.
nsivabalan commented on issue #6048:
URL: https://github.com/apache/hudi/issues/6048#issuecomment-1210080037
@noahtaite : can you delete entire "metadata" directory and restart your pipeline w/ 0.10.1 or 0.11.0 or later.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] yihua commented on issue #6048: [SUPPORT] S3 throttling while loading a table written with "hoodie.metadata.enable" = true
Posted by GitBox <gi...@apache.org>.
yihua commented on issue #6048:
URL: https://github.com/apache/hudi/issues/6048#issuecomment-1176947410
@noahtaite We made a foundational redesign to the metadata table with synchronous updates in 0.10.0 release: https://hudi.apache.org/releases/release-0.10.0#metadata-table. Could you try the latest release 0.11.1 and see if the problem is resolved? Could you also share the content under `table/.hoodie/metadata` where the metadata table resides?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] noahtaite commented on issue #6048: [SUPPORT] S3 throttling while loading a table written with "hoodie.metadata.enable" = true
Posted by GitBox <gi...@apache.org>.
noahtaite commented on issue #6048:
URL: https://github.com/apache/hudi/issues/6048#issuecomment-1261589951
Hi @nsivabalan - we had to table this one until EMR released Hudi 0.11. We now have access to EMR 6.7 which supports Hudi 0.11, but no EMR 5.x version (Spark 2) does, We're porting our code to Spark 3 and will test the metadata table performance. Apologies for the delay, and thank you for the support.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] noahtaite commented on issue #6048: [SUPPORT] S3 throttling while loading a table written with "hoodie.metadata.enable" = true
Posted by GitBox <gi...@apache.org>.
noahtaite commented on issue #6048:
URL: https://github.com/apache/hudi/issues/6048#issuecomment-1175210980
You can see the effect is a spike here after inserting 500 more data sources (for a total of 1500) and then calling load() to do schema validation for the last 500.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] noahtaite commented on issue #6048: [SUPPORT] S3 throttling while loading a table written with "hoodie.metadata.enable" = true
Posted by GitBox <gi...@apache.org>.
noahtaite commented on issue #6048:
URL: https://github.com/apache/hudi/issues/6048#issuecomment-1177685164
@yihua
Yes, we can try EMR 5.36 which is released with the following Hudi version: 0.10.1-amzn-1. Do we need to re-generate the metadata table, or will the existing one for our large experiment work ok?
Yes, here is the content of the metadata table:
```
2022-07-04 13:27:11 0 hudi_experiment/20220704/transactions.all_hudi/.hoodie/metadata/.hoodie/.aux/.bootstrap/.fileids_$folder$
2022-07-04 13:27:10 0 hudi_experiment/20220704/transactions.all_hudi/.hoodie/metadata/.hoodie/.aux/.bootstrap/.partitions_$folder$
2022-07-04 13:27:10 0 hudi_experiment/20220704/transactions.all_hudi/.hoodie/metadata/.hoodie/.aux/.bootstrap_$folder$
2022-07-04 13:27:09 0 hudi_experiment/20220704/transactions.all_hudi/.hoodie/metadata/.hoodie/.aux_$folder$
2022-07-04 13:27:09 0 hudi_experiment/20220704/transactions.all_hudi/.hoodie/metadata/.hoodie/.temp_$folder$
2022-07-04 13:27:26 2205 hudi_experiment/20220704/transactions.all_hudi/.hoodie/metadata/.hoodie/20220704141215.deltacommit
2022-07-04 13:27:22 1189 hudi_experiment/20220704/transactions.all_hudi/.hoodie/metadata/.hoodie/20220704141215.deltacommit.inflight
2022-07-04 13:27:21 0 hudi_experiment/20220704/transactions.all_hudi/.hoodie/metadata/.hoodie/20220704141215.deltacommit.requested
2022-07-04 14:33:09 2454 hudi_experiment/20220704/transactions.all_hudi/.hoodie/metadata/.hoodie/20220704162653.deltacommit
2022-07-04 14:33:04 1907 hudi_experiment/20220704/transactions.all_hudi/.hoodie/metadata/.hoodie/20220704162653.deltacommit.inflight
2022-07-04 14:33:03 0 hudi_experiment/20220704/transactions.all_hudi/.hoodie/metadata/.hoodie/20220704162653.deltacommit.requested
2022-07-04 15:44:30 2532 hudi_experiment/20220704/transactions.all_hudi/.hoodie/metadata/.hoodie/20220704170712.deltacommit
2022-07-04 15:44:25 1907 hudi_experiment/20220704/transactions.all_hudi/.hoodie/metadata/.hoodie/20220704170712.deltacommit.inflight
2022-07-04 15:44:24 0 hudi_experiment/20220704/transactions.all_hudi/.hoodie/metadata/.hoodie/20220704170712.deltacommit.requested
2022-07-04 16:36:17 2605 hudi_experiment/20220704/transactions.all_hudi/.hoodie/metadata/.hoodie/20220704181907.deltacommit
2022-07-04 16:36:13 1907 hudi_experiment/20220704/transactions.all_hudi/.hoodie/metadata/.hoodie/20220704181907.deltacommit.inflight
2022-07-04 16:36:12 0 hudi_experiment/20220704/transactions.all_hudi/.hoodie/metadata/.hoodie/20220704181907.deltacommit.requested
2022-07-05 03:08:03 2673 hudi_experiment/20220704/transactions.all_hudi/.hoodie/metadata/.hoodie/20220704190618.deltacommit
2022-07-05 03:07:59 1904 hudi_experiment/20220704/transactions.all_hudi/.hoodie/metadata/.hoodie/20220704190618.deltacommit.inflight
2022-07-05 03:07:58 0 hudi_experiment/20220704/transactions.all_hudi/.hoodie/metadata/.hoodie/20220704190618.deltacommit.requested
2022-07-05 03:08:15 2761 hudi_experiment/20220704/transactions.all_hudi/.hoodie/metadata/.hoodie/20220705031627.deltacommit
2022-07-05 03:08:09 1908 hudi_experiment/20220704/transactions.all_hudi/.hoodie/metadata/.hoodie/20220705031627.deltacommit.inflight
2022-07-05 03:08:07 0 hudi_experiment/20220704/transactions.all_hudi/.hoodie/metadata/.hoodie/20220705031627.deltacommit.requested
2022-07-05 12:55:26 2837 hudi_experiment/20220704/transactions.all_hudi/.hoodie/metadata/.hoodie/20220705122401.deltacommit
2022-07-05 12:55:21 1908 hudi_experiment/20220704/transactions.all_hudi/.hoodie/metadata/.hoodie/20220705122401.deltacommit.inflight
2022-07-05 12:55:19 0 hudi_experiment/20220704/transactions.all_hudi/.hoodie/metadata/.hoodie/20220705122401.deltacommit.requested
2022-07-04 13:27:09 0 hudi_experiment/20220704/transactions.all_hudi/.hoodie/metadata/.hoodie/archived_$folder$
2022-07-04 13:27:11 361 hudi_experiment/20220704/transactions.all_hudi/.hoodie/metadata/.hoodie/hoodie.properties
2022-07-04 13:27:08 0 hudi_experiment/20220704/transactions.all_hudi/.hoodie/metadata/.hoodie_$folder$
2022-07-04 14:33:08 294258 hudi_experiment/20220704/transactions.all_hudi/.hoodie/metadata/files/.2110f5ce-9f6b-4a12-9ff2-b74889a03aa5-0_20220704141215.log.1_0-66-77005
2022-07-04 15:44:30 315616 hudi_experiment/20220704/transactions.all_hudi/.hoodie/metadata/files/.2110f5ce-9f6b-4a12-9ff2-b74889a03aa5-0_20220704141215.log.2_0-66-148500
2022-07-04 16:36:16 314918 hudi_experiment/20220704/transactions.all_hudi/.hoodie/metadata/files/.2110f5ce-9f6b-4a12-9ff2-b74889a03aa5-0_20220704141215.log.3_0-69-86707
2022-07-05 03:08:03 5947 hudi_experiment/20220704/transactions.all_hudi/.hoodie/metadata/files/.2110f5ce-9f6b-4a12-9ff2-b74889a03aa5-0_20220704141215.log.4_0-69-239755
2022-07-05 03:08:14 971618 hudi_experiment/20220704/transactions.all_hudi/.hoodie/metadata/files/.2110f5ce-9f6b-4a12-9ff2-b74889a03aa5-0_20220704141215.log.5_0-81-242837
2022-07-05 12:55:25 985699 hudi_experiment/20220704/transactions.all_hudi/.hoodie/metadata/files/.2110f5ce-9f6b-4a12-9ff2-b74889a03aa5-0_20220704141215.log.6_0-74-271022
2022-07-04 13:27:23 93 hudi_experiment/20220704/transactions.all_hudi/.hoodie/metadata/files/.hoodie_partition_metadata
2022-07-04 13:27:25 469982 hudi_experiment/20220704/transactions.all_hudi/.hoodie/metadata/files/2110f5ce-9f6b-4a12-9ff2-b74889a03aa5-0_0-55-127122_20220704141215.hfile
2022-07-04 13:27:12 0 hudi_experiment/20220704/transactions.all_hudi/.hoodie/metadata/files_$folder$
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan commented on issue #6048: [SUPPORT] S3 throttling while loading a table written with "hoodie.metadata.enable" = true
Posted by GitBox <gi...@apache.org>.
nsivabalan commented on issue #6048:
URL: https://github.com/apache/hudi/issues/6048#issuecomment-1229348933
@noahtaite : any updates here please
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan commented on issue #6048: [SUPPORT] S3 throttling while loading a table written with "hoodie.metadata.enable" = true
Posted by GitBox <gi...@apache.org>.
nsivabalan commented on issue #6048:
URL: https://github.com/apache/hudi/issues/6048#issuecomment-1210080996
btw, in general we had some bugs around metadata table in earlier versions of hudi. Would recommend to enable metadata from 0.11.0.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan commented on issue #6048: [SUPPORT] S3 throttling while loading a table written with "hoodie.metadata.enable" = true
Posted by GitBox <gi...@apache.org>.
nsivabalan commented on issue #6048:
URL: https://github.com/apache/hudi/issues/6048#issuecomment-1302872527
@noahtaite : going ahead and closing this one for now. Feel free to raise a new issue if you are looking for further assistance.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org