You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "Balaji Varadarajan (Jira)" <ji...@apache.org> on 2020/10/01 11:47:00 UTC
[jira] [Created] (HUDI-1310) Corruption Block Handling too slow in
S3
Balaji Varadarajan created HUDI-1310:
----------------------------------------
Summary: Corruption Block Handling too slow in S3
Key: HUDI-1310
URL: https://issues.apache.org/jira/browse/HUDI-1310
Project: Apache Hudi
Issue Type: Sub-task
Components: Writer Core
Reporter: Balaji Varadarajan
The logic to figure out next valid starting block offset is too slow when run in S3.
I have bolded the log message that takes long time to appear.
36589 [Spring Shell] INFO org.apache.hudi.common.table.log.AbstractHoodieLogRecordScanner - Scanning log file HoodieLogFile\{pathStr='s3a://robinhood-encrypted-hudi-data-cove/dev_hudi_tables/balaji_varadarajan/benchmark_1M_10K_partitions/.hoodie/metadata/metadata_partition/.f02585bd-bb02-43f6-8bc8-cec71df87d1e-0_00000000000000.log.1_0-23-206045', fileLen=0}
36590 [Spring Shell] INFO org.apache.hudi.common.table.log.HoodieLogFileReader - Found corrupted block in file HoodieLogFile\{pathStr='s3a://robinhood-encrypted-hudi-data-cove/dev_hudi_tables/balaji_varadarajan/benchmark_1M_10K_partitions/.hoodie/metadata/metadata_partition/.f02585bd-bb02-43f6-8bc8-cec71df87d1e-0_00000000000000.log.1_0-23-206045', fileLen=0} with block size(3723305) running past EOF
36684 [Spring Shell] INFO org.apache.hudi.common.table.log.HoodieLogFileReader - Log HoodieLogFile\{pathStr='s3a://xxxxx/dev_hudi_tables/balaji_varadarajan/benchmark_1M_10K_partitions/.hoodie/metadata/metadata_partition/.f02585bd-bb02-43f6-8bc8-cec71df87d1e-0_00000000000000.log.1_0-23-206045', fileLen=0} has a corrupted block at 14
*44515 [Spring Shell] INFO org.apache.hudi.common.table.log.HoodieLogFileReader - Next available block in* HoodieLogFile\{pathStr='s3a://xxxxx/dev_hudi_tables/balaji_varadarajan/benchmark_1M_10K_partitions/.hoodie/metadata/metadata_partition/.f02585bd-bb02-43f6-8bc8-cec71df87d1e-0_00000000000000.log.1_0-23-206045', fileLen=0} starts at 3723319
44566 [Spring Shell] INFO org.apache.hudi.common.table.log.AbstractHoodieLogRecordScanner - Found a corrupt block in s3a://xxxxx/dev_hudi_tables/balaji_varadarajan/benchmark_1M_10K_partitions/.hoodie/metadata/metadata_partition/.f02585bd-bb02-43f6-8bc8-cec71df87d1e-0_00000000000000.log.1_0-23-206045
--
This message was sent by Atlassian Jira
(v8.3.4#803005)