You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by GitBox <gi...@apache.org> on 2021/10/08 19:22:30 UTC
[GitHub] [hudi] codejoyan opened a new issue #3770: [SUPPORT] Clustering on a metadata enabled table
codejoyan opened a new issue #3770:
URL: https://github.com/apache/hudi/issues/3770
Spark 3.1.2
Hudi 0.9.0
Looks like I have hit a bug. Listing down the steps for context.
1. Bootstrapped a COW table using bulk_insert and these properties
option(HoodieMetadataConfig.METADATA_ENABLE_PROP, "true").
option(HoodieMetadataConfig.METADATA_METRICS_ENABLE_PROP, "true")
2. Run async clustering on all the partitions
3. Overwrite a partition by using bulk insert with metadata enable and metadata metrics enable as true.
The step#3 failed with the below error and looks like the metadata table is corrupted. And now any other commit fails and I am stuck.
```
21/09/29 12:34:06 INFO org.apache.hudi.metadata.HoodieBackedTableMetadata: Opened metadata log files from [gs://udp-hudi-storage3/store_visit_scan_bootstrap_hudi/.hoodie/metadata/files/.3c7ae5cf-0a6b-446b-92bb-c341049fb9fd-0_20210928133730001.log.1_0-131-119, gs://udp-hudi-storage3/store_visit_scan_bootstrap_hudi/.hoodie/metadata/files/.3c7ae5cf-0a6b-446b-92bb-c341049fb9fd-0_20210928133730001.log.2_0-141-128, gs://udp-hudi-storage3/store_visit_scan_bootstrap_hudi/.hoodie/metadata/files/.3c7ae5cf-0a6b-446b-92bb-c341049fb9fd-0_20210928133730001.log.3_0-151-137, gs://udp-hudi-storage3/store_visit_scan_bootstrap_hudi/.hoodie/metadata/files/.3c7ae5cf-0a6b-446b-92bb-c341049fb9fd-0_20210928133730001.log.4_0-163-148, gs://udp-hudi-storage3/store_visit_scan_bootstrap_hudi/.hoodie/metadata/files/.3c7ae5cf-0a6b-446b-92bb-c341049fb9fd-0_20210928133730001.log.5_0-173-157, gs://udp-hudi-storage3/store_visit_scan_bootstrap_hudi/.hoodie/metadata/files/.3c7ae5cf-0a6b-446b-92bb-c341049fb9fd-0_20210
928133730001.log.6_0-183-166, gs://udp-hudi-storage3/store_visit_scan_bootstrap_hudi/.hoodie/metadata/files/.3c7ae5cf-0a6b-446b-92bb-c341049fb9fd-0_20210928133730001.log.7_0-193-175, gs://udp-hudi-storage3/store_visit_scan_bootstrap_hudi/.hoodie/metadata/files/.3c7ae5cf-0a6b-446b-92bb-c341049fb9fd-0_20210928133730001.log.8_0-203-184, gs://udp-hudi-storage3/store_visit_scan_bootstrap_hudi/.hoodie/metadata/files/.3c7ae5cf-0a6b-446b-92bb-c341049fb9fd-0_20210928133730001.log.9_0-213-193, gs://udp-hudi-storage3/store_visit_scan_bootstrap_hudi/.hoodie/metadata/files/.3c7ae5cf-0a6b-446b-92bb-c341049fb9fd-0_20210928133730001.log.10_0-223-202, gs://udp-hudi-storage3/store_visit_scan_bootstrap_hudi/.hoodie/metadata/files/.3c7ae5cf-0a6b-446b-92bb-c341049fb9fd-0_20210928133730001.log.11_0-233-211, gs://udp-hudi-storage3/store_visit_scan_bootstrap_hudi/.hoodie/metadata/files/.3c7ae5cf-0a6b-446b-92bb-c341049fb9fd-0_20210928133730001.log.12_0-243-220, gs://udp-hudi-storage3/store_visit_scan_bootst
rap_hudi/.hoodie/metadata/files/.3c7ae5cf-0a6b-446b-92bb-c341049fb9fd-0_20210928133730001.log.13_0-253-229, gs://udp-hudi-storage3/store_visit_scan_bootstrap_hudi/.hoodie/metadata/files/.3c7ae5cf-0a6b-446b-92bb-c341049fb9fd-0_20210928133730001.log.14_0-263-238, gs://udp-hudi-storage3/store_visit_scan_bootstrap_hudi/.hoodie/metadata/files/.3c7ae5cf-0a6b-446b-92bb-c341049fb9fd-0_20210928133730001.log.15_0-273-247, gs://udp-hudi-storage3/store_visit_scan_bootstrap_hudi/.hoodie/metadata/files/.3c7ae5cf-0a6b-446b-92bb-c341049fb9fd-0_20210928133730001.log.16_0-283-256, gs://udp-hudi-storage3/store_visit_scan_bootstrap_hudi/.hoodie/metadata/files/.3c7ae5cf-0a6b-446b-92bb-c341049fb9fd-0_20210928133730001.log.17_0-293-265, gs://udp-hudi-storage3/store_visit_scan_bootstrap_hudi/.hoodie/metadata/files/.3c7ae5cf-0a6b-446b-92bb-c341049fb9fd-0_20210928133730001.log.18_0-303-274, gs://udp-hudi-storage3/store_visit_scan_bootstrap_hudi/.hoodie/metadata/files/.3c7ae5cf-0a6b-446b-92bb-c341049fb9fd-0_2
0210928133730001.log.19_0-313-283, gs://udp-hudi-storage3/store_visit_scan_bootstrap_hudi/.hoodie/metadata/files/.3c7ae5cf-0a6b-446b-92bb-c341049fb9fd-0_20210928133730001.log.20_0-323-292, gs://udp-hudi-storage3/store_visit_scan_bootstrap_hudi/.hoodie/metadata/files/.3c7ae5cf-0a6b-446b-92bb-c341049fb9fd-0_20210928133730001.log.21_0-333-301, gs://udp-hudi-storage3/store_visit_scan_bootstrap_hudi/.hoodie/metadata/files/.3c7ae5cf-0a6b-446b-92bb-c341049fb9fd-0_20210928133730001.log.22_0-343-310, gs://udp-hudi-storage3/store_visit_scan_bootstrap_hudi/.hoodie/metadata/files/.3c7ae5cf-0a6b-446b-92bb-c341049fb9fd-0_20210928133730001.log.23_0-353-319] at instant (dataset instant=20210929122502, metadata instant=20210929093134) in 2627 ms
21/09/29 12:34:06 INFO org.apache.hudi.metadata.HoodieMetadataMetrics: Updating metadata metrics (scan.totalDuration=2762ms, scan.count=1)
21/09/29 12:34:06 INFO org.apache.hudi.metadata.HoodieMetadataMetrics: Updating metadata metrics (basefile_read.totalDuration=8ms, basefile_read.count=1)
21/09/29 12:34:06 INFO org.apache.hudi.metadata.HoodieBackedTableMetadata: Metadata read for key op_cmpny_cd=WMT-US/visit_date=2021-01-04 took [baseFileRead, logMerge] [8, 0] ms
21/09/29 12:34:06 INFO org.apache.hudi.metadata.HoodieMetadataMetrics: Updating metadata metrics (lookup_files.totalDuration=2771ms, lookup_files.count=1)
21/09/29 12:34:06 ERROR org.apache.spark.executor.Executor: Exception in task 64.1 in stage 376.0 (TID 3842)
org.apache.hudi.exception.HoodieMetadataException: Failed to retrieve files in partition gs://udp-hudi-storage3/store_visit_scan_bootstrap_hudi/op_cmpny_cd=WMT-US/visit_date=2021-01-04 from metadata
at org.apache.hudi.metadata.BaseTableMetadata.getAllFilesInPartition(BaseTableMetadata.java:128)
at org.apache.hudi.metadata.HoodieMetadataFileSystemView.listPartition(HoodieMetadataFileSystemView.java:65)
at org.apache.hudi.common.table.view.AbstractTableFileSystemView.lambda$ensurePartitionLoadedCorrectly$9(AbstractTableFileSystemView.java:280)
at java.util.concurrent.ConcurrentHashMap.computeIfAbsent(ConcurrentHashMap.java:1660)
at org.apache.hudi.common.table.view.AbstractTableFileSystemView.ensurePartitionLoadedCorrectly(AbstractTableFileSystemView.java:269)
at org.apache.hudi.common.table.view.AbstractTableFileSystemView.getAllFileGroupsIncludingReplaced(AbstractTableFileSystemView.java:682)
at org.apache.hudi.common.table.view.AbstractTableFileSystemView.getReplacedFileGroupsBefore(AbstractTableFileSystemView.java:696)
at org.apache.hudi.common.table.view.PriorityBasedFileSystemView.execute(PriorityBasedFileSystemView.java:101)
at org.apache.hudi.common.table.view.PriorityBasedFileSystemView.getReplacedFileGroupsBefore(PriorityBasedFileSystemView.java:204)
at org.apache.hudi.table.action.clean.CleanPlanner.getReplacedFilesEligibleToClean(CleanPlanner.java:337)
at org.apache.hudi.table.action.clean.CleanPlanner.getFilesToCleanKeepingLatestCommits(CleanPlanner.java:279)
at org.apache.hudi.table.action.clean.CleanPlanner.getDeletePaths(CleanPlanner.java:388)
at org.apache.hudi.table.action.clean.BaseCleanPlanActionExecutor.lambda$requestClean$1fb25f79$1(BaseCleanPlanActionExecutor.java:86)
at org.apache.spark.api.java.JavaPairRDD$.$anonfun$toScalaFunction$1(JavaPairRDD.scala:1070)
at scala.collection.Iterator$$anon$10.next(Iterator.scala:461)
at scala.collection.Iterator.foreach(Iterator.scala:943)
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan commented on issue #3770: [SUPPORT] Clustering on a metadata enabled table
Posted by GitBox <gi...@apache.org>.
nsivabalan commented on issue #3770:
URL: https://github.com/apache/hudi/issues/3770#issuecomment-939352973
can you disable metadata and get your unblocked? atleast wanted to ensure you are not completely blocked.
getting into the actual problem.
in (3), why do you say overwrite a partition by using bulk_insert. you can't overwrite a partition using bulk_insert. bulk_insert will just bulk import insert records. Can you please clarify.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan edited a comment on issue #3770: [SUPPORT] Clustering on a metadata enabled table
Posted by GitBox <gi...@apache.org>.
nsivabalan edited a comment on issue #3770:
URL: https://github.com/apache/hudi/issues/3770#issuecomment-939352973
can you disable metadata and get your unblocked? atleast wanted to ensure you are not completely blocked.
getting into the actual problem.
1. in (3), why do you say overwrite a partition by using bulk_insert. you can't overwrite a partition using bulk_insert. bulk_insert will just bulk import insert records. Can you please clarify.
2. by (2), did you mean, async clustering got scheduled and completed or just got scheduled. or while it was running, a new bulk_insert was attempted?
3. If (3) was attempted after clustering was complete, were you able to query the table and things were fine before (3).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan closed issue #3770: [SUPPORT] Clustering on a metadata enabled table
Posted by GitBox <gi...@apache.org>.
nsivabalan closed issue #3770:
URL: https://github.com/apache/hudi/issues/3770
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan commented on issue #3770: [SUPPORT] Clustering on a metadata enabled table
Posted by GitBox <gi...@apache.org>.
nsivabalan commented on issue #3770:
URL: https://github.com/apache/hudi/issues/3770#issuecomment-997296994
Clustering along with metadata wasn't tested well in previous versions and known to had bugs. So, would recommend to try it out with 0.10.0. Will close this issue out. If you have any more questions/issues, feel free to open a new github issue.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan edited a comment on issue #3770: [SUPPORT] Clustering on a metadata enabled table
Posted by GitBox <gi...@apache.org>.
nsivabalan edited a comment on issue #3770:
URL: https://github.com/apache/hudi/issues/3770#issuecomment-939352973
can you disable metadata and get your unblocked? atleast wanted to ensure you are not completely blocked.
getting into the actual problem.
1. in (3), why do you say overwrite a partition by using bulk_insert. you can't overwrite a partition using bulk_insert. bulk_insert will just bulk import insert records. Can you please clarify.
2. by (2), did you mean, async clustering got scheduled and completed or just got scheduled. or while it was running, a new bulk_insert was attempted?
3. If (3) was attempted after clustering was complete, were you able to query the table and things were fine before (3).
4. On similar lines, after (1), did you try querying the table and things were all good?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan commented on issue #3770: [SUPPORT] Clustering on a metadata enabled table
Posted by GitBox <gi...@apache.org>.
nsivabalan commented on issue #3770:
URL: https://github.com/apache/hudi/issues/3770#issuecomment-991850761
@codejoyan : Can you try this with 0.10.0 and let us know how it goes.
and respond to previous comment if applicable.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org