You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "Klring (via GitHub)" <gi...@apache.org> on 2023/05/17 09:59:35 UTC
[GitHub] [hudi] Klring opened a new issue, #8737: [SUPPORT]multi-writer and clustering
Klring opened a new issue, #8737:
URL: https://github.com/apache/hudi/issues/8737
**Describe the problem you faced**
We use Flink to write log data，the config is multi-writer + cow + cluster ，an error occurred in the cluster after running for a while.
The error indicates that a specific Parquet file does not exist.We have tried removing the cluster config in another write operation, but the problem still persists.How can I resolve it? Does Hudi support multi-writer and cluster operations simultaneously?
**Config**
```'table.type' = 'COPY_ON_WRITE',
'hoodie.datasource.write.partitionpath.field'= 'receive_date',
'hoodie.datasource.write.hive_style_partitioning' = 'true' ,
'write.operation'='insert',
'write.bucket_assign.tasks'='2',
'write.tasks'='4',
'write.task.max.size'='2048',
'write.merge.max_memory'='1024',
'write.batch.size'='256',
'metadata.enabled'='true',
'hoodie.enable.data.skipping'='true',
'clustering.schedule.enabled'='true' ,
'clustering.async.enabled'='true' ,
'hoodie.clustering.plan.strategy.small.file.limit'='629145600',
'hoodie.clustering.plan.strategy.target.file.max.bytes'='1073741824',
'hoodie.write.concurrency.mode'='optimistic_concurrency_control',
'hoodie.cleaner.policy.failed.writes'='LAZY',
'hoodie.write.lock.wait_time_ms'='300000',
'hoodie.write.lock.num_retries'='60',
'hoodie.write.lock.provider'='org.apache.hudi.hive.transaction.lock.HiveMetastoreBasedLockProvider',
'hoodie.write.lock.hivemetastore.uris'='thrift://xxxx:9083' ,
'hoodie.write.lock.hivemetastore.database'='log' ,
'hoodie.write.lock.hivemetastore.table'='log_test',
'clean.async.enabled'='true',
'clean.retain_file_versions'='1' ,
'cleaner.policy'='KEEP_LATEST_FILE_VERSIONS' ,
'clean.retain_commits' = '1' ,
'hoodie.cleaner.delete.bootstrap.base.file'='true',
'hoodie.archive.automatic'='true',
'hoodie.archive.merge.enable'='true',
'hoodie.archive.async'='true',
'hive_sync.enable'='true',
'hive_sync.mode'='hms',
'hive_sync.metastore.uris'='thrift://xxxx:9083',
'hive_sync.db'='log',
'hive_sync.table'='log_test',
'hive_sync.partition_fields'='receive_date',
'hive_sync.support_timestamp'='true'
**Environment Description**
* Hudi version :0.13.0
* Flink version :1.14.4
* Hive version :3.1.2
* Hadoop version :3.2.3
* Storage (HDFS/S3/GCS..) :OBS
* Running on Docker? (yes/no) :no
**Stacktrace**
```Add the stacktrace of the error.```
2023-05-17 11:11:05,232 ERROR org.apache.hudi.sink.clustering.ClusteringOperator [] - Executor executes action [Execute clustering for instant 20230517110938069 from task 1] error
org.apache.hudi.exception.HoodieClusteringException: Error reading input data for obs://platform-daka/warehouse/hive/log.db/log_test/receive_date=2023-05-16/bc26fb97-68ba-4634-8d36-a47e3c244f89-0_3-4-0_20230517110750167.parquet and []
at org.apache.hudi.sink.clustering.ClusteringOperator.lambda$null$4(ClusteringOperator.java:337) ~[flink_sls_applog-1.5.2-jar-with-dependencies.jar:?]
at java.lang.Iterable.spliterator(Iterable.java:101) ~[?:1.8.0_333]
at org.apache.hudi.sink.clustering.ClusteringOperator.lambda$readRecordsForGroupBaseFiles$5(ClusteringOperator.java:341) ~[flink_sls_applog-1.5.2-jar-with-dependencies.jar:?]
at java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193) ~[?:1.8.0_333]
at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1384) ~[?:1.8.0_333]
at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482) ~[?:1.8.0_333]
at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:472) ~[?:1.8.0_333]
at java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:708) ~[?:1.8.0_333]
at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234) ~[?:1.8.0_333]
at java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:499) ~[?:1.8.0_333]
at org.apache.hudi.sink.clustering.ClusteringOperator.readRecordsForGroupBaseFiles(ClusteringOperator.java:342) ~[flink_sls_applog-1.5.2-jar-with-dependencies.jar:?]
at org.apache.hudi.sink.clustering.ClusteringOperator.doClustering(ClusteringOperator.java:242) ~[flink_sls_applog-1.5.2-jar-with-dependencies.jar:?]
at org.apache.hudi.sink.clustering.ClusteringOperator.lambda$processElement$0(ClusteringOperator.java:194) ~[flink_sls_applog-1.5.2-jar-with-dependencies.jar:?]
at org.apache.hudi.sink.utils.NonThrownExecutor.lambda$wrapAction$0(NonThrownExecutor.java:130) ~[flink_sls_applog-1.5.2-jar-with-dependencies.jar:?]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_333]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_333]
at java.lang.Thread.run(Thread.java:750) [?:1.8.0_333]
Caused by: java.io.FileNotFoundException: getFileStatus on obs://platform-daka/warehouse/hive/log.db/log_test/receive_date=2023-05-16/bc26fb97-68ba-4634-8d36-a47e3c244f89-0_3-4-0_20230517110750167.parquet: status [404] - request id [0000018827B09A07930FB5D6E65CDE27] - error code [null] - error message [Request Error.] - trace :com.obs.services.exception.ObsException: Error message:Request Error.OBS servcie Error Message. -- ResponseCode: 404, ResponseStatus: Not Found, RequestId: 0000018827B09A07930FB5D6E65CDE27, HostId: 32AAAQAAEAABAAAQAAEAABAAAQAAEAABCSe68Sh+FOp7ECYlweelx1ks1zjfo/Qs
at org.apache.hadoop.fs.obs.OBSCommonUtils.translateException(OBSCommonUtils.java:409) ~[hadoop-huaweicloud-3.2.3-hw-46.jar:?]
at org.apache.hadoop.fs.obs.OBSCommonUtils.translateException(OBSCommonUtils.java:691) ~[hadoop-huaweicloud-3.2.3-hw-46.jar:?]
at org.apache.hadoop.fs.obs.OBSPosixBucketUtils.innerFsGetObjectStatus(OBSPosixBucketUtils.java:563) ~[hadoop-huaweicloud-3.2.3-hw-46.jar:?]
at org.apache.hadoop.fs.obs.OBSFileSystem.innerGetFileStatus(OBSFileSystem.java:1663) ~[hadoop-huaweicloud-3.2.3-hw-46.jar:?]
at org.apache.hadoop.fs.obs.OBSCommonUtils.innerGetFileStatusWithRetry(OBSCommonUtils.java:1662) ~[hadoop-huaweicloud-3.2.3-hw-46.jar:?]
at org.apache.hadoop.fs.obs.OBSFileSystem.getFileStatus(OBSFileSystem.java:1631) ~[hadoop-huaweicloud-3.2.3-hw-46.jar:?]
at org.apache.parquet.hadoop.ParquetReader$Builder.build(ParquetReader.java:337) ~[flink_sls_applog-1.5.2-jar-with-dependencies.jar:?]
at org.apache.hudi.io.storage.HoodieAvroParquetReader.getIndexedRecordIteratorInternal(HoodieAvroParquetReader.java:168) ~[flink_sls_applog-1.5.2-jar-with-dependencies.jar:?]
at org.apache.hudi.io.storage.HoodieAvroParquetReader.getIndexedRecordIterator(HoodieAvroParquetReader.java:94) ~[flink_sls_applog-1.5.2-jar-with-dependencies.jar:?]
at org.apache.hudi.io.storage.HoodieAvroParquetReader.getRecordIterator(HoodieAvroParquetReader.java:73) ~[flink_sls_applog-1.5.2-jar-with-dependencies.jar:?]
at org.apache.hudi.sink.clustering.ClusteringOperator.lambda$null$4(ClusteringOperator.java:334) ~[flink_sls_applog-1.5.2-jar-with-dependencies.jar:?]
... 16 more
Caused by: com.obs.services.exception.ObsException: Error message:Request Error.OBS servcie Error Message.
at com.obs.services.internal.utils.ServiceUtils.changeFromServiceException(ServiceUtils.java:620) ~[hadoop-huaweicloud-3.2.3-hw-46.jar:?]
at com.obs.services.AbstractClient.doActionWithResult(AbstractClient.java:399) ~[hadoop-huaweicloud-3.2.3-hw-46.jar:?]
at com.obs.services.AbstractObjectClient.getObjectMetadata(AbstractObjectClient.java:583) ~[hadoop-huaweicloud-3.2.3-hw-46.jar:?]
at com.obs.services.AbstractPFSClient.getAttribute(AbstractPFSClient.java:154) ~[hadoop-huaweicloud-3.2.3-hw-46.jar:?]
at org.apache.hadoop.fs.obs.OBSPosixBucketUtils.innerFsGetObjectStatus(OBSPosixBucketUtils.java:551) ~[hadoop-huaweicloud-3.2.3-hw-46.jar:?]
at org.apache.hadoop.fs.obs.OBSFileSystem.innerGetFileStatus(OBSFileSystem.java:1663) ~[hadoop-huaweicloud-3.2.3-hw-46.jar:?]
at org.apache.hadoop.fs.obs.OBSCommonUtils.innerGetFileStatusWithRetry(OBSCommonUtils.java:1662) ~[hadoop-huaweicloud-3.2.3-hw-46.jar:?]
at org.apache.hadoop.fs.obs.OBSFileSystem.getFileStatus(OBSFileSystem.java:1631) ~[hadoop-huaweicloud-3.2.3-hw-46.jar:?]
at org.apache.parquet.hadoop.ParquetReader$Builder.build(ParquetReader.java:337) ~[flink_sls_applog-1.5.2-jar-with-dependencies.jar:?]
at org.apache.hudi.io.storage.HoodieAvroParquetReader.getIndexedRecordIteratorInternal(HoodieAvroParquetReader.java:168) ~[flink_sls_applog-1.5.2-jar-with-dependencies.jar:?]
at org.apache.hudi.io.storage.HoodieAvroParquetReader.getIndexedRecordIterator(HoodieAvroParquetReader.java:94) ~[flink_sls_applog-1.5.2-jar-with-dependencies.jar:?]
at org.apache.hudi.io.storage.HoodieAvroParquetReader.getRecordIterator(HoodieAvroParquetReader.java:73) ~[flink_sls_applog-1.5.2-jar-with-dependencies.jar:?]
at org.apache.hudi.sink.clustering.ClusteringOperator.lambda$null$4(ClusteringOperator.java:334) ~[flink_sls_applog-1.5.2-jar-with-dependencies.jar:?]
... 16 more
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] Klring commented on issue #8737: [SUPPORT]multi-writer and clustering
Posted by "Klring (via GitHub)" <gi...@apache.org>.
Klring commented on issue #8737:
URL: https://github.com/apache/hudi/issues/8737#issuecomment-1556891841
> Do you set the config `write.client.id`,and try again
I checked the https://hudi.apache.org/docs/configurations page, but did not find the configuration "write.client.id".Is this configuration valid?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] danny0405 commented on issue #8737: [SUPPORT]multi-writer and clustering
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on issue #8737:
URL: https://github.com/apache/hudi/issues/8737#issuecomment-1558764414
Flink does not support multi-writer yet for you version that been used, @Klring , we plan to support this for 0.14.0, the config option `write.client.id` is introduced as a builtin option for Flink multiple writers.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] ad1happy2go commented on issue #8737: [SUPPORT]multi-writer and clustering
Posted by "ad1happy2go (via GitHub)" <gi...@apache.org>.
ad1happy2go commented on issue #8737:
URL: https://github.com/apache/hudi/issues/8737#issuecomment-1566073242
@Klring Feel free to close this issue in case you don't need any other support.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] Klring closed issue #8737: [SUPPORT]multi-writer and clustering
Posted by "Klring (via GitHub)" <gi...@apache.org>.
Klring closed issue #8737: [SUPPORT]multi-writer and clustering
URL: https://github.com/apache/hudi/issues/8737
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] c-f-cooper commented on issue #8737: [SUPPORT]multi-writer and clustering
Posted by "c-f-cooper (via GitHub)" <gi...@apache.org>.
c-f-cooper commented on issue #8737:
URL: https://github.com/apache/hudi/issues/8737#issuecomment-1556703427
Do you set the config `write.client.id`,and try again
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org