You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "Udit Mehrotra (Jira)" <ji...@apache.org> on 2021/08/03 23:17:00 UTC
[jira] [Commented] (HUDI-1170) File Listing during log file
rollback is affecting ingestion latency in S3
[ https://issues.apache.org/jira/browse/HUDI-1170?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17392593#comment-17392593 ]
Udit Mehrotra commented on HUDI-1170:
-------------------------------------
[~vbalaji] since there has been no update on this, rolling this over to Hudi 0.10.0. Let me know if you feel its a release blocker for 0.9.0.
> File Listing during log file rollback is affecting ingestion latency in S3
> --------------------------------------------------------------------------
>
> Key: HUDI-1170
> URL: https://issues.apache.org/jira/browse/HUDI-1170
> Project: Apache Hudi
> Issue Type: Improvement
> Components: Writer Core
> Affects Versions: 0.9.0
> Reporter: Balaji Varadarajan
> Priority: Blocker
> Fix For: 0.9.0
>
>
> (Source : [https://github.com/apache/hudi/issues/1852])
>
> : sun.net.www.protocol.https.HttpsURLConnectionImpl.getResponseCode(HttpsURLConnectionImpl.java:352)
> shaded.databricks.v20180920_b33d810.org.apache.hadoop.fs.azurebfs.services.AbfsHttpOperation.processResponse(AbfsHttpOperation.java:259)
> shaded.databricks.v20180920_b33d810.org.apache.hadoop.fs.azurebfs.services.AbfsRestOperation.executeHttpOperation(AbfsRestOperation.java:167)
> shaded.databricks.v20180920_b33d810.org.apache.hadoop.fs.azurebfs.services.AbfsRestOperation.execute(AbfsRestOperation.java:124)
> shaded.databricks.v20180920_b33d810.org.apache.hadoop.fs.azurebfs.services.AbfsClient.listPath(AbfsClient.java:180)
> shaded.databricks.v20180920_b33d810.org.apache.hadoop.fs.azurebfs.AzureBlobFileSystemStore.listFiles(AzureBlobFileSystemStore.java:549)
> shaded.databricks.v20180920_b33d810.org.apache.hadoop.fs.azurebfs.AzureBlobFileSystemStore.listStatus(AzureBlobFileSystemStore.java:628)
> shaded.databricks.v20180920_b33d810.org.apache.hadoop.fs.azurebfs.AzureBlobFileSystemStore.listStatus(AzureBlobFileSystemStore.java:532)
> shaded.databricks.v20180920_b33d810.org.apache.hadoop.fs.azurebfs.AzureBlobFileSystem.listStatus(AzureBlobFileSystem.java:344)
> org.apache.hadoop.fs.FileSystem.listStatus(FileSystem.java:1517)
> org.apache.hadoop.fs.FileSystem.listStatus(FileSystem.java:1557)
> org.apache.hudi.common.fs.HoodieWrapperFileSystem.listStatus(HoodieWrapperFileSystem.java:487)
> org.apache.hudi.common.fs.FSUtils.getAllLogFiles(FSUtils.java:409)
> org.apache.hudi.common.fs.FSUtils.getLatestLogVersion(FSUtils.java:420)
> org.apache.hudi.common.fs.FSUtils.computeNextLogVersion(FSUtils.java:434)
> org.apache.hudi.common.model.HoodieLogFile.rollOver(HoodieLogFile.java:115)
> org.apache.hudi.common.table.log.HoodieLogFormatWriter.(HoodieLogFormatWriter.java:101)
> org.apache.hudi.common.table.log.HoodieLogFormat$WriterBuilder.build(HoodieLogFormat.java:249)
> org.apache.hudi.io.HoodieAppendHandle.createLogWriter(HoodieAppendHandle.java:291)
> org.apache.hudi.io.HoodieAppendHandle.init(HoodieAppendHandle.java:141)
> org.apache.hudi.io.HoodieAppendHandle.doAppend(HoodieAppendHandle.java:197)
> org.apache.hudi.table.action.deltacommit.DeltaCommitActionExecutor.handleUpdate(DeltaCommitActionExecutor.java:77)
> org.apache.hudi.table.action.commit.BaseCommitActionExecutor.handleUpsertPartition(BaseCommitActionExecutor.java:246)
> org.apache.hudi.table.action.commit.BaseCommitActionExecutor.lambda$execute$caffe4c4$1(BaseCommitActionExecutor.java:102)
> org.apache.hudi.table.action.commit.BaseCommitActionExecutor$$Lambda$192/1449069739.call(Unknown Source)
> org.apache.spark.api.java.JavaRDDLike$$anonfun$mapPartitionsWithIndex$1.apply(JavaRDDLike.scala:105)
--
This message was sent by Atlassian Jira
(v8.3.4#803005)