You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "HEPBO3AH (via GitHub)" <gi...@apache.org> on 2023/03/19 20:32:33 UTC
[GitHub] [hudi] HEPBO3AH opened a new issue, #8234: [ISSUE] Clustering always rewrites the file even when there is nothing to cluster it with
HEPBO3AH opened a new issue, #8234:
URL: https://github.com/apache/hudi/issues/8234
**Describe the problem you faced**
When clustering triggers it's supposed to go through the selected partitions. If there are files that can be clustered, it creates new versions and marks the old ones to be cleaned.
However, if there is only a single file as the clustering candidate, it will be selected and rewritten even though the contents stays 100% the same.
**To Reproduce**
Steps to reproduce the behavior:
1. Write into table to create a clustering candidate
2. Run clustering
3. Observe new version of the file being created even when there isn't anything to cluster
**Expected behavior**
If there is nothing to cluster new version of the file shouldn't be created.
**Environment Description**
* Hudi version : 0.11 / 0.12.2 / 0.13
* Spark version : 3.1.2
* Hive version : 3.1.2
* Hadoop version : ?
* Storage (HDFS/S3/GCS..) : S3
* Running on Docker? (yes/no) : No
**Additional context**
Code sample:
```
def main(args: Array[String]): Unit = {
println("Start.")
val numberOfCores = "3"
val spark = SparkSession
.builder()
.master("local[" + numberOfCores + "]")
.appName("hudi-clustering-duplication")
.config("spark.eventLog.enabled", "true")
.config("spark.eventLog.dir", logPath)
.config("spark.serializer", "org.apache.spark.serializer.KryoSerializer")
.getOrCreate()
import spark.implicits._
val df = Seq(RandomStringUtils.randomAlphabetic(5)).toDF("id")
df
.write
.format("hudi")
.option(RECORDKEY_FIELD_NAME.key(), "id")
.option(PRECOMBINE_FIELD.key(), "id")
.option(TABLE_TYPE.key(), "COPY_ON_WRITE")
.option(KEYGENERATOR_CLASS_NAME.key(), "org.apache.hudi.keygen.NonpartitionedKeyGenerator")
.option("hoodie.table.name", "tbl")
.option("hoodie.copyonwrite.record.size.estimate", "20")
.option("hoodie.parquet.small.file.limit", "73400320") // 70MB
.option("hoodie.clustering.inline", "true")
.option("hoodie.clustering.inline.max.commits", "0")
.mode(SaveMode.Append)
.save(basePath + "/tbl")
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan commented on issue #8234: [ISSUE] Clustering always rewrites the file even when there is nothing to cluster it with
Posted by "nsivabalan (via GitHub)" <gi...@apache.org>.
nsivabalan commented on issue #8234:
URL: https://github.com/apache/hudi/issues/8234#issuecomment-1481926716
thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan closed issue #8234: [ISSUE] Clustering always rewrites the file even when there is nothing to cluster it with
Posted by "nsivabalan (via GitHub)" <gi...@apache.org>.
nsivabalan closed issue #8234: [ISSUE] Clustering always rewrites the file even when there is nothing to cluster it with
URL: https://github.com/apache/hudi/issues/8234
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] HEPBO3AH commented on issue #8234: [ISSUE] Clustering always rewrites the file even when there is nothing to cluster it with
Posted by "HEPBO3AH (via GitHub)" <gi...@apache.org>.
HEPBO3AH commented on issue #8234:
URL: https://github.com/apache/hudi/issues/8234#issuecomment-1477003396
Thanks @danny0405 ! Was wondering why is this behind the configuration? I don't see any obvious benefits to wanting to rewrite bunch of files.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] danny0405 commented on issue #8234: [ISSUE] Clustering always rewrites the file even when there is nothing to cluster it with
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on issue #8234:
URL: https://github.com/apache/hudi/issues/8234#issuecomment-1477284340
Yeah, we can enable it by default in future release.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] danny0405 commented on issue #8234: [ISSUE] Clustering always rewrites the file even when there is nothing to cluster it with
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on issue #8234:
URL: https://github.com/apache/hudi/issues/8234#issuecomment-1475530426
Glad that you perceive the details, that is why we introduce an option to control the behavior recently: https://github.com/apache/hudi/pull/7159
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org