You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by GitBox <gi...@apache.org> on 2022/06/06 15:06:10 UTC

[GitHub] [hudi] Rap70r opened a new issue, #5770: [SUPPORT] hoodie.parquet.max.file.size Property is Being Ignored

Rap70r opened a new issue, #5770:
URL: https://github.com/apache/hudi/issues/5770

   Hello,
   
   We are using Apache Hudi to apply COPY_ON_WRITE incremental updates into Parquet files on S3. We have configured the property hoodie.parquet.max.file.size to 18874368 (18 MB).
   On initial sync, the Hudi table is split into multiple parquet files of size set by this property, under the partition folders. However, after several incremental updates, the files get merged into one large file that exceeds 100MB in size.
   
   Here are the configs we are using:
   ```
   parallelism: 200
   operation: upsert
   storageType: COPY_ON_WRITE
   maxVersions: 1
   
   hoodie.datasource.hive_sync.enable: false
   hoodie.finalize.write.parallelism: 200
   hoodie.parquet.max.file.size: 18874368
   ```
   
   **Expected behavior**
   
   Hudi table should maintain parquet files of size defined by hoodie.parquet.max.file.size property.
   
   Are we missing a specific property that needs to be configured in order to maintain those file sizes?
   
   **Environment Description**
   
   * Hudi version : 0.11.0
   
   * Spark version : 3.1.2
   
   * Hive version : None
   
   * Hadoop version : 3.2.1
   
   * Storage  : S3
   
   * Running on Docker? : No
   
   
   Thank you
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] Rap70r commented on issue #5770: [SUPPORT] hoodie.parquet.max.file.size Property is Being Ignored

Posted by GitBox <gi...@apache.org>.
Rap70r commented on issue #5770:
URL: https://github.com/apache/hudi/issues/5770#issuecomment-1164766592

   I'm a bit confused. We are running Spark jobs on AWS EMR to merge changes against Hudi table on S3. We are passing the following parameter: `hoodie.parquet.max.file.size` which according to the documentation, it should maintain the size of files provided in this parameter. During initial load the files are split evenly according to that param. However, after several upserts, the files merge into a large one that greatly exceed the size specified in above parameter. Kindly point me to the configs in the documentation necessary to maintain the file size specified in above parameter for follow up upserts.  


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] adarsh112 commented on issue #5770: [SUPPORT] hoodie.parquet.max.file.size Property is Being Ignored

Posted by "adarsh112 (via GitHub)" <gi...@apache.org>.
adarsh112 commented on issue #5770:
URL: https://github.com/apache/hudi/issues/5770#issuecomment-1578109493

   I think Hudi treats them as small files and auto clusters it to the size set by the property hoodie.parquet.small.file.limit . So try changing hoodie.parquet.small.file.limit to 18 mb


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] Rap70r commented on issue #5770: [SUPPORT] hoodie.parquet.max.file.size Property is Being Ignored

Posted by GitBox <gi...@apache.org>.
Rap70r commented on issue #5770:
URL: https://github.com/apache/hudi/issues/5770#issuecomment-1163266839

   Hi @YuweiXiao, I'm not familiar with the clustering config you mentioned. Can you please provide details?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] Rap70r commented on issue #5770: [SUPPORT] hoodie.parquet.max.file.size Property is Being Ignored

Posted by GitBox <gi...@apache.org>.
Rap70r commented on issue #5770:
URL: https://github.com/apache/hudi/issues/5770#issuecomment-1149146233

   Hi @nsivabalan,
   
   What number should I set hoodie.parquet.block.size to? Should I set it to be same as hoodie.parquet.max.file.size?
   Also, when you say avg record size, what do you mean? These Hudi tables represent SQL Server tables we are syncing.
   
   Thank you


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] nsivabalan commented on issue #5770: [SUPPORT] hoodie.parquet.max.file.size Property is Being Ignored

Posted by GitBox <gi...@apache.org>.
nsivabalan commented on issue #5770:
URL: https://github.com/apache/hudi/issues/5770#issuecomment-1216221375

   @Rap70r : 
   We made some fixes around file sizing in 0.11.1. Can you try it out and let us know if you still see the issue. btw, hudi does not check for file size threshold limit for every record while writing records to files. we do it once every 100 or 1000 records to avoid perf hit. So, the size may not be honored to precision. 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] Rap70r closed issue #5770: [SUPPORT] hoodie.parquet.max.file.size Property is Being Ignored

Posted by GitBox <gi...@apache.org>.
Rap70r closed issue #5770: [SUPPORT] hoodie.parquet.max.file.size Property is Being Ignored
URL: https://github.com/apache/hudi/issues/5770


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] nsivabalan commented on issue #5770: [SUPPORT] hoodie.parquet.max.file.size Property is Being Ignored

Posted by GitBox <gi...@apache.org>.
nsivabalan commented on issue #5770:
URL: https://github.com/apache/hudi/issues/5770#issuecomment-1149117813

   above property should suffice. 
   whats the avg record size in your case? 
   Also, can you also try setting hoodie.parquet.block.size as well. 
   and let us know what do you see. 
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] YuweiXiao commented on issue #5770: [SUPPORT] hoodie.parquet.max.file.size Property is Being Ignored

Posted by GitBox <gi...@apache.org>.
YuweiXiao commented on issue #5770:
URL: https://github.com/apache/hudi/issues/5770#issuecomment-1163824258

   Clustering is a table service that re-organize the data files' layout, maybe it is not relevant in your case.
   
   About the `average record size`, hudi use this (`hoodie.copyonwrite.record.size.estimate`, default 1KB) to estimate the total file size in your initial write (no available commits to compute the average size).  
   
   The code for file size management in `UpsertPartitioner::assignInserts`. Maybe you could also check logs and see if it take effects.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] nsivabalan commented on issue #5770: [SUPPORT] hoodie.parquet.max.file.size Property is Being Ignored

Posted by GitBox <gi...@apache.org>.
nsivabalan commented on issue #5770:
URL: https://github.com/apache/hudi/issues/5770#issuecomment-1229354518

   @Rap70r : any updates please. feel free to close out the issue if you got it resolved.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] Rap70r commented on issue #5770: [SUPPORT] hoodie.parquet.max.file.size Property is Being Ignored

Posted by GitBox <gi...@apache.org>.
Rap70r commented on issue #5770:
URL: https://github.com/apache/hudi/issues/5770#issuecomment-1243826554

   Hi @nsivabalan,
   I will upgrade and reload the files and give it couple of weeks of testing and confirm again. I will close this ticket and re-open if necessary. I was just curious if there is a different parameter that needs to be configured other than hoodie.parquet.max.file.size in order to maintain maximum file size.
   
   Thank you


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] YuweiXiao commented on issue #5770: [SUPPORT] hoodie.parquet.max.file.size Property is Being Ignored

Posted by GitBox <gi...@apache.org>.
YuweiXiao commented on issue #5770:
URL: https://github.com/apache/hudi/issues/5770#issuecomment-1149433070

   Did you enable `clustering`, as you mentioned `files merged into larger file`.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] YuweiXiao commented on issue #5770: [SUPPORT] hoodie.parquet.max.file.size Property is Being Ignored

Posted by GitBox <gi...@apache.org>.
YuweiXiao commented on issue #5770:
URL: https://github.com/apache/hudi/issues/5770#issuecomment-1163823698

   Clustering is a table service that re-organize the data files' layout, maybe it is not relevant in your case.
   
   About the `average record size`, hudi use this (`hoodie.copyonwrite.record.size.estimate`, default 1KB) to estimate the total file size in your initial write (no available commits to compute the average size).  
   
   The code for file size management in `UpsertPartitioner::assignInserts`. Maybe you could also check logs and see if it take effects.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org