You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "c-f-cooper (via GitHub)" <gi...@apache.org> on 2023/05/06 07:19:48 UTC

[GitHub] [hudi] c-f-cooper opened a new issue, #8651: [SUPPORT]How to resolve small file?

c-f-cooper opened a new issue, #8651:
URL: https://github.com/apache/hudi/issues/8651

   **_Tips before filing an issue_**
   
   - Have you gone through our [FAQs](https://hudi.apache.org/learn/faq/)?
   
   - Join the mailing list to engage in conversations and get faster support at dev-subscribe@hudi.apache.org.
   
   - If you have triaged this as a bug, then file an [issue](https://issues.apache.org/jira/projects/HUDI/issues) directly.
   
   **Describe the problem you faced**
   
   we ingest log data into hudi,we use 'cow + insert' which means append mode.we found there are some small file on hdfs,so we use clustering to resolve the problem,but no effect!
   
   **To Reproduce**
   
   Steps to reproduce the behavior:
   
   1.
   2.
   3.
   4.
   
   **Expected behavior**
   
   A clear and concise description of what you expected to happen.
   
   **Environment Description**
   
   * Hudi version :
   0.13.0
   * Flink version :
   1.15..0
   * Hive version :
   3.1.2
   * Hadoop version :
   3.3.2
   * Storage (HDFS/S3/GCS..) :
   hdfs
   * Running on Docker? (yes/no) :
   k8s
   
   **Additional context**
   
   Add any other context about the problem here.
   
   **Stacktrace**
   
   ```Add the stacktrace of the error.```
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [hudi] ad1happy2go commented on issue #8651: [SUPPORT]How to resolve small file?

Posted by "ad1happy2go (via GitHub)" <gi...@apache.org>.
ad1happy2go commented on issue #8651:
URL: https://github.com/apache/hudi/issues/8651#issuecomment-1537127541

   @c-f-cooper Are you using Insert or Bulk Insert. Can you please share the clustering command and table configs you are using.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [SUPPORT]How to resolve small file? [hudi]

Posted by "vkhoroshko (via GitHub)" <gi...@apache.org>.
vkhoroshko commented on issue #8651:
URL: https://github.com/apache/hudi/issues/8651#issuecomment-1904324751

   Hello Guys,
   
   I believe I face the similar issue. 
   
   I create the CoW table with append only mode and expect auto file sizing feature to work (as depicted in the documentation):
   
   ```
   
   CREATE TABLE hudi_table(
       ts BIGINT,
       uuid VARCHAR(40) PRIMARY KEY NOT ENFORCED,
       rider VARCHAR(20),
       driver VARCHAR(20),
       fare DOUBLE,
       city VARCHAR(20)
   )
   WITH (
     'connector' = 'hudi',
     'path' = 'file:///opt/flink/hudi',
     'table.type' = 'COPY_ON_WRITE',
     'write.operation'='insert',
     'parquet.small.file.limit'='500000',
     'parquet.max.file.size'='1000000'
   );
   ```
   
   But for every insert (and commit) - a separate parquet file is created and they are never merged.
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [hudi] c-f-cooper commented on issue #8651: [SUPPORT]How to resolve small file?

Posted by "c-f-cooper (via GitHub)" <gi...@apache.org>.
c-f-cooper commented on issue #8651:
URL: https://github.com/apache/hudi/issues/8651#issuecomment-1537144331

   > What kind of clustering are you using, online or offline? For no effect do you mean no bigger Parquets are generated and replacing the existing files?
   
   we use online async clustering,there are alse some small file,and no bigger parquet file,I found the replacecommit file in the .hoodie directory.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [hudi] c-f-cooper commented on issue #8651: [SUPPORT]How to resolve small file?

Posted by "c-f-cooper (via GitHub)" <gi...@apache.org>.
c-f-cooper commented on issue #8651:
URL: https://github.com/apache/hudi/issues/8651#issuecomment-1537307015

   > > Do you do not enable the async clustering right? We have inline clustering, async clustering, and offline clustering, which one are you using?
   > 
   > we use async clustering,we use flink datastream api,we use two confin items,set`clustering.schedule.enabled`and `clustering.async.enabled` true.
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [hudi] c-f-cooper commented on issue #8651: [SUPPORT]How to resolve small file?

Posted by "c-f-cooper (via GitHub)" <gi...@apache.org>.
c-f-cooper commented on issue #8651:
URL: https://github.com/apache/hudi/issues/8651#issuecomment-1537144673

   > @c-f-cooper Are you using Insert or Bulk Insert. Can you please share the clustering command and table configs you are using.
   
   we use cow+insert mode,beside the clustering config,onther config use default config.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [hudi] ad1happy2go commented on issue #8651: [SUPPORT]How to resolve small file?

Posted by "ad1happy2go (via GitHub)" <gi...@apache.org>.
ad1happy2go commented on issue #8651:
URL: https://github.com/apache/hudi/issues/8651#issuecomment-1579181473

   @c-f-cooper Were you able to resolve this issue, was it same one as @danny0405 mentioned.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [hudi] danny0405 commented on issue #8651: [SUPPORT]How to resolve small file?

Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on issue #8651:
URL: https://github.com/apache/hudi/issues/8651#issuecomment-1555443384

   Good news, maybe it is because there are too many pending clustering tasks blocked in the queue.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [hudi] danny0405 commented on issue #8651: [SUPPORT]How to resolve small file?

Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on issue #8651:
URL: https://github.com/apache/hudi/issues/8651#issuecomment-1537295105

   Do you do not enable the async clustering right?
   We have inline clustering, async clustering, and offline clustering, which one are you using?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [hudi] c-f-cooper commented on issue #8651: [SUPPORT]How to resolve small file?

Posted by "c-f-cooper (via GitHub)" <gi...@apache.org>.
c-f-cooper commented on issue #8651:
URL: https://github.com/apache/hudi/issues/8651#issuecomment-1537306637

   > Do you do not enable the async clustering right? We have inline clustering, async clustering, and offline clustering, which one are you using?
   
   we use async clustering,we use flink datastream api,we use two confin items,set`clustering.schedule.enabled`and `clustering.async.enabled` is true.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [hudi] danny0405 commented on issue #8651: [SUPPORT]How to resolve small file?

Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on issue #8651:
URL: https://github.com/apache/hudi/issues/8651#issuecomment-1623645641

   > I guess it async clustering could solve this later on, but i saw this line in the [documentation](https://hudi.apache.org/docs/clustering#execute-clustering):
   
   It's a mistake, you can schedule the clustering plan directly in the separate clustering job.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [hudi] danny0405 commented on issue #8651: [SUPPORT]How to resolve small file?

Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on issue #8651:
URL: https://github.com/apache/hudi/issues/8651#issuecomment-1537115673

   What kind of clustering are you using, online or offline? For no effect do you mean no bigger Parquets are generated and replacing the existing files?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [hudi] c-f-cooper commented on issue #8651: [SUPPORT]How to resolve small file?

Posted by "c-f-cooper (via GitHub)" <gi...@apache.org>.
c-f-cooper commented on issue #8651:
URL: https://github.com/apache/hudi/issues/8651#issuecomment-1586527634

   > The file sizing should be still valid.
   
   I try it


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [hudi] c-f-cooper commented on issue #8651: [SUPPORT]How to resolve small file?

Posted by "c-f-cooper (via GitHub)" <gi...@apache.org>.
c-f-cooper commented on issue #8651:
URL: https://github.com/apache/hudi/issues/8651#issuecomment-1554718543

   > Can you add more parallelism to the clustering task?
   
   when i add parallelism to 16,the excutor of task excuted.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [hudi] danny0405 commented on issue #8651: [SUPPORT]How to resolve small file?

Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on issue #8651:
URL: https://github.com/apache/hudi/issues/8651#issuecomment-1586494277

   The file sizing should be still valid.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [hudi] c-f-cooper commented on issue #8651: [SUPPORT]How to resolve small file?

Posted by "c-f-cooper (via GitHub)" <gi...@apache.org>.
c-f-cooper commented on issue #8651:
URL: https://github.com/apache/hudi/issues/8651#issuecomment-1586428325

   > Hey there try adding this config i have tried and works for me
   > 
   > ```
   >     # Define a dictionary with the Fiel Size
   >     hudi_file_size = {
   >         "hoodie.parquet.max.file.size": 512 * 1024 * 1024,  # 512MB
   >         "hoodie.parquet.small.file.limit": 104857600,  # 100MB
   >     }
   > ```
   > 
   > ![image](https://user-images.githubusercontent.com/39345855/244869775-01472bb9-198a-44ab-964b-515bddd68eba.png)
   > 
   > Hope this helps
   
   The configs belongs to file sizing,is it effective in clustering?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [hudi] soumilshah1995 commented on issue #8651: [SUPPORT]How to resolve small file?

Posted by "soumilshah1995 (via GitHub)" <gi...@apache.org>.
soumilshah1995 commented on issue #8651:
URL: https://github.com/apache/hudi/issues/8651#issuecomment-1585641043

   Hey there try adding this config i have tried and works for me 
   
   ```
       # Define a dictionary with the Fiel Size
       hudi_file_size = {
           "hoodie.parquet.max.file.size": 512 * 1024 * 1024,  # 512MB
           "hoodie.parquet.small.file.limit": 104857600,  # 100MB
       }
   
   ```
   
   ![image](https://github.com/apache/hudi/assets/39345855/01472bb9-198a-44ab-964b-515bddd68eba)
   
   
   Hope this helps 
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [hudi] danny0405 commented on issue #8651: [SUPPORT]How to resolve small file?

Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on issue #8651:
URL: https://github.com/apache/hudi/issues/8651#issuecomment-1553992470

   Can you add more parallelism to the clustering task?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [hudi] jlloh commented on issue #8651: [SUPPORT]How to resolve small file?

Posted by "jlloh (via GitHub)" <gi...@apache.org>.
jlloh commented on issue #8651:
URL: https://github.com/apache/hudi/issues/8651#issuecomment-1623499726

   Sorry to hop on the thread, but @danny0405 I'm using a similar setup to OP (Flink + Async Clustering + COW + Insert), but writing to S3 instead of HDFS. I'm also getting small files, but I realised that the number of files written basically correspond to the environment parallelism of the flink environment.
   
   I tried tweaking `write.bucket_assign.tasks` but it doesn't seem to work, also tried tweaking the parquet size configurations, but doesn't seem to take effect. Flink still seems to write out as many files as the parallelism, in my case parallelism is 30, so it's writing out 30 files of ~5MB each. 
   
   I guess it async clustering could solve this later on, but i saw this line in the [documentation](https://hudi.apache.org/docs/clustering#execute-clustering):
   ```
   NOTE: Clustering can only be scheduled for tables / partitions not receiving any concurrent updates. In the future, concurrent updates use-case will be supported as well.
   ```
   Does this mean that if a partition is currently being written to (e.g. I do a daily partition), the clustering task won't be able to run to cluster the files until after the day has passed and the writer stops writing to the partition? I.e. clustering will be one day delayed.
   
   My dag for reference below:
   ![image](https://github.com/apache/hudi/assets/15816131/2b338868-1980-464a-9fc7-d418e73fa0e6)
   
   My table configuration:
   ![image](https://github.com/apache/hudi/assets/15816131/3c37cf03-5e13-4265-8201-28bc3c66428c)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [hudi] danny0405 commented on issue #8651: [SUPPORT]How to resolve small file?

Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on issue #8651:
URL: https://github.com/apache/hudi/issues/8651#issuecomment-1537696123

   Does the async clustering happens in normal progress? Can you past the DAG of the running job?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [hudi] c-f-cooper commented on issue #8651: [SUPPORT]How to resolve small file?

Posted by "c-f-cooper (via GitHub)" <gi...@apache.org>.
c-f-cooper commented on issue #8651:
URL: https://github.com/apache/hudi/issues/8651#issuecomment-1537736137

   yes,It has come into effect,the clustering task appears in DAG,and the replace_commit appears in .hoodie directory.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [hudi] c-f-cooper commented on issue #8651: [SUPPORT]How to resolve small file?

Posted by "c-f-cooper (via GitHub)" <gi...@apache.org>.
c-f-cooper commented on issue #8651:
URL: https://github.com/apache/hudi/issues/8651#issuecomment-1553931467

   ![image](https://github.com/apache/hudi/assets/25735549/8b816399-ede9-4a2a-97b5-d28e7ef3b1e4)
   
   ![023D4646-7D12-4606-8188-0F1A05DE47C5_1_102_o](https://github.com/apache/hudi/assets/25735549/af7586f4-5be8-4efd-a895-d56108c8c878)
   
   I found that the async cluster shedule done,but not execute @danny0405 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [SUPPORT]How to resolve small file? [hudi]

Posted by "c-f-cooper (via GitHub)" <gi...@apache.org>.
c-f-cooper closed issue #8651: [SUPPORT]How to resolve small file?
URL: https://github.com/apache/hudi/issues/8651


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org