You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "ummadinagarjuna (via GitHub)" <gi...@apache.org> on 2025/06/30 06:58:51 UTC

[I] [SUPPORT] [hudi]

ummadinagarjuna opened a new issue, #13502:
URL: https://github.com/apache/hudi/issues/13502

   **_Tips before filing an issue_**
   
   - Have you gone through our [FAQs](https://hudi.apache.org/learn/faq/)?
   
   - Join the mailing list to engage in conversations and get faster support at dev-subscribe@hudi.apache.org.
   
   - If you have triaged this as a bug, then file an [issue](https://issues.apache.org/jira/projects/HUDI/issues) directly.
   
   **Describe the problem you faced**
   
   A clear and concise description of the problem.
   
   **To Reproduce**
   
   Steps to reproduce the behavior:
   
   1. 3 file version not maintaining using below hudi  options  running glue jobs
   
   {
           'hoodie.datasource.write.table.type': 'COPY_ON_WRITE',
           'hoodie.datasource.hive_sync.enable': 'true',
           "hoodie.datasource.hive_sync.mode": "hms",
           'hoodie.datasource.hive_sync.sync_as_datasource': 'false',
           'hoodie.datasource.hive_sync.use_jdbc': 'false',
           'hoodie.datasource.hive_sync.support_timestamp': 'true',
           'hoodie.index.type': "GLOBAL_SIMPLE",
           'hoodie.cleaner.policy': "KEEP_LATEST_FILE_VERSIONS",
           # 'hoodie.cleaner.fileversions.retained': "5",
           # "hoodie.parquet.max.file.size": 536870912,
           # 'hoodie.cleaner.policy.failed.writes': 'LAZY',
           'hoodie.write.concurrency.mode': "optimistic_concurrency_control",
           'hoodie.write.lock.provider': 'org.apache.hudi.aws.transaction.lock.DynamoDBBasedLockProvider',
           'hoodie.write.lock.dynamodb.table': args['LOCK_TABLE_NAME'],
           'hoodie.write.lock.dynamodb.billing_mode': 'PAY_PER_REQUEST',
           'hoodie.write.lock.dynamodb.region': '{0}'.format(curr_region),
           'hoodie.write.lock.dynamodb.endpoint_url': 'dynamodb.{0}.amazonaws.com'.format(curr_region),
           'hoodie.insert.shuffle.parallelism': 200,
           'hoodie.upsert.shuffle.parallelism': 200,
           'hoodie.metadata.index.column.stats.enable': 'true',
           'hoodie.clean.automatic': "true",
           'hoodie.cleaner.fileversions.retained': "3",
           "hoodie.parquet.max.file.size": 30000000,
           "hoodie.parquet.small.file.limit": 20000000,  # 100 MB
           'hoodie.cleaner.policy.failed.writes': 'LAZY',
           'hoodie.cleaner.parallelism': 200,
           }
   
   ![Image](https://github.com/user-attachments/assets/b67a3c3b-0e60-4839-9ed6-a06dbf094a5c)
   
   **Expected behavior**
   need to maintain 3 file versions
   
   
   A clear and concise description of what you expected to happen.
   
   **Environment Description**
   
   * Hudi version :0.14.0
   
   * Spark version : 3.3
   
   * Hive version :
   
   * Hadoop version :
   
   * Storage (HDFS/S3/GCS..) :s3
   
   * Running on Docker? (yes/no) : No
   
   
   **Additional context**
   
   Add any other context about the problem here.
   
   **Stacktrace**
   
   ```Add the stacktrace of the error.```
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [SUPPORT] file versioning is happening as not expected for cow Tables hudi 0.14.0 [hudi]

Posted by "ad1happy2go (via GitHub)" <gi...@apache.org>.
ad1happy2go commented on issue #13502:
URL: https://github.com/apache/hudi/issues/13502#issuecomment-3024083483

   @ummadinagarjuna Can you elaborate on issue, as screenshot is not clear about what you want to say.
   
   Are you saying cleaner is not retaining 3 file versions for you or is it not cleaning properly and retaining more versions. You can also check on clean commits in timeline to look into it more.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [SUPPORT] file versioning is happening as not expected for cow Tables hudi 0.14.0 [hudi]

Posted by "ummadinagarjuna (via GitHub)" <gi...@apache.org>.
ummadinagarjuna commented on issue #13502:
URL: https://github.com/apache/hudi/issues/13502#issuecomment-3031989114

   yes for bronze layer insert and  silver layer upsert 
   
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [SUPPORT] file versioning is happening as not expected for cow Tables hudi 0.14.0 [hudi]

Posted by "ummadinagarjuna (via GitHub)" <gi...@apache.org>.
ummadinagarjuna commented on issue #13502:
URL: https://github.com/apache/hudi/issues/13502#issuecomment-3026262462

   clean commits doing ok, but I have given small file limit and max file limit ,trying to retain 3 file version but if I am publishing 10k,20k it will be creating more group files without reaching the limit of parquet file as given in screen shot


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [SUPPORT] file versioning is happening as not expected for cow Tables hudi 0.14.0 [hudi]

Posted by "ummadinagarjuna (via GitHub)" <gi...@apache.org>.
ummadinagarjuna commented on issue #13502:
URL: https://github.com/apache/hudi/issues/13502#issuecomment-3205080853

   Hi @ad1happy2go 
    need support on this 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [SUPPORT] file versioning is happening as not expected for cow Tables hudi 0.14.0 [hudi]

Posted by "ummadinagarjuna (via GitHub)" <gi...@apache.org>.
ummadinagarjuna commented on issue #13502:
URL: https://github.com/apache/hudi/issues/13502#issuecomment-3162478529

   Hi team any update on this 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [SUPPORT] file versioning is happening as not expected for cow Tables hudi 0.14.0 [hudi]

Posted by "ad1happy2go (via GitHub)" <gi...@apache.org>.
ad1happy2go commented on issue #13502:
URL: https://github.com/apache/hudi/issues/13502#issuecomment-3031982359

   @ummadinagarjuna So issue for you seems small file handling is not working for you so its creating so many new file groups. Are you using upsert?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org