You are viewing a plain text version of this content. The canonical link for it is here.

Posted to commits@hudi.apache.org by "Ethan Guo (Jira)" <ji...@apache.org> on 2024/03/28 01:15:00 UTC

[jira] [Updated] (HUDI-3872) Write tests for async clustering job to run concurrently w/ deltastreamer w/o overlap

     [ https://issues.apache.org/jira/browse/HUDI-3872?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Ethan Guo updated HUDI-3872:
----------------------------
    Component/s: table-service

> Write tests for async clustering job to run concurrently w/ deltastreamer w/o overlap
> -------------------------------------------------------------------------------------
>
>                 Key: HUDI-3872
>                 URL: https://issues.apache.org/jira/browse/HUDI-3872
>             Project: Apache Hudi
>          Issue Type: Test
>          Components: clustering, deltastreamer, table-service
>            Reporter: sivabalan narayanan
>            Priority: Major
>
> We want to test HoodieClusteringJob along w/ deltastreamer continuous job. But to test it reliably, we need to ensure regular writers don't overlap w/ clustering file groups. If not, one of them will fail. So, we need to come up w/ test data generation differently so that regular writers keep writing to new partition in every batch and so clustering can take care of older partitions and there won't be any over-stepping among these. 
>  
>  



--
This message was sent by Atlassian Jira
(v8.20.10#820010)