You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "zealjoanna (via GitHub)" <gi...@apache.org> on 2023/08/28 09:48:37 UTC
[GitHub] [hudi] zealjoanna opened a new issue, #9557: [SUPPORT] CDC file clean not work
zealjoanna opened a new issue, #9557:
URL: https://github.com/apache/hudi/issues/9557
**Describe the problem you faced**
i'm using CDC read for COW table, i want to keep last two commit by setting hoodie.cleaner.commits.retained = 1
then i found the parquet file cleaned as as I thought
but the cdc log file retained
**To Reproduce**
Steps to reproduce the behavior:
spark.sql("create table hudi_cow_pt_tbl (\n" +
" begin_lat double,\n" +
" begin_lon double,\n" +
" driver string,\n" +
" end_lat double,\n" +
" end_lon double,\n" +
" fare double,\n" +
" partitionpath String,\n" +
" rider string,\n" +
" ts String,\n" +
" uuid string)" +
" using hudi\n" +
"tblproperties (\n" +
"hoodie.table.cdc.enabled = 'true',\n" +
"hoodie.table.cdc.supplemental.logging = 'true',\n" +
"hoodie.datasource.query.incremental.format = 'cdc',\n" +
"type = 'cow',\n" +
"primaryKey = 'uuid',\n" +
"preCombineField = 'ts'\n " +
")\n" +
"partitioned by (partitionpath)\n" +
"location 'file:///D:/work/IDEA_PROJECT/spark_base/hudi_sql';")
spark.sql("set hoodie.clean.automatic = true;")
spark.sql("set hoodie.clean.max.commits = 1;")
spark.sql("set hoodie.cleaner.commits.retained = 1;")
spark.sql("insert into hudi_cow_pt_tbl " +
"select 2.0 as begin_lat , 0.1 as begin_lon, '' as driver ,0.2 as end_Slat ,0.3 as end_lon ,0.4 as fare,'china/wuhan' as partitionpath ,'rider-1' as rider ,1691560140000 as ts,'1' as uuid ")
spark.sql("insert into hudi_cow_pt_tbl " +
"select 2.0 as begin_lat , 0.1 as begin_lon, '' as driver ,0.2 as end_Slat ,0.3 as end_lon ,0.4 as fare,'china/wuhan' as partitionpath ,'rider-1' as rider ,1691560140000 as ts,'1' as uuid ")
spark.sql("insert into hudi_cow_pt_tbl " +
"select 2.0 as begin_lat , 0.1 as begin_lon, '' as driver ,0.2 as end_Slat ,0.3 as end_lon ,0.4 as fare,'china/wuhan' as partitionpath ,'rider-1' as rider ,1691560140000 as ts,'1' as uuid ")
spark.sql("insert into hudi_cow_pt_tbl " +
"select 2.0 as begin_lat , 0.1 as begin_lon, '' as driver ,0.2 as end_Slat ,0.3 as end_lon ,0.4 as fare,'china/wuhan' as partitionpath ,'rider-1' as rider ,1691560140000 as ts,'1' as uuid ")
**Expected behavior**
i expect two cdc log file remains and two parquet file remains
**Environment Description**
* Hudi version :1.31.1
* Spark version :3.3.2
* Hive version : no
* Hadoop version :2.8.3
* Storage (HDFS/S3/GCS..) :
* Running on Docker? (yes/no) :no
**Additional context**
**Stacktrace**
```Add the stacktrace of the error.```
here are some infos that i found
the cdc file path in metadata is not right
d5cec5e4-2cb0-42c2-bcd1-39f1fe400e12-0_0-15-11_20230825201900316.parquet -> [436825,false],
d5cec5e4-2cb0-42c2-bcd1-39f1fe400e12-0_0-15-11_20230825203147367.parquet -> [436824,false],
partitionpath=1/.d5cec5e4-2cb0-42c2-bcd1-39f1fe400e12-0_20230825201939193.log.1_0-15-11.cdc -> [1283,false],
partitionpath=1/.d5cec5e4-2cb0-42c2-bcd1-39f1fe400e12-0_20230825203147367.log.1_0-15-11.cdc -> [1283,false],
d5cec5e4-2cb0-42c2-bcd1-39f1fe400e12-0_0-15-11_20230825201939193.parquet -> [436821,false],
partitionpath=1/.d5cec5e4-2cb0-42c2-bcd1-39f1fe400e12-0_20230825201900316.log.1_0-15-11.cdc -> [1283,false], partitionpath=1/.d5cec5e4-2cb0-42c2-bcd1-39f1fe400e12-0_20230825202620629.log.1_0-15-11.cdc -> [1283,false]
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [SUPPORT] CDC file clean not work [hudi]
Posted by "zealjoanna (via GitHub)" <gi...@apache.org>.
zealjoanna commented on issue #9557:
URL: https://github.com/apache/hudi/issues/9557#issuecomment-1751573310
> @zealjoanna Do you still have spare time for contributing the fix ?
sorry for late replying , I've been quite busy lately and I'm not available for giving a PR
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] danny0405 commented on issue #9557: [SUPPORT] CDC file clean not work
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on issue #9557:
URL: https://github.com/apache/hudi/issues/9557#issuecomment-1696777012
Yeah, nice findings, can you fire a fix for it?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] zealjoanna commented on issue #9557: [SUPPORT] CDC file clean not work
Posted by "zealjoanna (via GitHub)" <gi...@apache.org>.
zealjoanna commented on issue #9557:
URL: https://github.com/apache/hudi/issues/9557#issuecomment-1696667766
the cdc file contains the partition_path while the parquet file use the filename
i guess that's the point
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [SUPPORT] CDC file clean not work [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on issue #9557:
URL: https://github.com/apache/hudi/issues/9557#issuecomment-1751561400
@zealjoanna Do you still have spare time for contributing the fix ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] ad1happy2go commented on issue #9557: [SUPPORT] CDC file clean not work
Posted by "ad1happy2go (via GitHub)" <gi...@apache.org>.
ad1happy2go commented on issue #9557:
URL: https://github.com/apache/hudi/issues/9557#issuecomment-1697760207
@zealjoanna Thanks a lot. Can you provide the JIRA ticket here which you are working for this so that we can track it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [SUPPORT] CDC file clean not work [hudi]
Posted by "Hans-Raintree (via GitHub)" <gi...@apache.org>.
Hans-Raintree commented on issue #9557:
URL: https://github.com/apache/hudi/issues/9557#issuecomment-1750498971
Hey @zealjoanna, @ad1happy2go has there been any movement on this? I'm facing the same issue.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [SUPPORT] CDC file clean not work [hudi]
Posted by "zealjoanna (via GitHub)" <gi...@apache.org>.
zealjoanna commented on issue #9557:
URL: https://github.com/apache/hudi/issues/9557#issuecomment-1751572036
> @zealjoanna Do you still have spare time for contributing the fix ?
>  the cdc file contains the partition_path while the parquet file use the filename i guess that's the point
if (cdcPathAndSizes != null && !cdcPathAndSizes.isEmpty()) {
for (Map.Entry<String, Long> cdcEntry : cdcPathAndSizes.entrySet()) {
String mapKey = FSUtils.getFileName(cdcEntry.getKey(),
partitionStatName);
Long mapValue = cdcEntry.getValue();
map.put(mapKey, mapValue);
}
}
this fix works
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [SUPPORT] CDC file clean not work [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 closed issue #9557: [SUPPORT] CDC file clean not work
URL: https://github.com/apache/hudi/issues/9557
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] zealjoanna commented on issue #9557: [SUPPORT] CDC file clean not work
Posted by "zealjoanna (via GitHub)" <gi...@apache.org>.
zealjoanna commented on issue #9557:
URL: https://github.com/apache/hudi/issues/9557#issuecomment-1697099022
> Yeah, nice findings, can you fire a fix for it?
thanks for replying, I temporarily fixed it locally.
After verification, I will make time to fire a fix
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org