You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "zdl11111 (via GitHub)" <gi...@apache.org> on 2023/11/08 09:19:43 UTC
[I] [SUPPORT] Some .commit files generated by compaction with 0 size [hudi]
zdl11111 opened a new issue, #10017:
URL: https://github.com/apache/hudi/issues/10017
**_Tips before filing an issue_**
- Have you gone through our [FAQs](https://hudi.apache.org/learn/faq/)? yes
- Join the mailing list to engage in conversations and get faster support at dev-subscribe@hudi.apache.org.
- If you have triaged this as a bug, then file an [issue](https://issues.apache.org/jira/projects/HUDI/issues) directly.
**Describe the problem you faced**
The write mode is upsert, but there are only insert operation to a MOR table.
After some insertion, the size of some .commit files generated by compaction are 0, but the size of corresponding .commit.requested are not 0, is it reasonable? I will appreciate it if anyone could provide any help.
**To Reproduce**
**Environment Description**
* Hudi version : 0.13.1
* Flink version : 1.14
* Storage (HDFS/S3/GCS..) : HDFS
* Running on Docker? (yes/no) : no
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [SUPPORT] Some .commit files generated by compaction with 0 size [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on issue #10017:
URL: https://github.com/apache/hudi/issues/10017#issuecomment-1803103939
Are the inflight and requested compaction metdata files empty also?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [SUPPORT] Some .commit files generated by compaction with 0 size [hudi]
Posted by "zdl11111 (via GitHub)" <gi...@apache.org>.
zdl11111 commented on issue #10017:
URL: https://github.com/apache/hudi/issues/10017#issuecomment-1803070116
Thanks for your help, and there is another question, for an existing partition, the new insertion record to an existing bucket will be marked as **numUpdateWrites**, so how could I confirm how many new records have been written in this delta commit? I will appreciate it if you could help me with it.
@ad1happy2go @danny0405
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [SUPPORT] Some .commit files generated by compaction with 0 size [hudi]
Posted by "zdl11111 (via GitHub)" <gi...@apache.org>.
zdl11111 closed issue #10017: [SUPPORT] Some .commit files generated by compaction with 0 size
URL: https://github.com/apache/hudi/issues/10017
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [SUPPORT] Some .commit files generated by compaction with 0 size [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on issue #10017:
URL: https://github.com/apache/hudi/issues/10017#issuecomment-1803093991
what release did you use? Since 0.14.0, for MOR + INSERT, we write parquets directly, you should use clustering instead of compaction.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [SUPPORT] Some .commit files generated by compaction with 0 size [hudi]
Posted by "nsivabalan (via GitHub)" <gi...@apache.org>.
nsivabalan commented on issue #10017:
URL: https://github.com/apache/hudi/issues/10017#issuecomment-2347902011
hey @zdl11111 : are you looking for more help on this. If not, can we close it out. If its still an issue, can you help us understand any progress we have made on this regard.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [SUPPORT] Some .commit files generated by compaction with 0 size [hudi]
Posted by "zdl11111 (via GitHub)" <gi...@apache.org>.
zdl11111 commented on issue #10017:
URL: https://github.com/apache/hudi/issues/10017#issuecomment-1803102881
> what release did you use?
Thanks for your information! But now I am using flink1.14 and hudi0.13.1
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [SUPPORT] Some .commit files generated by compaction with 0 size [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on issue #10017:
URL: https://github.com/apache/hudi/issues/10017#issuecomment-1804919943
Did you ever check the commit_sink operator logs to find any errors about the compaction commiting?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [SUPPORT] Some .commit files generated by compaction with 0 size [hudi]
Posted by "zdl11111 (via GitHub)" <gi...@apache.org>.
zdl11111 commented on issue #10017:
URL: https://github.com/apache/hudi/issues/10017#issuecomment-1803107881
> Are the inflight and requested compaction metdata files empty also?
The inflight file is empty but the requested file is not.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [SUPPORT] Some .commit files generated by compaction with 0 size [hudi]
Posted by "ad1happy2go (via GitHub)" <gi...@apache.org>.
ad1happy2go commented on issue #10017:
URL: https://github.com/apache/hudi/issues/10017#issuecomment-1803263076
@zdl11111 Can you also share sample code/data what you are trying if possible.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [SUPPORT] Some .commit files generated by compaction with 0 size [hudi]
Posted by "ad1happy2go (via GitHub)" <gi...@apache.org>.
ad1happy2go commented on issue #10017:
URL: https://github.com/apache/hudi/issues/10017#issuecomment-1801869228
@zdl11111 With Flink writers as we only write log files with ingestion so even if there are no updates we should have some log files to compact. So compact file size ideally should not be zero. cc @danny0405
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [SUPPORT] Some .commit files generated by compaction with 0 size [hudi]
Posted by "zdl11111 (via GitHub)" <gi...@apache.org>.
zdl11111 commented on issue #10017:
URL: https://github.com/apache/hudi/issues/10017#issuecomment-1803344542
> @zdl11111 Can you also share sample code/data what you are trying if possible.
Sure, actually, this is a random problem, we can also query the correct data from the table, I am just wondering whether it is reasonable. Here are the steps:
1. create a hudi table:
create table table1 (
c1 char(6), c2 varchar(10), c3 boolean,
c4 smallint, c5 integer , c6 bigint,
c7 float, c8 double,
c9 timestamp , c10 date, c11 decimal(10,4), updatecol int,
primary key (c5, c10) not enforced) partitioned by (c10, updatecol)
with ('connector' = 'hudi',
'table.type' = 'MERGE_ON_READ',
'path' = 'hdfs:/flink/table1'
'index.type' = 'BUCKET',
'hoodie.bucket.index.hash.field' = 'c5,c10',
'hoodie.bucket.index.num.buckets' = '4');
2. create a source table using 'datagen' connector
create table t_integer_gen (
c1 char(6), c2 varchar(10), c3 boolean,
c4 smallint, c5 integer, c6 bigint,
c7 float, c8 double,
c10 timestamp, c11 date, c12 decimal(10,4))
with (
'connector' = 'datagen',
'number-of-rows' = '100000',
'rows-per-second' = '2000',
'fields.c1.length' = '6',
'fields.c2.length' = '10'
);
3. insert into table1 select * from t_integer_gen;
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org