You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by GitBox <gi...@apache.org> on 2022/08/12 02:15:01 UTC
[GitHub] [hudi] bithw1 opened a new issue, #6378: [SUPPORT]What's the behavior when writing to COW table
bithw1 opened a new issue, #6378:
URL: https://github.com/apache/hudi/issues/6378
Hi,
Assume I have a `COW` table and I would like to write 5 records into this table, `one record per minute`
1. I would ask how many commits will be created for the above 5 operation, will 5 commits be created? That is, new commits will be created for each new record?
2. If I write to the COW in batch mode, say, write 1000 records in one time, then how many commits be created?
Thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] bithw1 commented on issue #6378: [SUPPORT]What's the behavior when writing to COW table
Posted by GitBox <gi...@apache.org>.
bithw1 commented on issue #6378:
URL: https://github.com/apache/hudi/issues/6378#issuecomment-1212673797
> 1. If you are in streaming mode, each miniBatch creates a commit. For example, a commit will be created during a checkpoint in the `Flink engine`.
> 2. If you write in batch mode, and write 1000 records at a time, only one commit will be created.
>
> Happy to help.
Thanks for the helpful reply!
For my case of the COW table and I would like to write 5 records into this table, one record per minute, I am using spark, and
not using streaming,
I am doing is:
0. start spark application
1. kick off 1st spark write job and write one record
2. sleep 1 minutes
3. kick off 2nd spark write job and write one record
....
n. stop spark application
Will each spark job create a commit? It looks to me that no matter how many records are written by one spark job, it will always create one commit?
Thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] BruceKellan commented on issue #6378: [SUPPORT]What's the behavior when writing to COW table
Posted by GitBox <gi...@apache.org>.
BruceKellan commented on issue #6378:
URL: https://github.com/apache/hudi/issues/6378#issuecomment-1212669377
1. If you are in streaming mode, each miniBatch creates a commit. For example, a commit will be created during a checkpoint in the `Flink engine`.
2. If you write in batch mode, and write 1000 records at a time, only one commit will be created.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] bithw1 commented on issue #6378: [SUPPORT]What's the behavior when writing to COW table
Posted by GitBox <gi...@apache.org>.
bithw1 commented on issue #6378:
URL: https://github.com/apache/hudi/issues/6378#issuecomment-1212684142
> > > 1. If you are in streaming mode, each miniBatch creates a commit. For example, a commit will be created during a checkpoint in the `Flink engine`.
> > > 2. If you write in batch mode, and write 1000 records at a time, only one commit will be created.
> > >
> > > Happy to help.
> >
> >
> > Thanks @BruceKellan for the helpful reply! I have a further question: for my case of the COW table and I would like to write 5 records into this table, one record per minute, I am using spark, and not using streaming,
> > I am doing is: 0. start spark application
> >
> > 1. kick off 1st spark write job and write one record
> > 2. sleep 1 minutes
> > 3. kick off 2nd spark write job and write one record
> > ....
> > n. stop spark application
> >
> > Will each spark job create a commit? It looks to me that no matter how many records are written by one spark job, it will always create one commit?
> > Thanks!
>
> Yes, you are actually using 5 batch spark application in this scenario.
Thanks @BruceKellan for the helpful answer, I am closing this support request!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] BruceKellan commented on issue #6378: [SUPPORT]What's the behavior when writing to COW table
Posted by GitBox <gi...@apache.org>.
BruceKellan commented on issue #6378:
URL: https://github.com/apache/hudi/issues/6378#issuecomment-1212674948
> > 1. If you are in streaming mode, each miniBatch creates a commit. For example, a commit will be created during a checkpoint in the `Flink engine`.
> > 2. If you write in batch mode, and write 1000 records at a time, only one commit will be created.
> >
> > Happy to help.
>
> Thanks @BruceKellan for the helpful reply! I have a further question: for my case of the COW table and I would like to write 5 records into this table, one record per minute, I am using spark, and not using streaming,
>
> I am doing is: 0. start spark application
>
> 1. kick off 1st spark write job and write one record
> 2. sleep 1 minutes
> 3. kick off 2nd spark write job and write one record
> ....
> n. stop spark application
>
> Will each spark job create a commit? It looks to me that no matter how many records are written by one spark job, it will always create one commit?
>
> Thanks!
Yes, you are actually using 5 batch spark application in this scenario.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] bithw1 closed issue #6378: [SUPPORT]What's the behavior when writing to COW table
Posted by GitBox <gi...@apache.org>.
bithw1 closed issue #6378: [SUPPORT]What's the behavior when writing to COW table
URL: https://github.com/apache/hudi/issues/6378
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org