You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hudi.apache.org by Sivabalan <n....@gmail.com> on 2020/03/03 07:04:00 UTC
Re: Need clarity on these test cases in TestHoodieDeltaStreamer
I will sync up with Pratyaksh offline on this.
On Thu, Feb 27, 2020 at 11:24 PM Pratyaksh Sharma <pr...@gmail.com>
wrote:
> Hi Balaji,
>
> Right now I am facing some different issue in the same test case. The
> number of records are not matching and assertion is failing. Once I am able
> to fix that as well, I will open the PR for sure. :)
>
> On Thu, Feb 27, 2020 at 11:17 PM Balaji Varadarajan
> <v....@ymail.com.invalid> wrote:
>
>>
>> Awesome Pratyaksh, would you mind opening a PR to documenting it.
>> Balaji.V
>>
>> Sent from Yahoo Mail for iPhone
>>
>>
>> On Wednesday, February 26, 2020, 11:14 PM, Pratyaksh Sharma <
>> pratyaksh13@gmail.com> wrote:
>>
>> Hi,
>>
>> I figured out the issue yesterday. Thank you for helping me out.
>>
>> On Thu, Feb 27, 2020 at 12:01 AM vbalaji@apache.org <vb...@apache.org>
>> wrote:
>>
>> >
>> > This change was done as part of adding delete API support :
>> >
>> https://github.com/apache/incubator-hudi/commit/7031445eb3cae5a4557786c7eb080944320609aa
>> >
>> > I don't remember the reason behind this.
>> > Sivabalan, Can you explain the reason when you get a chance.
>> > Thanks,Balaji.V
>> > On Wednesday, February 26, 2020, 06:03:53 AM PST, Pratyaksh Sharma <
>> > pratyaksh13@gmail.com> wrote:
>> >
>> > Anybody got a chance to look at this?
>> >
>> > On Mon, Feb 24, 2020 at 1:04 AM Pratyaksh Sharma <pratyaksh13@gmail.com
>> >
>> > wrote:
>> >
>> > > Hi,
>> > >
>> > > While working on one of my PRs, I am stuck with the following test
>> cases
>> > > in TestHoodieDeltaStreamer -
>> > > 1. testUpsertsCOWContinuousMode
>> > > 2. testUpsertsMORContinuousMode
>> > >
>> > > For both of them, at line [1] and [2], we are adding 200 to
>> totalRecords
>> > > while asserting record count and distance count respectively. I am
>> unable
>> > > to understand what do these 200 records correspond to. Any leads are
>> > > appreciated.
>> > >
>> > > I feel probably I am missing some piece of code where I need to do
>> > changes
>> > > for the above tests to pass.
>> > >
>> > > [1]
>> > >
>> >
>> https://github.com/apache/incubator-hudi/blob/078d4825d909b2c469398f31c97d2290687321a8/hudi-utilities/src/test/java/org/apache/hudi/utilities/TestHoodieDeltaStreamer.java#L425
>> > > .
>> > > [2]
>> > >
>> >
>> https://github.com/apache/incubator-hudi/blob/078d4825d909b2c469398f31c97d2290687321a8/hudi-utilities/src/test/java/org/apache/hudi/utilities/TestHoodieDeltaStreamer.java#L426
>> > > .
>> > >
>> > >
>> >
>>
>>
>>
>>
--
Regards,
-Sivabalan
Re: Need clarity on these test cases in TestHoodieDeltaStreamer
Posted by Pratyaksh Sharma <pr...@gmail.com>.
@Balaji
As part of bug fix, the addition of 200 records has been removed from our
code base. I guess there is no need of documenting this. If you still feel
there is a need, please let me know. Will do the needful.
On Wed, Mar 4, 2020 at 1:07 PM Pratyaksh Sharma <pr...@gmail.com>
wrote:
> Sure. I will share my updated changes with Siva today so that he can take
> a look. Then we will discuss further.
>
> On Tue, Mar 3, 2020 at 8:19 PM Vinoth Chandar <vi...@apache.org> wrote:
>
>> Please keep discussions on the mailing list, as much as possible :)
>>
>> On Mon, Mar 2, 2020 at 11:04 PM Sivabalan <n....@gmail.com> wrote:
>>
>>> I will sync up with Pratyaksh offline on this.
>>>
>>> On Thu, Feb 27, 2020 at 11:24 PM Pratyaksh Sharma <pratyaksh13@gmail.com
>>> >
>>> wrote:
>>>
>>> > Hi Balaji,
>>> >
>>> > Right now I am facing some different issue in the same test case. The
>>> > number of records are not matching and assertion is failing. Once I am
>>> able
>>> > to fix that as well, I will open the PR for sure. :)
>>> >
>>> > On Thu, Feb 27, 2020 at 11:17 PM Balaji Varadarajan
>>> > <v....@ymail.com.invalid> wrote:
>>> >
>>> >>
>>> >> Awesome Pratyaksh, would you mind opening a PR to documenting it.
>>> >> Balaji.V
>>> >>
>>> >> Sent from Yahoo Mail for iPhone
>>> >>
>>> >>
>>> >> On Wednesday, February 26, 2020, 11:14 PM, Pratyaksh Sharma <
>>> >> pratyaksh13@gmail.com> wrote:
>>> >>
>>> >> Hi,
>>> >>
>>> >> I figured out the issue yesterday. Thank you for helping me out.
>>> >>
>>> >> On Thu, Feb 27, 2020 at 12:01 AM vbalaji@apache.org <
>>> vbalaji@apache.org>
>>> >> wrote:
>>> >>
>>> >> >
>>> >> > This change was done as part of adding delete API support :
>>> >> >
>>> >>
>>> https://github.com/apache/incubator-hudi/commit/7031445eb3cae5a4557786c7eb080944320609aa
>>> >> >
>>> >> > I don't remember the reason behind this.
>>> >> > Sivabalan, Can you explain the reason when you get a chance.
>>> >> > Thanks,Balaji.V
>>> >> > On Wednesday, February 26, 2020, 06:03:53 AM PST, Pratyaksh
>>> Sharma <
>>> >> > pratyaksh13@gmail.com> wrote:
>>> >> >
>>> >> > Anybody got a chance to look at this?
>>> >> >
>>> >> > On Mon, Feb 24, 2020 at 1:04 AM Pratyaksh Sharma <
>>> pratyaksh13@gmail.com
>>> >> >
>>> >> > wrote:
>>> >> >
>>> >> > > Hi,
>>> >> > >
>>> >> > > While working on one of my PRs, I am stuck with the following test
>>> >> cases
>>> >> > > in TestHoodieDeltaStreamer -
>>> >> > > 1. testUpsertsCOWContinuousMode
>>> >> > > 2. testUpsertsMORContinuousMode
>>> >> > >
>>> >> > > For both of them, at line [1] and [2], we are adding 200 to
>>> >> totalRecords
>>> >> > > while asserting record count and distance count respectively. I am
>>> >> unable
>>> >> > > to understand what do these 200 records correspond to. Any leads
>>> are
>>> >> > > appreciated.
>>> >> > >
>>> >> > > I feel probably I am missing some piece of code where I need to do
>>> >> > changes
>>> >> > > for the above tests to pass.
>>> >> > >
>>> >> > > [1]
>>> >> > >
>>> >> >
>>> >>
>>> https://github.com/apache/incubator-hudi/blob/078d4825d909b2c469398f31c97d2290687321a8/hudi-utilities/src/test/java/org/apache/hudi/utilities/TestHoodieDeltaStreamer.java#L425
>>> >> > > .
>>> >> > > [2]
>>> >> > >
>>> >> >
>>> >>
>>> https://github.com/apache/incubator-hudi/blob/078d4825d909b2c469398f31c97d2290687321a8/hudi-utilities/src/test/java/org/apache/hudi/utilities/TestHoodieDeltaStreamer.java#L426
>>> >> > > .
>>> >> > >
>>> >> > >
>>> >> >
>>> >>
>>> >>
>>> >>
>>> >>
>>>
>>> --
>>> Regards,
>>> -Sivabalan
>>>
>>
Re: Need clarity on these test cases in TestHoodieDeltaStreamer
Posted by Pratyaksh Sharma <pr...@gmail.com>.
Sure. I will share my updated changes with Siva today so that he can take a
look. Then we will discuss further.
On Tue, Mar 3, 2020 at 8:19 PM Vinoth Chandar <vi...@apache.org> wrote:
> Please keep discussions on the mailing list, as much as possible :)
>
> On Mon, Mar 2, 2020 at 11:04 PM Sivabalan <n....@gmail.com> wrote:
>
>> I will sync up with Pratyaksh offline on this.
>>
>> On Thu, Feb 27, 2020 at 11:24 PM Pratyaksh Sharma <pr...@gmail.com>
>> wrote:
>>
>> > Hi Balaji,
>> >
>> > Right now I am facing some different issue in the same test case. The
>> > number of records are not matching and assertion is failing. Once I am
>> able
>> > to fix that as well, I will open the PR for sure. :)
>> >
>> > On Thu, Feb 27, 2020 at 11:17 PM Balaji Varadarajan
>> > <v....@ymail.com.invalid> wrote:
>> >
>> >>
>> >> Awesome Pratyaksh, would you mind opening a PR to documenting it.
>> >> Balaji.V
>> >>
>> >> Sent from Yahoo Mail for iPhone
>> >>
>> >>
>> >> On Wednesday, February 26, 2020, 11:14 PM, Pratyaksh Sharma <
>> >> pratyaksh13@gmail.com> wrote:
>> >>
>> >> Hi,
>> >>
>> >> I figured out the issue yesterday. Thank you for helping me out.
>> >>
>> >> On Thu, Feb 27, 2020 at 12:01 AM vbalaji@apache.org <
>> vbalaji@apache.org>
>> >> wrote:
>> >>
>> >> >
>> >> > This change was done as part of adding delete API support :
>> >> >
>> >>
>> https://github.com/apache/incubator-hudi/commit/7031445eb3cae5a4557786c7eb080944320609aa
>> >> >
>> >> > I don't remember the reason behind this.
>> >> > Sivabalan, Can you explain the reason when you get a chance.
>> >> > Thanks,Balaji.V
>> >> > On Wednesday, February 26, 2020, 06:03:53 AM PST, Pratyaksh
>> Sharma <
>> >> > pratyaksh13@gmail.com> wrote:
>> >> >
>> >> > Anybody got a chance to look at this?
>> >> >
>> >> > On Mon, Feb 24, 2020 at 1:04 AM Pratyaksh Sharma <
>> pratyaksh13@gmail.com
>> >> >
>> >> > wrote:
>> >> >
>> >> > > Hi,
>> >> > >
>> >> > > While working on one of my PRs, I am stuck with the following test
>> >> cases
>> >> > > in TestHoodieDeltaStreamer -
>> >> > > 1. testUpsertsCOWContinuousMode
>> >> > > 2. testUpsertsMORContinuousMode
>> >> > >
>> >> > > For both of them, at line [1] and [2], we are adding 200 to
>> >> totalRecords
>> >> > > while asserting record count and distance count respectively. I am
>> >> unable
>> >> > > to understand what do these 200 records correspond to. Any leads
>> are
>> >> > > appreciated.
>> >> > >
>> >> > > I feel probably I am missing some piece of code where I need to do
>> >> > changes
>> >> > > for the above tests to pass.
>> >> > >
>> >> > > [1]
>> >> > >
>> >> >
>> >>
>> https://github.com/apache/incubator-hudi/blob/078d4825d909b2c469398f31c97d2290687321a8/hudi-utilities/src/test/java/org/apache/hudi/utilities/TestHoodieDeltaStreamer.java#L425
>> >> > > .
>> >> > > [2]
>> >> > >
>> >> >
>> >>
>> https://github.com/apache/incubator-hudi/blob/078d4825d909b2c469398f31c97d2290687321a8/hudi-utilities/src/test/java/org/apache/hudi/utilities/TestHoodieDeltaStreamer.java#L426
>> >> > > .
>> >> > >
>> >> > >
>> >> >
>> >>
>> >>
>> >>
>> >>
>>
>> --
>> Regards,
>> -Sivabalan
>>
>
Re: Need clarity on these test cases in TestHoodieDeltaStreamer
Posted by Vinoth Chandar <vi...@apache.org>.
Please keep discussions on the mailing list, as much as possible :)
On Mon, Mar 2, 2020 at 11:04 PM Sivabalan <n....@gmail.com> wrote:
> I will sync up with Pratyaksh offline on this.
>
> On Thu, Feb 27, 2020 at 11:24 PM Pratyaksh Sharma <pr...@gmail.com>
> wrote:
>
> > Hi Balaji,
> >
> > Right now I am facing some different issue in the same test case. The
> > number of records are not matching and assertion is failing. Once I am
> able
> > to fix that as well, I will open the PR for sure. :)
> >
> > On Thu, Feb 27, 2020 at 11:17 PM Balaji Varadarajan
> > <v....@ymail.com.invalid> wrote:
> >
> >>
> >> Awesome Pratyaksh, would you mind opening a PR to documenting it.
> >> Balaji.V
> >>
> >> Sent from Yahoo Mail for iPhone
> >>
> >>
> >> On Wednesday, February 26, 2020, 11:14 PM, Pratyaksh Sharma <
> >> pratyaksh13@gmail.com> wrote:
> >>
> >> Hi,
> >>
> >> I figured out the issue yesterday. Thank you for helping me out.
> >>
> >> On Thu, Feb 27, 2020 at 12:01 AM vbalaji@apache.org <vbalaji@apache.org
> >
> >> wrote:
> >>
> >> >
> >> > This change was done as part of adding delete API support :
> >> >
> >>
> https://github.com/apache/incubator-hudi/commit/7031445eb3cae5a4557786c7eb080944320609aa
> >> >
> >> > I don't remember the reason behind this.
> >> > Sivabalan, Can you explain the reason when you get a chance.
> >> > Thanks,Balaji.V
> >> > On Wednesday, February 26, 2020, 06:03:53 AM PST, Pratyaksh Sharma
> <
> >> > pratyaksh13@gmail.com> wrote:
> >> >
> >> > Anybody got a chance to look at this?
> >> >
> >> > On Mon, Feb 24, 2020 at 1:04 AM Pratyaksh Sharma <
> pratyaksh13@gmail.com
> >> >
> >> > wrote:
> >> >
> >> > > Hi,
> >> > >
> >> > > While working on one of my PRs, I am stuck with the following test
> >> cases
> >> > > in TestHoodieDeltaStreamer -
> >> > > 1. testUpsertsCOWContinuousMode
> >> > > 2. testUpsertsMORContinuousMode
> >> > >
> >> > > For both of them, at line [1] and [2], we are adding 200 to
> >> totalRecords
> >> > > while asserting record count and distance count respectively. I am
> >> unable
> >> > > to understand what do these 200 records correspond to. Any leads are
> >> > > appreciated.
> >> > >
> >> > > I feel probably I am missing some piece of code where I need to do
> >> > changes
> >> > > for the above tests to pass.
> >> > >
> >> > > [1]
> >> > >
> >> >
> >>
> https://github.com/apache/incubator-hudi/blob/078d4825d909b2c469398f31c97d2290687321a8/hudi-utilities/src/test/java/org/apache/hudi/utilities/TestHoodieDeltaStreamer.java#L425
> >> > > .
> >> > > [2]
> >> > >
> >> >
> >>
> https://github.com/apache/incubator-hudi/blob/078d4825d909b2c469398f31c97d2290687321a8/hudi-utilities/src/test/java/org/apache/hudi/utilities/TestHoodieDeltaStreamer.java#L426
> >> > > .
> >> > >
> >> > >
> >> >
> >>
> >>
> >>
> >>
>
> --
> Regards,
> -Sivabalan
>