You are viewing a plain text version of this content. The canonical link for it is here.
Posted to hdfs-dev@hadoop.apache.org by Steve Loughran <st...@cloudera.com.INVALID> on 2024/06/06 19:58:28 UTC
[DISCUSS] lots of yetus failures in hdfs
PR's which trigger hdfs builds seem to hit a lot of hdfs test failures
https://github.com/apache/hadoop/pull/6675
Are these regressions or are the tests flaky?
I don't want commit patches which break things, yet hdfs tests seem
unreliable and so I'm dangerously tempted to +1 anyway...
Re: [DISCUSS] lots of yetus failures in hdfs
Posted by Ayush Saxena <ay...@gmail.com>.
The build issues in trunk are because of that surefire-upgrade, the
build crashing & not posting the result on the PR, so reverting that
should make things work.
Checking the PR that Steve mentioned, that is for branch-3.3 and most
of the failures are due to OOM "unable to create native thread", that
for some reason is quite random in nature and is with us like since
stone age and the only solution is retrigger or we can explore
increasing the memory like in HADOOP-18680
-Ayush
On Fri, 7 Jun 2024 at 13:51, Xiaoqiao He <he...@apache.org> wrote:
>
> Thanks Ayush for your information.
>
>> So, technically it should be an HDFS induced mess, else I would have
>> reverted by now.
>
> Do you mean revert https://github.com/apache/hadoop/pull/6664?
> If true, +1 to revert it and recover the build flow runs normally.
> cc @Shilun Fan
>
> Best Regards,
> - He Xiaoqiao
>
> On Fri, Jun 7, 2024 at 1:07 PM Ayush Saxena <ay...@gmail.com> wrote:
>>
>> There are two places where you can tune the memory [1] & [2]
>>
>> I haven't checked again, but I think it is the same old problem, I
>> mentioned that in [3] in the last paragraph where there was some
>> windows failure report. I did drop a comment [4] on that PR telling
>> about those issues, there were some comments on the Jira but I didn't
>> follow...
>>
>> So, technically it should be an HDFS induced mess, else I would have
>> reverted by now :-)
>>
>> Good Luck!!!
>>
>> -Ayush
>>
>>
>> [1] https://github.com/apache/hadoop/blob/2ee0bf953492b66765d3d2c902407fbf9bceddec/hadoop-project/pom.xml#L172
>> [2] https://github.com/apache/hadoop/blob/2ee0bf953492b66765d3d2c902407fbf9bceddec/dev-support/docker/Dockerfile#L77
>> [3] https://lists.apache.org/thread/hmzl61ow0sbs10p0hky17xxhsggbhc3g
>> [4] https://github.com/apache/hadoop/pull/6664#issuecomment-2082356393
>>
>> On Fri, 7 Jun 2024 at 08:04, Xiaoqiao He <he...@apache.org> wrote:
>> >
>> > Thanks Steve. Try to trigger CI manually and let's wait what it will say.
>> > BTW, the flaky tests seem not related to UT logic itself, but most of them
>> > throw OOM. Not sure if @Ayush Saxena knows how to re-config or tune
>> > the memory of Yetus?
>> >
>> > Best Regards,
>> > - He Xiaoqiao
>> >
>> > On Fri, Jun 7, 2024 at 3:59 AM Steve Loughran <st...@cloudera.com.invalid> wrote:
>> >>
>> >> PR's which trigger hdfs builds seem to hit a lot of hdfs test failures
>> >> https://github.com/apache/hadoop/pull/6675
>> >>
>> >> Are these regressions or are the tests flaky?
>> >>
>> >> I don't want commit patches which break things, yet hdfs tests seem
>> >> unreliable and so I'm dangerously tempted to +1 anyway...
---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-help@hadoop.apache.org
Re: [DISCUSS] lots of yetus failures in hdfs
Posted by Xiaoqiao He <he...@apache.org>.
Thanks Ayush for your information.
So, technically it should be an HDFS induced mess, else I would have
> reverted by now.
Do you mean revert https://github.com/apache/hadoop/pull/6664?
If true, +1 to revert it and recover the build flow runs normally.
cc @Shilun Fan <sl...@foxmail.com>
Best Regards,
- He Xiaoqiao
On Fri, Jun 7, 2024 at 1:07 PM Ayush Saxena <ay...@gmail.com> wrote:
> There are two places where you can tune the memory [1] & [2]
>
> I haven't checked again, but I think it is the same old problem, I
> mentioned that in [3] in the last paragraph where there was some
> windows failure report. I did drop a comment [4] on that PR telling
> about those issues, there were some comments on the Jira but I didn't
> follow...
>
> So, technically it should be an HDFS induced mess, else I would have
> reverted by now :-)
>
> Good Luck!!!
>
> -Ayush
>
>
> [1]
> https://github.com/apache/hadoop/blob/2ee0bf953492b66765d3d2c902407fbf9bceddec/hadoop-project/pom.xml#L172
> [2]
> https://github.com/apache/hadoop/blob/2ee0bf953492b66765d3d2c902407fbf9bceddec/dev-support/docker/Dockerfile#L77
> [3] https://lists.apache.org/thread/hmzl61ow0sbs10p0hky17xxhsggbhc3g
> [4] https://github.com/apache/hadoop/pull/6664#issuecomment-2082356393
>
> On Fri, 7 Jun 2024 at 08:04, Xiaoqiao He <he...@apache.org> wrote:
> >
> > Thanks Steve. Try to trigger CI manually and let's wait what it will say.
> > BTW, the flaky tests seem not related to UT logic itself, but most of
> them
> > throw OOM. Not sure if @Ayush Saxena knows how to re-config or tune
> > the memory of Yetus?
> >
> > Best Regards,
> > - He Xiaoqiao
> >
> > On Fri, Jun 7, 2024 at 3:59 AM Steve Loughran
> <st...@cloudera.com.invalid> wrote:
> >>
> >> PR's which trigger hdfs builds seem to hit a lot of hdfs test failures
> >> https://github.com/apache/hadoop/pull/6675
> >>
> >> Are these regressions or are the tests flaky?
> >>
> >> I don't want commit patches which break things, yet hdfs tests seem
> >> unreliable and so I'm dangerously tempted to +1 anyway...
>
Re: [DISCUSS] lots of yetus failures in hdfs
Posted by Ayush Saxena <ay...@gmail.com>.
There are two places where you can tune the memory [1] & [2]
I haven't checked again, but I think it is the same old problem, I
mentioned that in [3] in the last paragraph where there was some
windows failure report. I did drop a comment [4] on that PR telling
about those issues, there were some comments on the Jira but I didn't
follow...
So, technically it should be an HDFS induced mess, else I would have
reverted by now :-)
Good Luck!!!
-Ayush
[1] https://github.com/apache/hadoop/blob/2ee0bf953492b66765d3d2c902407fbf9bceddec/hadoop-project/pom.xml#L172
[2] https://github.com/apache/hadoop/blob/2ee0bf953492b66765d3d2c902407fbf9bceddec/dev-support/docker/Dockerfile#L77
[3] https://lists.apache.org/thread/hmzl61ow0sbs10p0hky17xxhsggbhc3g
[4] https://github.com/apache/hadoop/pull/6664#issuecomment-2082356393
On Fri, 7 Jun 2024 at 08:04, Xiaoqiao He <he...@apache.org> wrote:
>
> Thanks Steve. Try to trigger CI manually and let's wait what it will say.
> BTW, the flaky tests seem not related to UT logic itself, but most of them
> throw OOM. Not sure if @Ayush Saxena knows how to re-config or tune
> the memory of Yetus?
>
> Best Regards,
> - He Xiaoqiao
>
> On Fri, Jun 7, 2024 at 3:59 AM Steve Loughran <st...@cloudera.com.invalid> wrote:
>>
>> PR's which trigger hdfs builds seem to hit a lot of hdfs test failures
>> https://github.com/apache/hadoop/pull/6675
>>
>> Are these regressions or are the tests flaky?
>>
>> I don't want commit patches which break things, yet hdfs tests seem
>> unreliable and so I'm dangerously tempted to +1 anyway...
---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-help@hadoop.apache.org
Re: [DISCUSS] lots of yetus failures in hdfs
Posted by Xiaoqiao He <he...@apache.org>.
Thanks Steve. Try to trigger CI manually and let's wait what it will say.
BTW, the flaky tests seem not related to UT logic itself, but most of them
throw OOM. Not sure if @Ayush Saxena <ay...@gmail.com> knows how to
re-config or tune
the memory of Yetus?
Best Regards,
- He Xiaoqiao
On Fri, Jun 7, 2024 at 3:59 AM Steve Loughran <st...@cloudera.com.invalid>
wrote:
> PR's which trigger hdfs builds seem to hit a lot of hdfs test failures
> https://github.com/apache/hadoop/pull/6675
>
> Are these regressions or are the tests flaky?
>
> I don't want commit patches which break things, yet hdfs tests seem
> unreliable and so I'm dangerously tempted to +1 anyway...
>