You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hudi.apache.org by Sivabalan <n....@gmail.com> on 2020/08/30 14:59:59 UTC
DevX, Test infra Rgdn
As Hudi matures as a project, we need to get our devX and test infra rock
solid. Availability of test utils and base classes for ease of writing more
tests, stable integration tests, ease of debuggability, micro benchmarks,
performance test infra, automating checkstyle formatting, nightly snapshot
builds and so on.
We have identified and categorized these into different areas as below.
- Test fixes and some clean up. // There are a lot of jira tickets
lying around in this section.
- Test refactoring. // For ease of development, and reduce clutter, we need
to work on refactoring test infra like having more test utils, base classes
etc.
- More tests to improve coverage in some areas.
- CI stability and ease of debugging integration tests.
- Checkstyle, sl4j, warnings, spotless, etc.
- Micro benchmarks. // add benchmarking framework to hudi. and then
identify regressions on any key paths.
- Long running test suite
- Config clean ups in hudi client
- Perf test environment
- Nightly builds
As we plan out work in each of these sections, we are looking for help from
the community in getting these done. Plan is to put together a few umbrella
tickets for each of these areas and will have a coordinator. Coordinator
will be one who has expertise in the area of interest. Coordinator will
plan out the work in their resp area and will help drive the initiative
with help from the community depending on who volunteers to help out.
I understand the list is huge. Some work areas will be well defined and
should be able to get it done if we allocate enough time and resources. But
some are exploratory in nature and need some initial push to get the ball
rolling.
Very likely some of the work items in these would be well defined and
should be easy for new folks to contribute. We are not really having any
target timeframe in mind(as we had 1 month for bug bash), but would like to
get concrete work items done in decent time and have others ready by the
next major release(for eg, perf test env) depending on resources.
Let us know if you would be interested to help our community in this
regard.
--
Regards,
-Sivabalan
Re: DevX, Test infra Rgdn
Posted by Gary Li <ga...@outlook.com>.
+1. Well defined tasks will bring more contributors onboard. I believe there are a certain amount of Hudi users would love to contribute to the codebase.
Gary Li
________________________________
From: Bhavani Sudha <bh...@gmail.com>
Sent: Wednesday, September 2, 2020 3:04:26 AM
To: dev@hudi.apache.org <de...@hudi.apache.org>
Subject: Re: DevX, Test infra Rgdn
+1 This will definitely reduce time to capture regressions and pave way for
frequent release cycles.
On Mon, Aug 31, 2020 at 9:59 PM Balaji Varadarajan
<v....@ymail.com.invalid> wrote:
> +1. This would be a great contribution as all developers will benefit
> from this work.
> On Monday, August 31, 2020, 08:07:08 AM PDT, Vinoth Chandar <
> vinoth@apache.org> wrote:
>
> +1 this is a great way to also ramp on the code base
>
> On Sun, Aug 30, 2020 at 8:00 AM Sivabalan <n....@gmail.com> wrote:
>
> > As Hudi matures as a project, we need to get our devX and test infra rock
> > solid. Availability of test utils and base classes for ease of writing
> more
> > tests, stable integration tests, ease of debuggability, micro benchmarks,
> > performance test infra, automating checkstyle formatting, nightly
> snapshot
> > builds and so on.
> >
> > We have identified and categorized these into different areas as below.
> >
> > - Test fixes and some clean up. // There are a lot of jira tickets
> > lying around in this section.
> > - Test refactoring. // For ease of development, and reduce clutter, we
> need
> > to work on refactoring test infra like having more test utils, base
> classes
> > etc.
> > - More tests to improve coverage in some areas.
> > - CI stability and ease of debugging integration tests.
> > - Checkstyle, sl4j, warnings, spotless, etc.
> > - Micro benchmarks. // add benchmarking framework to hudi. and then
> > identify regressions on any key paths.
> > - Long running test suite
> > - Config clean ups in hudi client
> > - Perf test environment
> > - Nightly builds
> >
> > As we plan out work in each of these sections, we are looking for help
> from
> > the community in getting these done. Plan is to put together a few
> umbrella
> > tickets for each of these areas and will have a coordinator. Coordinator
> > will be one who has expertise in the area of interest. Coordinator will
> > plan out the work in their resp area and will help drive the initiative
> > with help from the community depending on who volunteers to help out.
> >
> > I understand the list is huge. Some work areas will be well defined and
> > should be able to get it done if we allocate enough time and resources.
> But
> > some are exploratory in nature and need some initial push to get the ball
> > rolling.
> >
> > Very likely some of the work items in these would be well defined and
> > should be easy for new folks to contribute. We are not really having any
> > target timeframe in mind(as we had 1 month for bug bash), but would like
> to
> > get concrete work items done in decent time and have others ready by the
> > next major release(for eg, perf test env) depending on resources.
> >
> > Let us know if you would be interested to help our community in this
> > regard.
> >
> > --
> > Regards,
> > -Sivabalan
> >
>
Re: DevX, Test infra Rgdn
Posted by Bhavani Sudha <bh...@gmail.com>.
+1 This will definitely reduce time to capture regressions and pave way for
frequent release cycles.
On Mon, Aug 31, 2020 at 9:59 PM Balaji Varadarajan
<v....@ymail.com.invalid> wrote:
> +1. This would be a great contribution as all developers will benefit
> from this work.
> On Monday, August 31, 2020, 08:07:08 AM PDT, Vinoth Chandar <
> vinoth@apache.org> wrote:
>
> +1 this is a great way to also ramp on the code base
>
> On Sun, Aug 30, 2020 at 8:00 AM Sivabalan <n....@gmail.com> wrote:
>
> > As Hudi matures as a project, we need to get our devX and test infra rock
> > solid. Availability of test utils and base classes for ease of writing
> more
> > tests, stable integration tests, ease of debuggability, micro benchmarks,
> > performance test infra, automating checkstyle formatting, nightly
> snapshot
> > builds and so on.
> >
> > We have identified and categorized these into different areas as below.
> >
> > - Test fixes and some clean up. // There are a lot of jira tickets
> > lying around in this section.
> > - Test refactoring. // For ease of development, and reduce clutter, we
> need
> > to work on refactoring test infra like having more test utils, base
> classes
> > etc.
> > - More tests to improve coverage in some areas.
> > - CI stability and ease of debugging integration tests.
> > - Checkstyle, sl4j, warnings, spotless, etc.
> > - Micro benchmarks. // add benchmarking framework to hudi. and then
> > identify regressions on any key paths.
> > - Long running test suite
> > - Config clean ups in hudi client
> > - Perf test environment
> > - Nightly builds
> >
> > As we plan out work in each of these sections, we are looking for help
> from
> > the community in getting these done. Plan is to put together a few
> umbrella
> > tickets for each of these areas and will have a coordinator. Coordinator
> > will be one who has expertise in the area of interest. Coordinator will
> > plan out the work in their resp area and will help drive the initiative
> > with help from the community depending on who volunteers to help out.
> >
> > I understand the list is huge. Some work areas will be well defined and
> > should be able to get it done if we allocate enough time and resources.
> But
> > some are exploratory in nature and need some initial push to get the ball
> > rolling.
> >
> > Very likely some of the work items in these would be well defined and
> > should be easy for new folks to contribute. We are not really having any
> > target timeframe in mind(as we had 1 month for bug bash), but would like
> to
> > get concrete work items done in decent time and have others ready by the
> > next major release(for eg, perf test env) depending on resources.
> >
> > Let us know if you would be interested to help our community in this
> > regard.
> >
> > --
> > Regards,
> > -Sivabalan
> >
>
Re: DevX, Test infra Rgdn
Posted by Balaji Varadarajan <v....@ymail.com.INVALID>.
+1. This would be a great contribution as all developers will benefit from this work.
On Monday, August 31, 2020, 08:07:08 AM PDT, Vinoth Chandar <vi...@apache.org> wrote:
+1 this is a great way to also ramp on the code base
On Sun, Aug 30, 2020 at 8:00 AM Sivabalan <n....@gmail.com> wrote:
> As Hudi matures as a project, we need to get our devX and test infra rock
> solid. Availability of test utils and base classes for ease of writing more
> tests, stable integration tests, ease of debuggability, micro benchmarks,
> performance test infra, automating checkstyle formatting, nightly snapshot
> builds and so on.
>
> We have identified and categorized these into different areas as below.
>
> - Test fixes and some clean up. // There are a lot of jira tickets
> lying around in this section.
> - Test refactoring. // For ease of development, and reduce clutter, we need
> to work on refactoring test infra like having more test utils, base classes
> etc.
> - More tests to improve coverage in some areas.
> - CI stability and ease of debugging integration tests.
> - Checkstyle, sl4j, warnings, spotless, etc.
> - Micro benchmarks. // add benchmarking framework to hudi. and then
> identify regressions on any key paths.
> - Long running test suite
> - Config clean ups in hudi client
> - Perf test environment
> - Nightly builds
>
> As we plan out work in each of these sections, we are looking for help from
> the community in getting these done. Plan is to put together a few umbrella
> tickets for each of these areas and will have a coordinator. Coordinator
> will be one who has expertise in the area of interest. Coordinator will
> plan out the work in their resp area and will help drive the initiative
> with help from the community depending on who volunteers to help out.
>
> I understand the list is huge. Some work areas will be well defined and
> should be able to get it done if we allocate enough time and resources. But
> some are exploratory in nature and need some initial push to get the ball
> rolling.
>
> Very likely some of the work items in these would be well defined and
> should be easy for new folks to contribute. We are not really having any
> target timeframe in mind(as we had 1 month for bug bash), but would like to
> get concrete work items done in decent time and have others ready by the
> next major release(for eg, perf test env) depending on resources.
>
> Let us know if you would be interested to help our community in this
> regard.
>
> --
> Regards,
> -Sivabalan
>
Re: DevX, Test infra Rgdn
Posted by Vinoth Chandar <vi...@apache.org>.
+1 this is a great way to also ramp on the code base
On Sun, Aug 30, 2020 at 8:00 AM Sivabalan <n....@gmail.com> wrote:
> As Hudi matures as a project, we need to get our devX and test infra rock
> solid. Availability of test utils and base classes for ease of writing more
> tests, stable integration tests, ease of debuggability, micro benchmarks,
> performance test infra, automating checkstyle formatting, nightly snapshot
> builds and so on.
>
> We have identified and categorized these into different areas as below.
>
> - Test fixes and some clean up. // There are a lot of jira tickets
> lying around in this section.
> - Test refactoring. // For ease of development, and reduce clutter, we need
> to work on refactoring test infra like having more test utils, base classes
> etc.
> - More tests to improve coverage in some areas.
> - CI stability and ease of debugging integration tests.
> - Checkstyle, sl4j, warnings, spotless, etc.
> - Micro benchmarks. // add benchmarking framework to hudi. and then
> identify regressions on any key paths.
> - Long running test suite
> - Config clean ups in hudi client
> - Perf test environment
> - Nightly builds
>
> As we plan out work in each of these sections, we are looking for help from
> the community in getting these done. Plan is to put together a few umbrella
> tickets for each of these areas and will have a coordinator. Coordinator
> will be one who has expertise in the area of interest. Coordinator will
> plan out the work in their resp area and will help drive the initiative
> with help from the community depending on who volunteers to help out.
>
> I understand the list is huge. Some work areas will be well defined and
> should be able to get it done if we allocate enough time and resources. But
> some are exploratory in nature and need some initial push to get the ball
> rolling.
>
> Very likely some of the work items in these would be well defined and
> should be easy for new folks to contribute. We are not really having any
> target timeframe in mind(as we had 1 month for bug bash), but would like to
> get concrete work items done in decent time and have others ready by the
> next major release(for eg, perf test env) depending on resources.
>
> Let us know if you would be interested to help our community in this
> regard.
>
> --
> Regards,
> -Sivabalan
>