You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@beam.apache.org by Anand Inguva via dev <de...@beam.apache.org> on 2023/04/12 04:35:28 UTC
[Python SDK] Use pre-released dependencies for Beam python unit testing
Hi all,
For Apache Beam Python we are considering using pre-released dependencies
for unit testing by using the --pre flag to install pre-released
dependencies of packages.
We believe that using pre-released dependencies may help us to identify and
resolve bugs more quickly, and to take advantage of new features or bug
fixes that are not yet available in stable releases. However, we also
understand that using pre-released dependencies may introduce new risks and
challenges, including potential code duplication and stability issues.
Before proceeding, we wanted to get your feedback on this approach.
1. Create a new PreCommit test suite and a PostCommit test suite that runs
tests by installing pre-released dependencies.
Pros:
- stable and pre-released test suites are separate and it will be easier
to debug if the pre-released test suite fails.
Cons:
- More test infra code to maintain. More tests to monitor.
2. Make use of the current PreCommit and PostCommit test suite and modify
it so that it installs pre-released dependencies.
Pros:
- Less infra code and less tests to monitor.
Cons:
- Leads to noisy test signals if the pre-release candidate is unstable.
I am in favor of approach 1 since this approach would ensure that any
issues encountered during pre-release testing do not impact the stable
release environment, and vice versa.
If you have experience or done any testing work using pre-released
dependencies, please let me know if you took any different approaches. It
will be really helpful.
Thanks,
Anand
Re: [Python SDK] Use pre-released dependencies for Beam python unit testing
Posted by Valentyn Tymofieiev via dev <de...@beam.apache.org>.
2. Make use of the current PreCommit and PostCommit test suite and modify
it so that it installs pre-released dependencies.
> Leads to noisy test signals if the pre-release candidate is unstable.
I am favor of option 2 since it's a simple solution that is easy to
implement and try out. The disadvantage rests on an assumption
that pre-released candidates would be unstable, which may not be the case.
We could try this and pivot if we find this create too much noise. @Jarek
Potiuk <ja...@potiuk.com> - curious, from your experience with Airflow
dependency management and testing, which option do you use (if you have a
similar scenario)?
On Wed, Apr 12, 2023 at 7:45 AM Danny McCormick via dev <de...@beam.apache.org>
wrote:
> Thanks for doing this Anand, I'm +1 on option 1 as well - I think having
> the clear signal of the normal suite succeeding and the prerelease one
> failing would be helpful and there shouldn't be too much additional code
> necessary. That makes it really easy to treat the prerelease suite as a (at
> least temporary) signal on needing upper bounds on our dependencies.
>
> Thanks,
> Danny
>
> On Wed, Apr 12, 2023 at 12:36 AM Anand Inguva via dev <de...@beam.apache.org>
> wrote:
>
>> Hi all,
>>
>> For Apache Beam Python we are considering using pre-released dependencies
>> for unit testing by using the --pre flag to install pre-released
>> dependencies of packages.
>>
>> We believe that using pre-released dependencies may help us to identify
>> and resolve bugs more quickly, and to take advantage of new features or bug
>> fixes that are not yet available in stable releases. However, we also
>> understand that using pre-released dependencies may introduce new risks and
>> challenges, including potential code duplication and stability issues.
>>
>> Before proceeding, we wanted to get your feedback on this approach.
>>
>> 1. Create a new PreCommit test suite and a PostCommit test suite that
>> runs tests by installing pre-released dependencies.
>>
>> Pros:
>>
>> - stable and pre-released test suites are separate and it will be
>> easier to debug if the pre-released test suite fails.
>>
>> Cons:
>>
>> - More test infra code to maintain. More tests to monitor.
>>
>>
>> 2. Make use of the current PreCommit and PostCommit test suite and modify
>> it so that it installs pre-released dependencies.
>>
>> Pros:
>>
>> - Less infra code and less tests to monitor.
>>
>> Cons:
>>
>> - Leads to noisy test signals if the pre-release candidate is
>> unstable.
>>
>> I am in favor of approach 1 since this approach would ensure that any
>> issues encountered during pre-release testing do not impact the stable
>> release environment, and vice versa.
>>
>> If you have experience or done any testing work using pre-released
>> dependencies, please let me know if you took any different approaches. It
>> will be really helpful.
>>
>> Thanks,
>> Anand
>>
>
Re: [Python SDK] Use pre-released dependencies for Beam python unit testing
Posted by Robert Bradshaw via dev <de...@beam.apache.org>.
The main concern here seems to be whether using pre-release candidates
would be too disruptive to our workflow. I think this is an easy
hypothesis to test out--we can give using prerelease candidates a try, and
if that indeed turns out to be a problem we can then do the work of
trying to put together a separate test suite (TBD the balance between
coverage and cost).
On Thu, Apr 20, 2023 at 10:20 AM Anand Inguva via dev <de...@beam.apache.org>
wrote:
> Friendly ping:)
>
> If anyone has experience testing the pre-released versions, your feedback
> would be much appreciated.
>
> Thanks,
> Anand
>
>
>
> On Wed, Apr 12, 2023 at 5:07 PM Yi Hu <ya...@google.com> wrote:
>
>> Sounds good, thanks!
>>
>> Best,
>> Yi
>>
>> On Wed, Apr 12, 2023 at 2:20 PM Anand Inguva <an...@google.com>
>> wrote:
>>
>>> @Yi Hu <ya...@google.com> I think adding them to Jenkins or github
>>> actions is okay with me. With Github actions, since we don't use self
>>> hosted runners yet, I worry that action workers might get queued up.
>>>
>>> Also, I plan to not run these on every commit but run it as a cron
>>> job(maybe once per day) and also as trigger phrases and only on the lowest
>>> and highest python version. Also, migrating this workflow to jenkins would
>>> be trivial in the future once beam starts the migration. For now, I think
>>> it might be best to run on jenkins.
>>>
>>> On Wed, Apr 12, 2023 at 1:32 PM Valentyn Tymofieiev <va...@google.com>
>>> wrote:
>>>
>>>> I think case in point dependency that would benefit from this testing
>>>> is grpcio, which includes pre-releases, and broke us and multiple of it's
>>>> released versions were yanked. https://pypi.org/project/grpcio/#history
>>>> .
>>>>
>>>> We can look at how grpcio affected Beam previously. Couple of issues:
>>>>
>>>> - https://github.com/grpc/grpc/issues/30446 -- affected XLang tests
>>>> - https://github.com/apache/beam/issues/23734 -- affected MacOS suites
>>>> - https://github.com/apache/beam/issues/22159 -- (not detected by us,
>>>> but potentially could have affected a performance test).
>>>>
>>>> I'm afraid a dedicated suite may not give us desired test coverage to
>>>> catch regression at RC stage.
>>>>
>>>> On Wed, Apr 12, 2023 at 10:19 AM Yi Hu via dev <de...@beam.apache.org>
>>>> wrote:
>>>>
>>>>> Thanks Anand,
>>>>>
>>>>> This would be very helpful to avoid experiencing multiple time (
>>>>> https://s.apache.org/beam-python-dependencies-pm). One thing to note
>>>>> is that Beam Jenkins CI is experiencing many issues recently, mostly due to
>>>>> that multiple Jenkins plugins does not scale (draining GitHub API call
>>>>> limit; disk usage, etc) so more PreCommit may add more pressures to Jenkins
>>>>> if going ahead with Option 1. As we have started GitHub Action migration,
>>>>> is it considered to add these new tests to GitHub Action?
>>>>>
>>>>> Best,
>>>>> Yi
>>>>>
>>>>> On Wed, Apr 12, 2023 at 10:46 AM Danny McCormick via dev <
>>>>> dev@beam.apache.org> wrote:
>>>>>
>>>>>> Thanks for doing this Anand, I'm +1 on option 1 as well - I think
>>>>>> having the clear signal of the normal suite succeeding and the prerelease
>>>>>> one failing would be helpful and there shouldn't be too much additional
>>>>>> code necessary. That makes it really easy to treat the prerelease suite as
>>>>>> a (at least temporary) signal on needing upper bounds on our dependencies.
>>>>>>
>>>>>> Thanks,
>>>>>> Danny
>>>>>>
>>>>>> On Wed, Apr 12, 2023 at 12:36 AM Anand Inguva via dev <
>>>>>> dev@beam.apache.org> wrote:
>>>>>>
>>>>>>> Hi all,
>>>>>>>
>>>>>>> For Apache Beam Python we are considering using pre-released
>>>>>>> dependencies for unit testing by using the --pre flag to install
>>>>>>> pre-released dependencies of packages.
>>>>>>>
>>>>>>> We believe that using pre-released dependencies may help us to
>>>>>>> identify and resolve bugs more quickly, and to take advantage of new
>>>>>>> features or bug fixes that are not yet available in stable releases.
>>>>>>> However, we also understand that using pre-released dependencies may
>>>>>>> introduce new risks and challenges, including potential code duplication
>>>>>>> and stability issues.
>>>>>>>
>>>>>>> Before proceeding, we wanted to get your feedback on this approach.
>>>>>>>
>>>>>>> 1. Create a new PreCommit test suite and a PostCommit test suite
>>>>>>> that runs tests by installing pre-released dependencies.
>>>>>>>
>>>>>>> Pros:
>>>>>>>
>>>>>>> - stable and pre-released test suites are separate and it will
>>>>>>> be easier to debug if the pre-released test suite fails.
>>>>>>>
>>>>>>> Cons:
>>>>>>>
>>>>>>> - More test infra code to maintain. More tests to monitor.
>>>>>>>
>>>>>>>
>>>>>>> 2. Make use of the current PreCommit and PostCommit test suite and
>>>>>>> modify it so that it installs pre-released dependencies.
>>>>>>>
>>>>>>> Pros:
>>>>>>>
>>>>>>> - Less infra code and less tests to monitor.
>>>>>>>
>>>>>>> Cons:
>>>>>>>
>>>>>>> - Leads to noisy test signals if the pre-release candidate is
>>>>>>> unstable.
>>>>>>>
>>>>>>> I am in favor of approach 1 since this approach would ensure that
>>>>>>> any issues encountered during pre-release testing do not impact the stable
>>>>>>> release environment, and vice versa.
>>>>>>>
>>>>>>> If you have experience or done any testing work using pre-released
>>>>>>> dependencies, please let me know if you took any different approaches. It
>>>>>>> will be really helpful.
>>>>>>>
>>>>>>> Thanks,
>>>>>>> Anand
>>>>>>>
>>>>>>
Re: [Python SDK] Use pre-released dependencies for Beam python unit testing
Posted by Anand Inguva via dev <de...@beam.apache.org>.
Friendly ping:)
If anyone has experience testing the pre-released versions, your feedback
would be much appreciated.
Thanks,
Anand
On Wed, Apr 12, 2023 at 5:07 PM Yi Hu <ya...@google.com> wrote:
> Sounds good, thanks!
>
> Best,
> Yi
>
> On Wed, Apr 12, 2023 at 2:20 PM Anand Inguva <an...@google.com>
> wrote:
>
>> @Yi Hu <ya...@google.com> I think adding them to Jenkins or github
>> actions is okay with me. With Github actions, since we don't use self
>> hosted runners yet, I worry that action workers might get queued up.
>>
>> Also, I plan to not run these on every commit but run it as a cron
>> job(maybe once per day) and also as trigger phrases and only on the lowest
>> and highest python version. Also, migrating this workflow to jenkins would
>> be trivial in the future once beam starts the migration. For now, I think
>> it might be best to run on jenkins.
>>
>> On Wed, Apr 12, 2023 at 1:32 PM Valentyn Tymofieiev <va...@google.com>
>> wrote:
>>
>>> I think case in point dependency that would benefit from this testing is
>>> grpcio, which includes pre-releases, and broke us and multiple of it's
>>> released versions were yanked. https://pypi.org/project/grpcio/#history
>>> .
>>>
>>> We can look at how grpcio affected Beam previously. Couple of issues:
>>>
>>> - https://github.com/grpc/grpc/issues/30446 -- affected XLang tests
>>> - https://github.com/apache/beam/issues/23734 -- affected MacOS suites
>>> - https://github.com/apache/beam/issues/22159 -- (not detected by us,
>>> but potentially could have affected a performance test).
>>>
>>> I'm afraid a dedicated suite may not give us desired test coverage to
>>> catch regression at RC stage.
>>>
>>> On Wed, Apr 12, 2023 at 10:19 AM Yi Hu via dev <de...@beam.apache.org>
>>> wrote:
>>>
>>>> Thanks Anand,
>>>>
>>>> This would be very helpful to avoid experiencing multiple time (
>>>> https://s.apache.org/beam-python-dependencies-pm). One thing to note
>>>> is that Beam Jenkins CI is experiencing many issues recently, mostly due to
>>>> that multiple Jenkins plugins does not scale (draining GitHub API call
>>>> limit; disk usage, etc) so more PreCommit may add more pressures to Jenkins
>>>> if going ahead with Option 1. As we have started GitHub Action migration,
>>>> is it considered to add these new tests to GitHub Action?
>>>>
>>>> Best,
>>>> Yi
>>>>
>>>> On Wed, Apr 12, 2023 at 10:46 AM Danny McCormick via dev <
>>>> dev@beam.apache.org> wrote:
>>>>
>>>>> Thanks for doing this Anand, I'm +1 on option 1 as well - I think
>>>>> having the clear signal of the normal suite succeeding and the prerelease
>>>>> one failing would be helpful and there shouldn't be too much additional
>>>>> code necessary. That makes it really easy to treat the prerelease suite as
>>>>> a (at least temporary) signal on needing upper bounds on our dependencies.
>>>>>
>>>>> Thanks,
>>>>> Danny
>>>>>
>>>>> On Wed, Apr 12, 2023 at 12:36 AM Anand Inguva via dev <
>>>>> dev@beam.apache.org> wrote:
>>>>>
>>>>>> Hi all,
>>>>>>
>>>>>> For Apache Beam Python we are considering using pre-released
>>>>>> dependencies for unit testing by using the --pre flag to install
>>>>>> pre-released dependencies of packages.
>>>>>>
>>>>>> We believe that using pre-released dependencies may help us to
>>>>>> identify and resolve bugs more quickly, and to take advantage of new
>>>>>> features or bug fixes that are not yet available in stable releases.
>>>>>> However, we also understand that using pre-released dependencies may
>>>>>> introduce new risks and challenges, including potential code duplication
>>>>>> and stability issues.
>>>>>>
>>>>>> Before proceeding, we wanted to get your feedback on this approach.
>>>>>>
>>>>>> 1. Create a new PreCommit test suite and a PostCommit test suite that
>>>>>> runs tests by installing pre-released dependencies.
>>>>>>
>>>>>> Pros:
>>>>>>
>>>>>> - stable and pre-released test suites are separate and it will be
>>>>>> easier to debug if the pre-released test suite fails.
>>>>>>
>>>>>> Cons:
>>>>>>
>>>>>> - More test infra code to maintain. More tests to monitor.
>>>>>>
>>>>>>
>>>>>> 2. Make use of the current PreCommit and PostCommit test suite and
>>>>>> modify it so that it installs pre-released dependencies.
>>>>>>
>>>>>> Pros:
>>>>>>
>>>>>> - Less infra code and less tests to monitor.
>>>>>>
>>>>>> Cons:
>>>>>>
>>>>>> - Leads to noisy test signals if the pre-release candidate is
>>>>>> unstable.
>>>>>>
>>>>>> I am in favor of approach 1 since this approach would ensure that any
>>>>>> issues encountered during pre-release testing do not impact the stable
>>>>>> release environment, and vice versa.
>>>>>>
>>>>>> If you have experience or done any testing work using pre-released
>>>>>> dependencies, please let me know if you took any different approaches. It
>>>>>> will be really helpful.
>>>>>>
>>>>>> Thanks,
>>>>>> Anand
>>>>>>
>>>>>
Re: [Python SDK] Use pre-released dependencies for Beam python unit testing
Posted by Yi Hu via dev <de...@beam.apache.org>.
Sounds good, thanks!
Best,
Yi
On Wed, Apr 12, 2023 at 2:20 PM Anand Inguva <an...@google.com> wrote:
> @Yi Hu <ya...@google.com> I think adding them to Jenkins or github
> actions is okay with me. With Github actions, since we don't use self
> hosted runners yet, I worry that action workers might get queued up.
>
> Also, I plan to not run these on every commit but run it as a cron
> job(maybe once per day) and also as trigger phrases and only on the lowest
> and highest python version. Also, migrating this workflow to jenkins would
> be trivial in the future once beam starts the migration. For now, I think
> it might be best to run on jenkins.
>
> On Wed, Apr 12, 2023 at 1:32 PM Valentyn Tymofieiev <va...@google.com>
> wrote:
>
>> I think case in point dependency that would benefit from this testing is
>> grpcio, which includes pre-releases, and broke us and multiple of it's
>> released versions were yanked. https://pypi.org/project/grpcio/#history .
>>
>> We can look at how grpcio affected Beam previously. Couple of issues:
>>
>> - https://github.com/grpc/grpc/issues/30446 -- affected XLang tests
>> - https://github.com/apache/beam/issues/23734 -- affected MacOS suites
>> - https://github.com/apache/beam/issues/22159 -- (not detected by us,
>> but potentially could have affected a performance test).
>>
>> I'm afraid a dedicated suite may not give us desired test coverage to
>> catch regression at RC stage.
>>
>> On Wed, Apr 12, 2023 at 10:19 AM Yi Hu via dev <de...@beam.apache.org>
>> wrote:
>>
>>> Thanks Anand,
>>>
>>> This would be very helpful to avoid experiencing multiple time (
>>> https://s.apache.org/beam-python-dependencies-pm). One thing to note is
>>> that Beam Jenkins CI is experiencing many issues recently, mostly due to
>>> that multiple Jenkins plugins does not scale (draining GitHub API call
>>> limit; disk usage, etc) so more PreCommit may add more pressures to Jenkins
>>> if going ahead with Option 1. As we have started GitHub Action migration,
>>> is it considered to add these new tests to GitHub Action?
>>>
>>> Best,
>>> Yi
>>>
>>> On Wed, Apr 12, 2023 at 10:46 AM Danny McCormick via dev <
>>> dev@beam.apache.org> wrote:
>>>
>>>> Thanks for doing this Anand, I'm +1 on option 1 as well - I think
>>>> having the clear signal of the normal suite succeeding and the prerelease
>>>> one failing would be helpful and there shouldn't be too much additional
>>>> code necessary. That makes it really easy to treat the prerelease suite as
>>>> a (at least temporary) signal on needing upper bounds on our dependencies.
>>>>
>>>> Thanks,
>>>> Danny
>>>>
>>>> On Wed, Apr 12, 2023 at 12:36 AM Anand Inguva via dev <
>>>> dev@beam.apache.org> wrote:
>>>>
>>>>> Hi all,
>>>>>
>>>>> For Apache Beam Python we are considering using pre-released
>>>>> dependencies for unit testing by using the --pre flag to install
>>>>> pre-released dependencies of packages.
>>>>>
>>>>> We believe that using pre-released dependencies may help us to
>>>>> identify and resolve bugs more quickly, and to take advantage of new
>>>>> features or bug fixes that are not yet available in stable releases.
>>>>> However, we also understand that using pre-released dependencies may
>>>>> introduce new risks and challenges, including potential code duplication
>>>>> and stability issues.
>>>>>
>>>>> Before proceeding, we wanted to get your feedback on this approach.
>>>>>
>>>>> 1. Create a new PreCommit test suite and a PostCommit test suite that
>>>>> runs tests by installing pre-released dependencies.
>>>>>
>>>>> Pros:
>>>>>
>>>>> - stable and pre-released test suites are separate and it will be
>>>>> easier to debug if the pre-released test suite fails.
>>>>>
>>>>> Cons:
>>>>>
>>>>> - More test infra code to maintain. More tests to monitor.
>>>>>
>>>>>
>>>>> 2. Make use of the current PreCommit and PostCommit test suite and
>>>>> modify it so that it installs pre-released dependencies.
>>>>>
>>>>> Pros:
>>>>>
>>>>> - Less infra code and less tests to monitor.
>>>>>
>>>>> Cons:
>>>>>
>>>>> - Leads to noisy test signals if the pre-release candidate is
>>>>> unstable.
>>>>>
>>>>> I am in favor of approach 1 since this approach would ensure that any
>>>>> issues encountered during pre-release testing do not impact the stable
>>>>> release environment, and vice versa.
>>>>>
>>>>> If you have experience or done any testing work using pre-released
>>>>> dependencies, please let me know if you took any different approaches. It
>>>>> will be really helpful.
>>>>>
>>>>> Thanks,
>>>>> Anand
>>>>>
>>>>
Re: [Python SDK] Use pre-released dependencies for Beam python unit testing
Posted by Anand Inguva via dev <de...@beam.apache.org>.
@Yi Hu <ya...@google.com> I think adding them to Jenkins or github actions
is okay with me. With Github actions, since we don't use self hosted
runners yet, I worry that action workers might get queued up.
Also, I plan to not run these on every commit but run it as a cron
job(maybe once per day) and also as trigger phrases and only on the lowest
and highest python version. Also, migrating this workflow to jenkins would
be trivial in the future once beam starts the migration. For now, I think
it might be best to run on jenkins.
On Wed, Apr 12, 2023 at 1:32 PM Valentyn Tymofieiev <va...@google.com>
wrote:
> I think case in point dependency that would benefit from this testing is
> grpcio, which includes pre-releases, and broke us and multiple of it's
> released versions were yanked. https://pypi.org/project/grpcio/#history .
>
> We can look at how grpcio affected Beam previously. Couple of issues:
>
> - https://github.com/grpc/grpc/issues/30446 -- affected XLang tests
> - https://github.com/apache/beam/issues/23734 -- affected MacOS suites
> - https://github.com/apache/beam/issues/22159 -- (not detected by us, but
> potentially could have affected a performance test).
>
> I'm afraid a dedicated suite may not give us desired test coverage to
> catch regression at RC stage.
>
> On Wed, Apr 12, 2023 at 10:19 AM Yi Hu via dev <de...@beam.apache.org>
> wrote:
>
>> Thanks Anand,
>>
>> This would be very helpful to avoid experiencing multiple time (
>> https://s.apache.org/beam-python-dependencies-pm). One thing to note is
>> that Beam Jenkins CI is experiencing many issues recently, mostly due to
>> that multiple Jenkins plugins does not scale (draining GitHub API call
>> limit; disk usage, etc) so more PreCommit may add more pressures to Jenkins
>> if going ahead with Option 1. As we have started GitHub Action migration,
>> is it considered to add these new tests to GitHub Action?
>>
>> Best,
>> Yi
>>
>> On Wed, Apr 12, 2023 at 10:46 AM Danny McCormick via dev <
>> dev@beam.apache.org> wrote:
>>
>>> Thanks for doing this Anand, I'm +1 on option 1 as well - I think having
>>> the clear signal of the normal suite succeeding and the prerelease one
>>> failing would be helpful and there shouldn't be too much additional code
>>> necessary. That makes it really easy to treat the prerelease suite as a (at
>>> least temporary) signal on needing upper bounds on our dependencies.
>>>
>>> Thanks,
>>> Danny
>>>
>>> On Wed, Apr 12, 2023 at 12:36 AM Anand Inguva via dev <
>>> dev@beam.apache.org> wrote:
>>>
>>>> Hi all,
>>>>
>>>> For Apache Beam Python we are considering using pre-released
>>>> dependencies for unit testing by using the --pre flag to install
>>>> pre-released dependencies of packages.
>>>>
>>>> We believe that using pre-released dependencies may help us to identify
>>>> and resolve bugs more quickly, and to take advantage of new features or bug
>>>> fixes that are not yet available in stable releases. However, we also
>>>> understand that using pre-released dependencies may introduce new risks and
>>>> challenges, including potential code duplication and stability issues.
>>>>
>>>> Before proceeding, we wanted to get your feedback on this approach.
>>>>
>>>> 1. Create a new PreCommit test suite and a PostCommit test suite that
>>>> runs tests by installing pre-released dependencies.
>>>>
>>>> Pros:
>>>>
>>>> - stable and pre-released test suites are separate and it will be
>>>> easier to debug if the pre-released test suite fails.
>>>>
>>>> Cons:
>>>>
>>>> - More test infra code to maintain. More tests to monitor.
>>>>
>>>>
>>>> 2. Make use of the current PreCommit and PostCommit test suite and
>>>> modify it so that it installs pre-released dependencies.
>>>>
>>>> Pros:
>>>>
>>>> - Less infra code and less tests to monitor.
>>>>
>>>> Cons:
>>>>
>>>> - Leads to noisy test signals if the pre-release candidate is
>>>> unstable.
>>>>
>>>> I am in favor of approach 1 since this approach would ensure that any
>>>> issues encountered during pre-release testing do not impact the stable
>>>> release environment, and vice versa.
>>>>
>>>> If you have experience or done any testing work using pre-released
>>>> dependencies, please let me know if you took any different approaches. It
>>>> will be really helpful.
>>>>
>>>> Thanks,
>>>> Anand
>>>>
>>>
Re: [Python SDK] Use pre-released dependencies for Beam python unit testing
Posted by Valentyn Tymofieiev via dev <de...@beam.apache.org>.
I think case in point dependency that would benefit from this testing is
grpcio, which includes pre-releases, and broke us and multiple of it's
released versions were yanked. https://pypi.org/project/grpcio/#history .
We can look at how grpcio affected Beam previously. Couple of issues:
- https://github.com/grpc/grpc/issues/30446 -- affected XLang tests
- https://github.com/apache/beam/issues/23734 -- affected MacOS suites
- https://github.com/apache/beam/issues/22159 -- (not detected by us, but
potentially could have affected a performance test).
I'm afraid a dedicated suite may not give us desired test coverage to catch
regression at RC stage.
On Wed, Apr 12, 2023 at 10:19 AM Yi Hu via dev <de...@beam.apache.org> wrote:
> Thanks Anand,
>
> This would be very helpful to avoid experiencing multiple time (
> https://s.apache.org/beam-python-dependencies-pm). One thing to note is
> that Beam Jenkins CI is experiencing many issues recently, mostly due to
> that multiple Jenkins plugins does not scale (draining GitHub API call
> limit; disk usage, etc) so more PreCommit may add more pressures to Jenkins
> if going ahead with Option 1. As we have started GitHub Action migration,
> is it considered to add these new tests to GitHub Action?
>
> Best,
> Yi
>
> On Wed, Apr 12, 2023 at 10:46 AM Danny McCormick via dev <
> dev@beam.apache.org> wrote:
>
>> Thanks for doing this Anand, I'm +1 on option 1 as well - I think having
>> the clear signal of the normal suite succeeding and the prerelease one
>> failing would be helpful and there shouldn't be too much additional code
>> necessary. That makes it really easy to treat the prerelease suite as a (at
>> least temporary) signal on needing upper bounds on our dependencies.
>>
>> Thanks,
>> Danny
>>
>> On Wed, Apr 12, 2023 at 12:36 AM Anand Inguva via dev <
>> dev@beam.apache.org> wrote:
>>
>>> Hi all,
>>>
>>> For Apache Beam Python we are considering using pre-released
>>> dependencies for unit testing by using the --pre flag to install
>>> pre-released dependencies of packages.
>>>
>>> We believe that using pre-released dependencies may help us to identify
>>> and resolve bugs more quickly, and to take advantage of new features or bug
>>> fixes that are not yet available in stable releases. However, we also
>>> understand that using pre-released dependencies may introduce new risks and
>>> challenges, including potential code duplication and stability issues.
>>>
>>> Before proceeding, we wanted to get your feedback on this approach.
>>>
>>> 1. Create a new PreCommit test suite and a PostCommit test suite that
>>> runs tests by installing pre-released dependencies.
>>>
>>> Pros:
>>>
>>> - stable and pre-released test suites are separate and it will be
>>> easier to debug if the pre-released test suite fails.
>>>
>>> Cons:
>>>
>>> - More test infra code to maintain. More tests to monitor.
>>>
>>>
>>> 2. Make use of the current PreCommit and PostCommit test suite and
>>> modify it so that it installs pre-released dependencies.
>>>
>>> Pros:
>>>
>>> - Less infra code and less tests to monitor.
>>>
>>> Cons:
>>>
>>> - Leads to noisy test signals if the pre-release candidate is
>>> unstable.
>>>
>>> I am in favor of approach 1 since this approach would ensure that any
>>> issues encountered during pre-release testing do not impact the stable
>>> release environment, and vice versa.
>>>
>>> If you have experience or done any testing work using pre-released
>>> dependencies, please let me know if you took any different approaches. It
>>> will be really helpful.
>>>
>>> Thanks,
>>> Anand
>>>
>>
Re: [Python SDK] Use pre-released dependencies for Beam python unit testing
Posted by Yi Hu via dev <de...@beam.apache.org>.
Thanks Anand,
This would be very helpful to avoid experiencing multiple time (
https://s.apache.org/beam-python-dependencies-pm). One thing to note is
that Beam Jenkins CI is experiencing many issues recently, mostly due to
that multiple Jenkins plugins does not scale (draining GitHub API call
limit; disk usage, etc) so more PreCommit may add more pressures to Jenkins
if going ahead with Option 1. As we have started GitHub Action migration,
is it considered to add these new tests to GitHub Action?
Best,
Yi
On Wed, Apr 12, 2023 at 10:46 AM Danny McCormick via dev <
dev@beam.apache.org> wrote:
> Thanks for doing this Anand, I'm +1 on option 1 as well - I think having
> the clear signal of the normal suite succeeding and the prerelease one
> failing would be helpful and there shouldn't be too much additional code
> necessary. That makes it really easy to treat the prerelease suite as a (at
> least temporary) signal on needing upper bounds on our dependencies.
>
> Thanks,
> Danny
>
> On Wed, Apr 12, 2023 at 12:36 AM Anand Inguva via dev <de...@beam.apache.org>
> wrote:
>
>> Hi all,
>>
>> For Apache Beam Python we are considering using pre-released dependencies
>> for unit testing by using the --pre flag to install pre-released
>> dependencies of packages.
>>
>> We believe that using pre-released dependencies may help us to identify
>> and resolve bugs more quickly, and to take advantage of new features or bug
>> fixes that are not yet available in stable releases. However, we also
>> understand that using pre-released dependencies may introduce new risks and
>> challenges, including potential code duplication and stability issues.
>>
>> Before proceeding, we wanted to get your feedback on this approach.
>>
>> 1. Create a new PreCommit test suite and a PostCommit test suite that
>> runs tests by installing pre-released dependencies.
>>
>> Pros:
>>
>> - stable and pre-released test suites are separate and it will be
>> easier to debug if the pre-released test suite fails.
>>
>> Cons:
>>
>> - More test infra code to maintain. More tests to monitor.
>>
>>
>> 2. Make use of the current PreCommit and PostCommit test suite and modify
>> it so that it installs pre-released dependencies.
>>
>> Pros:
>>
>> - Less infra code and less tests to monitor.
>>
>> Cons:
>>
>> - Leads to noisy test signals if the pre-release candidate is
>> unstable.
>>
>> I am in favor of approach 1 since this approach would ensure that any
>> issues encountered during pre-release testing do not impact the stable
>> release environment, and vice versa.
>>
>> If you have experience or done any testing work using pre-released
>> dependencies, please let me know if you took any different approaches. It
>> will be really helpful.
>>
>> Thanks,
>> Anand
>>
>
Re: [Python SDK] Use pre-released dependencies for Beam python unit testing
Posted by Danny McCormick via dev <de...@beam.apache.org>.
Thanks for doing this Anand, I'm +1 on option 1 as well - I think having
the clear signal of the normal suite succeeding and the prerelease one
failing would be helpful and there shouldn't be too much additional code
necessary. That makes it really easy to treat the prerelease suite as a (at
least temporary) signal on needing upper bounds on our dependencies.
Thanks,
Danny
On Wed, Apr 12, 2023 at 12:36 AM Anand Inguva via dev <de...@beam.apache.org>
wrote:
> Hi all,
>
> For Apache Beam Python we are considering using pre-released dependencies
> for unit testing by using the --pre flag to install pre-released
> dependencies of packages.
>
> We believe that using pre-released dependencies may help us to identify
> and resolve bugs more quickly, and to take advantage of new features or bug
> fixes that are not yet available in stable releases. However, we also
> understand that using pre-released dependencies may introduce new risks and
> challenges, including potential code duplication and stability issues.
>
> Before proceeding, we wanted to get your feedback on this approach.
>
> 1. Create a new PreCommit test suite and a PostCommit test suite that runs
> tests by installing pre-released dependencies.
>
> Pros:
>
> - stable and pre-released test suites are separate and it will be
> easier to debug if the pre-released test suite fails.
>
> Cons:
>
> - More test infra code to maintain. More tests to monitor.
>
>
> 2. Make use of the current PreCommit and PostCommit test suite and modify
> it so that it installs pre-released dependencies.
>
> Pros:
>
> - Less infra code and less tests to monitor.
>
> Cons:
>
> - Leads to noisy test signals if the pre-release candidate is
> unstable.
>
> I am in favor of approach 1 since this approach would ensure that any
> issues encountered during pre-release testing do not impact the stable
> release environment, and vice versa.
>
> If you have experience or done any testing work using pre-released
> dependencies, please let me know if you took any different approaches. It
> will be really helpful.
>
> Thanks,
> Anand
>