You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@heron.apache.org by Nicholas Nezis <ni...@gmail.com> on 2022/01/21 21:10:45 UTC
Python 3.8 as minimum
Heron Devs,
We are running into issues with the older Python 3.6 that seems to be
resolved in more modern versions of Python. To move things forward and get
our release working, I'm upgrading to Python 3.8 in our older Docker
container flavors.
Much of the work is being done off of Oliver's previous PR to upgrade us to
Python 3.9 support. We updated various Pex dependencies (and other
python dependencies such as PyYAML). Please review the latest changes in
the PR here: https://github.com/apache/incubator-heron/pull/3646
I'm setting the minimum Python in `bazel_configure.sh` to 3.8. Is everyone
ok with this change?
Some of the key changes:
Centos 7 -> Centos8 (Python 3.6 -> Python 3.9)
Ubuntu 18.04 (Python 3.6 -> Python 3.8)
I didn't really want to update Centos 7 to 8. But I don't think anyone is
using it. The Python 3.9 packages did not exist in the older image flavor.
I eventually want to upgrade this to Rocky or Alma Linux flavor or maybe
even RedHat's UBI image which could also help solve the OpenJDK issue (i.e.
ubi8/openjdk-11). But for now I'm trying to do the least amount of changes
to get us over the release finish line.
I've also cleaned up a lot of the documentation references to older flavors
that no longer exist (ubuntu14.04 and debian9).
Re: Python 3.8 as minimum
Posted by Ning Wang <wa...@gmail.com>.
Huge progress! Thanks!
On Mon, Jan 24, 2022 at 1:41 PM Saad Ur Rahman <sa...@apache.org>
wrote:
> Monumental work!
>
> On Mon, Jan 24, 2022 at 2:51 PM Josh Fischer <jo...@joshfischer.io> wrote:
>
> > +1 Nick. I’ll try to check it later this weekS
> >
> > On Mon, Jan 24, 2022 at 1:42 PM Nicholas Nezis <nicholas.nezis@gmail.com
> >
> > wrote:
> >
> > > We finally have a fully green Travis CI build with all of the new
> Python
> > > 3.8+ updates. The PR is here:
> > > https://github.com/apache/incubator-heron/pull/3646
> > >
> > > This PR started with Oliver's awesome updates he contributed back in
> > > December 2020. The PR is actually a merge of two different PRs (The
> other
> > > PR: https://github.com/apache/incubator-heron/pull/3642). There were a
> > > number of different changes needed to make it all work. For the items
> > that
> > > were not directly tied to this PR, I submitted separate PRs as best I
> > > could. This was not always possible. Below is a summary of what is in
> > this
> > > larger PR:
> > >
> > > - All of the awesome Python updates provided by Oliver
> > > - Tracker code was simplified with more modern Python libraries
> > > - Upgraded Travis image to Ubuntu 20.04 to get newer Python packages
> > > - Upgraded Python 3.8 as minimum version
> > > - Collapsed `--config` targets to `linux` and `darwin` (no more
> > > `ubuntu`, `centos` and `darwin` in bazel.rc config file)
> > > - Upgraded various Pex dependencies
> > > - Upgraded `rules_python` (0.0.2 -> 0.1.0)
> > > - Upgraded the `io_bazel_rules_docker` (0.14.4 -> 0.22.0). This was
> > > dependent on `rules_python`
> > > - Debian 9 docker image is now Debian 10
> > > - Centos 7 docker image is now Centos 8
> > >
> > >
> > > On Fri, Jan 21, 2022 at 7:08 PM Saad Ur Rahman <
> saadurrahman@apache.org>
> > > wrote:
> > >
> > > > I do not think there is an issue with consolidating `--config` to
> > `linux`
> > > > and `darwin`.
> > > >
> > > > For the libraries on Linux systems, you need to link in the libraries
> > > > manually, at least in the older compilers. For instance, if you are
> > using
> > > > the underlying POSIX threading API you will need to link in the
> > pthreads
> > > > library. Removing these libraries from linking may break the build on
> > > > Linux.
> > > >
> > > > On Fri, Jan 21, 2022 at 6:58 PM Nicholas Nezis <
> > nicholas.nezis@gmail.com
> > > >
> > > > wrote:
> > > >
> > > > > For the 3 libraries, it seems they are already included in macOS's
> > main
> > > > > system library. So keeping the Darwin/Linux split in bazel.rc still
> > > makes
> > > > > sense.
> > > > >
> > > > > On Fri, Jan 21, 2022 at 6:23 PM Nicholas Nezis <
> > > nicholas.nezis@gmail.com
> > > > >
> > > > > wrote:
> > > > >
> > > > > > Thanks guys. Making good progress.
> > > > > >
> > > > > > Found an issue with this `ci.sh` script.
> > > > > >
> > > > >
> > > >
> > >
> >
> https://github.com/apache/incubator-heron/blob/c23dba7a5d6982d38bb786fb54462bfa51688929/scripts/shutils/common.sh#L95-L108
> > > > > >
> > > > > > The platform was not detected properly. The strings never match
> > what
> > > > > > `python3 -mplatform` outputs. I've decided to simplify the whole
> > > flavor
> > > > > > thing in our Bazel build. From now on we'll have `linux` and
> > `darwin`
> > > > as
> > > > > > `--config` options. We'll still have different target platforms
> as
> > we
> > > > > build
> > > > > > docker images, but there have been no differences in config for
> > years
> > > > > since
> > > > > > I've joined the project. So I'm going to simplify things. Changes
> > > going
> > > > > > into the same PR.
> > > > > >
> > > > > > These are the only 3 lines that differ between linux and darwin
> in
> > > the
> > > > > > bazel.rc file. Do we even need these flags? Should the flags also
> > be
> > > on
> > > > > the
> > > > > > darwin build?
> > > > > >
> > > > >
> > > >
> > >
> >
> https://github.com/apache/incubator-heron/blob/c23dba7a5d6982d38bb786fb54462bfa51688929/tools/bazel.rc#L32-L34
> > > > > >
> > > > > > Hopefully this is also ok. My frustrations with old outdated
> stuff
> > is
> > > > > > making me simplify things in hopes it will be more maintainable.
> > > > > >
> > > > > > On Fri, Jan 21, 2022 at 4:53 PM Ning Wang <wa...@gmail.com>
> > > > wrote:
> > > > > >
> > > > > >> nvm. you mentioned it already and I missed it. My bad.
> > > > > >>
> > > > > >> no issues.
> > > > > >>
> > > > > >> On Fri, Jan 21, 2022 at 1:52 PM Ning Wang <wangninggm@gmail.com
> >
> > > > wrote:
> > > > > >>
> > > > > >> > Sounds good to me. Any reason for going to 3.8 instead of 3.9?
> > > > > >> >
> > > > > >> > On Fri, Jan 21, 2022 at 1:37 PM Josh Fischer <
> > josh@joshfischer.io
> > > >
> > > > > >> wrote:
> > > > > >> >
> > > > > >> >> No issues from me.
> > > > > >> >>
> > > > > >> >> On Fri, Jan 21, 2022 at 3:11 PM Nicholas Nezis <
> > > > > >> nicholas.nezis@gmail.com>
> > > > > >> >> wrote:
> > > > > >> >>
> > > > > >> >> > Heron Devs,
> > > > > >> >> >
> > > > > >> >> > We are running into issues with the older Python 3.6 that
> > seems
> > > > to
> > > > > be
> > > > > >> >> > resolved in more modern versions of Python. To move things
> > > > forward
> > > > > >> and
> > > > > >> >> get
> > > > > >> >> > our release working, I'm upgrading to Python 3.8 in our
> older
> > > > > Docker
> > > > > >> >> > container flavors.
> > > > > >> >> >
> > > > > >> >> > Much of the work is being done off of Oliver's previous PR
> to
> > > > > upgrade
> > > > > >> >> us to
> > > > > >> >> > Python 3.9 support. We updated various Pex dependencies
> (and
> > > > other
> > > > > >> >> > python dependencies such as PyYAML). Please review the
> latest
> > > > > >> changes in
> > > > > >> >> > the PR here:
> > > https://github.com/apache/incubator-heron/pull/3646
> > > > > >> >> >
> > > > > >> >> > I'm setting the minimum Python in `bazel_configure.sh` to
> > 3.8.
> > > Is
> > > > > >> >> everyone
> > > > > >> >> > ok with this change?
> > > > > >> >> >
> > > > > >> >> > Some of the key changes:
> > > > > >> >> > Centos 7 -> Centos8 (Python 3.6 -> Python 3.9)
> > > > > >> >> > Ubuntu 18.04 (Python 3.6 -> Python 3.8)
> > > > > >> >> >
> > > > > >> >> > I didn't really want to update Centos 7 to 8. But I don't
> > think
> > > > > >> anyone
> > > > > >> >> is
> > > > > >> >> > using it. The Python 3.9 packages did not exist in the
> older
> > > > image
> > > > > >> >> flavor.
> > > > > >> >> > I eventually want to upgrade this to Rocky or Alma Linux
> > flavor
> > > > or
> > > > > >> maybe
> > > > > >> >> > even RedHat's UBI image which could also help solve the
> > OpenJDK
> > > > > issue
> > > > > >> >> (i.e.
> > > > > >> >> > ubi8/openjdk-11). But for now I'm trying to do the least
> > amount
> > > > of
> > > > > >> >> changes
> > > > > >> >> > to get us over the release finish line.
> > > > > >> >> >
> > > > > >> >> > I've also cleaned up a lot of the documentation references
> to
> > > > older
> > > > > >> >> flavors
> > > > > >> >> > that no longer exist (ubuntu14.04 and debian9).
> > > > > >> >> >
> > > > > >> >>
> > > > > >> >
> > > > > >>
> > > > > >
> > > > >
> > > >
> > >
> > --
> > Sent from A Mobile Device
> >
>
Re: Python 3.8 as minimum
Posted by Saad Ur Rahman <sa...@apache.org>.
Monumental work!
On Mon, Jan 24, 2022 at 2:51 PM Josh Fischer <jo...@joshfischer.io> wrote:
> +1 Nick. I’ll try to check it later this weekS
>
> On Mon, Jan 24, 2022 at 1:42 PM Nicholas Nezis <ni...@gmail.com>
> wrote:
>
> > We finally have a fully green Travis CI build with all of the new Python
> > 3.8+ updates. The PR is here:
> > https://github.com/apache/incubator-heron/pull/3646
> >
> > This PR started with Oliver's awesome updates he contributed back in
> > December 2020. The PR is actually a merge of two different PRs (The other
> > PR: https://github.com/apache/incubator-heron/pull/3642). There were a
> > number of different changes needed to make it all work. For the items
> that
> > were not directly tied to this PR, I submitted separate PRs as best I
> > could. This was not always possible. Below is a summary of what is in
> this
> > larger PR:
> >
> > - All of the awesome Python updates provided by Oliver
> > - Tracker code was simplified with more modern Python libraries
> > - Upgraded Travis image to Ubuntu 20.04 to get newer Python packages
> > - Upgraded Python 3.8 as minimum version
> > - Collapsed `--config` targets to `linux` and `darwin` (no more
> > `ubuntu`, `centos` and `darwin` in bazel.rc config file)
> > - Upgraded various Pex dependencies
> > - Upgraded `rules_python` (0.0.2 -> 0.1.0)
> > - Upgraded the `io_bazel_rules_docker` (0.14.4 -> 0.22.0). This was
> > dependent on `rules_python`
> > - Debian 9 docker image is now Debian 10
> > - Centos 7 docker image is now Centos 8
> >
> >
> > On Fri, Jan 21, 2022 at 7:08 PM Saad Ur Rahman <sa...@apache.org>
> > wrote:
> >
> > > I do not think there is an issue with consolidating `--config` to
> `linux`
> > > and `darwin`.
> > >
> > > For the libraries on Linux systems, you need to link in the libraries
> > > manually, at least in the older compilers. For instance, if you are
> using
> > > the underlying POSIX threading API you will need to link in the
> pthreads
> > > library. Removing these libraries from linking may break the build on
> > > Linux.
> > >
> > > On Fri, Jan 21, 2022 at 6:58 PM Nicholas Nezis <
> nicholas.nezis@gmail.com
> > >
> > > wrote:
> > >
> > > > For the 3 libraries, it seems they are already included in macOS's
> main
> > > > system library. So keeping the Darwin/Linux split in bazel.rc still
> > makes
> > > > sense.
> > > >
> > > > On Fri, Jan 21, 2022 at 6:23 PM Nicholas Nezis <
> > nicholas.nezis@gmail.com
> > > >
> > > > wrote:
> > > >
> > > > > Thanks guys. Making good progress.
> > > > >
> > > > > Found an issue with this `ci.sh` script.
> > > > >
> > > >
> > >
> >
> https://github.com/apache/incubator-heron/blob/c23dba7a5d6982d38bb786fb54462bfa51688929/scripts/shutils/common.sh#L95-L108
> > > > >
> > > > > The platform was not detected properly. The strings never match
> what
> > > > > `python3 -mplatform` outputs. I've decided to simplify the whole
> > flavor
> > > > > thing in our Bazel build. From now on we'll have `linux` and
> `darwin`
> > > as
> > > > > `--config` options. We'll still have different target platforms as
> we
> > > > build
> > > > > docker images, but there have been no differences in config for
> years
> > > > since
> > > > > I've joined the project. So I'm going to simplify things. Changes
> > going
> > > > > into the same PR.
> > > > >
> > > > > These are the only 3 lines that differ between linux and darwin in
> > the
> > > > > bazel.rc file. Do we even need these flags? Should the flags also
> be
> > on
> > > > the
> > > > > darwin build?
> > > > >
> > > >
> > >
> >
> https://github.com/apache/incubator-heron/blob/c23dba7a5d6982d38bb786fb54462bfa51688929/tools/bazel.rc#L32-L34
> > > > >
> > > > > Hopefully this is also ok. My frustrations with old outdated stuff
> is
> > > > > making me simplify things in hopes it will be more maintainable.
> > > > >
> > > > > On Fri, Jan 21, 2022 at 4:53 PM Ning Wang <wa...@gmail.com>
> > > wrote:
> > > > >
> > > > >> nvm. you mentioned it already and I missed it. My bad.
> > > > >>
> > > > >> no issues.
> > > > >>
> > > > >> On Fri, Jan 21, 2022 at 1:52 PM Ning Wang <wa...@gmail.com>
> > > wrote:
> > > > >>
> > > > >> > Sounds good to me. Any reason for going to 3.8 instead of 3.9?
> > > > >> >
> > > > >> > On Fri, Jan 21, 2022 at 1:37 PM Josh Fischer <
> josh@joshfischer.io
> > >
> > > > >> wrote:
> > > > >> >
> > > > >> >> No issues from me.
> > > > >> >>
> > > > >> >> On Fri, Jan 21, 2022 at 3:11 PM Nicholas Nezis <
> > > > >> nicholas.nezis@gmail.com>
> > > > >> >> wrote:
> > > > >> >>
> > > > >> >> > Heron Devs,
> > > > >> >> >
> > > > >> >> > We are running into issues with the older Python 3.6 that
> seems
> > > to
> > > > be
> > > > >> >> > resolved in more modern versions of Python. To move things
> > > forward
> > > > >> and
> > > > >> >> get
> > > > >> >> > our release working, I'm upgrading to Python 3.8 in our older
> > > > Docker
> > > > >> >> > container flavors.
> > > > >> >> >
> > > > >> >> > Much of the work is being done off of Oliver's previous PR to
> > > > upgrade
> > > > >> >> us to
> > > > >> >> > Python 3.9 support. We updated various Pex dependencies (and
> > > other
> > > > >> >> > python dependencies such as PyYAML). Please review the latest
> > > > >> changes in
> > > > >> >> > the PR here:
> > https://github.com/apache/incubator-heron/pull/3646
> > > > >> >> >
> > > > >> >> > I'm setting the minimum Python in `bazel_configure.sh` to
> 3.8.
> > Is
> > > > >> >> everyone
> > > > >> >> > ok with this change?
> > > > >> >> >
> > > > >> >> > Some of the key changes:
> > > > >> >> > Centos 7 -> Centos8 (Python 3.6 -> Python 3.9)
> > > > >> >> > Ubuntu 18.04 (Python 3.6 -> Python 3.8)
> > > > >> >> >
> > > > >> >> > I didn't really want to update Centos 7 to 8. But I don't
> think
> > > > >> anyone
> > > > >> >> is
> > > > >> >> > using it. The Python 3.9 packages did not exist in the older
> > > image
> > > > >> >> flavor.
> > > > >> >> > I eventually want to upgrade this to Rocky or Alma Linux
> flavor
> > > or
> > > > >> maybe
> > > > >> >> > even RedHat's UBI image which could also help solve the
> OpenJDK
> > > > issue
> > > > >> >> (i.e.
> > > > >> >> > ubi8/openjdk-11). But for now I'm trying to do the least
> amount
> > > of
> > > > >> >> changes
> > > > >> >> > to get us over the release finish line.
> > > > >> >> >
> > > > >> >> > I've also cleaned up a lot of the documentation references to
> > > older
> > > > >> >> flavors
> > > > >> >> > that no longer exist (ubuntu14.04 and debian9).
> > > > >> >> >
> > > > >> >>
> > > > >> >
> > > > >>
> > > > >
> > > >
> > >
> >
> --
> Sent from A Mobile Device
>
Re: Python 3.8 as minimum
Posted by Josh Fischer <jo...@joshfischer.io>.
+1 Nick. I’ll try to check it later this weekS
On Mon, Jan 24, 2022 at 1:42 PM Nicholas Nezis <ni...@gmail.com>
wrote:
> We finally have a fully green Travis CI build with all of the new Python
> 3.8+ updates. The PR is here:
> https://github.com/apache/incubator-heron/pull/3646
>
> This PR started with Oliver's awesome updates he contributed back in
> December 2020. The PR is actually a merge of two different PRs (The other
> PR: https://github.com/apache/incubator-heron/pull/3642). There were a
> number of different changes needed to make it all work. For the items that
> were not directly tied to this PR, I submitted separate PRs as best I
> could. This was not always possible. Below is a summary of what is in this
> larger PR:
>
> - All of the awesome Python updates provided by Oliver
> - Tracker code was simplified with more modern Python libraries
> - Upgraded Travis image to Ubuntu 20.04 to get newer Python packages
> - Upgraded Python 3.8 as minimum version
> - Collapsed `--config` targets to `linux` and `darwin` (no more
> `ubuntu`, `centos` and `darwin` in bazel.rc config file)
> - Upgraded various Pex dependencies
> - Upgraded `rules_python` (0.0.2 -> 0.1.0)
> - Upgraded the `io_bazel_rules_docker` (0.14.4 -> 0.22.0). This was
> dependent on `rules_python`
> - Debian 9 docker image is now Debian 10
> - Centos 7 docker image is now Centos 8
>
>
> On Fri, Jan 21, 2022 at 7:08 PM Saad Ur Rahman <sa...@apache.org>
> wrote:
>
> > I do not think there is an issue with consolidating `--config` to `linux`
> > and `darwin`.
> >
> > For the libraries on Linux systems, you need to link in the libraries
> > manually, at least in the older compilers. For instance, if you are using
> > the underlying POSIX threading API you will need to link in the pthreads
> > library. Removing these libraries from linking may break the build on
> > Linux.
> >
> > On Fri, Jan 21, 2022 at 6:58 PM Nicholas Nezis <nicholas.nezis@gmail.com
> >
> > wrote:
> >
> > > For the 3 libraries, it seems they are already included in macOS's main
> > > system library. So keeping the Darwin/Linux split in bazel.rc still
> makes
> > > sense.
> > >
> > > On Fri, Jan 21, 2022 at 6:23 PM Nicholas Nezis <
> nicholas.nezis@gmail.com
> > >
> > > wrote:
> > >
> > > > Thanks guys. Making good progress.
> > > >
> > > > Found an issue with this `ci.sh` script.
> > > >
> > >
> >
> https://github.com/apache/incubator-heron/blob/c23dba7a5d6982d38bb786fb54462bfa51688929/scripts/shutils/common.sh#L95-L108
> > > >
> > > > The platform was not detected properly. The strings never match what
> > > > `python3 -mplatform` outputs. I've decided to simplify the whole
> flavor
> > > > thing in our Bazel build. From now on we'll have `linux` and `darwin`
> > as
> > > > `--config` options. We'll still have different target platforms as we
> > > build
> > > > docker images, but there have been no differences in config for years
> > > since
> > > > I've joined the project. So I'm going to simplify things. Changes
> going
> > > > into the same PR.
> > > >
> > > > These are the only 3 lines that differ between linux and darwin in
> the
> > > > bazel.rc file. Do we even need these flags? Should the flags also be
> on
> > > the
> > > > darwin build?
> > > >
> > >
> >
> https://github.com/apache/incubator-heron/blob/c23dba7a5d6982d38bb786fb54462bfa51688929/tools/bazel.rc#L32-L34
> > > >
> > > > Hopefully this is also ok. My frustrations with old outdated stuff is
> > > > making me simplify things in hopes it will be more maintainable.
> > > >
> > > > On Fri, Jan 21, 2022 at 4:53 PM Ning Wang <wa...@gmail.com>
> > wrote:
> > > >
> > > >> nvm. you mentioned it already and I missed it. My bad.
> > > >>
> > > >> no issues.
> > > >>
> > > >> On Fri, Jan 21, 2022 at 1:52 PM Ning Wang <wa...@gmail.com>
> > wrote:
> > > >>
> > > >> > Sounds good to me. Any reason for going to 3.8 instead of 3.9?
> > > >> >
> > > >> > On Fri, Jan 21, 2022 at 1:37 PM Josh Fischer <josh@joshfischer.io
> >
> > > >> wrote:
> > > >> >
> > > >> >> No issues from me.
> > > >> >>
> > > >> >> On Fri, Jan 21, 2022 at 3:11 PM Nicholas Nezis <
> > > >> nicholas.nezis@gmail.com>
> > > >> >> wrote:
> > > >> >>
> > > >> >> > Heron Devs,
> > > >> >> >
> > > >> >> > We are running into issues with the older Python 3.6 that seems
> > to
> > > be
> > > >> >> > resolved in more modern versions of Python. To move things
> > forward
> > > >> and
> > > >> >> get
> > > >> >> > our release working, I'm upgrading to Python 3.8 in our older
> > > Docker
> > > >> >> > container flavors.
> > > >> >> >
> > > >> >> > Much of the work is being done off of Oliver's previous PR to
> > > upgrade
> > > >> >> us to
> > > >> >> > Python 3.9 support. We updated various Pex dependencies (and
> > other
> > > >> >> > python dependencies such as PyYAML). Please review the latest
> > > >> changes in
> > > >> >> > the PR here:
> https://github.com/apache/incubator-heron/pull/3646
> > > >> >> >
> > > >> >> > I'm setting the minimum Python in `bazel_configure.sh` to 3.8.
> Is
> > > >> >> everyone
> > > >> >> > ok with this change?
> > > >> >> >
> > > >> >> > Some of the key changes:
> > > >> >> > Centos 7 -> Centos8 (Python 3.6 -> Python 3.9)
> > > >> >> > Ubuntu 18.04 (Python 3.6 -> Python 3.8)
> > > >> >> >
> > > >> >> > I didn't really want to update Centos 7 to 8. But I don't think
> > > >> anyone
> > > >> >> is
> > > >> >> > using it. The Python 3.9 packages did not exist in the older
> > image
> > > >> >> flavor.
> > > >> >> > I eventually want to upgrade this to Rocky or Alma Linux flavor
> > or
> > > >> maybe
> > > >> >> > even RedHat's UBI image which could also help solve the OpenJDK
> > > issue
> > > >> >> (i.e.
> > > >> >> > ubi8/openjdk-11). But for now I'm trying to do the least amount
> > of
> > > >> >> changes
> > > >> >> > to get us over the release finish line.
> > > >> >> >
> > > >> >> > I've also cleaned up a lot of the documentation references to
> > older
> > > >> >> flavors
> > > >> >> > that no longer exist (ubuntu14.04 and debian9).
> > > >> >> >
> > > >> >>
> > > >> >
> > > >>
> > > >
> > >
> >
>
--
Sent from A Mobile Device
Re: Python 3.8 as minimum
Posted by Nicholas Nezis <ni...@gmail.com>.
We finally have a fully green Travis CI build with all of the new Python
3.8+ updates. The PR is here:
https://github.com/apache/incubator-heron/pull/3646
This PR started with Oliver's awesome updates he contributed back in
December 2020. The PR is actually a merge of two different PRs (The other
PR: https://github.com/apache/incubator-heron/pull/3642). There were a
number of different changes needed to make it all work. For the items that
were not directly tied to this PR, I submitted separate PRs as best I
could. This was not always possible. Below is a summary of what is in this
larger PR:
- All of the awesome Python updates provided by Oliver
- Tracker code was simplified with more modern Python libraries
- Upgraded Travis image to Ubuntu 20.04 to get newer Python packages
- Upgraded Python 3.8 as minimum version
- Collapsed `--config` targets to `linux` and `darwin` (no more
`ubuntu`, `centos` and `darwin` in bazel.rc config file)
- Upgraded various Pex dependencies
- Upgraded `rules_python` (0.0.2 -> 0.1.0)
- Upgraded the `io_bazel_rules_docker` (0.14.4 -> 0.22.0). This was
dependent on `rules_python`
- Debian 9 docker image is now Debian 10
- Centos 7 docker image is now Centos 8
On Fri, Jan 21, 2022 at 7:08 PM Saad Ur Rahman <sa...@apache.org>
wrote:
> I do not think there is an issue with consolidating `--config` to `linux`
> and `darwin`.
>
> For the libraries on Linux systems, you need to link in the libraries
> manually, at least in the older compilers. For instance, if you are using
> the underlying POSIX threading API you will need to link in the pthreads
> library. Removing these libraries from linking may break the build on
> Linux.
>
> On Fri, Jan 21, 2022 at 6:58 PM Nicholas Nezis <ni...@gmail.com>
> wrote:
>
> > For the 3 libraries, it seems they are already included in macOS's main
> > system library. So keeping the Darwin/Linux split in bazel.rc still makes
> > sense.
> >
> > On Fri, Jan 21, 2022 at 6:23 PM Nicholas Nezis <nicholas.nezis@gmail.com
> >
> > wrote:
> >
> > > Thanks guys. Making good progress.
> > >
> > > Found an issue with this `ci.sh` script.
> > >
> >
> https://github.com/apache/incubator-heron/blob/c23dba7a5d6982d38bb786fb54462bfa51688929/scripts/shutils/common.sh#L95-L108
> > >
> > > The platform was not detected properly. The strings never match what
> > > `python3 -mplatform` outputs. I've decided to simplify the whole flavor
> > > thing in our Bazel build. From now on we'll have `linux` and `darwin`
> as
> > > `--config` options. We'll still have different target platforms as we
> > build
> > > docker images, but there have been no differences in config for years
> > since
> > > I've joined the project. So I'm going to simplify things. Changes going
> > > into the same PR.
> > >
> > > These are the only 3 lines that differ between linux and darwin in the
> > > bazel.rc file. Do we even need these flags? Should the flags also be on
> > the
> > > darwin build?
> > >
> >
> https://github.com/apache/incubator-heron/blob/c23dba7a5d6982d38bb786fb54462bfa51688929/tools/bazel.rc#L32-L34
> > >
> > > Hopefully this is also ok. My frustrations with old outdated stuff is
> > > making me simplify things in hopes it will be more maintainable.
> > >
> > > On Fri, Jan 21, 2022 at 4:53 PM Ning Wang <wa...@gmail.com>
> wrote:
> > >
> > >> nvm. you mentioned it already and I missed it. My bad.
> > >>
> > >> no issues.
> > >>
> > >> On Fri, Jan 21, 2022 at 1:52 PM Ning Wang <wa...@gmail.com>
> wrote:
> > >>
> > >> > Sounds good to me. Any reason for going to 3.8 instead of 3.9?
> > >> >
> > >> > On Fri, Jan 21, 2022 at 1:37 PM Josh Fischer <jo...@joshfischer.io>
> > >> wrote:
> > >> >
> > >> >> No issues from me.
> > >> >>
> > >> >> On Fri, Jan 21, 2022 at 3:11 PM Nicholas Nezis <
> > >> nicholas.nezis@gmail.com>
> > >> >> wrote:
> > >> >>
> > >> >> > Heron Devs,
> > >> >> >
> > >> >> > We are running into issues with the older Python 3.6 that seems
> to
> > be
> > >> >> > resolved in more modern versions of Python. To move things
> forward
> > >> and
> > >> >> get
> > >> >> > our release working, I'm upgrading to Python 3.8 in our older
> > Docker
> > >> >> > container flavors.
> > >> >> >
> > >> >> > Much of the work is being done off of Oliver's previous PR to
> > upgrade
> > >> >> us to
> > >> >> > Python 3.9 support. We updated various Pex dependencies (and
> other
> > >> >> > python dependencies such as PyYAML). Please review the latest
> > >> changes in
> > >> >> > the PR here: https://github.com/apache/incubator-heron/pull/3646
> > >> >> >
> > >> >> > I'm setting the minimum Python in `bazel_configure.sh` to 3.8. Is
> > >> >> everyone
> > >> >> > ok with this change?
> > >> >> >
> > >> >> > Some of the key changes:
> > >> >> > Centos 7 -> Centos8 (Python 3.6 -> Python 3.9)
> > >> >> > Ubuntu 18.04 (Python 3.6 -> Python 3.8)
> > >> >> >
> > >> >> > I didn't really want to update Centos 7 to 8. But I don't think
> > >> anyone
> > >> >> is
> > >> >> > using it. The Python 3.9 packages did not exist in the older
> image
> > >> >> flavor.
> > >> >> > I eventually want to upgrade this to Rocky or Alma Linux flavor
> or
> > >> maybe
> > >> >> > even RedHat's UBI image which could also help solve the OpenJDK
> > issue
> > >> >> (i.e.
> > >> >> > ubi8/openjdk-11). But for now I'm trying to do the least amount
> of
> > >> >> changes
> > >> >> > to get us over the release finish line.
> > >> >> >
> > >> >> > I've also cleaned up a lot of the documentation references to
> older
> > >> >> flavors
> > >> >> > that no longer exist (ubuntu14.04 and debian9).
> > >> >> >
> > >> >>
> > >> >
> > >>
> > >
> >
>
Re: Python 3.8 as minimum
Posted by Saad Ur Rahman <sa...@apache.org>.
I do not think there is an issue with consolidating `--config` to `linux`
and `darwin`.
For the libraries on Linux systems, you need to link in the libraries
manually, at least in the older compilers. For instance, if you are using
the underlying POSIX threading API you will need to link in the pthreads
library. Removing these libraries from linking may break the build on Linux.
On Fri, Jan 21, 2022 at 6:58 PM Nicholas Nezis <ni...@gmail.com>
wrote:
> For the 3 libraries, it seems they are already included in macOS's main
> system library. So keeping the Darwin/Linux split in bazel.rc still makes
> sense.
>
> On Fri, Jan 21, 2022 at 6:23 PM Nicholas Nezis <ni...@gmail.com>
> wrote:
>
> > Thanks guys. Making good progress.
> >
> > Found an issue with this `ci.sh` script.
> >
> https://github.com/apache/incubator-heron/blob/c23dba7a5d6982d38bb786fb54462bfa51688929/scripts/shutils/common.sh#L95-L108
> >
> > The platform was not detected properly. The strings never match what
> > `python3 -mplatform` outputs. I've decided to simplify the whole flavor
> > thing in our Bazel build. From now on we'll have `linux` and `darwin` as
> > `--config` options. We'll still have different target platforms as we
> build
> > docker images, but there have been no differences in config for years
> since
> > I've joined the project. So I'm going to simplify things. Changes going
> > into the same PR.
> >
> > These are the only 3 lines that differ between linux and darwin in the
> > bazel.rc file. Do we even need these flags? Should the flags also be on
> the
> > darwin build?
> >
> https://github.com/apache/incubator-heron/blob/c23dba7a5d6982d38bb786fb54462bfa51688929/tools/bazel.rc#L32-L34
> >
> > Hopefully this is also ok. My frustrations with old outdated stuff is
> > making me simplify things in hopes it will be more maintainable.
> >
> > On Fri, Jan 21, 2022 at 4:53 PM Ning Wang <wa...@gmail.com> wrote:
> >
> >> nvm. you mentioned it already and I missed it. My bad.
> >>
> >> no issues.
> >>
> >> On Fri, Jan 21, 2022 at 1:52 PM Ning Wang <wa...@gmail.com> wrote:
> >>
> >> > Sounds good to me. Any reason for going to 3.8 instead of 3.9?
> >> >
> >> > On Fri, Jan 21, 2022 at 1:37 PM Josh Fischer <jo...@joshfischer.io>
> >> wrote:
> >> >
> >> >> No issues from me.
> >> >>
> >> >> On Fri, Jan 21, 2022 at 3:11 PM Nicholas Nezis <
> >> nicholas.nezis@gmail.com>
> >> >> wrote:
> >> >>
> >> >> > Heron Devs,
> >> >> >
> >> >> > We are running into issues with the older Python 3.6 that seems to
> be
> >> >> > resolved in more modern versions of Python. To move things forward
> >> and
> >> >> get
> >> >> > our release working, I'm upgrading to Python 3.8 in our older
> Docker
> >> >> > container flavors.
> >> >> >
> >> >> > Much of the work is being done off of Oliver's previous PR to
> upgrade
> >> >> us to
> >> >> > Python 3.9 support. We updated various Pex dependencies (and other
> >> >> > python dependencies such as PyYAML). Please review the latest
> >> changes in
> >> >> > the PR here: https://github.com/apache/incubator-heron/pull/3646
> >> >> >
> >> >> > I'm setting the minimum Python in `bazel_configure.sh` to 3.8. Is
> >> >> everyone
> >> >> > ok with this change?
> >> >> >
> >> >> > Some of the key changes:
> >> >> > Centos 7 -> Centos8 (Python 3.6 -> Python 3.9)
> >> >> > Ubuntu 18.04 (Python 3.6 -> Python 3.8)
> >> >> >
> >> >> > I didn't really want to update Centos 7 to 8. But I don't think
> >> anyone
> >> >> is
> >> >> > using it. The Python 3.9 packages did not exist in the older image
> >> >> flavor.
> >> >> > I eventually want to upgrade this to Rocky or Alma Linux flavor or
> >> maybe
> >> >> > even RedHat's UBI image which could also help solve the OpenJDK
> issue
> >> >> (i.e.
> >> >> > ubi8/openjdk-11). But for now I'm trying to do the least amount of
> >> >> changes
> >> >> > to get us over the release finish line.
> >> >> >
> >> >> > I've also cleaned up a lot of the documentation references to older
> >> >> flavors
> >> >> > that no longer exist (ubuntu14.04 and debian9).
> >> >> >
> >> >>
> >> >
> >>
> >
>
Re: Python 3.8 as minimum
Posted by Nicholas Nezis <ni...@gmail.com>.
For the 3 libraries, it seems they are already included in macOS's main
system library. So keeping the Darwin/Linux split in bazel.rc still makes
sense.
On Fri, Jan 21, 2022 at 6:23 PM Nicholas Nezis <ni...@gmail.com>
wrote:
> Thanks guys. Making good progress.
>
> Found an issue with this `ci.sh` script.
> https://github.com/apache/incubator-heron/blob/c23dba7a5d6982d38bb786fb54462bfa51688929/scripts/shutils/common.sh#L95-L108
>
> The platform was not detected properly. The strings never match what
> `python3 -mplatform` outputs. I've decided to simplify the whole flavor
> thing in our Bazel build. From now on we'll have `linux` and `darwin` as
> `--config` options. We'll still have different target platforms as we build
> docker images, but there have been no differences in config for years since
> I've joined the project. So I'm going to simplify things. Changes going
> into the same PR.
>
> These are the only 3 lines that differ between linux and darwin in the
> bazel.rc file. Do we even need these flags? Should the flags also be on the
> darwin build?
> https://github.com/apache/incubator-heron/blob/c23dba7a5d6982d38bb786fb54462bfa51688929/tools/bazel.rc#L32-L34
>
> Hopefully this is also ok. My frustrations with old outdated stuff is
> making me simplify things in hopes it will be more maintainable.
>
> On Fri, Jan 21, 2022 at 4:53 PM Ning Wang <wa...@gmail.com> wrote:
>
>> nvm. you mentioned it already and I missed it. My bad.
>>
>> no issues.
>>
>> On Fri, Jan 21, 2022 at 1:52 PM Ning Wang <wa...@gmail.com> wrote:
>>
>> > Sounds good to me. Any reason for going to 3.8 instead of 3.9?
>> >
>> > On Fri, Jan 21, 2022 at 1:37 PM Josh Fischer <jo...@joshfischer.io>
>> wrote:
>> >
>> >> No issues from me.
>> >>
>> >> On Fri, Jan 21, 2022 at 3:11 PM Nicholas Nezis <
>> nicholas.nezis@gmail.com>
>> >> wrote:
>> >>
>> >> > Heron Devs,
>> >> >
>> >> > We are running into issues with the older Python 3.6 that seems to be
>> >> > resolved in more modern versions of Python. To move things forward
>> and
>> >> get
>> >> > our release working, I'm upgrading to Python 3.8 in our older Docker
>> >> > container flavors.
>> >> >
>> >> > Much of the work is being done off of Oliver's previous PR to upgrade
>> >> us to
>> >> > Python 3.9 support. We updated various Pex dependencies (and other
>> >> > python dependencies such as PyYAML). Please review the latest
>> changes in
>> >> > the PR here: https://github.com/apache/incubator-heron/pull/3646
>> >> >
>> >> > I'm setting the minimum Python in `bazel_configure.sh` to 3.8. Is
>> >> everyone
>> >> > ok with this change?
>> >> >
>> >> > Some of the key changes:
>> >> > Centos 7 -> Centos8 (Python 3.6 -> Python 3.9)
>> >> > Ubuntu 18.04 (Python 3.6 -> Python 3.8)
>> >> >
>> >> > I didn't really want to update Centos 7 to 8. But I don't think
>> anyone
>> >> is
>> >> > using it. The Python 3.9 packages did not exist in the older image
>> >> flavor.
>> >> > I eventually want to upgrade this to Rocky or Alma Linux flavor or
>> maybe
>> >> > even RedHat's UBI image which could also help solve the OpenJDK issue
>> >> (i.e.
>> >> > ubi8/openjdk-11). But for now I'm trying to do the least amount of
>> >> changes
>> >> > to get us over the release finish line.
>> >> >
>> >> > I've also cleaned up a lot of the documentation references to older
>> >> flavors
>> >> > that no longer exist (ubuntu14.04 and debian9).
>> >> >
>> >>
>> >
>>
>
Re: Python 3.8 as minimum
Posted by Nicholas Nezis <ni...@gmail.com>.
Thanks guys. Making good progress.
Found an issue with this `ci.sh` script.
https://github.com/apache/incubator-heron/blob/c23dba7a5d6982d38bb786fb54462bfa51688929/scripts/shutils/common.sh#L95-L108
The platform was not detected properly. The strings never match what
`python3 -mplatform` outputs. I've decided to simplify the whole flavor
thing in our Bazel build. From now on we'll have `linux` and `darwin` as
`--config` options. We'll still have different target platforms as we build
docker images, but there have been no differences in config for years since
I've joined the project. So I'm going to simplify things. Changes going
into the same PR.
These are the only 3 lines that differ between linux and darwin in the
bazel.rc file. Do we even need these flags? Should the flags also be on the
darwin build?
https://github.com/apache/incubator-heron/blob/c23dba7a5d6982d38bb786fb54462bfa51688929/tools/bazel.rc#L32-L34
Hopefully this is also ok. My frustrations with old outdated stuff is
making me simplify things in hopes it will be more maintainable.
On Fri, Jan 21, 2022 at 4:53 PM Ning Wang <wa...@gmail.com> wrote:
> nvm. you mentioned it already and I missed it. My bad.
>
> no issues.
>
> On Fri, Jan 21, 2022 at 1:52 PM Ning Wang <wa...@gmail.com> wrote:
>
> > Sounds good to me. Any reason for going to 3.8 instead of 3.9?
> >
> > On Fri, Jan 21, 2022 at 1:37 PM Josh Fischer <jo...@joshfischer.io>
> wrote:
> >
> >> No issues from me.
> >>
> >> On Fri, Jan 21, 2022 at 3:11 PM Nicholas Nezis <
> nicholas.nezis@gmail.com>
> >> wrote:
> >>
> >> > Heron Devs,
> >> >
> >> > We are running into issues with the older Python 3.6 that seems to be
> >> > resolved in more modern versions of Python. To move things forward and
> >> get
> >> > our release working, I'm upgrading to Python 3.8 in our older Docker
> >> > container flavors.
> >> >
> >> > Much of the work is being done off of Oliver's previous PR to upgrade
> >> us to
> >> > Python 3.9 support. We updated various Pex dependencies (and other
> >> > python dependencies such as PyYAML). Please review the latest changes
> in
> >> > the PR here: https://github.com/apache/incubator-heron/pull/3646
> >> >
> >> > I'm setting the minimum Python in `bazel_configure.sh` to 3.8. Is
> >> everyone
> >> > ok with this change?
> >> >
> >> > Some of the key changes:
> >> > Centos 7 -> Centos8 (Python 3.6 -> Python 3.9)
> >> > Ubuntu 18.04 (Python 3.6 -> Python 3.8)
> >> >
> >> > I didn't really want to update Centos 7 to 8. But I don't think anyone
> >> is
> >> > using it. The Python 3.9 packages did not exist in the older image
> >> flavor.
> >> > I eventually want to upgrade this to Rocky or Alma Linux flavor or
> maybe
> >> > even RedHat's UBI image which could also help solve the OpenJDK issue
> >> (i.e.
> >> > ubi8/openjdk-11). But for now I'm trying to do the least amount of
> >> changes
> >> > to get us over the release finish line.
> >> >
> >> > I've also cleaned up a lot of the documentation references to older
> >> flavors
> >> > that no longer exist (ubuntu14.04 and debian9).
> >> >
> >>
> >
>
Re: Python 3.8 as minimum
Posted by Ning Wang <wa...@gmail.com>.
nvm. you mentioned it already and I missed it. My bad.
no issues.
On Fri, Jan 21, 2022 at 1:52 PM Ning Wang <wa...@gmail.com> wrote:
> Sounds good to me. Any reason for going to 3.8 instead of 3.9?
>
> On Fri, Jan 21, 2022 at 1:37 PM Josh Fischer <jo...@joshfischer.io> wrote:
>
>> No issues from me.
>>
>> On Fri, Jan 21, 2022 at 3:11 PM Nicholas Nezis <ni...@gmail.com>
>> wrote:
>>
>> > Heron Devs,
>> >
>> > We are running into issues with the older Python 3.6 that seems to be
>> > resolved in more modern versions of Python. To move things forward and
>> get
>> > our release working, I'm upgrading to Python 3.8 in our older Docker
>> > container flavors.
>> >
>> > Much of the work is being done off of Oliver's previous PR to upgrade
>> us to
>> > Python 3.9 support. We updated various Pex dependencies (and other
>> > python dependencies such as PyYAML). Please review the latest changes in
>> > the PR here: https://github.com/apache/incubator-heron/pull/3646
>> >
>> > I'm setting the minimum Python in `bazel_configure.sh` to 3.8. Is
>> everyone
>> > ok with this change?
>> >
>> > Some of the key changes:
>> > Centos 7 -> Centos8 (Python 3.6 -> Python 3.9)
>> > Ubuntu 18.04 (Python 3.6 -> Python 3.8)
>> >
>> > I didn't really want to update Centos 7 to 8. But I don't think anyone
>> is
>> > using it. The Python 3.9 packages did not exist in the older image
>> flavor.
>> > I eventually want to upgrade this to Rocky or Alma Linux flavor or maybe
>> > even RedHat's UBI image which could also help solve the OpenJDK issue
>> (i.e.
>> > ubi8/openjdk-11). But for now I'm trying to do the least amount of
>> changes
>> > to get us over the release finish line.
>> >
>> > I've also cleaned up a lot of the documentation references to older
>> flavors
>> > that no longer exist (ubuntu14.04 and debian9).
>> >
>>
>
Re: Python 3.8 as minimum
Posted by Ning Wang <wa...@gmail.com>.
Sounds good to me. Any reason for going to 3.8 instead of 3.9?
On Fri, Jan 21, 2022 at 1:37 PM Josh Fischer <jo...@joshfischer.io> wrote:
> No issues from me.
>
> On Fri, Jan 21, 2022 at 3:11 PM Nicholas Nezis <ni...@gmail.com>
> wrote:
>
> > Heron Devs,
> >
> > We are running into issues with the older Python 3.6 that seems to be
> > resolved in more modern versions of Python. To move things forward and
> get
> > our release working, I'm upgrading to Python 3.8 in our older Docker
> > container flavors.
> >
> > Much of the work is being done off of Oliver's previous PR to upgrade us
> to
> > Python 3.9 support. We updated various Pex dependencies (and other
> > python dependencies such as PyYAML). Please review the latest changes in
> > the PR here: https://github.com/apache/incubator-heron/pull/3646
> >
> > I'm setting the minimum Python in `bazel_configure.sh` to 3.8. Is
> everyone
> > ok with this change?
> >
> > Some of the key changes:
> > Centos 7 -> Centos8 (Python 3.6 -> Python 3.9)
> > Ubuntu 18.04 (Python 3.6 -> Python 3.8)
> >
> > I didn't really want to update Centos 7 to 8. But I don't think anyone is
> > using it. The Python 3.9 packages did not exist in the older image
> flavor.
> > I eventually want to upgrade this to Rocky or Alma Linux flavor or maybe
> > even RedHat's UBI image which could also help solve the OpenJDK issue
> (i.e.
> > ubi8/openjdk-11). But for now I'm trying to do the least amount of
> changes
> > to get us over the release finish line.
> >
> > I've also cleaned up a lot of the documentation references to older
> flavors
> > that no longer exist (ubuntu14.04 and debian9).
> >
>
Re: Python 3.8 as minimum
Posted by Saad Ur Rahman <sa...@apache.org>.
+1
On Fri, Jan 21, 2022 at 4:37 PM Josh Fischer <jo...@joshfischer.io> wrote:
> No issues from me.
>
> On Fri, Jan 21, 2022 at 3:11 PM Nicholas Nezis <ni...@gmail.com>
> wrote:
>
> > Heron Devs,
> >
> > We are running into issues with the older Python 3.6 that seems to be
> > resolved in more modern versions of Python. To move things forward and
> get
> > our release working, I'm upgrading to Python 3.8 in our older Docker
> > container flavors.
> >
> > Much of the work is being done off of Oliver's previous PR to upgrade us
> to
> > Python 3.9 support. We updated various Pex dependencies (and other
> > python dependencies such as PyYAML). Please review the latest changes in
> > the PR here: https://github.com/apache/incubator-heron/pull/3646
> >
> > I'm setting the minimum Python in `bazel_configure.sh` to 3.8. Is
> everyone
> > ok with this change?
> >
> > Some of the key changes:
> > Centos 7 -> Centos8 (Python 3.6 -> Python 3.9)
> > Ubuntu 18.04 (Python 3.6 -> Python 3.8)
> >
> > I didn't really want to update Centos 7 to 8. But I don't think anyone is
> > using it. The Python 3.9 packages did not exist in the older image
> flavor.
> > I eventually want to upgrade this to Rocky or Alma Linux flavor or maybe
> > even RedHat's UBI image which could also help solve the OpenJDK issue
> (i.e.
> > ubi8/openjdk-11). But for now I'm trying to do the least amount of
> changes
> > to get us over the release finish line.
> >
> > I've also cleaned up a lot of the documentation references to older
> flavors
> > that no longer exist (ubuntu14.04 and debian9).
> >
>
Re: Python 3.8 as minimum
Posted by Josh Fischer <jo...@joshfischer.io>.
No issues from me.
On Fri, Jan 21, 2022 at 3:11 PM Nicholas Nezis <ni...@gmail.com>
wrote:
> Heron Devs,
>
> We are running into issues with the older Python 3.6 that seems to be
> resolved in more modern versions of Python. To move things forward and get
> our release working, I'm upgrading to Python 3.8 in our older Docker
> container flavors.
>
> Much of the work is being done off of Oliver's previous PR to upgrade us to
> Python 3.9 support. We updated various Pex dependencies (and other
> python dependencies such as PyYAML). Please review the latest changes in
> the PR here: https://github.com/apache/incubator-heron/pull/3646
>
> I'm setting the minimum Python in `bazel_configure.sh` to 3.8. Is everyone
> ok with this change?
>
> Some of the key changes:
> Centos 7 -> Centos8 (Python 3.6 -> Python 3.9)
> Ubuntu 18.04 (Python 3.6 -> Python 3.8)
>
> I didn't really want to update Centos 7 to 8. But I don't think anyone is
> using it. The Python 3.9 packages did not exist in the older image flavor.
> I eventually want to upgrade this to Rocky or Alma Linux flavor or maybe
> even RedHat's UBI image which could also help solve the OpenJDK issue (i.e.
> ubi8/openjdk-11). But for now I'm trying to do the least amount of changes
> to get us over the release finish line.
>
> I've also cleaned up a lot of the documentation references to older flavors
> that no longer exist (ubuntu14.04 and debian9).
>