You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@heron.apache.org by Josh Fischer <jo...@joshfischer.io> on 2018/12/08 18:54:39 UTC
Bazel Version Compatibility Concerns
Hey All,
I've spent little bit of time over the past day or two working to allow
Heron to be built with newer versions of Bazel (at this time v-0.20.0).
I've managed to restructure certain files within the Heron to project to
comply with what Bazel is expecting.. I think I have most of it worked out
with one big exception (I think) of the python pex rules that were written
by the community and seem to be several months old. I do not know enough
about PEX to update this custom rule and verify it works as expected.. I am
afraid that if we don't address this issue soon will be stuck with an older
version of Bazel moving forward.. Another thought I had: "is it required
that we use pex"? Is there a way we could get away from it and just use
supported official Bazel rules if this issue is too hard to fix or maintain?
Here is my output when building
bazel build --config=darwin heron/...
--incompatible_remove_native_http_archive=false
INFO: Invocation ID: 372ad837-104a-4aef-9e35-5069e5e76ded
INFO: Build options have changed, discarding analysis cache.
ERROR: /Users/joshfischer/Source/heron/heron/statemgrs/tests/python/BUILD:22:1:
in eggs attribute of pex_binary rule
//heron/statemgrs/tests/python:zkstatemanager_unittest_runner:
'@pytest_whl//file:file' does not produce any pex_binary eggs files
(expected .egg or .whl). Since this rule was created by the macro
'pex_pytest', the error might have been caused by the macro implementation
in /Users/joshfischer/Source/heron/tools/rules/pex/pex_rules.bzl:462:14
ERROR: /Users/joshfischer/Source/heron/heron/statemgrs/tests/python/BUILD:22:1:
in eggs attribute of pex_binary rule
//heron/statemgrs/tests/python:zkstatemanager_unittest_runner:
'@py_whl//file:file' does not produce any pex_binary eggs files (expected
.egg or .whl). Since this rule was created by the macro 'pex_pytest', the
error might have been caused by the macro implementation in
/Users/joshfischer/Source/heron/tools/rules/pex/pex_rules.bzl:462:14
ERROR: Analysis of target
'//heron/statemgrs/tests/python:zkstatemanager_unittest_runner' failed;
build aborted: Analysis of target
'//heron/statemgrs/tests/python:zkstatemanager_unittest_runner' failed;
build aborted
ignore the flag --incompatible_remove_native_http_archive=false I'm using
that ignore a warning with http_archive targets temporarily while I work
through this python issue. I have a fork of heron at
*https://github.com/joshfischer1108/incubator-heron
<https://github.com/joshfischer1108/incubator-heron>. *The branch I am
working off of within that repo is support-bazel-0.20.0
<https://github.com/joshfischer1108/incubator-heron/tree/support-bazel-0.20.0>.
If anyone has any experience with python and could possibly look over this
and help me get Heron up to speed with the most recent version of Bazel I
would greatly appreciate it.. Just a FYI, as this is still a work in
progress you'll notice I have some duplicate folder structures and files in
the repo right now.. This on purpose.. Bazel has made some changes that
were breaking..
Thanks,
Josh
Re: Bazel Version Compatibility Concerns
Posted by Oliver Bristow <ol...@oliverbristow.co.uk>.
+1
On Tue, 18 Dec 2018 at 11:13, Josh Fischer <jo...@joshfischer.io> wrote:
> Hey Oliver,
>
> Thanks for looking into this. I apologize for my bad choice of words
> describing how the rule was used.. I think you are correct, it looks as the
> rule was copy and pasted.
>
> I would prefer to change as little as possible for the Bazel upgrade. I
> really wanted someone with more Python experience to point my in the right
> direction (which I think you have done)..
>
> Maybe the first thing to do is swap out the parts we need to update the
> rule.. If that doesn't work then we can investigate moving away from PEX.
> Everyone ok with that?
>
> -Josh
>
> On Tue, Dec 18, 2018 at 11:10 AM Ning Wang <wa...@gmail.com> wrote:
>
> > Thanks for the work!
> >
> > I feel moving away from pex might be ok, as long as the final output can
> > run in the same way. There are quite a few modules in Python: python API,
> > cli, executor, tracker, UI, etc.
> >
> > It seems this is a blocker to the bazel upgrade? In that case maybe we
> > should focus on python package first. Does any bazel has internal python
> > packaging support?
> >
> >
> >
> > On Tue, Dec 18, 2018 at 8:58 AM Oliver Bristow <
> oliver@oliverbristow.co.uk
> > >
> > wrote:
> >
> > > Hey Josh,
> > >
> > > I had a look into this/pex/subpar a bit more:
> > >
> > > - external pex rules aren't used (e.g. benley/bazel_rules_pex
> > > <https://github.com/benley/bazel_rules_pex> - although it looks
> like
> > > the
> > > heron tools/rules/pex/pex_rules.bzl
> > > <
> > >
> >
> https://github.com/apache/incubator-heron/blob/master/tools/rules/pex/pex_rules.bzl
> > > >
> > > was copied+pasted from the same source at some point - the earliest
> > > copy in
> > > heron doesn't appear in the remote repo so I guess with changes).
> > > Although
> > > the public repo hasn't been touched in months, there isn't an issue
> > > (yet?)
> > > for bazel 0.20.0
> > > - content of pex files are imported from by the pex_loader module
> > > maintained in heron - I couldn't find the code via google, but maybe
> > > it's
> > > something that could live in pex
> > > <https://github.com/pantsbuild/pex/tree/master/pex> - the pex
> format
> > > appears stable (or inactive?) so hopefully we'll be ok for now
> > > - pex's design documentation <
> > > https://www.pantsbuild.org/pex_design.html>
> > > doesn't list subpar as an alternative - I made an issue for
> > > clarification
> > > <https://github.com/pantsbuild/pantsbuild.github.io/issues/10>
> > >
> > >
> > > I'm under the impression that switching out pex for par would be quite
> a
> > > bit of work (and not sure yet if it introduces more limitations),
> > including
> > > a making a par_loader and also deciding on maintaining both for a while
> > or
> > > dropping pex. Moving away from the loaders may be nice so we aren't
> tied
> > to
> > > a language if possible, but I haven't looked into the how it is used, I
> > > think elsewhere the interface is a program argument/flag and protobuf
> so
> > > that might be an option.
> > >
> > > I don't know if there's any required customisation within the heron
> copy
> > of
> > > pex_rules.bzl (I hope not), so it may be worth trying to drop in the
> > public
> > > rules and using those, then at least the rules should be easy to update
> > and
> > > there's a better shot at community support.
> > >
> > > Does using bazel_pex_rules repo sound passable?
> > >
> > > Thanks,
> > > Oliver
> > >
> > > On Sat, 8 Dec 2018 at 13:43, Josh Fischer <jo...@joshfischer.io> wrote:
> > >
> > > > I'm open to any suggestions.. I'm not much of a python developer, so
> my
> > > > knowledge is limited in this part of the project. I appreciate the
> > > > thoughts and ideas. Thanks Oliver. Let us know what you come up
> with.
> > > >
> > > > On Sat, Dec 8, 2018 at 1:34 PM Oliver Bristow <
> > > oliver@oliverbristow.co.uk>
> > > > wrote:
> > > >
> > > > > Google's subpar may be an option as a replacement for .pex - I
> > imagine
> > > > it'd
> > > > > have better support in Bazel/generally. Has anyone got any thoughts
> > on
> > > > > pex/subpar? I can ponder more and look up stuff next week
> > > > >
> > > > > On Sat, 8 Dec 2018, 12:57 Josh Fischer <josh@joshfischer.io wrote:
> > > > >
> > > > > > Hey All,
> > > > > >
> > > > > > I've spent little bit of time over the past day or two working to
> > > allow
> > > > > > Heron to be built with newer versions of Bazel (at this time
> > > v-0.20.0).
> > > > > > I've managed to restructure certain files within the Heron to
> > > project
> > > > to
> > > > > > comply with what Bazel is expecting.. I think I have most of it
> > > worked
> > > > > out
> > > > > > with one big exception (I think) of the python pex rules that
> were
> > > > > written
> > > > > > by the community and seem to be several months old. I do not know
> > > > enough
> > > > > > about PEX to update this custom rule and verify it works as
> > > expected..
> > > > I
> > > > > am
> > > > > > afraid that if we don't address this issue soon will be stuck
> with
> > an
> > > > > older
> > > > > > version of Bazel moving forward.. Another thought I had: "is it
> > > > > required
> > > > > > that we use pex"? Is there a way we could get away from it and
> > just
> > > > use
> > > > > > supported official Bazel rules if this issue is too hard to fix
> or
> > > > > > maintain?
> > > > > >
> > > > > > Here is my output when building
> > > > > >
> > > > > > bazel build --config=darwin heron/...
> > > > > > --incompatible_remove_native_http_archive=false
> > > > > >
> > > > > > INFO: Invocation ID: 372ad837-104a-4aef-9e35-5069e5e76ded
> > > > > >
> > > > > > INFO: Build options have changed, discarding analysis cache.
> > > > > >
> > > > > > ERROR:
> > > > > >
> > > >
> > /Users/joshfischer/Source/heron/heron/statemgrs/tests/python/BUILD:22:1:
> > > > > > in eggs attribute of pex_binary rule
> > > > > > //heron/statemgrs/tests/python:zkstatemanager_unittest_runner:
> > > > > > '@pytest_whl//file:file' does not produce any pex_binary eggs
> files
> > > > > > (expected .egg or .whl). Since this rule was created by the macro
> > > > > > 'pex_pytest', the error might have been caused by the macro
> > > > > implementation
> > > > > > in
> > > /Users/joshfischer/Source/heron/tools/rules/pex/pex_rules.bzl:462:14
> > > > > >
> > > > > > ERROR:
> > > > > >
> > > >
> > /Users/joshfischer/Source/heron/heron/statemgrs/tests/python/BUILD:22:1:
> > > > > > in eggs attribute of pex_binary rule
> > > > > > //heron/statemgrs/tests/python:zkstatemanager_unittest_runner:
> > > > > > '@py_whl//file:file' does not produce any pex_binary eggs files
> > > > (expected
> > > > > > .egg or .whl). Since this rule was created by the macro
> > 'pex_pytest',
> > > > the
> > > > > > error might have been caused by the macro implementation in
> > > > > >
> > /Users/joshfischer/Source/heron/tools/rules/pex/pex_rules.bzl:462:14
> > > > > >
> > > > > > ERROR: Analysis of target
> > > > > > '//heron/statemgrs/tests/python:zkstatemanager_unittest_runner'
> > > failed;
> > > > > > build aborted: Analysis of target
> > > > > > '//heron/statemgrs/tests/python:zkstatemanager_unittest_runner'
> > > failed;
> > > > > > build aborted
> > > > > >
> > > > > > ignore the flag --incompatible_remove_native_http_archive=false
> > I'm
> > > > > using
> > > > > > that ignore a warning with http_archive targets temporarily
> while I
> > > > work
> > > > > > through this python issue. I have a fork of heron at
> > > > > > *https://github.com/joshfischer1108/incubator-heron
> > > > > > <https://github.com/joshfischer1108/incubator-heron>. *The
> > branch I
> > > > am
> > > > > > working off of within that repo is support-bazel-0.20.0
> > > > > > <
> > > > > >
> > > > >
> > > >
> > >
> >
> https://github.com/joshfischer1108/incubator-heron/tree/support-bazel-0.20.0
> > > > > > >.
> > > > > > If anyone has any experience with python and could possibly look
> > over
> > > > > this
> > > > > > and help me get Heron up to speed with the most recent version of
> > > > Bazel I
> > > > > > would greatly appreciate it.. Just a FYI, as this is still a work
> > in
> > > > > > progress you'll notice I have some duplicate folder structures
> and
> > > > files
> > > > > in
> > > > > > the repo right now.. This on purpose.. Bazel has made some
> changes
> > > that
> > > > > > were breaking..
> > > > > >
> > > > > >
> > > > > > Thanks,
> > > > > >
> > > > > > Josh
> > > > > >
> > > > >
> > > >
> > >
> >
>
Re: Bazel Version Compatibility Concerns
Posted by Josh Fischer <jo...@joshfischer.io>.
Last night I updated the Pex rules with what is in the github repository at
https://github.com/benley/bazel_rules_pex
I'm getting an error stating that the macro implementation does produce any
pex_binary eggs
Here is my command and some output:
ERROR: /Users/joshfischer/Source/heron/heron/statemgrs/tests/python/BUILD:22:1:
in eggs attribute of pex_binary rule
//heron/statemgrs/tests/python:zkstatemanager_unittest_runner:
'@pytest_whl//file:file' does not produce any pex_binary eggs files
(expected .egg or .whl). Since this rule was created by the macro
'pex_pytest', the error might have been caused by the macro implementation
in /Users/joshfischer/Source/heron/tools/rules/pex/pex_rules.bzl:496:14
ERROR: /Users/joshfischer/Source/heron/heron/statemgrs/tests/python/BUILD:22:1:
in eggs attribute of pex_binary rule
//heron/statemgrs/tests/python:zkstatemanager_unittest_runner:
'@py_whl//file:file' does not produce any pex_binary eggs files (expected
.egg or .whl). Since this rule was created by the macro 'pex_pytest', the
error might have been caused by the macro implementation in
/Users/joshfischer/Source/heron/tools/rules/pex/pex_rules.bzl:496:14
ERROR: /Users/joshfischer/Source/heron/heron/statemgrs/tests/python/BUILD:22:1:
in eggs attribute of pex_binary rule
//heron/statemgrs/tests/python:zkstatemanager_unittest_runner:
'@setuptools_src//file:file' does not produce any pex_binary eggs files
(expected .egg or .whl). Since this rule was created by the macro
'pex_pytest', the error might have been caused by the macro implementation
in /Users/joshfischer/Source/heron/tools/rules/pex/pex_rules.bzl:496:14
INFO: SHA256 (
http://central.maven.org/maven2/org/apache/httpcomponents/httpclient/4.5.2/httpclient-4.5.2-tests.jar)
= d87c8dcfd609085090e5ced0249c5ac064c5d50bda29eb8633794fd91789d163
ERROR: Analysis of target
'//heron/statemgrs/tests/python:zkstatemanager_unittest_runner' failed;
build aborted: Analysis of target
'//heron/statemgrs/tests/python:zkstatemanager_unittest_runner' failed;
build aborted
Think we should continue down the path of trying to fix this or go a
different route?
On Wed, Dec 19, 2018 at 2:35 AM thinker0 <th...@gmail.com> wrote:
> +1
>
> 2018년 12월 19일 (수) 오전 2:42, Ning Wang <wa...@gmail.com>님이 작성:
>
> > +1
> >
> > On Tue, Dec 18, 2018 at 9:13 AM Josh Fischer <jo...@joshfischer.io>
> wrote:
> >
> > > Hey Oliver,
> > >
> > > Thanks for looking into this. I apologize for my bad choice of words
> > > describing how the rule was used.. I think you are correct, it looks as
> > the
> > > rule was copy and pasted.
> > >
> > > I would prefer to change as little as possible for the Bazel upgrade.
> I
> > > really wanted someone with more Python experience to point my in the
> > right
> > > direction (which I think you have done)..
> > >
> > > Maybe the first thing to do is swap out the parts we need to update the
> > > rule.. If that doesn't work then we can investigate moving away from
> PEX.
> > > Everyone ok with that?
> > >
> > > -Josh
> > >
> > > On Tue, Dec 18, 2018 at 11:10 AM Ning Wang <wa...@gmail.com>
> wrote:
> > >
> > > > Thanks for the work!
> > > >
> > > > I feel moving away from pex might be ok, as long as the final output
> > can
> > > > run in the same way. There are quite a few modules in Python: python
> > API,
> > > > cli, executor, tracker, UI, etc.
> > > >
> > > > It seems this is a blocker to the bazel upgrade? In that case maybe
> we
> > > > should focus on python package first. Does any bazel has internal
> > python
> > > > packaging support?
> > > >
> > > >
> > > >
> > > > On Tue, Dec 18, 2018 at 8:58 AM Oliver Bristow <
> > > oliver@oliverbristow.co.uk
> > > > >
> > > > wrote:
> > > >
> > > > > Hey Josh,
> > > > >
> > > > > I had a look into this/pex/subpar a bit more:
> > > > >
> > > > > - external pex rules aren't used (e.g. benley/bazel_rules_pex
> > > > > <https://github.com/benley/bazel_rules_pex> - although it looks
> > > like
> > > > > the
> > > > > heron tools/rules/pex/pex_rules.bzl
> > > > > <
> > > > >
> > > >
> > >
> >
> https://github.com/apache/incubator-heron/blob/master/tools/rules/pex/pex_rules.bzl
> > > > > >
> > > > > was copied+pasted from the same source at some point - the
> > earliest
> > > > > copy in
> > > > > heron doesn't appear in the remote repo so I guess with
> changes).
> > > > > Although
> > > > > the public repo hasn't been touched in months, there isn't an
> > issue
> > > > > (yet?)
> > > > > for bazel 0.20.0
> > > > > - content of pex files are imported from by the pex_loader
> module
> > > > > maintained in heron - I couldn't find the code via google, but
> > maybe
> > > > > it's
> > > > > something that could live in pex
> > > > > <https://github.com/pantsbuild/pex/tree/master/pex> - the pex
> > > format
> > > > > appears stable (or inactive?) so hopefully we'll be ok for now
> > > > > - pex's design documentation <
> > > > > https://www.pantsbuild.org/pex_design.html>
> > > > > doesn't list subpar as an alternative - I made an issue for
> > > > > clarification
> > > > > <https://github.com/pantsbuild/pantsbuild.github.io/issues/10>
> > > > >
> > > > >
> > > > > I'm under the impression that switching out pex for par would be
> > quite
> > > a
> > > > > bit of work (and not sure yet if it introduces more limitations),
> > > > including
> > > > > a making a par_loader and also deciding on maintaining both for a
> > while
> > > > or
> > > > > dropping pex. Moving away from the loaders may be nice so we aren't
> > > tied
> > > > to
> > > > > a language if possible, but I haven't looked into the how it is
> > used, I
> > > > > think elsewhere the interface is a program argument/flag and
> protobuf
> > > so
> > > > > that might be an option.
> > > > >
> > > > > I don't know if there's any required customisation within the heron
> > > copy
> > > > of
> > > > > pex_rules.bzl (I hope not), so it may be worth trying to drop in
> the
> > > > public
> > > > > rules and using those, then at least the rules should be easy to
> > update
> > > > and
> > > > > there's a better shot at community support.
> > > > >
> > > > > Does using bazel_pex_rules repo sound passable?
> > > > >
> > > > > Thanks,
> > > > > Oliver
> > > > >
> > > > > On Sat, 8 Dec 2018 at 13:43, Josh Fischer <jo...@joshfischer.io>
> > wrote:
> > > > >
> > > > > > I'm open to any suggestions.. I'm not much of a python developer,
> > so
> > > my
> > > > > > knowledge is limited in this part of the project. I appreciate
> the
> > > > > > thoughts and ideas. Thanks Oliver. Let us know what you come up
> > > with.
> > > > > >
> > > > > > On Sat, Dec 8, 2018 at 1:34 PM Oliver Bristow <
> > > > > oliver@oliverbristow.co.uk>
> > > > > > wrote:
> > > > > >
> > > > > > > Google's subpar may be an option as a replacement for .pex - I
> > > > imagine
> > > > > > it'd
> > > > > > > have better support in Bazel/generally. Has anyone got any
> > thoughts
> > > > on
> > > > > > > pex/subpar? I can ponder more and look up stuff next week
> > > > > > >
> > > > > > > On Sat, 8 Dec 2018, 12:57 Josh Fischer <josh@joshfischer.io
> > wrote:
> > > > > > >
> > > > > > > > Hey All,
> > > > > > > >
> > > > > > > > I've spent little bit of time over the past day or two
> working
> > to
> > > > > allow
> > > > > > > > Heron to be built with newer versions of Bazel (at this time
> > > > > v-0.20.0).
> > > > > > > > I've managed to restructure certain files within the Heron
> to
> > > > > project
> > > > > > to
> > > > > > > > comply with what Bazel is expecting.. I think I have most of
> it
> > > > > worked
> > > > > > > out
> > > > > > > > with one big exception (I think) of the python pex rules that
> > > were
> > > > > > > written
> > > > > > > > by the community and seem to be several months old. I do not
> > know
> > > > > > enough
> > > > > > > > about PEX to update this custom rule and verify it works as
> > > > > expected..
> > > > > > I
> > > > > > > am
> > > > > > > > afraid that if we don't address this issue soon will be stuck
> > > with
> > > > an
> > > > > > > older
> > > > > > > > version of Bazel moving forward.. Another thought I had:
> "is
> > it
> > > > > > > required
> > > > > > > > that we use pex"? Is there a way we could get away from it
> and
> > > > just
> > > > > > use
> > > > > > > > supported official Bazel rules if this issue is too hard to
> fix
> > > or
> > > > > > > > maintain?
> > > > > > > >
> > > > > > > > Here is my output when building
> > > > > > > >
> > > > > > > > bazel build --config=darwin heron/...
> > > > > > > > --incompatible_remove_native_http_archive=false
> > > > > > > >
> > > > > > > > INFO: Invocation ID: 372ad837-104a-4aef-9e35-5069e5e76ded
> > > > > > > >
> > > > > > > > INFO: Build options have changed, discarding analysis cache.
> > > > > > > >
> > > > > > > > ERROR:
> > > > > > > >
> > > > > >
> > > >
> > /Users/joshfischer/Source/heron/heron/statemgrs/tests/python/BUILD:22:1:
> > > > > > > > in eggs attribute of pex_binary rule
> > > > > > > >
> //heron/statemgrs/tests/python:zkstatemanager_unittest_runner:
> > > > > > > > '@pytest_whl//file:file' does not produce any pex_binary eggs
> > > files
> > > > > > > > (expected .egg or .whl). Since this rule was created by the
> > macro
> > > > > > > > 'pex_pytest', the error might have been caused by the macro
> > > > > > > implementation
> > > > > > > > in
> > > > >
> /Users/joshfischer/Source/heron/tools/rules/pex/pex_rules.bzl:462:14
> > > > > > > >
> > > > > > > > ERROR:
> > > > > > > >
> > > > > >
> > > >
> > /Users/joshfischer/Source/heron/heron/statemgrs/tests/python/BUILD:22:1:
> > > > > > > > in eggs attribute of pex_binary rule
> > > > > > > >
> //heron/statemgrs/tests/python:zkstatemanager_unittest_runner:
> > > > > > > > '@py_whl//file:file' does not produce any pex_binary eggs
> files
> > > > > > (expected
> > > > > > > > .egg or .whl). Since this rule was created by the macro
> > > > 'pex_pytest',
> > > > > > the
> > > > > > > > error might have been caused by the macro implementation in
> > > > > > > >
> > > > /Users/joshfischer/Source/heron/tools/rules/pex/pex_rules.bzl:462:14
> > > > > > > >
> > > > > > > > ERROR: Analysis of target
> > > > > > > >
> '//heron/statemgrs/tests/python:zkstatemanager_unittest_runner'
> > > > > failed;
> > > > > > > > build aborted: Analysis of target
> > > > > > > >
> '//heron/statemgrs/tests/python:zkstatemanager_unittest_runner'
> > > > > failed;
> > > > > > > > build aborted
> > > > > > > >
> > > > > > > > ignore the flag
> --incompatible_remove_native_http_archive=false
> > > > I'm
> > > > > > > using
> > > > > > > > that ignore a warning with http_archive targets temporarily
> > > while I
> > > > > > work
> > > > > > > > through this python issue. I have a fork of heron at
> > > > > > > > *https://github.com/joshfischer1108/incubator-heron
> > > > > > > > <https://github.com/joshfischer1108/incubator-heron>. *The
> > > > branch I
> > > > > > am
> > > > > > > > working off of within that repo is support-bazel-0.20.0
> > > > > > > > <
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://github.com/joshfischer1108/incubator-heron/tree/support-bazel-0.20.0
> > > > > > > > >.
> > > > > > > > If anyone has any experience with python and could possibly
> > look
> > > > over
> > > > > > > this
> > > > > > > > and help me get Heron up to speed with the most recent
> version
> > of
> > > > > > Bazel I
> > > > > > > > would greatly appreciate it.. Just a FYI, as this is still a
> > work
> > > > in
> > > > > > > > progress you'll notice I have some duplicate folder
> structures
> > > and
> > > > > > files
> > > > > > > in
> > > > > > > > the repo right now.. This on purpose.. Bazel has made some
> > > changes
> > > > > that
> > > > > > > > were breaking..
> > > > > > > >
> > > > > > > >
> > > > > > > > Thanks,
> > > > > > > >
> > > > > > > > Josh
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
Re: Bazel Version Compatibility Concerns
Posted by thinker0 <th...@gmail.com>.
+1
2018년 12월 19일 (수) 오전 2:42, Ning Wang <wa...@gmail.com>님이 작성:
> +1
>
> On Tue, Dec 18, 2018 at 9:13 AM Josh Fischer <jo...@joshfischer.io> wrote:
>
> > Hey Oliver,
> >
> > Thanks for looking into this. I apologize for my bad choice of words
> > describing how the rule was used.. I think you are correct, it looks as
> the
> > rule was copy and pasted.
> >
> > I would prefer to change as little as possible for the Bazel upgrade. I
> > really wanted someone with more Python experience to point my in the
> right
> > direction (which I think you have done)..
> >
> > Maybe the first thing to do is swap out the parts we need to update the
> > rule.. If that doesn't work then we can investigate moving away from PEX.
> > Everyone ok with that?
> >
> > -Josh
> >
> > On Tue, Dec 18, 2018 at 11:10 AM Ning Wang <wa...@gmail.com> wrote:
> >
> > > Thanks for the work!
> > >
> > > I feel moving away from pex might be ok, as long as the final output
> can
> > > run in the same way. There are quite a few modules in Python: python
> API,
> > > cli, executor, tracker, UI, etc.
> > >
> > > It seems this is a blocker to the bazel upgrade? In that case maybe we
> > > should focus on python package first. Does any bazel has internal
> python
> > > packaging support?
> > >
> > >
> > >
> > > On Tue, Dec 18, 2018 at 8:58 AM Oliver Bristow <
> > oliver@oliverbristow.co.uk
> > > >
> > > wrote:
> > >
> > > > Hey Josh,
> > > >
> > > > I had a look into this/pex/subpar a bit more:
> > > >
> > > > - external pex rules aren't used (e.g. benley/bazel_rules_pex
> > > > <https://github.com/benley/bazel_rules_pex> - although it looks
> > like
> > > > the
> > > > heron tools/rules/pex/pex_rules.bzl
> > > > <
> > > >
> > >
> >
> https://github.com/apache/incubator-heron/blob/master/tools/rules/pex/pex_rules.bzl
> > > > >
> > > > was copied+pasted from the same source at some point - the
> earliest
> > > > copy in
> > > > heron doesn't appear in the remote repo so I guess with changes).
> > > > Although
> > > > the public repo hasn't been touched in months, there isn't an
> issue
> > > > (yet?)
> > > > for bazel 0.20.0
> > > > - content of pex files are imported from by the pex_loader module
> > > > maintained in heron - I couldn't find the code via google, but
> maybe
> > > > it's
> > > > something that could live in pex
> > > > <https://github.com/pantsbuild/pex/tree/master/pex> - the pex
> > format
> > > > appears stable (or inactive?) so hopefully we'll be ok for now
> > > > - pex's design documentation <
> > > > https://www.pantsbuild.org/pex_design.html>
> > > > doesn't list subpar as an alternative - I made an issue for
> > > > clarification
> > > > <https://github.com/pantsbuild/pantsbuild.github.io/issues/10>
> > > >
> > > >
> > > > I'm under the impression that switching out pex for par would be
> quite
> > a
> > > > bit of work (and not sure yet if it introduces more limitations),
> > > including
> > > > a making a par_loader and also deciding on maintaining both for a
> while
> > > or
> > > > dropping pex. Moving away from the loaders may be nice so we aren't
> > tied
> > > to
> > > > a language if possible, but I haven't looked into the how it is
> used, I
> > > > think elsewhere the interface is a program argument/flag and protobuf
> > so
> > > > that might be an option.
> > > >
> > > > I don't know if there's any required customisation within the heron
> > copy
> > > of
> > > > pex_rules.bzl (I hope not), so it may be worth trying to drop in the
> > > public
> > > > rules and using those, then at least the rules should be easy to
> update
> > > and
> > > > there's a better shot at community support.
> > > >
> > > > Does using bazel_pex_rules repo sound passable?
> > > >
> > > > Thanks,
> > > > Oliver
> > > >
> > > > On Sat, 8 Dec 2018 at 13:43, Josh Fischer <jo...@joshfischer.io>
> wrote:
> > > >
> > > > > I'm open to any suggestions.. I'm not much of a python developer,
> so
> > my
> > > > > knowledge is limited in this part of the project. I appreciate the
> > > > > thoughts and ideas. Thanks Oliver. Let us know what you come up
> > with.
> > > > >
> > > > > On Sat, Dec 8, 2018 at 1:34 PM Oliver Bristow <
> > > > oliver@oliverbristow.co.uk>
> > > > > wrote:
> > > > >
> > > > > > Google's subpar may be an option as a replacement for .pex - I
> > > imagine
> > > > > it'd
> > > > > > have better support in Bazel/generally. Has anyone got any
> thoughts
> > > on
> > > > > > pex/subpar? I can ponder more and look up stuff next week
> > > > > >
> > > > > > On Sat, 8 Dec 2018, 12:57 Josh Fischer <josh@joshfischer.io
> wrote:
> > > > > >
> > > > > > > Hey All,
> > > > > > >
> > > > > > > I've spent little bit of time over the past day or two working
> to
> > > > allow
> > > > > > > Heron to be built with newer versions of Bazel (at this time
> > > > v-0.20.0).
> > > > > > > I've managed to restructure certain files within the Heron to
> > > > project
> > > > > to
> > > > > > > comply with what Bazel is expecting.. I think I have most of it
> > > > worked
> > > > > > out
> > > > > > > with one big exception (I think) of the python pex rules that
> > were
> > > > > > written
> > > > > > > by the community and seem to be several months old. I do not
> know
> > > > > enough
> > > > > > > about PEX to update this custom rule and verify it works as
> > > > expected..
> > > > > I
> > > > > > am
> > > > > > > afraid that if we don't address this issue soon will be stuck
> > with
> > > an
> > > > > > older
> > > > > > > version of Bazel moving forward.. Another thought I had: "is
> it
> > > > > > required
> > > > > > > that we use pex"? Is there a way we could get away from it and
> > > just
> > > > > use
> > > > > > > supported official Bazel rules if this issue is too hard to fix
> > or
> > > > > > > maintain?
> > > > > > >
> > > > > > > Here is my output when building
> > > > > > >
> > > > > > > bazel build --config=darwin heron/...
> > > > > > > --incompatible_remove_native_http_archive=false
> > > > > > >
> > > > > > > INFO: Invocation ID: 372ad837-104a-4aef-9e35-5069e5e76ded
> > > > > > >
> > > > > > > INFO: Build options have changed, discarding analysis cache.
> > > > > > >
> > > > > > > ERROR:
> > > > > > >
> > > > >
> > >
> /Users/joshfischer/Source/heron/heron/statemgrs/tests/python/BUILD:22:1:
> > > > > > > in eggs attribute of pex_binary rule
> > > > > > > //heron/statemgrs/tests/python:zkstatemanager_unittest_runner:
> > > > > > > '@pytest_whl//file:file' does not produce any pex_binary eggs
> > files
> > > > > > > (expected .egg or .whl). Since this rule was created by the
> macro
> > > > > > > 'pex_pytest', the error might have been caused by the macro
> > > > > > implementation
> > > > > > > in
> > > > /Users/joshfischer/Source/heron/tools/rules/pex/pex_rules.bzl:462:14
> > > > > > >
> > > > > > > ERROR:
> > > > > > >
> > > > >
> > >
> /Users/joshfischer/Source/heron/heron/statemgrs/tests/python/BUILD:22:1:
> > > > > > > in eggs attribute of pex_binary rule
> > > > > > > //heron/statemgrs/tests/python:zkstatemanager_unittest_runner:
> > > > > > > '@py_whl//file:file' does not produce any pex_binary eggs files
> > > > > (expected
> > > > > > > .egg or .whl). Since this rule was created by the macro
> > > 'pex_pytest',
> > > > > the
> > > > > > > error might have been caused by the macro implementation in
> > > > > > >
> > > /Users/joshfischer/Source/heron/tools/rules/pex/pex_rules.bzl:462:14
> > > > > > >
> > > > > > > ERROR: Analysis of target
> > > > > > > '//heron/statemgrs/tests/python:zkstatemanager_unittest_runner'
> > > > failed;
> > > > > > > build aborted: Analysis of target
> > > > > > > '//heron/statemgrs/tests/python:zkstatemanager_unittest_runner'
> > > > failed;
> > > > > > > build aborted
> > > > > > >
> > > > > > > ignore the flag --incompatible_remove_native_http_archive=false
> > > I'm
> > > > > > using
> > > > > > > that ignore a warning with http_archive targets temporarily
> > while I
> > > > > work
> > > > > > > through this python issue. I have a fork of heron at
> > > > > > > *https://github.com/joshfischer1108/incubator-heron
> > > > > > > <https://github.com/joshfischer1108/incubator-heron>. *The
> > > branch I
> > > > > am
> > > > > > > working off of within that repo is support-bazel-0.20.0
> > > > > > > <
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://github.com/joshfischer1108/incubator-heron/tree/support-bazel-0.20.0
> > > > > > > >.
> > > > > > > If anyone has any experience with python and could possibly
> look
> > > over
> > > > > > this
> > > > > > > and help me get Heron up to speed with the most recent version
> of
> > > > > Bazel I
> > > > > > > would greatly appreciate it.. Just a FYI, as this is still a
> work
> > > in
> > > > > > > progress you'll notice I have some duplicate folder structures
> > and
> > > > > files
> > > > > > in
> > > > > > > the repo right now.. This on purpose.. Bazel has made some
> > changes
> > > > that
> > > > > > > were breaking..
> > > > > > >
> > > > > > >
> > > > > > > Thanks,
> > > > > > >
> > > > > > > Josh
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
Re: Bazel Version Compatibility Concerns
Posted by Ning Wang <wa...@gmail.com>.
+1
On Tue, Dec 18, 2018 at 9:13 AM Josh Fischer <jo...@joshfischer.io> wrote:
> Hey Oliver,
>
> Thanks for looking into this. I apologize for my bad choice of words
> describing how the rule was used.. I think you are correct, it looks as the
> rule was copy and pasted.
>
> I would prefer to change as little as possible for the Bazel upgrade. I
> really wanted someone with more Python experience to point my in the right
> direction (which I think you have done)..
>
> Maybe the first thing to do is swap out the parts we need to update the
> rule.. If that doesn't work then we can investigate moving away from PEX.
> Everyone ok with that?
>
> -Josh
>
> On Tue, Dec 18, 2018 at 11:10 AM Ning Wang <wa...@gmail.com> wrote:
>
> > Thanks for the work!
> >
> > I feel moving away from pex might be ok, as long as the final output can
> > run in the same way. There are quite a few modules in Python: python API,
> > cli, executor, tracker, UI, etc.
> >
> > It seems this is a blocker to the bazel upgrade? In that case maybe we
> > should focus on python package first. Does any bazel has internal python
> > packaging support?
> >
> >
> >
> > On Tue, Dec 18, 2018 at 8:58 AM Oliver Bristow <
> oliver@oliverbristow.co.uk
> > >
> > wrote:
> >
> > > Hey Josh,
> > >
> > > I had a look into this/pex/subpar a bit more:
> > >
> > > - external pex rules aren't used (e.g. benley/bazel_rules_pex
> > > <https://github.com/benley/bazel_rules_pex> - although it looks
> like
> > > the
> > > heron tools/rules/pex/pex_rules.bzl
> > > <
> > >
> >
> https://github.com/apache/incubator-heron/blob/master/tools/rules/pex/pex_rules.bzl
> > > >
> > > was copied+pasted from the same source at some point - the earliest
> > > copy in
> > > heron doesn't appear in the remote repo so I guess with changes).
> > > Although
> > > the public repo hasn't been touched in months, there isn't an issue
> > > (yet?)
> > > for bazel 0.20.0
> > > - content of pex files are imported from by the pex_loader module
> > > maintained in heron - I couldn't find the code via google, but maybe
> > > it's
> > > something that could live in pex
> > > <https://github.com/pantsbuild/pex/tree/master/pex> - the pex
> format
> > > appears stable (or inactive?) so hopefully we'll be ok for now
> > > - pex's design documentation <
> > > https://www.pantsbuild.org/pex_design.html>
> > > doesn't list subpar as an alternative - I made an issue for
> > > clarification
> > > <https://github.com/pantsbuild/pantsbuild.github.io/issues/10>
> > >
> > >
> > > I'm under the impression that switching out pex for par would be quite
> a
> > > bit of work (and not sure yet if it introduces more limitations),
> > including
> > > a making a par_loader and also deciding on maintaining both for a while
> > or
> > > dropping pex. Moving away from the loaders may be nice so we aren't
> tied
> > to
> > > a language if possible, but I haven't looked into the how it is used, I
> > > think elsewhere the interface is a program argument/flag and protobuf
> so
> > > that might be an option.
> > >
> > > I don't know if there's any required customisation within the heron
> copy
> > of
> > > pex_rules.bzl (I hope not), so it may be worth trying to drop in the
> > public
> > > rules and using those, then at least the rules should be easy to update
> > and
> > > there's a better shot at community support.
> > >
> > > Does using bazel_pex_rules repo sound passable?
> > >
> > > Thanks,
> > > Oliver
> > >
> > > On Sat, 8 Dec 2018 at 13:43, Josh Fischer <jo...@joshfischer.io> wrote:
> > >
> > > > I'm open to any suggestions.. I'm not much of a python developer, so
> my
> > > > knowledge is limited in this part of the project. I appreciate the
> > > > thoughts and ideas. Thanks Oliver. Let us know what you come up
> with.
> > > >
> > > > On Sat, Dec 8, 2018 at 1:34 PM Oliver Bristow <
> > > oliver@oliverbristow.co.uk>
> > > > wrote:
> > > >
> > > > > Google's subpar may be an option as a replacement for .pex - I
> > imagine
> > > > it'd
> > > > > have better support in Bazel/generally. Has anyone got any thoughts
> > on
> > > > > pex/subpar? I can ponder more and look up stuff next week
> > > > >
> > > > > On Sat, 8 Dec 2018, 12:57 Josh Fischer <josh@joshfischer.io wrote:
> > > > >
> > > > > > Hey All,
> > > > > >
> > > > > > I've spent little bit of time over the past day or two working to
> > > allow
> > > > > > Heron to be built with newer versions of Bazel (at this time
> > > v-0.20.0).
> > > > > > I've managed to restructure certain files within the Heron to
> > > project
> > > > to
> > > > > > comply with what Bazel is expecting.. I think I have most of it
> > > worked
> > > > > out
> > > > > > with one big exception (I think) of the python pex rules that
> were
> > > > > written
> > > > > > by the community and seem to be several months old. I do not know
> > > > enough
> > > > > > about PEX to update this custom rule and verify it works as
> > > expected..
> > > > I
> > > > > am
> > > > > > afraid that if we don't address this issue soon will be stuck
> with
> > an
> > > > > older
> > > > > > version of Bazel moving forward.. Another thought I had: "is it
> > > > > required
> > > > > > that we use pex"? Is there a way we could get away from it and
> > just
> > > > use
> > > > > > supported official Bazel rules if this issue is too hard to fix
> or
> > > > > > maintain?
> > > > > >
> > > > > > Here is my output when building
> > > > > >
> > > > > > bazel build --config=darwin heron/...
> > > > > > --incompatible_remove_native_http_archive=false
> > > > > >
> > > > > > INFO: Invocation ID: 372ad837-104a-4aef-9e35-5069e5e76ded
> > > > > >
> > > > > > INFO: Build options have changed, discarding analysis cache.
> > > > > >
> > > > > > ERROR:
> > > > > >
> > > >
> > /Users/joshfischer/Source/heron/heron/statemgrs/tests/python/BUILD:22:1:
> > > > > > in eggs attribute of pex_binary rule
> > > > > > //heron/statemgrs/tests/python:zkstatemanager_unittest_runner:
> > > > > > '@pytest_whl//file:file' does not produce any pex_binary eggs
> files
> > > > > > (expected .egg or .whl). Since this rule was created by the macro
> > > > > > 'pex_pytest', the error might have been caused by the macro
> > > > > implementation
> > > > > > in
> > > /Users/joshfischer/Source/heron/tools/rules/pex/pex_rules.bzl:462:14
> > > > > >
> > > > > > ERROR:
> > > > > >
> > > >
> > /Users/joshfischer/Source/heron/heron/statemgrs/tests/python/BUILD:22:1:
> > > > > > in eggs attribute of pex_binary rule
> > > > > > //heron/statemgrs/tests/python:zkstatemanager_unittest_runner:
> > > > > > '@py_whl//file:file' does not produce any pex_binary eggs files
> > > > (expected
> > > > > > .egg or .whl). Since this rule was created by the macro
> > 'pex_pytest',
> > > > the
> > > > > > error might have been caused by the macro implementation in
> > > > > >
> > /Users/joshfischer/Source/heron/tools/rules/pex/pex_rules.bzl:462:14
> > > > > >
> > > > > > ERROR: Analysis of target
> > > > > > '//heron/statemgrs/tests/python:zkstatemanager_unittest_runner'
> > > failed;
> > > > > > build aborted: Analysis of target
> > > > > > '//heron/statemgrs/tests/python:zkstatemanager_unittest_runner'
> > > failed;
> > > > > > build aborted
> > > > > >
> > > > > > ignore the flag --incompatible_remove_native_http_archive=false
> > I'm
> > > > > using
> > > > > > that ignore a warning with http_archive targets temporarily
> while I
> > > > work
> > > > > > through this python issue. I have a fork of heron at
> > > > > > *https://github.com/joshfischer1108/incubator-heron
> > > > > > <https://github.com/joshfischer1108/incubator-heron>. *The
> > branch I
> > > > am
> > > > > > working off of within that repo is support-bazel-0.20.0
> > > > > > <
> > > > > >
> > > > >
> > > >
> > >
> >
> https://github.com/joshfischer1108/incubator-heron/tree/support-bazel-0.20.0
> > > > > > >.
> > > > > > If anyone has any experience with python and could possibly look
> > over
> > > > > this
> > > > > > and help me get Heron up to speed with the most recent version of
> > > > Bazel I
> > > > > > would greatly appreciate it.. Just a FYI, as this is still a work
> > in
> > > > > > progress you'll notice I have some duplicate folder structures
> and
> > > > files
> > > > > in
> > > > > > the repo right now.. This on purpose.. Bazel has made some
> changes
> > > that
> > > > > > were breaking..
> > > > > >
> > > > > >
> > > > > > Thanks,
> > > > > >
> > > > > > Josh
> > > > > >
> > > > >
> > > >
> > >
> >
>
Re: Bazel Version Compatibility Concerns
Posted by Josh Fischer <jo...@joshfischer.io>.
Hey Oliver,
Thanks for looking into this. I apologize for my bad choice of words
describing how the rule was used.. I think you are correct, it looks as the
rule was copy and pasted.
I would prefer to change as little as possible for the Bazel upgrade. I
really wanted someone with more Python experience to point my in the right
direction (which I think you have done)..
Maybe the first thing to do is swap out the parts we need to update the
rule.. If that doesn't work then we can investigate moving away from PEX.
Everyone ok with that?
-Josh
On Tue, Dec 18, 2018 at 11:10 AM Ning Wang <wa...@gmail.com> wrote:
> Thanks for the work!
>
> I feel moving away from pex might be ok, as long as the final output can
> run in the same way. There are quite a few modules in Python: python API,
> cli, executor, tracker, UI, etc.
>
> It seems this is a blocker to the bazel upgrade? In that case maybe we
> should focus on python package first. Does any bazel has internal python
> packaging support?
>
>
>
> On Tue, Dec 18, 2018 at 8:58 AM Oliver Bristow <oliver@oliverbristow.co.uk
> >
> wrote:
>
> > Hey Josh,
> >
> > I had a look into this/pex/subpar a bit more:
> >
> > - external pex rules aren't used (e.g. benley/bazel_rules_pex
> > <https://github.com/benley/bazel_rules_pex> - although it looks like
> > the
> > heron tools/rules/pex/pex_rules.bzl
> > <
> >
> https://github.com/apache/incubator-heron/blob/master/tools/rules/pex/pex_rules.bzl
> > >
> > was copied+pasted from the same source at some point - the earliest
> > copy in
> > heron doesn't appear in the remote repo so I guess with changes).
> > Although
> > the public repo hasn't been touched in months, there isn't an issue
> > (yet?)
> > for bazel 0.20.0
> > - content of pex files are imported from by the pex_loader module
> > maintained in heron - I couldn't find the code via google, but maybe
> > it's
> > something that could live in pex
> > <https://github.com/pantsbuild/pex/tree/master/pex> - the pex format
> > appears stable (or inactive?) so hopefully we'll be ok for now
> > - pex's design documentation <
> > https://www.pantsbuild.org/pex_design.html>
> > doesn't list subpar as an alternative - I made an issue for
> > clarification
> > <https://github.com/pantsbuild/pantsbuild.github.io/issues/10>
> >
> >
> > I'm under the impression that switching out pex for par would be quite a
> > bit of work (and not sure yet if it introduces more limitations),
> including
> > a making a par_loader and also deciding on maintaining both for a while
> or
> > dropping pex. Moving away from the loaders may be nice so we aren't tied
> to
> > a language if possible, but I haven't looked into the how it is used, I
> > think elsewhere the interface is a program argument/flag and protobuf so
> > that might be an option.
> >
> > I don't know if there's any required customisation within the heron copy
> of
> > pex_rules.bzl (I hope not), so it may be worth trying to drop in the
> public
> > rules and using those, then at least the rules should be easy to update
> and
> > there's a better shot at community support.
> >
> > Does using bazel_pex_rules repo sound passable?
> >
> > Thanks,
> > Oliver
> >
> > On Sat, 8 Dec 2018 at 13:43, Josh Fischer <jo...@joshfischer.io> wrote:
> >
> > > I'm open to any suggestions.. I'm not much of a python developer, so my
> > > knowledge is limited in this part of the project. I appreciate the
> > > thoughts and ideas. Thanks Oliver. Let us know what you come up with.
> > >
> > > On Sat, Dec 8, 2018 at 1:34 PM Oliver Bristow <
> > oliver@oliverbristow.co.uk>
> > > wrote:
> > >
> > > > Google's subpar may be an option as a replacement for .pex - I
> imagine
> > > it'd
> > > > have better support in Bazel/generally. Has anyone got any thoughts
> on
> > > > pex/subpar? I can ponder more and look up stuff next week
> > > >
> > > > On Sat, 8 Dec 2018, 12:57 Josh Fischer <josh@joshfischer.io wrote:
> > > >
> > > > > Hey All,
> > > > >
> > > > > I've spent little bit of time over the past day or two working to
> > allow
> > > > > Heron to be built with newer versions of Bazel (at this time
> > v-0.20.0).
> > > > > I've managed to restructure certain files within the Heron to
> > project
> > > to
> > > > > comply with what Bazel is expecting.. I think I have most of it
> > worked
> > > > out
> > > > > with one big exception (I think) of the python pex rules that were
> > > > written
> > > > > by the community and seem to be several months old. I do not know
> > > enough
> > > > > about PEX to update this custom rule and verify it works as
> > expected..
> > > I
> > > > am
> > > > > afraid that if we don't address this issue soon will be stuck with
> an
> > > > older
> > > > > version of Bazel moving forward.. Another thought I had: "is it
> > > > required
> > > > > that we use pex"? Is there a way we could get away from it and
> just
> > > use
> > > > > supported official Bazel rules if this issue is too hard to fix or
> > > > > maintain?
> > > > >
> > > > > Here is my output when building
> > > > >
> > > > > bazel build --config=darwin heron/...
> > > > > --incompatible_remove_native_http_archive=false
> > > > >
> > > > > INFO: Invocation ID: 372ad837-104a-4aef-9e35-5069e5e76ded
> > > > >
> > > > > INFO: Build options have changed, discarding analysis cache.
> > > > >
> > > > > ERROR:
> > > > >
> > >
> /Users/joshfischer/Source/heron/heron/statemgrs/tests/python/BUILD:22:1:
> > > > > in eggs attribute of pex_binary rule
> > > > > //heron/statemgrs/tests/python:zkstatemanager_unittest_runner:
> > > > > '@pytest_whl//file:file' does not produce any pex_binary eggs files
> > > > > (expected .egg or .whl). Since this rule was created by the macro
> > > > > 'pex_pytest', the error might have been caused by the macro
> > > > implementation
> > > > > in
> > /Users/joshfischer/Source/heron/tools/rules/pex/pex_rules.bzl:462:14
> > > > >
> > > > > ERROR:
> > > > >
> > >
> /Users/joshfischer/Source/heron/heron/statemgrs/tests/python/BUILD:22:1:
> > > > > in eggs attribute of pex_binary rule
> > > > > //heron/statemgrs/tests/python:zkstatemanager_unittest_runner:
> > > > > '@py_whl//file:file' does not produce any pex_binary eggs files
> > > (expected
> > > > > .egg or .whl). Since this rule was created by the macro
> 'pex_pytest',
> > > the
> > > > > error might have been caused by the macro implementation in
> > > > >
> /Users/joshfischer/Source/heron/tools/rules/pex/pex_rules.bzl:462:14
> > > > >
> > > > > ERROR: Analysis of target
> > > > > '//heron/statemgrs/tests/python:zkstatemanager_unittest_runner'
> > failed;
> > > > > build aborted: Analysis of target
> > > > > '//heron/statemgrs/tests/python:zkstatemanager_unittest_runner'
> > failed;
> > > > > build aborted
> > > > >
> > > > > ignore the flag --incompatible_remove_native_http_archive=false
> I'm
> > > > using
> > > > > that ignore a warning with http_archive targets temporarily while I
> > > work
> > > > > through this python issue. I have a fork of heron at
> > > > > *https://github.com/joshfischer1108/incubator-heron
> > > > > <https://github.com/joshfischer1108/incubator-heron>. *The
> branch I
> > > am
> > > > > working off of within that repo is support-bazel-0.20.0
> > > > > <
> > > > >
> > > >
> > >
> >
> https://github.com/joshfischer1108/incubator-heron/tree/support-bazel-0.20.0
> > > > > >.
> > > > > If anyone has any experience with python and could possibly look
> over
> > > > this
> > > > > and help me get Heron up to speed with the most recent version of
> > > Bazel I
> > > > > would greatly appreciate it.. Just a FYI, as this is still a work
> in
> > > > > progress you'll notice I have some duplicate folder structures and
> > > files
> > > > in
> > > > > the repo right now.. This on purpose.. Bazel has made some changes
> > that
> > > > > were breaking..
> > > > >
> > > > >
> > > > > Thanks,
> > > > >
> > > > > Josh
> > > > >
> > > >
> > >
> >
>
Re: Bazel Version Compatibility Concerns
Posted by Ning Wang <wa...@gmail.com>.
Thanks for the work!
I feel moving away from pex might be ok, as long as the final output can
run in the same way. There are quite a few modules in Python: python API,
cli, executor, tracker, UI, etc.
It seems this is a blocker to the bazel upgrade? In that case maybe we
should focus on python package first. Does any bazel has internal python
packaging support?
On Tue, Dec 18, 2018 at 8:58 AM Oliver Bristow <ol...@oliverbristow.co.uk>
wrote:
> Hey Josh,
>
> I had a look into this/pex/subpar a bit more:
>
> - external pex rules aren't used (e.g. benley/bazel_rules_pex
> <https://github.com/benley/bazel_rules_pex> - although it looks like
> the
> heron tools/rules/pex/pex_rules.bzl
> <
> https://github.com/apache/incubator-heron/blob/master/tools/rules/pex/pex_rules.bzl
> >
> was copied+pasted from the same source at some point - the earliest
> copy in
> heron doesn't appear in the remote repo so I guess with changes).
> Although
> the public repo hasn't been touched in months, there isn't an issue
> (yet?)
> for bazel 0.20.0
> - content of pex files are imported from by the pex_loader module
> maintained in heron - I couldn't find the code via google, but maybe
> it's
> something that could live in pex
> <https://github.com/pantsbuild/pex/tree/master/pex> - the pex format
> appears stable (or inactive?) so hopefully we'll be ok for now
> - pex's design documentation <
> https://www.pantsbuild.org/pex_design.html>
> doesn't list subpar as an alternative - I made an issue for
> clarification
> <https://github.com/pantsbuild/pantsbuild.github.io/issues/10>
>
>
> I'm under the impression that switching out pex for par would be quite a
> bit of work (and not sure yet if it introduces more limitations), including
> a making a par_loader and also deciding on maintaining both for a while or
> dropping pex. Moving away from the loaders may be nice so we aren't tied to
> a language if possible, but I haven't looked into the how it is used, I
> think elsewhere the interface is a program argument/flag and protobuf so
> that might be an option.
>
> I don't know if there's any required customisation within the heron copy of
> pex_rules.bzl (I hope not), so it may be worth trying to drop in the public
> rules and using those, then at least the rules should be easy to update and
> there's a better shot at community support.
>
> Does using bazel_pex_rules repo sound passable?
>
> Thanks,
> Oliver
>
> On Sat, 8 Dec 2018 at 13:43, Josh Fischer <jo...@joshfischer.io> wrote:
>
> > I'm open to any suggestions.. I'm not much of a python developer, so my
> > knowledge is limited in this part of the project. I appreciate the
> > thoughts and ideas. Thanks Oliver. Let us know what you come up with.
> >
> > On Sat, Dec 8, 2018 at 1:34 PM Oliver Bristow <
> oliver@oliverbristow.co.uk>
> > wrote:
> >
> > > Google's subpar may be an option as a replacement for .pex - I imagine
> > it'd
> > > have better support in Bazel/generally. Has anyone got any thoughts on
> > > pex/subpar? I can ponder more and look up stuff next week
> > >
> > > On Sat, 8 Dec 2018, 12:57 Josh Fischer <josh@joshfischer.io wrote:
> > >
> > > > Hey All,
> > > >
> > > > I've spent little bit of time over the past day or two working to
> allow
> > > > Heron to be built with newer versions of Bazel (at this time
> v-0.20.0).
> > > > I've managed to restructure certain files within the Heron to
> project
> > to
> > > > comply with what Bazel is expecting.. I think I have most of it
> worked
> > > out
> > > > with one big exception (I think) of the python pex rules that were
> > > written
> > > > by the community and seem to be several months old. I do not know
> > enough
> > > > about PEX to update this custom rule and verify it works as
> expected..
> > I
> > > am
> > > > afraid that if we don't address this issue soon will be stuck with an
> > > older
> > > > version of Bazel moving forward.. Another thought I had: "is it
> > > required
> > > > that we use pex"? Is there a way we could get away from it and just
> > use
> > > > supported official Bazel rules if this issue is too hard to fix or
> > > > maintain?
> > > >
> > > > Here is my output when building
> > > >
> > > > bazel build --config=darwin heron/...
> > > > --incompatible_remove_native_http_archive=false
> > > >
> > > > INFO: Invocation ID: 372ad837-104a-4aef-9e35-5069e5e76ded
> > > >
> > > > INFO: Build options have changed, discarding analysis cache.
> > > >
> > > > ERROR:
> > > >
> > /Users/joshfischer/Source/heron/heron/statemgrs/tests/python/BUILD:22:1:
> > > > in eggs attribute of pex_binary rule
> > > > //heron/statemgrs/tests/python:zkstatemanager_unittest_runner:
> > > > '@pytest_whl//file:file' does not produce any pex_binary eggs files
> > > > (expected .egg or .whl). Since this rule was created by the macro
> > > > 'pex_pytest', the error might have been caused by the macro
> > > implementation
> > > > in
> /Users/joshfischer/Source/heron/tools/rules/pex/pex_rules.bzl:462:14
> > > >
> > > > ERROR:
> > > >
> > /Users/joshfischer/Source/heron/heron/statemgrs/tests/python/BUILD:22:1:
> > > > in eggs attribute of pex_binary rule
> > > > //heron/statemgrs/tests/python:zkstatemanager_unittest_runner:
> > > > '@py_whl//file:file' does not produce any pex_binary eggs files
> > (expected
> > > > .egg or .whl). Since this rule was created by the macro 'pex_pytest',
> > the
> > > > error might have been caused by the macro implementation in
> > > > /Users/joshfischer/Source/heron/tools/rules/pex/pex_rules.bzl:462:14
> > > >
> > > > ERROR: Analysis of target
> > > > '//heron/statemgrs/tests/python:zkstatemanager_unittest_runner'
> failed;
> > > > build aborted: Analysis of target
> > > > '//heron/statemgrs/tests/python:zkstatemanager_unittest_runner'
> failed;
> > > > build aborted
> > > >
> > > > ignore the flag --incompatible_remove_native_http_archive=false I'm
> > > using
> > > > that ignore a warning with http_archive targets temporarily while I
> > work
> > > > through this python issue. I have a fork of heron at
> > > > *https://github.com/joshfischer1108/incubator-heron
> > > > <https://github.com/joshfischer1108/incubator-heron>. *The branch I
> > am
> > > > working off of within that repo is support-bazel-0.20.0
> > > > <
> > > >
> > >
> >
> https://github.com/joshfischer1108/incubator-heron/tree/support-bazel-0.20.0
> > > > >.
> > > > If anyone has any experience with python and could possibly look over
> > > this
> > > > and help me get Heron up to speed with the most recent version of
> > Bazel I
> > > > would greatly appreciate it.. Just a FYI, as this is still a work in
> > > > progress you'll notice I have some duplicate folder structures and
> > files
> > > in
> > > > the repo right now.. This on purpose.. Bazel has made some changes
> that
> > > > were breaking..
> > > >
> > > >
> > > > Thanks,
> > > >
> > > > Josh
> > > >
> > >
> >
>
Re: Bazel Version Compatibility Concerns
Posted by Oliver Bristow <ol...@oliverbristow.co.uk>.
Hey Josh,
I had a look into this/pex/subpar a bit more:
- external pex rules aren't used (e.g. benley/bazel_rules_pex
<https://github.com/benley/bazel_rules_pex> - although it looks like the
heron tools/rules/pex/pex_rules.bzl
<https://github.com/apache/incubator-heron/blob/master/tools/rules/pex/pex_rules.bzl>
was copied+pasted from the same source at some point - the earliest copy in
heron doesn't appear in the remote repo so I guess with changes). Although
the public repo hasn't been touched in months, there isn't an issue (yet?)
for bazel 0.20.0
- content of pex files are imported from by the pex_loader module
maintained in heron - I couldn't find the code via google, but maybe it's
something that could live in pex
<https://github.com/pantsbuild/pex/tree/master/pex> - the pex format
appears stable (or inactive?) so hopefully we'll be ok for now
- pex's design documentation <https://www.pantsbuild.org/pex_design.html>
doesn't list subpar as an alternative - I made an issue for clarification
<https://github.com/pantsbuild/pantsbuild.github.io/issues/10>
I'm under the impression that switching out pex for par would be quite a
bit of work (and not sure yet if it introduces more limitations), including
a making a par_loader and also deciding on maintaining both for a while or
dropping pex. Moving away from the loaders may be nice so we aren't tied to
a language if possible, but I haven't looked into the how it is used, I
think elsewhere the interface is a program argument/flag and protobuf so
that might be an option.
I don't know if there's any required customisation within the heron copy of
pex_rules.bzl (I hope not), so it may be worth trying to drop in the public
rules and using those, then at least the rules should be easy to update and
there's a better shot at community support.
Does using bazel_pex_rules repo sound passable?
Thanks,
Oliver
On Sat, 8 Dec 2018 at 13:43, Josh Fischer <jo...@joshfischer.io> wrote:
> I'm open to any suggestions.. I'm not much of a python developer, so my
> knowledge is limited in this part of the project. I appreciate the
> thoughts and ideas. Thanks Oliver. Let us know what you come up with.
>
> On Sat, Dec 8, 2018 at 1:34 PM Oliver Bristow <ol...@oliverbristow.co.uk>
> wrote:
>
> > Google's subpar may be an option as a replacement for .pex - I imagine
> it'd
> > have better support in Bazel/generally. Has anyone got any thoughts on
> > pex/subpar? I can ponder more and look up stuff next week
> >
> > On Sat, 8 Dec 2018, 12:57 Josh Fischer <josh@joshfischer.io wrote:
> >
> > > Hey All,
> > >
> > > I've spent little bit of time over the past day or two working to allow
> > > Heron to be built with newer versions of Bazel (at this time v-0.20.0).
> > > I've managed to restructure certain files within the Heron to project
> to
> > > comply with what Bazel is expecting.. I think I have most of it worked
> > out
> > > with one big exception (I think) of the python pex rules that were
> > written
> > > by the community and seem to be several months old. I do not know
> enough
> > > about PEX to update this custom rule and verify it works as expected..
> I
> > am
> > > afraid that if we don't address this issue soon will be stuck with an
> > older
> > > version of Bazel moving forward.. Another thought I had: "is it
> > required
> > > that we use pex"? Is there a way we could get away from it and just
> use
> > > supported official Bazel rules if this issue is too hard to fix or
> > > maintain?
> > >
> > > Here is my output when building
> > >
> > > bazel build --config=darwin heron/...
> > > --incompatible_remove_native_http_archive=false
> > >
> > > INFO: Invocation ID: 372ad837-104a-4aef-9e35-5069e5e76ded
> > >
> > > INFO: Build options have changed, discarding analysis cache.
> > >
> > > ERROR:
> > >
> /Users/joshfischer/Source/heron/heron/statemgrs/tests/python/BUILD:22:1:
> > > in eggs attribute of pex_binary rule
> > > //heron/statemgrs/tests/python:zkstatemanager_unittest_runner:
> > > '@pytest_whl//file:file' does not produce any pex_binary eggs files
> > > (expected .egg or .whl). Since this rule was created by the macro
> > > 'pex_pytest', the error might have been caused by the macro
> > implementation
> > > in /Users/joshfischer/Source/heron/tools/rules/pex/pex_rules.bzl:462:14
> > >
> > > ERROR:
> > >
> /Users/joshfischer/Source/heron/heron/statemgrs/tests/python/BUILD:22:1:
> > > in eggs attribute of pex_binary rule
> > > //heron/statemgrs/tests/python:zkstatemanager_unittest_runner:
> > > '@py_whl//file:file' does not produce any pex_binary eggs files
> (expected
> > > .egg or .whl). Since this rule was created by the macro 'pex_pytest',
> the
> > > error might have been caused by the macro implementation in
> > > /Users/joshfischer/Source/heron/tools/rules/pex/pex_rules.bzl:462:14
> > >
> > > ERROR: Analysis of target
> > > '//heron/statemgrs/tests/python:zkstatemanager_unittest_runner' failed;
> > > build aborted: Analysis of target
> > > '//heron/statemgrs/tests/python:zkstatemanager_unittest_runner' failed;
> > > build aborted
> > >
> > > ignore the flag --incompatible_remove_native_http_archive=false I'm
> > using
> > > that ignore a warning with http_archive targets temporarily while I
> work
> > > through this python issue. I have a fork of heron at
> > > *https://github.com/joshfischer1108/incubator-heron
> > > <https://github.com/joshfischer1108/incubator-heron>. *The branch I
> am
> > > working off of within that repo is support-bazel-0.20.0
> > > <
> > >
> >
> https://github.com/joshfischer1108/incubator-heron/tree/support-bazel-0.20.0
> > > >.
> > > If anyone has any experience with python and could possibly look over
> > this
> > > and help me get Heron up to speed with the most recent version of
> Bazel I
> > > would greatly appreciate it.. Just a FYI, as this is still a work in
> > > progress you'll notice I have some duplicate folder structures and
> files
> > in
> > > the repo right now.. This on purpose.. Bazel has made some changes that
> > > were breaking..
> > >
> > >
> > > Thanks,
> > >
> > > Josh
> > >
> >
>
Re: Bazel Version Compatibility Concerns
Posted by Josh Fischer <jo...@joshfischer.io>.
I'm open to any suggestions.. I'm not much of a python developer, so my
knowledge is limited in this part of the project. I appreciate the
thoughts and ideas. Thanks Oliver. Let us know what you come up with.
On Sat, Dec 8, 2018 at 1:34 PM Oliver Bristow <ol...@oliverbristow.co.uk>
wrote:
> Google's subpar may be an option as a replacement for .pex - I imagine it'd
> have better support in Bazel/generally. Has anyone got any thoughts on
> pex/subpar? I can ponder more and look up stuff next week
>
> On Sat, 8 Dec 2018, 12:57 Josh Fischer <josh@joshfischer.io wrote:
>
> > Hey All,
> >
> > I've spent little bit of time over the past day or two working to allow
> > Heron to be built with newer versions of Bazel (at this time v-0.20.0).
> > I've managed to restructure certain files within the Heron to project to
> > comply with what Bazel is expecting.. I think I have most of it worked
> out
> > with one big exception (I think) of the python pex rules that were
> written
> > by the community and seem to be several months old. I do not know enough
> > about PEX to update this custom rule and verify it works as expected.. I
> am
> > afraid that if we don't address this issue soon will be stuck with an
> older
> > version of Bazel moving forward.. Another thought I had: "is it
> required
> > that we use pex"? Is there a way we could get away from it and just use
> > supported official Bazel rules if this issue is too hard to fix or
> > maintain?
> >
> > Here is my output when building
> >
> > bazel build --config=darwin heron/...
> > --incompatible_remove_native_http_archive=false
> >
> > INFO: Invocation ID: 372ad837-104a-4aef-9e35-5069e5e76ded
> >
> > INFO: Build options have changed, discarding analysis cache.
> >
> > ERROR:
> > /Users/joshfischer/Source/heron/heron/statemgrs/tests/python/BUILD:22:1:
> > in eggs attribute of pex_binary rule
> > //heron/statemgrs/tests/python:zkstatemanager_unittest_runner:
> > '@pytest_whl//file:file' does not produce any pex_binary eggs files
> > (expected .egg or .whl). Since this rule was created by the macro
> > 'pex_pytest', the error might have been caused by the macro
> implementation
> > in /Users/joshfischer/Source/heron/tools/rules/pex/pex_rules.bzl:462:14
> >
> > ERROR:
> > /Users/joshfischer/Source/heron/heron/statemgrs/tests/python/BUILD:22:1:
> > in eggs attribute of pex_binary rule
> > //heron/statemgrs/tests/python:zkstatemanager_unittest_runner:
> > '@py_whl//file:file' does not produce any pex_binary eggs files (expected
> > .egg or .whl). Since this rule was created by the macro 'pex_pytest', the
> > error might have been caused by the macro implementation in
> > /Users/joshfischer/Source/heron/tools/rules/pex/pex_rules.bzl:462:14
> >
> > ERROR: Analysis of target
> > '//heron/statemgrs/tests/python:zkstatemanager_unittest_runner' failed;
> > build aborted: Analysis of target
> > '//heron/statemgrs/tests/python:zkstatemanager_unittest_runner' failed;
> > build aborted
> >
> > ignore the flag --incompatible_remove_native_http_archive=false I'm
> using
> > that ignore a warning with http_archive targets temporarily while I work
> > through this python issue. I have a fork of heron at
> > *https://github.com/joshfischer1108/incubator-heron
> > <https://github.com/joshfischer1108/incubator-heron>. *The branch I am
> > working off of within that repo is support-bazel-0.20.0
> > <
> >
> https://github.com/joshfischer1108/incubator-heron/tree/support-bazel-0.20.0
> > >.
> > If anyone has any experience with python and could possibly look over
> this
> > and help me get Heron up to speed with the most recent version of Bazel I
> > would greatly appreciate it.. Just a FYI, as this is still a work in
> > progress you'll notice I have some duplicate folder structures and files
> in
> > the repo right now.. This on purpose.. Bazel has made some changes that
> > were breaking..
> >
> >
> > Thanks,
> >
> > Josh
> >
>
Re: Bazel Version Compatibility Concerns
Posted by Oliver Bristow <ol...@oliverbristow.co.uk>.
Google's subpar may be an option as a replacement for .pex - I imagine it'd
have better support in Bazel/generally. Has anyone got any thoughts on
pex/subpar? I can ponder more and look up stuff next week
On Sat, 8 Dec 2018, 12:57 Josh Fischer <josh@joshfischer.io wrote:
> Hey All,
>
> I've spent little bit of time over the past day or two working to allow
> Heron to be built with newer versions of Bazel (at this time v-0.20.0).
> I've managed to restructure certain files within the Heron to project to
> comply with what Bazel is expecting.. I think I have most of it worked out
> with one big exception (I think) of the python pex rules that were written
> by the community and seem to be several months old. I do not know enough
> about PEX to update this custom rule and verify it works as expected.. I am
> afraid that if we don't address this issue soon will be stuck with an older
> version of Bazel moving forward.. Another thought I had: "is it required
> that we use pex"? Is there a way we could get away from it and just use
> supported official Bazel rules if this issue is too hard to fix or
> maintain?
>
> Here is my output when building
>
> bazel build --config=darwin heron/...
> --incompatible_remove_native_http_archive=false
>
> INFO: Invocation ID: 372ad837-104a-4aef-9e35-5069e5e76ded
>
> INFO: Build options have changed, discarding analysis cache.
>
> ERROR:
> /Users/joshfischer/Source/heron/heron/statemgrs/tests/python/BUILD:22:1:
> in eggs attribute of pex_binary rule
> //heron/statemgrs/tests/python:zkstatemanager_unittest_runner:
> '@pytest_whl//file:file' does not produce any pex_binary eggs files
> (expected .egg or .whl). Since this rule was created by the macro
> 'pex_pytest', the error might have been caused by the macro implementation
> in /Users/joshfischer/Source/heron/tools/rules/pex/pex_rules.bzl:462:14
>
> ERROR:
> /Users/joshfischer/Source/heron/heron/statemgrs/tests/python/BUILD:22:1:
> in eggs attribute of pex_binary rule
> //heron/statemgrs/tests/python:zkstatemanager_unittest_runner:
> '@py_whl//file:file' does not produce any pex_binary eggs files (expected
> .egg or .whl). Since this rule was created by the macro 'pex_pytest', the
> error might have been caused by the macro implementation in
> /Users/joshfischer/Source/heron/tools/rules/pex/pex_rules.bzl:462:14
>
> ERROR: Analysis of target
> '//heron/statemgrs/tests/python:zkstatemanager_unittest_runner' failed;
> build aborted: Analysis of target
> '//heron/statemgrs/tests/python:zkstatemanager_unittest_runner' failed;
> build aborted
>
> ignore the flag --incompatible_remove_native_http_archive=false I'm using
> that ignore a warning with http_archive targets temporarily while I work
> through this python issue. I have a fork of heron at
> *https://github.com/joshfischer1108/incubator-heron
> <https://github.com/joshfischer1108/incubator-heron>. *The branch I am
> working off of within that repo is support-bazel-0.20.0
> <
> https://github.com/joshfischer1108/incubator-heron/tree/support-bazel-0.20.0
> >.
> If anyone has any experience with python and could possibly look over this
> and help me get Heron up to speed with the most recent version of Bazel I
> would greatly appreciate it.. Just a FYI, as this is still a work in
> progress you'll notice I have some duplicate folder structures and files in
> the repo right now.. This on purpose.. Bazel has made some changes that
> were breaking..
>
>
> Thanks,
>
> Josh
>