You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@heron.apache.org by GitBox <gi...@apache.org> on 2020/01/22 06:31:26 UTC
[GitHub] [incubator-heron] nicknezis opened a new pull request #3438: WIP:
Use new `maven_install` method for managing external jvm dependencies
nicknezis opened a new pull request #3438: WIP: Use new `maven_install` method for managing external jvm dependencies
URL: https://github.com/apache/incubator-heron/pull/3438
Update the Bazel build scripts to use this new capability: https://github.com/bazelbuild/rules_jvm_external
The goal would be to better manage the project's dependencies and not have to manually update the various transitive dependencies. This tool will use a dependency's pom.xml to dynamically generate the Bazel targets for each of the dependencies.
This approach started with the `buildozer` migration which did a lot of formatting cleanup, but ultimately didn't work. After the `buildozer` cleanup, I had to manually update the WORKSPACE and BUILD files to use the new `maven_install` feature.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-heron] Code0x58 commented on issue #3438: WIP: Upgrade
to Bazel 2.0.0
Posted by GitBox <gi...@apache.org>.
Code0x58 commented on issue #3438: WIP: Upgrade to Bazel 2.0.0
URL: https://github.com/apache/incubator-heron/pull/3438#issuecomment-581572644
One thing that springs to mind looking at this PR, is that adding a Bazel linting/formatting stage to travis could be a good idea. That when buildozer hopefully won't be responsible for such large diffs when it is used
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-heron] nicknezis commented on issue #3438: WIP: Use new
`maven_install` method for managing external jvm dependencies
Posted by GitBox <gi...@apache.org>.
nicknezis commented on issue #3438: WIP: Use new `maven_install` method for managing external jvm dependencies
URL: https://github.com/apache/incubator-heron/pull/3438#issuecomment-577499017
I made a new pull request to better describe some of the issues I was running into when trying to use this feature. I'm able to get things working at this point as shown by the green CI build.
When I try to Pin the list of dependencies as described [here](https://github.com/bazelbuild/rules_jvm_external#pinning-artifacts-and-integration-with-bazels-downloader), I run into my first issue. It complains about needing an attribute `netrc` which doesn't exist in Bazel 0.26.0. I can solve this issue by upgrading to 0.29.1, but it introduces other issues. I'm working on fixing those issues and trying to update things to the latest Bazel. As I dig into the issues to find fixes, many of the issues are actually due to the old Bazel and old OS versions.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-heron] nicknezis commented on issue #3438: Upgrade to
Bazel 2.0.0
Posted by GitBox <gi...@apache.org>.
nicknezis commented on issue #3438: Upgrade to Bazel 2.0.0
URL: https://github.com/apache/incubator-heron/pull/3438#issuecomment-613708557
Closing this. The work has moved to #3510
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-heron] nicknezis commented on issue #3438: WIP: Upgrade
to Bazel 2.0.0
Posted by GitBox <gi...@apache.org>.
nicknezis commented on issue #3438: WIP: Upgrade to Bazel 2.0.0
URL: https://github.com/apache/incubator-heron/pull/3438#issuecomment-581680534
@Code0x58 That's a great idea! I'm currently stumped with Python tests failing. It seems they are not able to find the protobuf generated python code. I assume I broke something in my edits.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-heron] nicknezis commented on issue #3438: WIP: Use new
`maven_install` method for managing external jvm dependencies
Posted by GitBox <gi...@apache.org>.
nicknezis commented on issue #3438: WIP: Use new `maven_install` method for managing external jvm dependencies
URL: https://github.com/apache/incubator-heron/pull/3438#issuecomment-577500019
An example of an odd issue I've run into when trying to use newer Bazel versions... I get errors about `bdist_wheel` not being a valid command. I couldn't find where in the Bazel files the `wheel` dependency was being installed. I thought it would be in the `pex_rules` `BUILD` file. It wasn't until later that I stumbled across the Travis CI config file which is actually doing a `pip install wheel`. If this is a dependency we should properly call it out. I'm not sure if the issue is only now impacting me because of the combination of Bazel upgrade and Python 3 upgrade. Either way I will try to resolve these issues, but wanted to give an example of the types of issues I'm running into.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-heron] Code0x58 edited a comment on issue #3438: WIP:
Upgrade to Bazel 2.0.0
Posted by GitBox <gi...@apache.org>.
Code0x58 edited a comment on issue #3438: WIP: Upgrade to Bazel 2.0.0
URL: https://github.com/apache/incubator-heron/pull/3438#issuecomment-581572644
One thing that springs to mind looking at this PR, is that adding a Bazel linting/formatting stage to travis could be a good idea. That whay buildozer hopefully won't be responsible for such large diffs when it is used
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-heron] nicknezis commented on issue #3438: WIP: Upgrade
to Bazel 2.0.0
Posted by GitBox <gi...@apache.org>.
nicknezis commented on issue #3438: WIP: Upgrade to Bazel 2.0.0
URL: https://github.com/apache/incubator-heron/pull/3438#issuecomment-579226133
@nwangtw Yes once I saw the `pip install wheel` in the Travis CI config, that helped a lot. I was not able to build in the Docker containers and thought it was something I had broken with the newer Bazel. I'm thinking the yum or apt packages might be more consistent way to install the pip and wheel dependencies because it will be easier to keep the containers and travis setup in sync.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-heron] nwangtw commented on issue #3438: WIP: Upgrade to
Bazel 2.0.0
Posted by GitBox <gi...@apache.org>.
nwangtw commented on issue #3438: WIP: Upgrade to Bazel 2.0.0
URL: https://github.com/apache/incubator-heron/pull/3438#issuecomment-579144987
Thanks!
Yeah. Wheel was installed and then used as a python tool instead of a library. I remember it is used to build platform independent CLI binary although I am not 100% sure. It could be nice if we can avoid the pip install but it might not be really necessary. Your call and whichever is easier should be good. :)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-heron] nicknezis closed pull request #3438: Upgrade to
Bazel 2.0.0
Posted by GitBox <gi...@apache.org>.
nicknezis closed pull request #3438: Upgrade to Bazel 2.0.0
URL: https://github.com/apache/incubator-heron/pull/3438
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services