You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@heron.apache.org by GitBox <gi...@apache.org> on 2022/01/28 19:46:11 UTC
[GitHub] [incubator-heron] surahman opened a new pull request #3768: Native Python Build Rules - Phase 1
surahman opened a new pull request #3768:
URL: https://github.com/apache/incubator-heron/pull/3768
**Sequential update of build rules towards native Python.**
Each change should pass completely on the CI before moving to the next.
- [x] `WORKSPACE`: added `pip install` entries.
- [x] `newgenproto.bzl`: removed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@heron.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-heron] thinker0 commented on pull request #3768: Native Python Build Rules - Phase 1
Posted by GitBox <gi...@apache.org>.
thinker0 commented on pull request #3768:
URL: https://github.com/apache/incubator-heron/pull/3768#issuecomment-1078513974
https://github.com/apache/incubator-heron/pull/3797/files#diff-5493ff8e9397811510e780de47c57abb70137f1afe85d1519130dc3679d60ce5R288-R290
protobuf-3.16.1 https://github.com/advisories/GHSA-wrvw-hg22-4m67, https://nvd.nist.gov/vuln/detail/CVE-2021-22569
https://github.com/apache/incubator-heron/pull/3797
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@heron.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-heron] surahman edited a comment on pull request #3768: Native Python Build Rules - Phase 1
Posted by GitBox <gi...@apache.org>.
surahman edited a comment on pull request #3768:
URL: https://github.com/apache/incubator-heron/pull/3768#issuecomment-1037680370
> I think this is still in draft and the checklist not complete, so not sure if this is approvable yet. Just wanted to make sure it wasn't merged in by accident. But I guess the Draft status should help with that.
Yes, this is still a WIP. There is a long way to go to completion and the protobuf is currently blocking progress.
> I'd like to better understand the refactoring of the protobuf files to this location: https://github.com/surahman/Heron-Protobuf
>
> @surahman Will this remove the need for the protobuf files to live in this repo? Or was this just a temporary home to figure out the build tooling?
It is to isolate the development of the protobuf build script and directory structures. It has sped up and greatly declutter the protobuf build script development. The directory structures and build scripts would need to be transplanted into the current Heron build scripts.
The issues to consider are whether the import/includes and directory structure within the protobuf libraries these scripts assemble will match what Heron requires. This could become very complicated if it does not.
_**Edit:**_
For clarity, everything outside of the files in [this directory](https://github.com/surahman/Heron-Protobuf/tree/main/proto) is extraneous and can be classified as build artifacts. The only relevant files are in this tree and will need to be integrated into Heron:
```bash
└── proto
├── api
│ ├── BUILD.bazel
│ └── topology.proto
├── ckptmgr
│ ├── BUILD.bazel
│ └── ckptmgr.proto
├── scheduler
│ ├── BUILD.bazel
│ └── scheduler.proto
├── stmgr
│ ├── BUILD.bazel
│ └── stmgr.proto
├── system
│ ├── BUILD.bazel
│ ├── common.proto
│ ├── execution_state.proto
│ ├── metrics.proto
│ ├── packing_plan.proto
│ ├── physical_plan.proto
│ ├── stats.proto
│ └── tuple.proto
├── testing
│ ├── BUILD.bazel
│ └── networktests.proto
└── manager
├── BUILD.bazel
└── tmanager.proto
```
The protobuf is central to all communication between processes in Heron and if it is broken things will fail, so great care and scrutiny are required.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@heron.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-heron] surahman commented on pull request #3768: Native Python Build Rules - Phase 1
Posted by GitBox <gi...@apache.org>.
surahman commented on pull request #3768:
URL: https://github.com/apache/incubator-heron/pull/3768#issuecomment-1037680370
> I think this is still in draft and the checklist not complete, so not sure if this is approvable yet. Just wanted to make sure it wasn't merged in by accident. But I guess the Draft status should help with that.
Yes, this is still a WiP. There is a long way to go to completion and the protobuf is currently blocking progress.
> I'd like to better understand the refactoring of the protobuf files to this location: https://github.com/surahman/Heron-Protobuf
>
> @surahman Will this remove the need for the protobuf files to live in this repo? Or was this just a temporary home to figure out the build tooling?
It is to isolate development of the protobuf build script and directory structures. It has speed up and greatly declutter the protobuf build script development. The directory structures and build scripts would need to be transplanted into the current Heron build scripts.
The issues to consider are whether the import/includes and directory structure within the protobuf libraries these scripts assemble will match what Heron requires. This could become very complicated if it does not.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@heron.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-heron] surahman commented on pull request #3768: Native Python Build Rules - Phase 1
Posted by GitBox <gi...@apache.org>.
surahman commented on pull request #3768:
URL: https://github.com/apache/incubator-heron/pull/3768#issuecomment-1037680370
> I think this is still in draft and the checklist not complete, so not sure if this is approvable yet. Just wanted to make sure it wasn't merged in by accident. But I guess the Draft status should help with that.
Yes, this is still a WiP. There is a long way to go to completion and the protobuf is currently blocking progress.
> I'd like to better understand the refactoring of the protobuf files to this location: https://github.com/surahman/Heron-Protobuf
>
> @surahman Will this remove the need for the protobuf files to live in this repo? Or was this just a temporary home to figure out the build tooling?
It is to isolate development of the protobuf build script and directory structures. It has speed up and greatly declutter the protobuf build script development. The directory structures and build scripts would need to be transplanted into the current Heron build scripts.
The issues to consider are whether the import/includes and directory structure within the protobuf libraries these scripts assemble will match what Heron requires. This could become very complicated if it does not.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@heron.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-heron] nicknezis commented on pull request #3768: Native Python Build Rules - Phase 1
Posted by GitBox <gi...@apache.org>.
nicknezis commented on pull request #3768:
URL: https://github.com/apache/incubator-heron/pull/3768#issuecomment-1037493922
I think this is still in draft and the checklist not complete, so not sure if this is approvable yet. Just wanted to make sure it wasn't merged in by accident. But I guess the Draft status should help with that.
I'd like to better understand the refactoring of the protobuf files to this location: https://github.com/surahman/Heron-Protobuf
@surahman Will this remove the need for the protobuf files to live in this repo? Or was this just a temporary home to figure out the build tooling?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@heron.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-heron] surahman edited a comment on pull request #3768: Native Python Build Rules - Phase 1
Posted by GitBox <gi...@apache.org>.
surahman edited a comment on pull request #3768:
URL: https://github.com/apache/incubator-heron/pull/3768#issuecomment-1037680370
> I think this is still in draft and the checklist not complete, so not sure if this is approvable yet. Just wanted to make sure it wasn't merged in by accident. But I guess the Draft status should help with that.
Yes, this is still a WIP. There is a long way to go to completion and the protobuf is currently blocking progress.
> I'd like to better understand the refactoring of the protobuf files to this location: https://github.com/surahman/Heron-Protobuf
>
> @surahman Will this remove the need for the protobuf files to live in this repo? Or was this just a temporary home to figure out the build tooling?
It is to isolate the development of the protobuf build script and directory structures. It has sped up and greatly declutter the protobuf build script development. The directory structures and build scripts would need to be transplanted into the current Heron build scripts.
The issues to consider are whether the import/includes and directory structure within the protobuf libraries these scripts assemble will match what Heron requires. This could become very complicated if it does not.
_**Edit:**_
For clarity, everything outside of the files in [this directory](https://github.com/surahman/Heron-Protobuf/tree/main/proto) is extraneous and can be classified as build artifacts. The only relevant files are in this tree and will need to be integrated into Heron:
```bash
└── proto
├── api
│ ├── BUILD.bazel
│ └── topology.proto
├── ckptmgr
│ ├── BUILD.bazel
│ └── ckptmgr.proto
├── scheduler
│ ├── BUILD.bazel
│ └── scheduler.proto
├── stmgr
│ ├── BUILD.bazel
│ └── stmgr.proto
├── system
│ ├── BUILD.bazel
│ ├── common.proto
│ ├── execution_state.proto
│ ├── metrics.proto
│ ├── packing_plan.proto
│ ├── physical_plan.proto
│ ├── stats.proto
│ └── tuple.proto
├── testing
│ ├── BUILD.bazel
│ └── networktests.proto
└── manager
├── BUILD.bazel
└── tmanager.proto
```
The protobuf is central to all communication between processes in Heron and if it is broken things will fail, so great care and scrutiny are required.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@heron.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-heron] surahman commented on pull request #3768: Native Python Build Rules - Phase 1
Posted by GitBox <gi...@apache.org>.
surahman commented on pull request #3768:
URL: https://github.com/apache/incubator-heron/pull/3768#issuecomment-1078600401
I don't think the changes in #3797 will affect the Protobuf IDLs that I have put together. This PR is blocked until more people can review and confirm the Protobuf IDLs are correct.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@heron.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-heron] nicknezis commented on pull request #3768: Native Python Build Rules - Phase 1
Posted by GitBox <gi...@apache.org>.
nicknezis commented on pull request #3768:
URL: https://github.com/apache/incubator-heron/pull/3768#issuecomment-1037493922
I think this is still in draft and the checklist not complete, so not sure if this is approvable yet. Just wanted to make sure it wasn't merged in by accident. But I guess the Draft status should help with that.
I'd like to better understand the refactoring of the protobuf files to this location: https://github.com/surahman/Heron-Protobuf
@surahman Will this remove the need for the protobuf files to live in this repo? Or was this just a temporary home to figure out the build tooling?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@heron.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org