You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@heron.apache.org by Nicholas Nezis <ni...@gmail.com> on 2022/02/27 07:23:51 UTC
Removal of --config setting
While working through a bug we ran into with MacOS failing to compile, I
got a better understanding of some of the Bazel settings. In another thread
with Josh, who is compiling on the Apple M1 for the first time, he ran into
issues with Linux vs OSX.
It dawned on me that we were explicitly setting the config to modify a
number of settings. Some of them include the following items:
1. Platform libraries (Linux C libraries)
2. Platform + CPU decision on which binary to use (Helm binary)
3. OS decision (MacOS unique steps vs Linux as the default)
4. Optimization setting (We just hardcoded to C03, but Bazel reported as
"fastbuild")
5. Stylecheck toggle (darwin vs darwin_nostyle)
In a recent PR, I collapsed all of the Linux flavors to a standard
`--config=linux`. This was born out of my frustration with the build
scripts not jiving with Bazel standards. But it was not actually solving
the problem. I'm hoping that this PR will help resolve some of the issues.
https://github.com/apache/incubator-heron/pull/3779
Changes:
1. No longer need to set `--config`. Bazel will detect the OS automatically.
2. Default is `fastbuild`, but you can add `-c opt` to build optimized
build.
a. You can even build both side by side with Bazel making a folder like
darwin-fastbuild and darwin-opt
3. If you want to run stylecheck, you can add `--config=stylecheck`
4. The Linux libraries (-lm, -lpthread, -lrt) are included in
`tools/platform/BUILD`
5. As we add support for ARM, we can add more options in
`tools/platform/BUILD`
6. Scripts outside of Bazel were updated to remove `--config`.
7. Scripts outside of Bazel will specify `-c opt` or `--config=stylecheck`
when needed.
Re: Removal of --config setting
Posted by Saad Ur Rahman <sa...@apache.org>.
That seems reasonable. Great work on the PR.
On Thu, Mar 24, 2022 at 10:45 PM Nicholas Nezis <ni...@gmail.com>
wrote:
> Ok, I think this PR has been around long enough and has a couple other sets
> of eyes on it. I made tweaks based on the feedback. I will merge it later
> tonight. Feel free to take a final look. Thanks to those who helped review
> the changes.
>
> https://github.com/apache/incubator-heron/pull/3779
>
> On Sun, Feb 27, 2022 at 4:38 PM Nicholas Nezis <ni...@gmail.com>
> wrote:
>
> > Ok, so a lot more Bazel related cleanup on that PR.
> > https://github.com/apache/incubator-heron/pull/3779
> >
> > 1. Removed Docker specific `bazel.rc`
> > a. No longer needed now that we have JDK 11 and it works better in
> Docker
> > 2. Removed Travis specific cross-compile toolchain
> > a. No longer needed now that Travis CI is using a modern OS version
> > 3. Removed Travis specific `bazel.rc`
> > a. I don't think this is needed. This was due to the old OS and limited
> > VM resources. Can put back if necessary, but the tests seem to be passing
> > just fine. Even faster than before.
> > 4. Removed ancient `Applatix` CI scripts
> > a. Not used or maintained. They were acquired by Intuit in 2018.
> > 5. Updated documentation
> > a. Including a switch from installing Bazel to Bazelisk. This will help
> > with documentation maintenance.
> >
> > On Sun, Feb 27, 2022 at 2:23 AM Nicholas Nezis <nicholas.nezis@gmail.com
> >
> > wrote:
> >
> >> While working through a bug we ran into with MacOS failing to compile, I
> >> got a better understanding of some of the Bazel settings. In another
> thread
> >> with Josh, who is compiling on the Apple M1 for the first time, he ran
> into
> >> issues with Linux vs OSX.
> >>
> >> It dawned on me that we were explicitly setting the config to modify a
> >> number of settings. Some of them include the following items:
> >> 1. Platform libraries (Linux C libraries)
> >> 2. Platform + CPU decision on which binary to use (Helm binary)
> >> 3. OS decision (MacOS unique steps vs Linux as the default)
> >> 4. Optimization setting (We just hardcoded to C03, but Bazel reported as
> >> "fastbuild")
> >> 5. Stylecheck toggle (darwin vs darwin_nostyle)
> >>
> >> In a recent PR, I collapsed all of the Linux flavors to a standard
> >> `--config=linux`. This was born out of my frustration with the build
> >> scripts not jiving with Bazel standards. But it was not actually solving
> >> the problem. I'm hoping that this PR will help resolve some of the
> issues.
> >>
> >> https://github.com/apache/incubator-heron/pull/3779
> >>
> >> Changes:
> >> 1. No longer need to set `--config`. Bazel will detect the OS
> >> automatically.
> >> 2. Default is `fastbuild`, but you can add `-c opt` to build optimized
> >> build.
> >> a. You can even build both side by side with Bazel making a folder
> like
> >> darwin-fastbuild and darwin-opt
> >> 3. If you want to run stylecheck, you can add `--config=stylecheck`
> >> 4. The Linux libraries (-lm, -lpthread, -lrt) are included in
> >> `tools/platform/BUILD`
> >> 5. As we add support for ARM, we can add more options in
> >> `tools/platform/BUILD`
> >> 6. Scripts outside of Bazel were updated to remove `--config`.
> >> 7. Scripts outside of Bazel will specify `-c opt` or
> >> `--config=stylecheck` when needed.
> >>
> >
>
Re: Removal of --config setting
Posted by Nicholas Nezis <ni...@gmail.com>.
Just realized there was some feedback in Slack that wasn't on the PR. Going
to address those issues before merging.
On Thu, Mar 24, 2022 at 10:44 PM Nicholas Nezis <ni...@gmail.com>
wrote:
> Ok, I think this PR has been around long enough and has a couple other
> sets of eyes on it. I made tweaks based on the feedback. I will merge it
> later tonight. Feel free to take a final look. Thanks to those who helped
> review the changes.
>
> https://github.com/apache/incubator-heron/pull/3779
>
> On Sun, Feb 27, 2022 at 4:38 PM Nicholas Nezis <ni...@gmail.com>
> wrote:
>
>> Ok, so a lot more Bazel related cleanup on that PR.
>> https://github.com/apache/incubator-heron/pull/3779
>>
>> 1. Removed Docker specific `bazel.rc`
>> a. No longer needed now that we have JDK 11 and it works better in
>> Docker
>> 2. Removed Travis specific cross-compile toolchain
>> a. No longer needed now that Travis CI is using a modern OS version
>> 3. Removed Travis specific `bazel.rc`
>> a. I don't think this is needed. This was due to the old OS and limited
>> VM resources. Can put back if necessary, but the tests seem to be passing
>> just fine. Even faster than before.
>> 4. Removed ancient `Applatix` CI scripts
>> a. Not used or maintained. They were acquired by Intuit in 2018.
>> 5. Updated documentation
>> a. Including a switch from installing Bazel to Bazelisk. This will help
>> with documentation maintenance.
>>
>> On Sun, Feb 27, 2022 at 2:23 AM Nicholas Nezis <ni...@gmail.com>
>> wrote:
>>
>>> While working through a bug we ran into with MacOS failing to compile, I
>>> got a better understanding of some of the Bazel settings. In another thread
>>> with Josh, who is compiling on the Apple M1 for the first time, he ran into
>>> issues with Linux vs OSX.
>>>
>>> It dawned on me that we were explicitly setting the config to modify a
>>> number of settings. Some of them include the following items:
>>> 1. Platform libraries (Linux C libraries)
>>> 2. Platform + CPU decision on which binary to use (Helm binary)
>>> 3. OS decision (MacOS unique steps vs Linux as the default)
>>> 4. Optimization setting (We just hardcoded to C03, but Bazel reported as
>>> "fastbuild")
>>> 5. Stylecheck toggle (darwin vs darwin_nostyle)
>>>
>>> In a recent PR, I collapsed all of the Linux flavors to a standard
>>> `--config=linux`. This was born out of my frustration with the build
>>> scripts not jiving with Bazel standards. But it was not actually solving
>>> the problem. I'm hoping that this PR will help resolve some of the issues.
>>>
>>> https://github.com/apache/incubator-heron/pull/3779
>>>
>>> Changes:
>>> 1. No longer need to set `--config`. Bazel will detect the OS
>>> automatically.
>>> 2. Default is `fastbuild`, but you can add `-c opt` to build optimized
>>> build.
>>> a. You can even build both side by side with Bazel making a folder
>>> like darwin-fastbuild and darwin-opt
>>> 3. If you want to run stylecheck, you can add `--config=stylecheck`
>>> 4. The Linux libraries (-lm, -lpthread, -lrt) are included in
>>> `tools/platform/BUILD`
>>> 5. As we add support for ARM, we can add more options in
>>> `tools/platform/BUILD`
>>> 6. Scripts outside of Bazel were updated to remove `--config`.
>>> 7. Scripts outside of Bazel will specify `-c opt` or
>>> `--config=stylecheck` when needed.
>>>
>>
Re: Removal of --config setting
Posted by Nicholas Nezis <ni...@gmail.com>.
Ok, I think this PR has been around long enough and has a couple other sets
of eyes on it. I made tweaks based on the feedback. I will merge it later
tonight. Feel free to take a final look. Thanks to those who helped review
the changes.
https://github.com/apache/incubator-heron/pull/3779
On Sun, Feb 27, 2022 at 4:38 PM Nicholas Nezis <ni...@gmail.com>
wrote:
> Ok, so a lot more Bazel related cleanup on that PR.
> https://github.com/apache/incubator-heron/pull/3779
>
> 1. Removed Docker specific `bazel.rc`
> a. No longer needed now that we have JDK 11 and it works better in Docker
> 2. Removed Travis specific cross-compile toolchain
> a. No longer needed now that Travis CI is using a modern OS version
> 3. Removed Travis specific `bazel.rc`
> a. I don't think this is needed. This was due to the old OS and limited
> VM resources. Can put back if necessary, but the tests seem to be passing
> just fine. Even faster than before.
> 4. Removed ancient `Applatix` CI scripts
> a. Not used or maintained. They were acquired by Intuit in 2018.
> 5. Updated documentation
> a. Including a switch from installing Bazel to Bazelisk. This will help
> with documentation maintenance.
>
> On Sun, Feb 27, 2022 at 2:23 AM Nicholas Nezis <ni...@gmail.com>
> wrote:
>
>> While working through a bug we ran into with MacOS failing to compile, I
>> got a better understanding of some of the Bazel settings. In another thread
>> with Josh, who is compiling on the Apple M1 for the first time, he ran into
>> issues with Linux vs OSX.
>>
>> It dawned on me that we were explicitly setting the config to modify a
>> number of settings. Some of them include the following items:
>> 1. Platform libraries (Linux C libraries)
>> 2. Platform + CPU decision on which binary to use (Helm binary)
>> 3. OS decision (MacOS unique steps vs Linux as the default)
>> 4. Optimization setting (We just hardcoded to C03, but Bazel reported as
>> "fastbuild")
>> 5. Stylecheck toggle (darwin vs darwin_nostyle)
>>
>> In a recent PR, I collapsed all of the Linux flavors to a standard
>> `--config=linux`. This was born out of my frustration with the build
>> scripts not jiving with Bazel standards. But it was not actually solving
>> the problem. I'm hoping that this PR will help resolve some of the issues.
>>
>> https://github.com/apache/incubator-heron/pull/3779
>>
>> Changes:
>> 1. No longer need to set `--config`. Bazel will detect the OS
>> automatically.
>> 2. Default is `fastbuild`, but you can add `-c opt` to build optimized
>> build.
>> a. You can even build both side by side with Bazel making a folder like
>> darwin-fastbuild and darwin-opt
>> 3. If you want to run stylecheck, you can add `--config=stylecheck`
>> 4. The Linux libraries (-lm, -lpthread, -lrt) are included in
>> `tools/platform/BUILD`
>> 5. As we add support for ARM, we can add more options in
>> `tools/platform/BUILD`
>> 6. Scripts outside of Bazel were updated to remove `--config`.
>> 7. Scripts outside of Bazel will specify `-c opt` or
>> `--config=stylecheck` when needed.
>>
>
Re: Removal of --config setting
Posted by Nicholas Nezis <ni...@gmail.com>.
Ok, so a lot more Bazel related cleanup on that PR.
https://github.com/apache/incubator-heron/pull/3779
1. Removed Docker specific `bazel.rc`
a. No longer needed now that we have JDK 11 and it works better in Docker
2. Removed Travis specific cross-compile toolchain
a. No longer needed now that Travis CI is using a modern OS version
3. Removed Travis specific `bazel.rc`
a. I don't think this is needed. This was due to the old OS and limited
VM resources. Can put back if necessary, but the tests seem to be passing
just fine. Even faster than before.
4. Removed ancient `Applatix` CI scripts
a. Not used or maintained. They were acquired by Intuit in 2018.
5. Updated documentation
a. Including a switch from installing Bazel to Bazelisk. This will help
with documentation maintenance.
On Sun, Feb 27, 2022 at 2:23 AM Nicholas Nezis <ni...@gmail.com>
wrote:
> While working through a bug we ran into with MacOS failing to compile, I
> got a better understanding of some of the Bazel settings. In another thread
> with Josh, who is compiling on the Apple M1 for the first time, he ran into
> issues with Linux vs OSX.
>
> It dawned on me that we were explicitly setting the config to modify a
> number of settings. Some of them include the following items:
> 1. Platform libraries (Linux C libraries)
> 2. Platform + CPU decision on which binary to use (Helm binary)
> 3. OS decision (MacOS unique steps vs Linux as the default)
> 4. Optimization setting (We just hardcoded to C03, but Bazel reported as
> "fastbuild")
> 5. Stylecheck toggle (darwin vs darwin_nostyle)
>
> In a recent PR, I collapsed all of the Linux flavors to a standard
> `--config=linux`. This was born out of my frustration with the build
> scripts not jiving with Bazel standards. But it was not actually solving
> the problem. I'm hoping that this PR will help resolve some of the issues.
>
> https://github.com/apache/incubator-heron/pull/3779
>
> Changes:
> 1. No longer need to set `--config`. Bazel will detect the OS
> automatically.
> 2. Default is `fastbuild`, but you can add `-c opt` to build optimized
> build.
> a. You can even build both side by side with Bazel making a folder like
> darwin-fastbuild and darwin-opt
> 3. If you want to run stylecheck, you can add `--config=stylecheck`
> 4. The Linux libraries (-lm, -lpthread, -lrt) are included in
> `tools/platform/BUILD`
> 5. As we add support for ARM, we can add more options in
> `tools/platform/BUILD`
> 6. Scripts outside of Bazel were updated to remove `--config`.
> 7. Scripts outside of Bazel will specify `-c opt` or
> `--config=stylecheck` when needed.
>