You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by GitBox <gi...@apache.org> on 2021/10/26 15:41:52 UTC
[GitHub] [iceberg] nastra opened a new pull request #3304: Run Benchmarks via GH actions
nastra opened a new pull request #3304:
URL: https://github.com/apache/iceberg/pull/3304
@rdblue this is a proposal that allows running the existing Benchmarks via GH actions by manually triggering the `JMH Benchmarks` action.
I have been doing some experiments on my own fork and the benchmark results can be seen in https://github.com/nastra/iceberg/actions/workflows/jmh-bechmarks.yml (note the **benchmark-results** artifact at the bottom with the uploaded results)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] nastra commented on a change in pull request #3304: Run Benchmarks via GH actions
Posted by GitBox <gi...@apache.org>.
nastra commented on a change in pull request #3304:
URL: https://github.com/apache/iceberg/pull/3304#discussion_r733349478
##########
File path: .github/workflows/jmh-bechmarks.yml
##########
@@ -0,0 +1,89 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+name: "JMH Benchmarks"
+on:
+ workflow_dispatch:
+ inputs:
+ repo:
+ description: 'Repo name with owner, such as apache/iceberg'
+ required: true
+ ref:
+ description: 'The branch name'
+ required: true
+ benchmarks:
+ description: 'A list of comma-separated double-quoted Benchmark names, such as "IcebergSourceFlatParquetDataReadBenchmark", "IcebergSourceFlatParquetDataFilterBenchmark"'
+ required: true
Review comment:
This is all user input that you enter when you trigger the benchmark

--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] kbendick commented on a change in pull request #3304: Run Benchmarks via GH actions
Posted by GitBox <gi...@apache.org>.
kbendick commented on a change in pull request #3304:
URL: https://github.com/apache/iceberg/pull/3304#discussion_r731268291
##########
File path: .github/workflows/java-ci-bechmarks.yml
##########
@@ -0,0 +1,90 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+name: "Java CI Benchmark"
+on:
+ pull_request:
+ types: [ labeled, opened, synchronize, reopened ]
+ paths-ignore:
+ - '.github/workflows/python-ci.yml'
+ - 'python/**'
+ - 'python_legacy/**'
+ - 'site/**'
+
+jobs:
+ matrix:
+ runs-on: ubuntu-latest
+ outputs:
+ matrix: ${{ steps.set-matrix.outputs.matrix }}
+ foundlabel: ${{ steps.set-matrix.outputs.foundlabel }}
+ steps:
+ - uses: actions/checkout@v2
+ - id: set-matrix
+ run: |
+ cat <<EOF > labels.json
+ ${{ toJson(github.event.pull_request.labels.*) }}
+ EOF
+ matrix=$(cat labels.json | jq '.[] | select(.name | startswith("benchmark-")) | .name' | jq .)
+ matrix=$(echo $matrix | sed 's/ /,/g' | sed 's/"/\"/g')
+ echo "::set-output name=matrix::[$matrix]"
+ echo "::set-output name=foundlabel::$(echo "[$matrix]" | jq 'if . | length > 0 then true else false end')"
+
+ show-matrix:
+ needs: matrix
+ runs-on: ubuntu-latest
+ steps:
+ - run: |
+ echo ${{ needs.matrix.outputs.matrix }}
+ echo ${{ needs.matrix.outputs.foundlabel }}
+
+ run-benchmark:
+ if: ${{ needs.matrix.outputs.foundlabel == 'true' }}
+ needs: matrix
+ runs-on: ubuntu-latest
+ strategy:
+ matrix:
+ benchmark: ${{ fromJson(needs.matrix.outputs.matrix) }}
+ env:
+ SPARK_LOCAL_IP: localhost
+ steps:
+ - uses: actions/checkout@v2
+ - uses: actions/setup-java@v1
+ with:
+ java-version: 11
+ - uses: actions/cache@v2
+ with:
+ path: ~/.gradle/caches
+ key: ${{ runner.os }}-gradle-${{ hashFiles('**/*.gradle') }}
+ restore-keys: ${{ runner.os }}-gradle
+ - run: echo -e "$(ip addr show eth0 | grep "inet\b" | awk '{print $2}' | cut -d/ -f1)\t$(hostname -f) $(hostname -s)" | sudo tee -a /etc/hosts
+
+ - name: Get Benchmark to run
+ id: get_benchmark
+ run: |
+ benchmark=$(echo ${{ matrix.benchmark }} | sed 's/benchmark-//g')
+ echo "::set-output name=BENCHMARK::${benchmark}Benchmark"
+
+ - name: Run Benchmark
+ run: ./gradlew :iceberg-spark:iceberg-spark3:jmh -PjmhIncludeRegex=${{ steps.get_benchmark.outputs.BENCHMARK }} -PjmhOutputPath=benchmark/${{ steps.get_benchmark.outputs.BENCHMARK }}.txt
Review comment:
Does this `jmhOutputPath` include a randomizing value or will it overwrite files instead? Is that even a concern with a github action?
The next upload step makes me think it might be to getting later logs but I'm not an expert in GH Actions by any means. 🙂
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] nastra commented on pull request #3304: Run Benchmarks via GH actions
Posted by GitBox <gi...@apache.org>.
nastra commented on pull request #3304:
URL: https://github.com/apache/iceberg/pull/3304#issuecomment-946460705
@kbendick @rdblue I pushed another commit that doesn't rely on labels and essentially creates an own Action that can be manually triggered with the respective benchmarks that should be executed. I also updated the PR description with the results from example runs on my own fork.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] nastra commented on a change in pull request #3304: Run Benchmarks via GH actions
Posted by GitBox <gi...@apache.org>.
nastra commented on a change in pull request #3304:
URL: https://github.com/apache/iceberg/pull/3304#discussion_r733350158
##########
File path: .github/workflows/jmh-bechmarks.yml
##########
@@ -0,0 +1,89 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+name: "JMH Benchmarks"
+on:
+ workflow_dispatch:
+ inputs:
+ repo:
+ description: 'Repo name with owner, such as apache/iceberg'
+ required: true
Review comment:
no, this is user input, since people most likely would be running benchmarks on their forks, they would enter their forked repo `nastra/iceberg`, the branch name and the benchmarks to run. The entered benchmarks are then executed as a matrix.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on pull request #3304: Run Benchmarks via GH actions
Posted by GitBox <gi...@apache.org>.
rdblue commented on pull request #3304:
URL: https://github.com/apache/iceberg/pull/3304#issuecomment-956927184
Thanks, @nastra!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue merged pull request #3304: Run Benchmarks via GH actions
Posted by GitBox <gi...@apache.org>.
rdblue merged pull request #3304:
URL: https://github.com/apache/iceberg/pull/3304
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue merged pull request #3304: Run Benchmarks via GH actions
Posted by GitBox <gi...@apache.org>.
rdblue merged pull request #3304:
URL: https://github.com/apache/iceberg/pull/3304
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] kbendick commented on a change in pull request #3304: Run Benchmarks via GH actions
Posted by GitBox <gi...@apache.org>.
kbendick commented on a change in pull request #3304:
URL: https://github.com/apache/iceberg/pull/3304#discussion_r735166717
##########
File path: .github/workflows/jmh-bechmarks.yml
##########
@@ -0,0 +1,89 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+name: "JMH Benchmarks"
+on:
+ workflow_dispatch:
+ inputs:
+ repo:
+ description: 'Repo name with owner, such as apache/iceberg'
+ required: true
Review comment:
Ah ok. Do we have any desire to force it to run in a fork? I’m thinking of resource conservation.
Though I guess only a select group of people (a small superset of committers or possibly just committers) can run actions in the main repo so probably not a concern.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] kbendick commented on a change in pull request #3304: Run Benchmarks via GH actions
Posted by GitBox <gi...@apache.org>.
kbendick commented on a change in pull request #3304:
URL: https://github.com/apache/iceberg/pull/3304#discussion_r732988240
##########
File path: .github/workflows/jmh-bechmarks.yml
##########
@@ -0,0 +1,89 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+name: "JMH Benchmarks"
+on:
+ workflow_dispatch:
+ inputs:
+ repo:
+ description: 'Repo name with owner, such as apache/iceberg'
+ required: true
+ ref:
+ description: 'The branch name'
+ required: true
+ benchmarks:
+ description: 'A list of comma-separated double-quoted Benchmark names, such as "IcebergSourceFlatParquetDataReadBenchmark", "IcebergSourceFlatParquetDataFilterBenchmark"'
+ required: true
Review comment:
Where do these inputs come from? How do we enter them? I tried looking at your runs but couldn't see. These are perhaps parsed from the three dot input box on the github action page?
If you could provide an example that would be great.
##########
File path: .github/workflows/jmh-bechmarks.yml
##########
@@ -0,0 +1,89 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+name: "JMH Benchmarks"
+on:
+ workflow_dispatch:
+ inputs:
+ repo:
+ description: 'Repo name with owner, such as apache/iceberg'
+ required: true
Review comment:
Does this get added to the input automatically by github to be where you run it from?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] nastra commented on pull request #3304: Run Benchmarks via GH actions
Posted by GitBox <gi...@apache.org>.
nastra commented on pull request #3304:
URL: https://github.com/apache/iceberg/pull/3304#issuecomment-950593362
> > > My biggest concern is that the benchmarks can take a while to run and the stability of the GH Action hosted runners is somewhat unknown (in regards to variability in performance runs)
> >
> >
> > @kbendick I agree and I share the same concern. However, I think it it still better to have a hosted way of running benchmarks, rather than having people to run them locally as the variability in performance will be equally bad, since you most likely will be still doing dev work while you run benchmarks locally.
> > One solution to that problem would be some dedicated GH runners with better CPUs and more memory. Would be interesting to know if that's possible to get from Apache INFRA
>
> I agree with your point here. And at least with time, the GH Actions env will hopefully have some sort of known variability (vs two different local dev envs that can be totally different in terms of available resources and what else the user is doing there).
>
> For that reason, and the fact that this can / should be run in a fork, I think this is a good idea.
>
> In my opinion, if / when we include this, we should:
>
> 1. Ensure we document the input parameters well - Can be done in a follow up.
> 2. Document that the benchmarks shouldn’t be taken as a great source of truth but rather something to help guide developer understanding of code changes (e.g. hopefully people don’t run these benchmarks, then run the same thing somewhere else that’s more stable with a different framework etc, and make production choices off of that). - Also can be a follow up.
> 3. Look into what impact, if any, running GH Actions in forks has on the resource pool for the whole `Apache` GitHub account. I know all projects share a limited set of concurrent runners, but I don’t know whether or not a fork of those projects has the same concerns. - I don’t think that forks use up the ASF’s limited GH Action concurrency / resources, but I admittedly don’t know. I only bring it up as the benchmarks can take quite some time to run.
> 4. Look into the possibility of hosted runners. I know that the Flink project has people run the test suite against Azure (and that the Azure account / resources are free). Possibly that’s an avenue worth exploring? I doubt ASF infra can provide us with much in the way of better guarantees on the GH Action runners. If they can, wonderful. But otherwise, taking a look at how the Flink project has devs set up Azure accounts that are free for running tests might be something to look into (maybe they just do this for free for open-source projects?).
>
> Overall, thanks for working on this. I’m always a little leery of providing much prominence to the JMH tests, but they’re much better than nothing and some power users make great use of them (i.e. - you slightly_smiling_face). I’ve just switched jobs and so don’t have a ton of free cycles, but once I do possibly we can collaborate on a slightly more standardized performance benchmark? I’ve had a WIP / rough version of one I’ve been working with for a while but haven’t found time to get it over the finish line.
I agree with your observations and I'm planning to look at all of those as follow-ups. I'd like to first get this merged so that we can experiment a bit more and see how stable things really are.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] nastra commented on pull request #3304: Run Benchmarks via GH actions
Posted by GitBox <gi...@apache.org>.
nastra commented on pull request #3304:
URL: https://github.com/apache/iceberg/pull/3304#issuecomment-952590374
@kbendick thanks for reviewing. I pushed a comment with some documentation.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on pull request #3304: Run Benchmarks via GH actions
Posted by GitBox <gi...@apache.org>.
rdblue commented on pull request #3304:
URL: https://github.com/apache/iceberg/pull/3304#issuecomment-956927184
Thanks, @nastra!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] kbendick commented on pull request #3304: Run Benchmarks via GH actions
Posted by GitBox <gi...@apache.org>.
kbendick commented on pull request #3304:
URL: https://github.com/apache/iceberg/pull/3304#issuecomment-950392422
> > My biggest concern is that the benchmarks can take a while to run and the stability of the GH Action hosted runners is somewhat unknown (in regards to variability in performance runs)
>
> @kbendick I agree and I share the same concern. However, I think it it still better to have a hosted way of running benchmarks, rather than having people to run them locally as the variability in performance will be equally bad, since you most likely will be still doing dev work while you run benchmarks locally.
>
> One solution to that problem would be some dedicated GH runners with better CPUs and more memory. Would be interesting to know if that's possible to get from Apache INFRA
I agree with your point here. And at least with time, the GH Actions env will hopefully have some sort known variability (vs two different local dev envs that can be totally different in terms of available resources and what else the user is doing there).
For the reason, and the fact that this can / should be run in a fork, I think this is a good idea.
In my opinion, if / when we include this, we should:
1) Ensure we document the input parameters well - Can be done in a follow up.
2) Document that the benchmarks shouldn’t be taken as a great source of truth but rather something to help guide developer understanding of code changes (e.g. hopefully people don’t run these benchmarks, then run the same thing somewhere else that’s more stable with a different framework etc, and make production choices off of that). - Also can be a follow up.
3) Look into the impact that running GH Actions in forks has on the resource pool for the whole `Apache` GitHub account. I know all projects share a limited set of concurrent runners, but I don’t know whether or not a fork of those projects has the same concerns. - I don’t think that forks use up the ASF’s limited GH Action concurrency / resources, but I admittedly don’t know. I only bring it up as the benchmarks can take quite some time to run.
4) Look into the possibility of hosted runners. I know that the Flink project has people run the test suite against Azure (and that the Azure account / resources are free). Possibly that’s an avenue worth exploring? I doubt ASF infra can provide us with much in the way of better guarantees on the GH Action runners. If they can, wonderful. But otherwise, taking a look at how the Flink project has devs set up Azure accounts that are free for running tests might be something to look into (maybe they just do this for free for open-source projects?).
Overall, thanks for working on this. I’m always a little leery of providing much prominence to the JMH tests, but they’re much better than nothing and some power users make great use of them (i.e. - you 🙂). I’ve just switched jobs and so don’t have a ton of free cycles, but once I do possibly we can collaborate on a slightly more standardized performance benchmark? I’ve had a WIP / rough version of one I’ve been working with for a while but haven’t found time to get it over the finish line.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue merged pull request #3304: Run Benchmarks via GH actions
Posted by GitBox <gi...@apache.org>.
rdblue merged pull request #3304:
URL: https://github.com/apache/iceberg/pull/3304
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] nastra commented on a change in pull request #3304: Run Benchmarks via GH actions
Posted by GitBox <gi...@apache.org>.
nastra commented on a change in pull request #3304:
URL: https://github.com/apache/iceberg/pull/3304#discussion_r731531191
##########
File path: .github/workflows/java-ci-bechmarks.yml
##########
@@ -0,0 +1,90 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+name: "Java CI Benchmark"
+on:
+ pull_request:
+ types: [ labeled, opened, synchronize, reopened ]
+ paths-ignore:
+ - '.github/workflows/python-ci.yml'
+ - 'python/**'
+ - 'python_legacy/**'
+ - 'site/**'
+
+jobs:
+ matrix:
+ runs-on: ubuntu-latest
+ outputs:
+ matrix: ${{ steps.set-matrix.outputs.matrix }}
+ foundlabel: ${{ steps.set-matrix.outputs.foundlabel }}
+ steps:
+ - uses: actions/checkout@v2
+ - id: set-matrix
+ run: |
+ cat <<EOF > labels.json
+ ${{ toJson(github.event.pull_request.labels.*) }}
+ EOF
+ matrix=$(cat labels.json | jq '.[] | select(.name | startswith("benchmark-")) | .name' | jq .)
+ matrix=$(echo $matrix | sed 's/ /,/g' | sed 's/"/\"/g')
+ echo "::set-output name=matrix::[$matrix]"
+ echo "::set-output name=foundlabel::$(echo "[$matrix]" | jq 'if . | length > 0 then true else false end')"
+
+ show-matrix:
+ needs: matrix
+ runs-on: ubuntu-latest
+ steps:
+ - run: |
+ echo ${{ needs.matrix.outputs.matrix }}
+ echo ${{ needs.matrix.outputs.foundlabel }}
+
+ run-benchmark:
+ if: ${{ needs.matrix.outputs.foundlabel == 'true' }}
+ needs: matrix
+ runs-on: ubuntu-latest
+ strategy:
+ matrix:
+ benchmark: ${{ fromJson(needs.matrix.outputs.matrix) }}
+ env:
+ SPARK_LOCAL_IP: localhost
+ steps:
+ - uses: actions/checkout@v2
+ - uses: actions/setup-java@v1
+ with:
+ java-version: 11
+ - uses: actions/cache@v2
+ with:
+ path: ~/.gradle/caches
+ key: ${{ runner.os }}-gradle-${{ hashFiles('**/*.gradle') }}
+ restore-keys: ${{ runner.os }}-gradle
+ - run: echo -e "$(ip addr show eth0 | grep "inet\b" | awk '{print $2}' | cut -d/ -f1)\t$(hostname -f) $(hostname -s)" | sudo tee -a /etc/hosts
+
+ - name: Get Benchmark to run
+ id: get_benchmark
+ run: |
+ benchmark=$(echo ${{ matrix.benchmark }} | sed 's/benchmark-//g')
+ echo "::set-output name=BENCHMARK::${benchmark}Benchmark"
+
+ - name: Run Benchmark
+ run: ./gradlew :iceberg-spark:iceberg-spark3:jmh -PjmhIncludeRegex=${{ steps.get_benchmark.outputs.BENCHMARK }} -PjmhOutputPath=benchmark/${{ steps.get_benchmark.outputs.BENCHMARK }}.txt
Review comment:
we don't need a randomized value here, as every benchmark execution in the matrix will have its own name, thus the gathered benchmark results will be unique. You can also check the results from my fork run in https://github.com/nastra/iceberg/actions/runs/1354529194 and see that this isn't an issue.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] nastra commented on pull request #3304: Run Benchmarks via GH actions
Posted by GitBox <gi...@apache.org>.
nastra commented on pull request #3304:
URL: https://github.com/apache/iceberg/pull/3304#issuecomment-948675715
> My biggest concern is that the benchmarks can take a while to run and the stability of the GH Action hosted runners is somewhat unknown (in regards to variability in performance runs)
@kbendick I agree and I share the same concern. However, I think it it still better to have a hosted way of running benchmarks, rather than having people to run them locally as the variability in performance will be equally bad, since you most likely will be still doing dev work while you run benchmarks locally.
One solution to that problem would be some dedicated GH runners with better CPUs and more memory. Would be interesting to know if that's possible to get from Apache INFRA
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] nastra commented on a change in pull request #3304: Run Benchmarks via GH actions
Posted by GitBox <gi...@apache.org>.
nastra commented on a change in pull request #3304:
URL: https://github.com/apache/iceberg/pull/3304#discussion_r735309812
##########
File path: .github/workflows/jmh-bechmarks.yml
##########
@@ -0,0 +1,89 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+name: "JMH Benchmarks"
+on:
+ workflow_dispatch:
+ inputs:
+ repo:
+ description: 'Repo name with owner, such as apache/iceberg'
+ required: true
Review comment:
this is probably not an issue as most people will most likely run it on their fork anyway
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] nastra commented on pull request #3304: Run Benchmarks via GH actions
Posted by GitBox <gi...@apache.org>.
nastra commented on pull request #3304:
URL: https://github.com/apache/iceberg/pull/3304#issuecomment-946409702
> That seems like a lot tags to add for benchmarks. We would probably need to keep those up to date, and it would clutter the PR list.
>
> Is it possible to either reduce the number of tags to groups, like arrow-benchmarks and spark-benchmarks, or to change this to run benchmarks when specific files change instead? It seems like those would be easier ways to configure this. I'm a fan of using Github actions, though!
I agree that the number of tags required are quite long and have to be kept in sync with new benchmarks being added. However, I was thinking that this might be acceptable for the beginning until we know that all benchmarks are actually stable enough to run on GH actions.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] kbendick commented on pull request #3304: Run Benchmarks via GH actions
Posted by GitBox <gi...@apache.org>.
kbendick commented on pull request #3304:
URL: https://github.com/apache/iceberg/pull/3304#issuecomment-946416898
Can we possibly make it a separate action entirely? Some of them can take
quite a while to run.
We might want to just follow up on those issues if they become problems. Im
not a huge fan of running these tests via CI as I'm just admittedly not a
huge fan of JMH tests.
But I think this is good enough for now. Perhaps you can look for an
existing JMH workflow or some pattern to help form groupings or easily
allow us to.
+1 overall from me if we feel like having this. But I do see the utility.
On Mon, Oct 18, 2021 at 11:38 PM Eduard Tudenhöfner <
***@***.***> wrote:
> That seems like a lot tags to add for benchmarks. We would probably need
> to keep those up to date, and it would clutter the PR list.
>
> Is it possible to either reduce the number of tags to groups, like
> arrow-benchmarks and spark-benchmarks, or to change this to run benchmarks
> when specific files change instead? It seems like those would be easier
> ways to configure this. I'm a fan of using Github actions, though!
>
> I agree that the number of tags required are quite long and have to be
> kept in sync with new benchmarks being added. However, I was thinking that
> this might be acceptable for the beginning until we know that all
> benchmarks are actually stable enough to run on GH actions.
>
> —
> You are receiving this because you commented.
> Reply to this email directly, view it on GitHub
> <https://github.com/apache/iceberg/pull/3304#issuecomment-946409702>, or
> unsubscribe
> <https://github.com/notifications/unsubscribe-auth/ACLAXEXABFUCVMY43IKARE3UHUG45ANCNFSM5GGPEBIQ>
> .
> Triage notifications on the go with GitHub Mobile for iOS
> <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675>
> or Android
> <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>.
>
>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on pull request #3304: Run Benchmarks via GH actions
Posted by GitBox <gi...@apache.org>.
rdblue commented on pull request #3304:
URL: https://github.com/apache/iceberg/pull/3304#issuecomment-946125926
That seems like a lot tags to add for benchmarks. We would probably need to keep those up to date, and it would clutter the PR list.
Is it possible to either reduce the number of tags to groups, like arrow-benchmarks and spark-benchmarks, or to change this to run benchmarks when specific files change instead? It seems like those would be easier ways to configure this. I'm a fan of using Github actions, though!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] nastra closed pull request #3304: Run Benchmarks via GH actions
Posted by GitBox <gi...@apache.org>.
nastra closed pull request #3304:
URL: https://github.com/apache/iceberg/pull/3304
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on pull request #3304: Run Benchmarks via GH actions
Posted by GitBox <gi...@apache.org>.
rdblue commented on pull request #3304:
URL: https://github.com/apache/iceberg/pull/3304#issuecomment-956927184
Thanks, @nastra!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] kbendick edited a comment on pull request #3304: Run Benchmarks via GH actions
Posted by GitBox <gi...@apache.org>.
kbendick edited a comment on pull request #3304:
URL: https://github.com/apache/iceberg/pull/3304#issuecomment-950392422
> > My biggest concern is that the benchmarks can take a while to run and the stability of the GH Action hosted runners is somewhat unknown (in regards to variability in performance runs)
>
> @kbendick I agree and I share the same concern. However, I think it it still better to have a hosted way of running benchmarks, rather than having people to run them locally as the variability in performance will be equally bad, since you most likely will be still doing dev work while you run benchmarks locally.
>
> One solution to that problem would be some dedicated GH runners with better CPUs and more memory. Would be interesting to know if that's possible to get from Apache INFRA
I agree with your point here. And at least with time, the GH Actions env will hopefully have some sort of known variability (vs two different local dev envs that can be totally different in terms of available resources and what else the user is doing there).
For that reason, and the fact that this can / should be run in a fork, I think this is a good idea.
In my opinion, if / when we include this, we should:
1) Ensure we document the input parameters well - Can be done in a follow up.
2) Document that the benchmarks shouldn’t be taken as a great source of truth but rather something to help guide developer understanding of code changes (e.g. hopefully people don’t run these benchmarks, then run the same thing somewhere else that’s more stable with a different framework etc, and make production choices off of that). - Also can be a follow up.
3) Look into what impact, if any, running GH Actions in forks has on the resource pool for the whole `Apache` GitHub account. I know all projects share a limited set of concurrent runners, but I don’t know whether or not a fork of those projects has the same concerns. - I don’t think that forks use up the ASF’s limited GH Action concurrency / resources, but I admittedly don’t know. I only bring it up as the benchmarks can take quite some time to run.
4) Look into the possibility of hosted runners. I know that the Flink project has people run the test suite against Azure (and that the Azure account / resources are free). Possibly that’s an avenue worth exploring? I doubt ASF infra can provide us with much in the way of better guarantees on the GH Action runners. If they can, wonderful. But otherwise, taking a look at how the Flink project has devs set up Azure accounts that are free for running tests might be something to look into (maybe they just do this for free for open-source projects?).
Overall, thanks for working on this. I’m always a little leery of providing much prominence to the JMH tests, but they’re much better than nothing and some power users make great use of them (i.e. - you 🙂). I’ve just switched jobs and so don’t have a ton of free cycles, but once I do possibly we can collaborate on a slightly more standardized performance benchmark? I’ve had a WIP / rough version of one I’ve been working with for a while but haven’t found time to get it over the finish line.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] kbendick commented on pull request #3304: Run Benchmarks via GH actions
Posted by GitBox <gi...@apache.org>.
kbendick commented on pull request #3304:
URL: https://github.com/apache/iceberg/pull/3304#issuecomment-946420479
Actually, looking at it now, it is quite a clutter of tags, even for just a
few. So I withdraw that +1. Sorry.
A hack that I’ve seen is to set the GitHub action configuration via a
string in the three dot menu (it has an input similar to the one you’re
using now). Like a comma separated input list maybe. Might not be safe for
user input but I guess tags aren't really either and I have no problem with
those.
It's not really a huge concern, but I can look into locking down the
permissions for the generated tokens for the actions like I did for the
labeler. It's supposed to be a good idea in repos like this that are
commonly forked. https://github.com/apache/iceberg/pull/3007
On Mon, Oct 18, 2021 at 11:51 PM Kyle Bendickson ***@***.***> wrote:
> Can we possibly make it a separate action entirely? Some of them can take
> quite a while to run.
>
> We might want to just follow up on those issues if they become problems.
> Im not a huge fan of running these tests via CI as I'm just admittedly not
> a huge fan of JMH tests.
>
> But I think this is good enough for now. Perhaps you can look for an
> existing JMH workflow or some pattern to help form groupings or easily
> allow us to.
>
> +1 overall from me if we feel like having this. But I do see the utility.
>
> On Mon, Oct 18, 2021 at 11:38 PM Eduard Tudenhöfner <
> ***@***.***> wrote:
>
>> That seems like a lot tags to add for benchmarks. We would probably need
>> to keep those up to date, and it would clutter the PR list.
>>
>> Is it possible to either reduce the number of tags to groups, like
>> arrow-benchmarks and spark-benchmarks, or to change this to run benchmarks
>> when specific files change instead? It seems like those would be easier
>> ways to configure this. I'm a fan of using Github actions, though!
>>
>> I agree that the number of tags required are quite long and have to be
>> kept in sync with new benchmarks being added. However, I was thinking that
>> this might be acceptable for the beginning until we know that all
>> benchmarks are actually stable enough to run on GH actions.
>>
>> —
>> You are receiving this because you commented.
>> Reply to this email directly, view it on GitHub
>> <https://github.com/apache/iceberg/pull/3304#issuecomment-946409702>, or
>> unsubscribe
>> <https://github.com/notifications/unsubscribe-auth/ACLAXEXABFUCVMY43IKARE3UHUG45ANCNFSM5GGPEBIQ>
>> .
>> Triage notifications on the go with GitHub Mobile for iOS
>> <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675>
>> or Android
>> <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>.
>>
>>
>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] nastra commented on pull request #3304: Run Benchmarks via GH actions
Posted by GitBox <gi...@apache.org>.
nastra commented on pull request #3304:
URL: https://github.com/apache/iceberg/pull/3304#issuecomment-946441070
After giving it some more thought, I'll experiment a bit with the `workflow_dispatch` event which would allow running the benchmark s completely separate from the PRs by giving it a list of Benchmark names
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org