You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by GitBox <gi...@apache.org> on 2021/06/18 21:24:06 UTC

[GitHub] [arrow-datafusion] edrevo opened a new pull request #586: Add Keda autoscaling for ballista in k8s

edrevo opened a new pull request #586:
URL: https://github.com/apache/arrow-datafusion/pull/586


   # Which issue does this PR close?
   Closes #585.
   
   # What changes are included in this PR?
   This PR provides very primitive support for auto-scaling: it works like an on-off switch. If there are no active tasks, the number of executors drops to 0. If there are active tasks, the number of executors will increase to the maximum number of allowed executors configured in Keda.
   
   Further work could refine this so that the number of executors is proportional to the amount of outstanding tasks or something like that, but it is not trivial since we may end up killing executors that hold valuable shuffle data even if the aren't processing anything.
   
   Changes in this PR:
   
   - The scheduler now implements the necessary gRPC interface for Keda.
   - k8s user guide has been updated
   
   I have tested this manually and seen it scale to 0 and back up when there is a query.
   
   # Are there any user-facing changes?
   No


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [arrow-datafusion] houqp commented on pull request #586: Add Keda autoscaling for ballista in k8s

Posted by GitBox <gi...@apache.org>.
houqp commented on pull request #586:
URL: https://github.com/apache/arrow-datafusion/pull/586#issuecomment-864672018


   3rd party work should have their original license and copyright maintained, see https://www.apache.org/legal/src-headers.html#3party. I think adding a header like below should be good enough:
   
   ```
      Copyright 2020 The KEDA Authors.
   
      and others that have contributed code to the public domain.
   
      Licensed under the Apache License, Version 2.0 (the "License"); 
      you may not use this file except in compliance with the License.
      You may obtain a copy of the License at.
   
          http://www.apache.org/licenses/LICENSE-2.0
   
      Unless required by applicable law or agreed to in writing, software
      distributed under the License is distributed on an "AS IS" BASIS,
      WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
      See the License for the specific language governing permissions and
      limitations under the License.
   ```
   
   Copied from the bottom of https://github.com/kedacore/keda/blob/main/LICENSE.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [arrow-datafusion] edrevo commented on pull request #586: Add Keda autoscaling for ballista in k8s

Posted by GitBox <gi...@apache.org>.
edrevo commented on pull request #586:
URL: https://github.com/apache/arrow-datafusion/pull/586#issuecomment-863999860






-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [arrow-datafusion] edrevo commented on pull request #586: Add Keda autoscaling for ballista in k8s

Posted by GitBox <gi...@apache.org>.
edrevo commented on pull request #586:
URL: https://github.com/apache/arrow-datafusion/pull/586#issuecomment-866968825


   merge conflicts resovled


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [arrow-datafusion] edrevo commented on pull request #586: Add Keda autoscaling for ballista in k8s

Posted by GitBox <gi...@apache.org>.
edrevo commented on pull request #586:
URL: https://github.com/apache/arrow-datafusion/pull/586#issuecomment-869455003


   merge conflicts resolved


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [arrow-datafusion] andygrove merged pull request #586: Add Keda autoscaling for ballista in k8s

Posted by GitBox <gi...@apache.org>.
andygrove merged pull request #586:
URL: https://github.com/apache/arrow-datafusion/pull/586


   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [arrow-datafusion] codecov-commenter commented on pull request #586: Add Keda autoscaling for ballista in k8s

Posted by GitBox <gi...@apache.org>.
codecov-commenter commented on pull request #586:
URL: https://github.com/apache/arrow-datafusion/pull/586#issuecomment-864010199


   # [Codecov](https://codecov.io/gh/apache/arrow-datafusion/pull/586?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
   > Merging [#586](https://codecov.io/gh/apache/arrow-datafusion/pull/586?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (580e692) into [master](https://codecov.io/gh/apache/arrow-datafusion/commit/51e5445fa51cef4f72df5db7804906a729fc5aa6?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (51e5445) will **decrease** coverage by `0.05%`.
   > The diff coverage is `17.64%`.
   
   [![Impacted file tree graph](https://codecov.io/gh/apache/arrow-datafusion/pull/586/graphs/tree.svg?width=650&height=150&src=pr&token=JXwWBKD3D9&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)](https://codecov.io/gh/apache/arrow-datafusion/pull/586?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
   
   ```diff
   @@            Coverage Diff             @@
   ##           master     #586      +/-   ##
   ==========================================
   - Coverage   76.02%   75.96%   -0.06%     
   ==========================================
     Files         156      156              
     Lines       27063    27086      +23     
   ==========================================
   + Hits        20575    20577       +2     
   - Misses       6488     6509      +21     
   ```
   
   
   | [Impacted Files](https://codecov.io/gh/apache/arrow-datafusion/pull/586?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
   |---|---|---|
   | [ballista/rust/scheduler/src/lib.rs](https://codecov.io/gh/apache/arrow-datafusion/pull/586/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-YmFsbGlzdGEvcnVzdC9zY2hlZHVsZXIvc3JjL2xpYi5ycw==) | `19.17% <0.00%> (-1.65%)` | :arrow_down: |
   | [ballista/rust/scheduler/src/main.rs](https://codecov.io/gh/apache/arrow-datafusion/pull/586/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-YmFsbGlzdGEvcnVzdC9zY2hlZHVsZXIvc3JjL21haW4ucnM=) | `0.00% <0.00%> (ø)` | |
   | [ballista/rust/scheduler/src/state/mod.rs](https://codecov.io/gh/apache/arrow-datafusion/pull/586/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-YmFsbGlzdGEvcnVzdC9zY2hlZHVsZXIvc3JjL3N0YXRlL21vZC5ycw==) | `71.01% <54.54%> (+0.52%)` | :arrow_up: |
   
   ------
   
   [Continue to review full report at Codecov](https://codecov.io/gh/apache/arrow-datafusion/pull/586?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
   > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
   > `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
   > Powered by [Codecov](https://codecov.io/gh/apache/arrow-datafusion/pull/586?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [51e5445...580e692](https://codecov.io/gh/apache/arrow-datafusion/pull/586?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [arrow-datafusion] andygrove commented on pull request #586: Add Keda autoscaling for ballista in k8s

Posted by GitBox <gi...@apache.org>.
andygrove commented on pull request #586:
URL: https://github.com/apache/arrow-datafusion/pull/586#issuecomment-869023244


   Hi @edrevo could you rebase this one when you get a chance.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [arrow-datafusion] andygrove merged pull request #586: Add Keda autoscaling for ballista in k8s

Posted by GitBox <gi...@apache.org>.
andygrove merged pull request #586:
URL: https://github.com/apache/arrow-datafusion/pull/586


   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org