You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by GitBox <gi...@apache.org> on 2021/04/17 11:29:31 UTC

[GitHub] [arrow] Dandandan opened a new pull request #10077: [Rust][DataFusion] Reproduce topkexec failure

Dandandan opened a new pull request #10077:
URL: https://github.com/apache/arrow/pull/10077


   This reproduces the test failure for the `TopKExec` test that @andygrove was seeing in combination with a 24-core machine.
   
   Also FYI @alamb .


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [arrow] github-actions[bot] commented on pull request #10077: [Rust][DataFusion] Reproduce topkexec failure

Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on pull request #10077:
URL: https://github.com/apache/arrow/pull/10077#issuecomment-821808634


   <!--
     Licensed to the Apache Software Foundation (ASF) under one
     or more contributor license agreements.  See the NOTICE file
     distributed with this work for additional information
     regarding copyright ownership.  The ASF licenses this file
     to you under the Apache License, Version 2.0 (the
     "License"); you may not use this file except in compliance
     with the License.  You may obtain a copy of the License at
   
       http://www.apache.org/licenses/LICENSE-2.0
   
     Unless required by applicable law or agreed to in writing,
     software distributed under the License is distributed on an
     "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
     KIND, either express or implied.  See the License for the
     specific language governing permissions and limitations
     under the License.
   -->
   
   Thanks for opening a pull request!
   
   If this is not a [minor PR](https://github.com/apache/arrow/blob/master/CONTRIBUTING.md#Minor-Fixes). Could you open an issue for this pull request on JIRA? https://issues.apache.org/jira/browse/ARROW
   
   Opening JIRAs ahead of time contributes to the [Openness](http://theapacheway.com/open/#:~:text=Openness%20allows%20new%20users%20the,must%20happen%20in%20the%20open.) of the Apache Arrow project.
   
   Then could you also rename pull request title in the following format?
   
       ARROW-${JIRA_ID}: [${COMPONENT}] ${SUMMARY}
   
   or
   
       MINOR: [${COMPONENT}] ${SUMMARY}
   
   See also:
   
     * [Other pull requests](https://github.com/apache/arrow/pulls/)
     * [Contribution Guidelines - How to contribute patches](https://arrow.apache.org/docs/developers/contributing.html#how-to-contribute-patches)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [arrow] andygrove closed pull request #10077: ARROW-12421: [Rust] [DataFusion] Fix topkexec failure

Posted by GitBox <gi...@apache.org>.
andygrove closed pull request #10077:
URL: https://github.com/apache/arrow/pull/10077


   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [arrow] Dandandan commented on pull request #10077: ARROW-12421: [Rust] [DataFusion] Fix topkexec failure

Posted by GitBox <gi...@apache.org>.
Dandandan commented on pull request #10077:
URL: https://github.com/apache/arrow/pull/10077#issuecomment-821971600


   Thanks @alamb @andygrove .
   
   It would still be valuable to see if we can improve the example, or even add it to DataFusion once we have a good implementation (without the extension stuff of course)!


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [arrow] andygrove commented on pull request #10077: [Rust][DataFusion] Reproduce topkexec failure

Posted by GitBox <gi...@apache.org>.
andygrove commented on pull request #10077:
URL: https://github.com/apache/arrow/pull/10077#issuecomment-821849479


   Thanks @Dandandan :rocket: 
   
   So if the issue was just in the test, I should change my vote on the release to a +1?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [arrow] Dandandan commented on pull request #10077: [Rust][DataFusion] Reproduce topkexec failure

Posted by GitBox <gi...@apache.org>.
Dandandan commented on pull request #10077:
URL: https://github.com/apache/arrow/pull/10077#issuecomment-821851791


   @andygrove as far as I can see the issue was in the test and faulty TopKExec implementation only, not really in repartion or the optimization of the repartion node.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [arrow] codecov-commenter commented on pull request #10077: [Rust][DataFusion] Reproduce topkexec failure

Posted by GitBox <gi...@apache.org>.
codecov-commenter commented on pull request #10077:
URL: https://github.com/apache/arrow/pull/10077#issuecomment-821849781


   # [Codecov](https://codecov.io/gh/apache/arrow/pull/10077?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
   > Merging [#10077](https://codecov.io/gh/apache/arrow/pull/10077?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (7d058ea) into [master](https://codecov.io/gh/apache/arrow/commit/9c1e5bd19347635ea9f373bcf93f2cea0231d50a?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (9c1e5bd) will **increase** coverage by `0.01%`.
   > The diff coverage is `83.33%`.
   
   [![Impacted file tree graph](https://codecov.io/gh/apache/arrow/pull/10077/graphs/tree.svg?width=650&height=150&src=pr&token=LpTCFbqVT1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)](https://codecov.io/gh/apache/arrow/pull/10077?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
   
   ```diff
   @@            Coverage Diff             @@
   ##           master   #10077      +/-   ##
   ==========================================
   + Coverage   78.90%   78.92%   +0.01%     
   ==========================================
     Files         286      286              
     Lines       64717    64713       -4     
   ==========================================
   + Hits        51068    51075       +7     
   + Misses      13649    13638      -11     
   ```
   
   
   | [Impacted Files](https://codecov.io/gh/apache/arrow/pull/10077?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
   |---|---|---|
   | [rust/datafusion/src/execution/context.rs](https://codecov.io/gh/apache/arrow/pull/10077/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cnVzdC9kYXRhZnVzaW9uL3NyYy9leGVjdXRpb24vY29udGV4dC5ycw==) | `92.60% <ø> (ø)` | |
   | [rust/datafusion/tests/user\_defined\_plan.rs](https://codecov.io/gh/apache/arrow/pull/10077/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cnVzdC9kYXRhZnVzaW9uL3Rlc3RzL3VzZXJfZGVmaW5lZF9wbGFuLnJz) | `87.74% <83.33%> (+0.32%)` | :arrow_up: |
   | [rust/datafusion/src/scalar.rs](https://codecov.io/gh/apache/arrow/pull/10077/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cnVzdC9kYXRhZnVzaW9uL3NyYy9zY2FsYXIucnM=) | `50.12% <0.00%> (+0.73%)` | :arrow_up: |
   | [rust/arrow/src/array/array.rs](https://codecov.io/gh/apache/arrow/pull/10077/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cnVzdC9hcnJvdy9zcmMvYXJyYXkvYXJyYXkucnM=) | `77.60% <0.00%> (+0.79%)` | :arrow_up: |
   | [...t/datafusion/src/physical\_plan/coalesce\_batches.rs](https://codecov.io/gh/apache/arrow/pull/10077/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cnVzdC9kYXRhZnVzaW9uL3NyYy9waHlzaWNhbF9wbGFuL2NvYWxlc2NlX2JhdGNoZXMucnM=) | `84.95% <0.00%> (+0.88%)` | :arrow_up: |
   | [rust/arrow/src/array/equal/utils.rs](https://codecov.io/gh/apache/arrow/pull/10077/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cnVzdC9hcnJvdy9zcmMvYXJyYXkvZXF1YWwvdXRpbHMucnM=) | `76.00% <0.00%> (+1.00%)` | :arrow_up: |
   | [...t/datafusion/src/physical\_optimizer/repartition.rs](https://codecov.io/gh/apache/arrow/pull/10077/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cnVzdC9kYXRhZnVzaW9uL3NyYy9waHlzaWNhbF9vcHRpbWl6ZXIvcmVwYXJ0aXRpb24ucnM=) | `96.72% <0.00%> (+4.91%)` | :arrow_up: |
   
   ------
   
   [Continue to review full report at Codecov](https://codecov.io/gh/apache/arrow/pull/10077?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
   > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
   > `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
   > Powered by [Codecov](https://codecov.io/gh/apache/arrow/pull/10077?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [9c1e5bd...7d058ea](https://codecov.io/gh/apache/arrow/pull/10077?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [arrow] Dandandan commented on pull request #10077: [Rust][DataFusion] Reproduce topkexec failure

Posted by GitBox <gi...@apache.org>.
Dandandan commented on pull request #10077:
URL: https://github.com/apache/arrow/pull/10077#issuecomment-821854443


   I probably won't get to that anymore today. Thanks, would be great


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [arrow] andygrove commented on pull request #10077: [Rust][DataFusion] Reproduce topkexec failure

Posted by GitBox <gi...@apache.org>.
andygrove commented on pull request #10077:
URL: https://github.com/apache/arrow/pull/10077#issuecomment-821852991


   ok, we probably need to get the flatbuffer dependency change in anyway, so
   we should re-introduce the repartition optimization. Do you want to create
   a PR for that? If not, I can get to it later today.
   
   On Sat, Apr 17, 2021 at 10:45 AM Daniël Heres ***@***.***>
   wrote:
   
   > @andygrove <https://github.com/andygrove> as far as I can see the issue
   > was in the test and faulty TopKExec implementation only, not really in
   > repartion or the optimization of the repartion node.
   >
   > —
   > You are receiving this because you were mentioned.
   > Reply to this email directly, view it on GitHub
   > <https://github.com/apache/arrow/pull/10077#issuecomment-821851791>, or
   > unsubscribe
   > <https://github.com/notifications/unsubscribe-auth/AAHEBRA26KAMAGCFYYAUZ4DTJG3JPANCNFSM43C7OCUA>
   > .
   >
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [arrow] alamb commented on pull request #10077: ARROW-12421: [Rust] [DataFusion] Fix topkexec failure

Posted by GitBox <gi...@apache.org>.
alamb commented on pull request #10077:
URL: https://github.com/apache/arrow/pull/10077#issuecomment-821970804


   Thanks @Dandandan 
   
   > @andygrove as far as I can see the issue was in the test and faulty TopKExec implementation only, not really in repartion or the optimization of the repartion node.
   
   For the record, the `TopKExec` is an example (not part of the DataFusion codebase) 
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [arrow] Dandandan commented on pull request #10077: [Rust][DataFusion] Reproduce topkexec failure

Posted by GitBox <gi...@apache.org>.
Dandandan commented on pull request #10077:
URL: https://github.com/apache/arrow/pull/10077#issuecomment-821847190


   @alamb @andygrove I pushed a version with a fix (I believe) for the `TopKExec` example. Not sure if the original version with `try_fold` could also be fixed to work, but this works at least. It now has a `.clone` on the map which is not a big deal for small `k` but probably would be better to avoid that. 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [arrow] github-actions[bot] commented on pull request #10077: ARROW-12421: [Rust] [DataFusion] Fix topkexec failure

Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on pull request #10077:
URL: https://github.com/apache/arrow/pull/10077#issuecomment-821897556


   https://issues.apache.org/jira/browse/ARROW-12421


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org