You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by GitBox <gi...@apache.org> on 2021/04/05 21:53:00 UTC
[GitHub] [arrow] alamb opened a new pull request #9897: ARROW-12214: [Rust][DataFusion] Add tests for limit
alamb opened a new pull request #9897:
URL: https://github.com/apache/arrow/pull/9897
I was debugging another issue (not a bug in DataFusion I don't think) but noticed there wasn't any coverage for LIMIT in exec.rs, so I figured I would add some.
(well really I was writing a test to trigger what I thought was a bug in DataFusion -- lol)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] alamb commented on a change in pull request #9897: ARROW-12214: [Rust][DataFusion] Add tests for limit
Posted by GitBox <gi...@apache.org>.
alamb commented on a change in pull request #9897:
URL: https://github.com/apache/arrow/pull/9897#discussion_r608578648
##########
File path: rust/datafusion/src/execution/context.rs
##########
@@ -1721,6 +1721,46 @@ mod tests {
Ok(())
}
+ #[tokio::test]
+ async fn limit() -> Result<()> {
+ let tmp_dir = TempDir::new()?;
+ let mut ctx = create_ctx(&tmp_dir, 1)?;
+ ctx.register_table("t", table_with_sequence(1, 1000).unwrap())
Review comment:
https://issues.apache.org/jira/browse/ARROW-12235 is one possibly problem I found
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] alamb commented on a change in pull request #9897: ARROW-12214: [Rust][DataFusion] Add tests for limit
Posted by GitBox <gi...@apache.org>.
alamb commented on a change in pull request #9897:
URL: https://github.com/apache/arrow/pull/9897#discussion_r607728739
##########
File path: rust/datafusion/src/execution/context.rs
##########
@@ -1721,6 +1721,46 @@ mod tests {
Ok(())
}
+ #[tokio::test]
+ async fn limit() -> Result<()> {
+ let tmp_dir = TempDir::new()?;
+ let mut ctx = create_ctx(&tmp_dir, 1)?;
+ ctx.register_table("t", table_with_sequence(1, 1000).unwrap())
Review comment:
I am not sure --- I have a program (https://github.com/influxdata/influxdb_iox/pull/1117) that I made for testing some stuff as we roll out IOx and there is definitely a problem with limit, though I haven't figured out if it is a problem in DataFusion or not
For this plan:
```
> explain select * from chunks limit 1;
+--------------+-----------------------------------------------------------------------------------------------------------------------------------------+
| plan_type | plan |
+--------------+-----------------------------------------------------------------------------------------------------------------------------------------+
| logical_plan | Limit: 1 |
| | Projection: #database_name, #id, #partition_key, #storage, #estimated_bytes, #time_of_first_write, #time_of_last_write, #time_closing |
| | TableScan: chunks projection=None |
+--------------+-----------------------------------------------------------------------------------------------------------------------------------------+
1 rows in set. Query took 0 seconds.
```
Produces no rows (and no columns) when there is a limit:
```
> select * from chunks limit 1;
++
||
++
++
0 rows in set. Query took 0 seconds.
```
But there is definitely data in that table:
```
> select * from chunks;
+-----------------------------------+-----+---------------------+---------------------+-----------------+-------------------------------+-------------------------------+-------------------------------+
| database_name | id | partition_key | storage | estimated_bytes | time_of_first_write | time_of_last_write | time_closing |
+-----------------------------------+-----+---------------------+---------------------+-----------------+-------------------------------+-------------------------------+-------------------------------+
| 844910ece80be8bc_05a7a51565539000 | 0 | 2021-04-05 21:00:00 | OpenMutableBuffer | 259733 | 2021-04-05 21:29:38.978576237 | 2021-04-05 21:49:47.995408514 | |
....
| 844910ece80be8bc_eaec8df57a81a1e9 | 1 | 2021-04-05 21:00:00 | OpenMutableBuffer | 6408933 | 2021-04-05 21:44:31.507950286 | 2021-04-05 21:55:37.226659960 | |
+-----------------------------------+-----+---------------------+---------------------+-----------------+-------------------------------+-------------------------------+-------------------------------+
226 rows in set. Query took 0 seconds.
```
I am still looking into it
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] Dandandan commented on a change in pull request #9897: ARROW-12214: [Rust][DataFusion] Add tests for limit
Posted by GitBox <gi...@apache.org>.
Dandandan commented on a change in pull request #9897:
URL: https://github.com/apache/arrow/pull/9897#discussion_r607551532
##########
File path: rust/datafusion/src/execution/context.rs
##########
@@ -1721,6 +1721,46 @@ mod tests {
Ok(())
}
+ #[tokio::test]
+ async fn limit() -> Result<()> {
+ let tmp_dir = TempDir::new()?;
+ let mut ctx = create_ctx(&tmp_dir, 1)?;
+ ctx.register_table("t", table_with_sequence(1, 1000).unwrap())
Review comment:
I was looking at `Limit` yesterday - it seems to poll new batches even when the limit has been reached (and throw away the result in the end)? Not wrong - but quite inefficient of course :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] github-actions[bot] commented on pull request #9897: ARROW-12214: [Rust][DataFusion] Add tests for limit
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on pull request #9897:
URL: https://github.com/apache/arrow/pull/9897#issuecomment-813692830
https://issues.apache.org/jira/browse/ARROW-12214
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] alamb commented on a change in pull request #9897: ARROW-12214: [Rust][DataFusion] Add tests for limit
Posted by GitBox <gi...@apache.org>.
alamb commented on a change in pull request #9897:
URL: https://github.com/apache/arrow/pull/9897#discussion_r608659790
##########
File path: rust/datafusion/src/execution/context.rs
##########
@@ -1721,6 +1721,46 @@ mod tests {
Ok(())
}
+ #[tokio::test]
+ async fn limit() -> Result<()> {
+ let tmp_dir = TempDir::new()?;
+ let mut ctx = create_ctx(&tmp_dir, 1)?;
+ ctx.register_table("t", table_with_sequence(1, 1000).unwrap())
Review comment:
> I was looking at Limit yesterday - it seems to poll new batches even when the limit has been reached (and throw away the result in the end)? Not wrong - but quite inefficient of course :)
@Dandandan I spent some time looking into this and you were absolutely correct. I have a PR (draft) up with a proposed fix: https://github.com/apache/arrow/pull/9926
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] alamb closed pull request #9897: ARROW-12214: [Rust][DataFusion] Add tests for limit
Posted by GitBox <gi...@apache.org>.
alamb closed pull request #9897:
URL: https://github.com/apache/arrow/pull/9897
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org