You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by "alamb (via GitHub)" <gi...@apache.org> on 2023/08/21 21:56:44 UTC
[GitHub] [arrow-datafusion] alamb opened a new issue, #7357: Trying to detach `ExecutionPlan` and `DataSource`
alamb opened a new issue, #7357:
URL: https://github.com/apache/arrow-datafusion/issues/7357
### Is your feature request related to a problem or challenge?
I am bringing the conversation from this thread into its own ticket so it isn't lost so deep down in the weeds: https://github.com/apache/arrow-datafusion/pull/7175#issuecomment-1684293835
Basically https://github.com/apache/arrow-datafusion/pull/7175 introduced an (explicit) dependency between datafusion core and the physical plans (I am trying to cut out the physical plans into their own crate in https://github.com/apache/arrow-datafusion/issues/1754 )
### Describe the solution you'd like
I would like to remove the dependency from physical_plan --> datasource (currently datasource depends on physical_plan) specifically this line:
https://github.com/apache/arrow-datafusion/blob/6aa423b70b4eb1f28165919fbfb4efe323669bc4/datafusion/core/src/physical_plan/mod.rs#L26
as @not-my-profile has pointed out the connections are non trivial and sometimes implicit, so the sooner we force the API to stay clean the better off we'll be
### Describe alternatives you've considered
I am not sure -- it seems like we should be able to pull the file config function into its own trait, and then downcast to as_any as @not-my-profile suggested in https://github.com/apache/arrow-datafusion/pull/7175#issuecomment-1684547594 but I was having trouble making that work
### Additional context
_No response_
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] not-my-profile commented on issue #7357: Trying to detach `ExecutionPlan` and `DataSource`
Posted by "not-my-profile (via GitHub)" <gi...@apache.org>.
not-my-profile commented on issue #7357:
URL: https://github.com/apache/arrow-datafusion/issues/7357#issuecomment-1707201455
You can certainly do that ... it's just ugly. Ideally all of the code related to avro would be in one module that's gated with `#![cfg(feature = "avro")]`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] not-my-profile commented on issue #7357: Trying to detach `ExecutionPlan` and `DataSource`
Posted by "not-my-profile (via GitHub)" <gi...@apache.org>.
not-my-profile commented on issue #7357:
URL: https://github.com/apache/arrow-datafusion/issues/7357#issuecomment-1707169485
Sorry for only responding now. As I explained in #7175 my motivation was "to feature-gate AvroExec in a followup commit, so that you cannot get bitten by AvroExec::execute returning an NotImplemented error if the "avro" feature isn't enabled".
I unfortunately currently don't really have time to look into this. I see that in #7425 you're again adding the downcasting, which is hard to feature gate and just not extensible ... I think it should really be possible to achieve both a trait-based solution as well as the decoupling you speak of but I'm not familiar enough with the codebase to be able to suggest the solution and as I said don't really have time to read through the code.
If you could give me a minimal self-contained example of the APIs that have to work together I could think of a solution, e.g:
```rust
mod physical_plan {
trait ExecutionPlan {
}
// whatever
}
mod datasource {
// whatever
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] tustvold commented on issue #7357: Trying to detach `ExecutionPlan` and `DataSource`
Posted by "tustvold (via GitHub)" <gi...@apache.org>.
tustvold commented on issue #7357:
URL: https://github.com/apache/arrow-datafusion/issues/7357#issuecomment-1707193574
> you cannot get bitten by AvroExec::execute returning an NotImplemented error if the "avro" feature isn't enabled
I might be missing something but why can't you do
```
fn config(a: &dyn ExecutionPlan) -> Option<FileScanConfig> {
#[cfg(feature = "avro")]
if let Some(v) = a.as_any().downcast_ref::<AvroExec>() {
return Some(...)
}
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] alamb commented on issue #7357: Trying to detach `ExecutionPlan` and `DataSource`
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #7357:
URL: https://github.com/apache/arrow-datafusion/issues/7357#issuecomment-1687102708
Specifically, the API added in https://github.com/apache/arrow-datafusion/pull/7175 recursively walks the ExecutionPlan tree and I can't figure out how to get rust to let me try and downcast from `&dyn ExecutionPlan` to some other trait (I am not sure it is possible).
@not-my-profile I wonder if you can share a little bit more about your usecase so that I can try to find some API that works for you that doesn't have this explicit back dependency
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] alamb commented on issue #7357: Trying to detach `ExecutionPlan` and `DataSource`
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #7357:
URL: https://github.com/apache/arrow-datafusion/issues/7357#issuecomment-1707262827
> You can certainly do that ... it's just ugly. Ideally all of the code related to avro would be in one module that's gated with #![cfg(feature = "avro")]. And also of course hard coded downcasting isn't extensible by API users.
I agree it is ugly. I wish I had a better solution.
cleaning up the Avro code into a single module would be very nice indeed. Maybe we can make some PRs to head in that direction
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] tustvold closed issue #7357: Detach `ExecutionPlan` and `DataSource`
Posted by "tustvold (via GitHub)" <gi...@apache.org>.
tustvold closed issue #7357: Detach `ExecutionPlan` and `DataSource`
URL: https://github.com/apache/arrow-datafusion/issues/7357
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org