You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by "smallzhongfeng (via GitHub)" <gi...@apache.org> on 2023/11/01 12:04:06 UTC
[I] Run `datafusion-examples` when start CI [arrow-datafusion]
smallzhongfeng opened a new issue, #8018:
URL: https://github.com/apache/arrow-datafusion/issues/8018
### Is your feature request related to a problem or challenge?
Currently, we have not verified the accuracy of the test classes in the datafusion-example directory, so we should test it while running ci. Reduce the probability of coding errors
### Describe the solution you'd like
The current idea may be relatively simple. I will create two directories, one named external_storage and one named flight_sql. External_storage will contain test cases that require external storage, and flight_sql will contain flight-related test cases. When I run my tests I skip these two directories.
### Describe alternatives you've considered
_No response_
### Additional context
_No response_
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Run `datafusion-examples` when start CI [arrow-datafusion]
Posted by "smallzhongfeng (via GitHub)" <gi...@apache.org>.
smallzhongfeng commented on issue #8018:
URL: https://github.com/apache/arrow-datafusion/issues/8018#issuecomment-1790743830
> > External_storage will contain test cases that require external storage, and flight_sql will contain flight-related test cases. When I run my tests I skip these two directories.
>
> Another option would be to use crate features -- and have two new features like `flight` and `external_storage` that default to on, but are not enabled in CI
Sounds great! I try the way you said, but it seems fail. It throw
```
consider adding a `main` function to `datafusion-examples/examples/dataframe-to-s3.rs`
```
I don’t know if I understand what you mean. I added the feature to the test class dataframe-to-s3.rs. like this:
```
#[cfg(feature = "external_service")]
#[tokio::main]
async fn main() -> Result<()> {
// create local execution context
let ctx = SessionContext::new();
...
```
It couldn't find the specific runtime. I was wondering if I need to encapsulate some logic into a method and then define the feature on this method.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Run `datafusion-examples` when start CI [arrow-datafusion]
Posted by "devinjdangelo (via GitHub)" <gi...@apache.org>.
devinjdangelo commented on issue #8018:
URL: https://github.com/apache/arrow-datafusion/issues/8018#issuecomment-1788974880
This sounds like a good idea to me, but will take a little work in at least one case...
The example showing writing to S3 won't work in CI out of the box since it requires specifying a bucket and credentials with write access. Public write access is not appropriate here, so we would need a set up to securely pass creds to the ci pipeline.
We could also just skip this one example.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Run `datafusion-examples` when start CI [arrow-datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #8018:
URL: https://github.com/apache/arrow-datafusion/issues/8018#issuecomment-1790596102
> External_storage will contain test cases that require external storage, and flight_sql will contain flight-related test cases. When I run my tests I skip these two directories.
Another option would be to use crate features -- and have two new features like `flight` and `external_storage` that default to on, but are not enabled in CI
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Run `datafusion-examples` when start CI [arrow-datafusion]
Posted by "smallzhongfeng (via GitHub)" <gi...@apache.org>.
smallzhongfeng commented on issue #8018:
URL: https://github.com/apache/arrow-datafusion/issues/8018#issuecomment-1788842288
cc @alamb let me know what you think. Thanks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Run `datafusion-examples` when start CI [arrow-datafusion]
Posted by "smallzhongfeng (via GitHub)" <gi...@apache.org>.
smallzhongfeng commented on issue #8018:
URL: https://github.com/apache/arrow-datafusion/issues/8018#issuecomment-1790292103
> This sounds like a good idea to me, but will take a little work in at least one case...
>
> The example showing writing to S3 won't work in CI out of the box since it requires specifying a bucket and credentials with write access. Public write access is not appropriate here, so we would need a set up to securely pass creds to the ci pipeline.
>
> We could also just skip this one example.
Yes, I also think I should skip these tests that rely on external storage.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Run `datafusion-examples` when start CI [arrow-datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb closed issue #8018: Run `datafusion-examples` when start CI
URL: https://github.com/apache/arrow-datafusion/issues/8018
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org