You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by GitBox <gi...@apache.org> on 2021/11/12 14:32:24 UTC
[GitHub] [arrow] paleolimbot opened a new pull request #11691: ARROW-14640 [R] reading data from S3
paleolimbot opened a new pull request #11691:
URL: https://github.com/apache/arrow/pull/11691
This PR adds the ability for `proxy_options` to be passed from R (`S3FileSystem$create()`) to C++ (`fs::S3Options.proxy_options`). This functionality is [available in Python](https://arrow.apache.org/docs/python/generated/pyarrow.fs.S3FileSystem.html#pyarrow.fs.S3FileSystem) but not in R.
Some issues I'm not sure about:
- I can test the failing case where there's an invalid URI passed but I'm not sure how to test the passing case (valid proxy URI specified).
- Should the `HTTP_PROXY` and/or the `HTTPS_PROXY` environment variables be read to set the default value (as suggested by the issue opener)? I can see how this would be useful but I'm too new to this to have an opinion about whether we should or shouldn't implement it at the R level.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] jorisvandenbossche commented on pull request #11691: ARROW-14640 [R] reading data from S3
Posted by GitBox <gi...@apache.org>.
jorisvandenbossche commented on pull request #11691:
URL: https://github.com/apache/arrow/pull/11691#issuecomment-968956933
> I can test the failing case where there's an invalid URI passed but I'm not sure how to test the passing case (valid proxy URI specified).
This was added to C++/Python in https://github.com/apache/arrow/pull/9996, and it seems we also only tested construction of the filesystem object with the keyword, and not whether it is actually using the proxy server.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] github-actions[bot] commented on pull request #11691: ARROW-14640 [R] reading data from S3
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on pull request #11691:
URL: https://github.com/apache/arrow/pull/11691#issuecomment-967162031
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] paleolimbot commented on pull request #11691: ARROW-14640 [R] reading data from S3
Posted by GitBox <gi...@apache.org>.
paleolimbot commented on pull request #11691:
URL: https://github.com/apache/arrow/pull/11691#issuecomment-971637002
Ok...both R and Python appear to work with a proxy!
``` r
with_squid_proxy <- function(expr) {
# brew install squid
# apt install squid
squid_proc <- processx::process$new("squid", args = "--foreground")
Sys.sleep(1)
stopifnot(squid_proc$is_alive())
on.exit({
squid_proc$interrupt()
})
force(expr)
}
# make sure proxy works with httr
with_squid_proxy({
httr::GET(
"http://httpbin.org/get",
httr::use_proxy("localhost", 3128), httr::verbose()
)
})
#> Response [http://httpbin.org/get]
#> Date: 2021-11-17 14:29
#> Status: 200
#> Content-Type: application/json
#> Size: 417 B
#> {
#> "args": {},
#> "headers": {
#> "Accept": "application/json, text/xml, application/xml, */*",
#> "Accept-Encoding": "deflate, gzip",
#> "Cache-Control": "max-age=259200",
#> "Host": "httpbin.org",
#> "User-Agent": "libcurl/7.64.1 r-curl/4.3.2 httr/1.4.2",
#> "X-Amzn-Trace-Id": "Root=1-619511bc-29ddd2da0df9bec865ef42ec"
#> },
#> ...
# remotes::install_github("apache/arrow/r#11691")
library(arrow, warn.conflicts = FALSE)
# make sure r/arrow works without proxy
arrow::s3_bucket("ursa-labs-taxi-data")$ls()
#> [1] "2009" "2010" "2011" "2012" "2013" "2014" "2015" "2016" "2017" "2018"
#> [11] "2019"
# make sure pyarrow works without proxy
reticulate::py_run_string("
from pyarrow.fs import S3FileSystem
from pyarrow import fs
s3 = S3FileSystem(region='us-east-2')
print(s3.get_file_info(fs.FileSelector('ursa-labs-taxi-data')))")
# make sure arrow fails with no proxy
s3_bucket("ursa-labs-taxi-data", proxy_options = "http://localhost:3128")$ls()
#> Error: IOError: When listing objects under key '' in bucket 'ursa-labs-taxi-data': AWS Error [code 99]: curlCode: 7, Couldn't connect to server
#> /Users/deweydunnington/Desktop/rscratch/arrow/cpp/src/arrow/filesystem/s3fs.cc:1861 TreeWalker::Walk(client_, io_context_, bucket, key, kListObjectsMaxKeys, handle_results, handle_error, handle_recursion)
#> /Users/deweydunnington/Desktop/rscratch/arrow/cpp/src/arrow/filesystem/s3fs.cc:2217 impl_->Walk(select, base_path.bucket, base_path.key, &results)
#> /Users/deweydunnington/Desktop/rscratch/arrow/cpp/src/arrow/filesystem/filesystem.cc:323 base_fs_->GetFileInfo(selector)
# make sure arrow succeeds with a good proxy
with_squid_proxy({
s3_bucket("ursa-labs-taxi-data", proxy_options = "http://localhost:3128")$ls()
})
#> [1] "2009" "2010" "2011" "2012" "2013" "2014" "2015" "2016" "2017" "2018"
#> [11] "2019"
# make sure pyarrow fails with no proxy
reticulate::py_run_string("
from pyarrow.fs import S3FileSystem
from pyarrow import fs
s3 = S3FileSystem(region='us-east-2', proxy_options='http://localhost:3128')
print(s3.get_file_info(fs.FileSelector('ursa-labs-taxi-data')))")
#> Error in py_run_string_impl(code, local, convert): OSError: When listing objects under key '' in bucket 'ursa-labs-taxi-data': AWS Error [code 99]: curlCode: 7, Couldn't connect to server
#>
#> Detailed traceback:
#> File "<string>", line 5, in <module>
#> File "pyarrow/_fs.pyx", line 431, in pyarrow._fs.FileSystem.get_file_info
#> File "pyarrow/error.pxi", line 143, in pyarrow.lib.pyarrow_internal_check_status
#> File "pyarrow/error.pxi", line 114, in pyarrow.lib.check_status
# make sure pyarrow succeeds with a good proxy
with_squid_proxy({
reticulate::py_run_string("
from pyarrow.fs import S3FileSystem
from pyarrow import fs
s3 = S3FileSystem(region='us-east-2', proxy_options='http://localhost:3128')
print(s3.get_file_info(fs.FileSelector('ursa-labs-taxi-data')))")
})
```
<sup>Created on 2021-11-17 by the [reprex package](https://reprex.tidyverse.org) (v2.0.1)</sup>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] paleolimbot commented on pull request #11691: ARROW-14640 [R] reading data from S3
Posted by GitBox <gi...@apache.org>.
paleolimbot commented on pull request #11691:
URL: https://github.com/apache/arrow/pull/11691#issuecomment-970738495
Thanks for this! I'll find a way to test this locally to make sure it works at least once...it's probably more in the scope of the C++ library to test (but I haven't looked for the tests yet!).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] ursabot edited a comment on pull request #11691: ARROW-14640 [R] reading data from S3
Posted by GitBox <gi...@apache.org>.
ursabot edited a comment on pull request #11691:
URL: https://github.com/apache/arrow/pull/11691#issuecomment-971773102
Benchmark runs are scheduled for baseline = 463b5dd1f58344aeec72946d6fae0f42b013a161 and contender = acf1d491cad1045ce71e31576081308f1b6897e9. acf1d491cad1045ce71e31576081308f1b6897e9 is a master commit associated with this PR. Results will be available as each benchmark for each run completes.
Conbench compare runs links:
[Finished :arrow_down:0.0% :arrow_up:0.0%] [ec2-t3-xlarge-us-east-2](https://conbench.ursa.dev/compare/runs/8a3b01213b574077b2de8dfa449edf4d...3c035e4c80cc4a4586bbd7a72b2581a8/)
[Finished :arrow_down:0.0% :arrow_up:0.0%] [ursa-i9-9960x](https://conbench.ursa.dev/compare/runs/222f4c677d3c4323907a497aae28004f...50b3020179644d0b87bbbf699985ea2c/)
[Finished :arrow_down:0.27% :arrow_up:0.0%] [ursa-thinkcentre-m75q](https://conbench.ursa.dev/compare/runs/6ed25ab4f15146d9b9bd3daabd7de931...c683e9a5795146f9b5a1234a3917feb8/)
Supported benchmarks:
ursa-i9-9960x: langs = Python, R, JavaScript
ursa-thinkcentre-m75q: langs = C++, Java
ec2-t3-xlarge-us-east-2: cloud = True
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] ursabot commented on pull request #11691: ARROW-14640 [R] reading data from S3
Posted by GitBox <gi...@apache.org>.
ursabot commented on pull request #11691:
URL: https://github.com/apache/arrow/pull/11691#issuecomment-971773102
Benchmark runs are scheduled for baseline = 463b5dd1f58344aeec72946d6fae0f42b013a161 and contender = acf1d491cad1045ce71e31576081308f1b6897e9. acf1d491cad1045ce71e31576081308f1b6897e9 is a master commit associated with this PR. Results will be available as each benchmark for each run completes.
Conbench compare runs links:
[Scheduled] [ec2-t3-xlarge-us-east-2](https://conbench.ursa.dev/compare/runs/8a3b01213b574077b2de8dfa449edf4d...3c035e4c80cc4a4586bbd7a72b2581a8/)
[Scheduled] [ursa-i9-9960x](https://conbench.ursa.dev/compare/runs/222f4c677d3c4323907a497aae28004f...50b3020179644d0b87bbbf699985ea2c/)
[Scheduled] [ursa-thinkcentre-m75q](https://conbench.ursa.dev/compare/runs/6ed25ab4f15146d9b9bd3daabd7de931...c683e9a5795146f9b5a1234a3917feb8/)
Supported benchmarks:
ursa-i9-9960x: langs = Python, R, JavaScript
ursa-thinkcentre-m75q: langs = C++, Java
ec2-t3-xlarge-us-east-2: cloud = True
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] pitrou commented on pull request #11691: ARROW-14640 [R] reading data from S3
Posted by GitBox <gi...@apache.org>.
pitrou commented on pull request #11691:
URL: https://github.com/apache/arrow/pull/11691#issuecomment-971769533
@jonkeane Any concern here?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] ursabot edited a comment on pull request #11691: ARROW-14640 [R] reading data from S3
Posted by GitBox <gi...@apache.org>.
ursabot edited a comment on pull request #11691:
URL: https://github.com/apache/arrow/pull/11691#issuecomment-971773102
Benchmark runs are scheduled for baseline = 463b5dd1f58344aeec72946d6fae0f42b013a161 and contender = acf1d491cad1045ce71e31576081308f1b6897e9. acf1d491cad1045ce71e31576081308f1b6897e9 is a master commit associated with this PR. Results will be available as each benchmark for each run completes.
Conbench compare runs links:
[Finished :arrow_down:0.0% :arrow_up:0.0%] [ec2-t3-xlarge-us-east-2](https://conbench.ursa.dev/compare/runs/8a3b01213b574077b2de8dfa449edf4d...3c035e4c80cc4a4586bbd7a72b2581a8/)
[Scheduled] [ursa-i9-9960x](https://conbench.ursa.dev/compare/runs/222f4c677d3c4323907a497aae28004f...50b3020179644d0b87bbbf699985ea2c/)
[Finished :arrow_down:0.27% :arrow_up:0.0%] [ursa-thinkcentre-m75q](https://conbench.ursa.dev/compare/runs/6ed25ab4f15146d9b9bd3daabd7de931...c683e9a5795146f9b5a1234a3917feb8/)
Supported benchmarks:
ursa-i9-9960x: langs = Python, R, JavaScript
ursa-thinkcentre-m75q: langs = C++, Java
ec2-t3-xlarge-us-east-2: cloud = True
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] ursabot edited a comment on pull request #11691: ARROW-14640 [R] reading data from S3
Posted by GitBox <gi...@apache.org>.
ursabot edited a comment on pull request #11691:
URL: https://github.com/apache/arrow/pull/11691#issuecomment-971773102
Benchmark runs are scheduled for baseline = 463b5dd1f58344aeec72946d6fae0f42b013a161 and contender = acf1d491cad1045ce71e31576081308f1b6897e9. acf1d491cad1045ce71e31576081308f1b6897e9 is a master commit associated with this PR. Results will be available as each benchmark for each run completes.
Conbench compare runs links:
[Finished :arrow_down:0.0% :arrow_up:0.0%] [ec2-t3-xlarge-us-east-2](https://conbench.ursa.dev/compare/runs/8a3b01213b574077b2de8dfa449edf4d...3c035e4c80cc4a4586bbd7a72b2581a8/)
[Scheduled] [ursa-i9-9960x](https://conbench.ursa.dev/compare/runs/222f4c677d3c4323907a497aae28004f...50b3020179644d0b87bbbf699985ea2c/)
[Scheduled] [ursa-thinkcentre-m75q](https://conbench.ursa.dev/compare/runs/6ed25ab4f15146d9b9bd3daabd7de931...c683e9a5795146f9b5a1234a3917feb8/)
Supported benchmarks:
ursa-i9-9960x: langs = Python, R, JavaScript
ursa-thinkcentre-m75q: langs = C++, Java
ec2-t3-xlarge-us-east-2: cloud = True
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] jonkeane closed pull request #11691: ARROW-14640 [R] reading data from S3
Posted by GitBox <gi...@apache.org>.
jonkeane closed pull request #11691:
URL: https://github.com/apache/arrow/pull/11691
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org