You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by GitBox <gi...@apache.org> on 2022/08/08 17:20:24 UTC
[GitHub] [arrow-rs] roeap opened a new pull request, #2374: feat: add token provider authorizatiojn to azure store
roeap opened a new pull request, #2374:
URL: https://github.com/apache/arrow-rs/pull/2374
# Which issue does this PR close?
Closes #2373
cc @tustvold @alamb
# Rationale for this change
Allow a wider range of use by offering more ob the azure native auth methods. While this change might eventually becoem irrelevant, once we move away from SDKs, it is quite helpful in the meantime.
# What changes are included in this PR?
Add an option to provide a `TokenProvider` credential via the `MicrosoftAzureBuilder`.
# Are there any user-facing changes?
users now have more authorization options.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-rs] roeap commented on a diff in pull request #2374: feat: add token provider authorization to azure store
Posted by GitBox <gi...@apache.org>.
roeap commented on code in PR #2374:
URL: https://github.com/apache/arrow-rs/pull/2374#discussion_r942933533
##########
object_store/src/azure.rs:
##########
@@ -277,79 +277,95 @@ impl ObjectStore for MicrosoftAzure {
}
async fn get(&self, location: &Path) -> Result<GetResult> {
- let blob = self
+ let loc = location.clone();
+ let mut stream = self
.container_client
- .as_blob_client(location.as_ref())
+ .blob_client(location.as_ref())
.get()
- .execute()
- .await
- .map_err(|err| {
- if check_err_not_found(&err) {
- return Error::NotFound {
- source: err,
- path: location.to_string(),
- };
- };
- Error::UnableToGetData {
- source: err,
- container: self.container_name.clone(),
- path: location.to_string(),
- }
- })?;
+ .into_stream()
+ .and_then(|chunk| chunk.data.collect())
+ .map_err(move |err| match err.kind() {
+ AzureErrorKind::HttpResponse {
+ status: StatusCode::NotFound,
+ ..
+ } => crate::Error::NotFound {
+ source: Box::new(err),
+ path: loc.to_string(),
+ },
+ _ => crate::Error::Generic {
+ source: Box::new(err),
+ store: "MicrosoftAzure",
+ },
+ })
+ .boxed();
- Ok(GetResult::Stream(
- futures::stream::once(async move { Ok(blob.data) }).boxed(),
- ))
+ let first = stream.next().await.transpose()?.unwrap_or_default();
+ Ok(GetResult::Stream(Box::pin(
+ futures::stream::once(async { Ok(first) }).chain(stream),
+ )))
Review Comment:
Somehow this felt a bit strange to collect the first element just to add it to a stream right after, but I saw no other obvious way to have the method raise an error. Before this we are never actually touching any results...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-rs] roeap commented on pull request #2374: feat: add token provider authorization to azure store
Posted by GitBox <gi...@apache.org>.
roeap commented on PR #2374:
URL: https://github.com/apache/arrow-rs/pull/2374#issuecomment-1211322592
> There do appear to be failing lints and tests btw
All tests and lints should be passing now.
> Very exciting, btw if you haven't seen #2352 that moves away from rusuto for S3, so the Azure SDKs are the only ones that still remain.
Very exciting indeed - one more reason to get this done and have everything nice and clean :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-rs] roeap commented on pull request #2374: feat: add token provider authorization to azure store
Posted by GitBox <gi...@apache.org>.
roeap commented on PR #2374:
URL: https://github.com/apache/arrow-rs/pull/2374#issuecomment-1208556432
@tustvold - had to convert this back to draft, since the currently released version of the crates do not support identity based credentials just yet ... Maybe I'll wait until this gets released, and do the updates then, or just get it over with and migrate the implementation to a custom one in this crate ...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-rs] roeap commented on a diff in pull request #2374: feat: add token provider authorization to azure store
Posted by GitBox <gi...@apache.org>.
roeap commented on code in PR #2374:
URL: https://github.com/apache/arrow-rs/pull/2374#discussion_r940489188
##########
object_store/src/azure.rs:
##########
@@ -627,6 +640,15 @@ impl MicrosoftAzureBuilder {
self
}
+ /// Set a TokenCredential to be used for authentication (required - one of token_credential ot access key)
+ pub fn with_token_credential(
+ mut self,
+ token_credential: Arc<dyn TokenCredential>,
Review Comment:
I haven't looked at the S§ implementation, but I seem to remember, that we do have some sense of a token credential as well. The problem I found with that, is that "get_token" accepts a client, while by now the client is somewhat hidden in the azure client. Also for gcs that was not a trait. Maybe thats the way to go?
An alternative might be to expose all (some) options required to initialize the token credentials. I guess we would always need that information regardless of the underlying implementation ..
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-rs] ursabot commented on pull request #2374: feat: add token provider authorization to azure store
Posted by GitBox <gi...@apache.org>.
ursabot commented on PR #2374:
URL: https://github.com/apache/arrow-rs/pull/2374#issuecomment-1214223110
Benchmark runs are scheduled for baseline = f0d7d0b8d41e5a13005f15b41c8f09f01e09ebed and contender = 12a9d84f0c7f8a5db20bfb7b0d7c2baf98d686aa. 12a9d84f0c7f8a5db20bfb7b0d7c2baf98d686aa is a master commit associated with this PR. Results will be available as each benchmark for each run completes.
Conbench compare runs links:
[Skipped :warning: Benchmarking of arrow-rs-commits is not supported on ec2-t3-xlarge-us-east-2] [ec2-t3-xlarge-us-east-2](https://conbench.ursa.dev/compare/runs/c0ec351edd1d49849d7fd61faf28b041...e64f0200aa374790b166f930085831ea/)
[Skipped :warning: Benchmarking of arrow-rs-commits is not supported on test-mac-arm] [test-mac-arm](https://conbench.ursa.dev/compare/runs/14505a4f6120420e99847ec092ddbe1f...fbcde1a0fabe44d7be6b7f5ab91a7982/)
[Skipped :warning: Benchmarking of arrow-rs-commits is not supported on ursa-i9-9960x] [ursa-i9-9960x](https://conbench.ursa.dev/compare/runs/bb89fa40edda4c6d8d12361acf1b1367...62988d6760eb4ec290da87a09dc28f25/)
[Skipped :warning: Benchmarking of arrow-rs-commits is not supported on ursa-thinkcentre-m75q] [ursa-thinkcentre-m75q](https://conbench.ursa.dev/compare/runs/624351bc25004ffbb52f389216d20f78...8734e8c455424b13b9a107d97b8804c1/)
Buildkite builds:
Supported benchmarks:
ec2-t3-xlarge-us-east-2: Supported benchmark langs: Python, R. Runs only benchmarks with cloud = True
test-mac-arm: Supported benchmark langs: C++, Python, R
ursa-i9-9960x: Supported benchmark langs: Python, R, JavaScript
ursa-thinkcentre-m75q: Supported benchmark langs: C++, Java
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-rs] tustvold commented on a diff in pull request #2374: feat: add token provider authorization to azure store
Posted by GitBox <gi...@apache.org>.
tustvold commented on code in PR #2374:
URL: https://github.com/apache/arrow-rs/pull/2374#discussion_r940483326
##########
object_store/src/azure.rs:
##########
@@ -615,7 +628,7 @@ impl MicrosoftAzureBuilder {
self
}
- /// Set the Azure Access Key (required)
+ /// Set the Azure Access Key (required - one of token_credential ot access key)
Review Comment:
```suggestion
/// Set the Azure Access Key (required - one of token_credential or access key)
```
##########
object_store/src/azure.rs:
##########
@@ -627,6 +640,15 @@ impl MicrosoftAzureBuilder {
self
}
+ /// Set a TokenCredential to be used for authentication (required - one of token_credential ot access key)
Review Comment:
```suggestion
/// Set a TokenCredential to be used for authentication (required - one of token_credential or access key)
```
##########
object_store/src/azure.rs:
##########
@@ -627,6 +640,15 @@ impl MicrosoftAzureBuilder {
self
}
+ /// Set a TokenCredential to be used for authentication (required - one of token_credential ot access key)
+ pub fn with_token_credential(
+ mut self,
+ token_credential: Arc<dyn TokenCredential>,
Review Comment:
Is there some way we could avoid leaking the azure_core type into the public interface, this will not only make it easier to use, but also survive a move away from the SDK?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-rs] tustvold commented on a diff in pull request #2374: feat: add token provider authorization to azure store
Posted by GitBox <gi...@apache.org>.
tustvold commented on code in PR #2374:
URL: https://github.com/apache/arrow-rs/pull/2374#discussion_r940490993
##########
object_store/src/azure.rs:
##########
@@ -627,6 +640,15 @@ impl MicrosoftAzureBuilder {
self
}
+ /// Set a TokenCredential to be used for authentication (required - one of token_credential ot access key)
+ pub fn with_token_credential(
+ mut self,
+ token_credential: Arc<dyn TokenCredential>,
Review Comment:
I'm not familiar enough with the various authentication options available in Azure-land to know the implications of asking this, but if we can expose the information for the various options that would be ideal imo...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-rs] roeap commented on a diff in pull request #2374: feat: add token provider authorization to azure store
Posted by GitBox <gi...@apache.org>.
roeap commented on code in PR #2374:
URL: https://github.com/apache/arrow-rs/pull/2374#discussion_r942683747
##########
object_store/src/azure.rs:
##########
@@ -277,79 +277,92 @@ impl ObjectStore for MicrosoftAzure {
}
async fn get(&self, location: &Path) -> Result<GetResult> {
- let blob = self
+ let loc = location.clone();
+ let stream = self
.container_client
- .as_blob_client(location.as_ref())
+ .blob_client(location.as_ref())
.get()
- .execute()
- .await
- .map_err(|err| {
- if check_err_not_found(&err) {
- return Error::NotFound {
- source: err,
- path: location.to_string(),
- };
- };
- Error::UnableToGetData {
- source: err,
- container: self.container_name.clone(),
- path: location.to_string(),
- }
- })?;
+ .into_stream()
+ .and_then(|chunk| chunk.data.collect())
+ .map_err(move |err| match err.kind() {
+ AzureErrorKind::HttpResponse {
+ status: StatusCode::NotFound,
+ ..
+ } => crate::Error::NotFound {
+ source: Box::new(err),
+ path: loc.to_string(),
+ },
+ _ => crate::Error::Generic {
+ source: Box::new(err),
+ store: "MicrosoftAzure",
Review Comment:
it feels like I am missing something really obvious here, but I struggled to get the container name included in this through various variations of cloning etc. but the compiler just kept on complaining about lifetimes ...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-rs] tustvold merged pull request #2374: feat: add token provider authorization to azure store
Posted by GitBox <gi...@apache.org>.
tustvold merged PR #2374:
URL: https://github.com/apache/arrow-rs/pull/2374
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-rs] roeap commented on a diff in pull request #2374: feat: add token provider authorization to azure store
Posted by GitBox <gi...@apache.org>.
roeap commented on code in PR #2374:
URL: https://github.com/apache/arrow-rs/pull/2374#discussion_r940492627
##########
object_store/src/azure.rs:
##########
@@ -627,6 +640,15 @@ impl MicrosoftAzureBuilder {
self
}
+ /// Set a TokenCredential to be used for authentication (required - one of token_credential ot access key)
+ pub fn with_token_credential(
+ mut self,
+ token_credential: Arc<dyn TokenCredential>,
Review Comment:
There are a few, but I can make that work ... also just realized I did this against the upcoming version, so I have to do some method renaming anyhow...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-rs] roeap commented on pull request #2374: feat: add token provider authorization to azure store
Posted by GitBox <gi...@apache.org>.
roeap commented on PR #2374:
URL: https://github.com/apache/arrow-rs/pull/2374#issuecomment-1211000843
The latest version of the Azure crates finally got released. Since there were many api changes between the last attempt (#2193) I did not update that PR but included it all here. The relevant update - authorizing via identity - is also part of the latest azure release.
Working on this made me realize once more that we really want to use our own, more focussed implementation of API interactions. I will take a look at that asap. in the meantime i was hoping to get the token auth in the next release, so while a bit finicky in some places, using the SDKs was more realistic to get doe in time (hopefully :)).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-rs] tustvold commented on pull request #2374: feat: add token provider authorization to azure store
Posted by GitBox <gi...@apache.org>.
tustvold commented on PR #2374:
URL: https://github.com/apache/arrow-rs/pull/2374#issuecomment-1211017441
There do appear to be failing lints and tests btw
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-rs] tustvold commented on a diff in pull request #2374: feat: add token provider authorization to azure store
Posted by GitBox <gi...@apache.org>.
tustvold commented on code in PR #2374:
URL: https://github.com/apache/arrow-rs/pull/2374#discussion_r942647896
##########
object_store/Cargo.toml:
##########
@@ -35,47 +31,109 @@ all-features = true
[dependencies] # In alphabetical order
async-trait = "0.1.53"
# Microsoft Azure Blob storage integration
-azure_core = { version = "0.2", optional = true, default-features = false, features = ["enable_reqwest_rustls"] }
-azure_storage = { version = "0.2", optional = true, default-features = false, features = ["account"] }
-azure_storage_blobs = { version = "0.2", optional = true, default-features = false, features = ["enable_reqwest_rustls"] }
+azure_core = { version = "0.4", optional = true, default-features = false, features = [
Review Comment:
FYI, I'm not sure what is formatting these, but I think it makes the file harder to read...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-rs] roeap commented on a diff in pull request #2374: feat: add token provider authorization to azure store
Posted by GitBox <gi...@apache.org>.
roeap commented on code in PR #2374:
URL: https://github.com/apache/arrow-rs/pull/2374#discussion_r942648959
##########
object_store/Cargo.toml:
##########
@@ -35,47 +31,109 @@ all-features = true
[dependencies] # In alphabetical order
async-trait = "0.1.53"
# Microsoft Azure Blob storage integration
-azure_core = { version = "0.2", optional = true, default-features = false, features = ["enable_reqwest_rustls"] }
-azure_storage = { version = "0.2", optional = true, default-features = false, features = ["account"] }
-azure_storage_blobs = { version = "0.2", optional = true, default-features = false, features = ["enable_reqwest_rustls"] }
+azure_core = { version = "0.4", optional = true, default-features = false, features = [
Review Comment:
I'll make my prettier less eager :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-rs] tustvold commented on a diff in pull request #2374: feat: add token provider authorization to azure store
Posted by GitBox <gi...@apache.org>.
tustvold commented on code in PR #2374:
URL: https://github.com/apache/arrow-rs/pull/2374#discussion_r942702801
##########
object_store/src/azure.rs:
##########
@@ -277,79 +277,92 @@ impl ObjectStore for MicrosoftAzure {
}
async fn get(&self, location: &Path) -> Result<GetResult> {
- let blob = self
+ let loc = location.clone();
+ let stream = self
.container_client
- .as_blob_client(location.as_ref())
+ .blob_client(location.as_ref())
.get()
- .execute()
- .await
- .map_err(|err| {
- if check_err_not_found(&err) {
- return Error::NotFound {
- source: err,
- path: location.to_string(),
- };
- };
- Error::UnableToGetData {
- source: err,
- container: self.container_name.clone(),
- path: location.to_string(),
- }
- })?;
+ .into_stream()
+ .and_then(|chunk| chunk.data.collect())
+ .map_err(move |err| match err.kind() {
+ AzureErrorKind::HttpResponse {
+ status: StatusCode::NotFound,
+ ..
+ } => crate::Error::NotFound {
Review Comment:
:heart:
##########
object_store/src/azure.rs:
##########
@@ -673,25 +700,41 @@ impl MicrosoftAzureBuilder {
(true, storage_client)
} else {
- (
- false,
- StorageAccountClient::new_access_key(
- Arc::clone(&http_client),
+ let client = if bearer_token.is_some() {
+ Ok(StorageClient::new_bearer_token(
&account,
- &access_key,
- ),
- )
+ bearer_token.unwrap(),
+ ))
+ } else if access_key.is_some() {
+ Ok(StorageClient::new_access_key(&account, access_key.unwrap()))
+ } else if client_id.is_some()
Review Comment:
You could use something like
```
if let (Some(client_id), Some(client_secret), Some(tenant_id)) = (tenant_id, client_id, client_secret) {
```
And avoid the unwraps
##########
object_store/src/azure.rs:
##########
@@ -673,25 +700,41 @@ impl MicrosoftAzureBuilder {
(true, storage_client)
} else {
- (
- false,
- StorageAccountClient::new_access_key(
- Arc::clone(&http_client),
+ let client = if bearer_token.is_some() {
Review Comment:
`if let Some(bearer_token) = bearer_token` might be a nicer way to write this as you can avoid the unwrap below
##########
object_store/src/azure.rs:
##########
@@ -277,79 +277,92 @@ impl ObjectStore for MicrosoftAzure {
}
async fn get(&self, location: &Path) -> Result<GetResult> {
- let blob = self
+ let loc = location.clone();
+ let stream = self
.container_client
- .as_blob_client(location.as_ref())
+ .blob_client(location.as_ref())
.get()
- .execute()
- .await
- .map_err(|err| {
- if check_err_not_found(&err) {
- return Error::NotFound {
- source: err,
- path: location.to_string(),
- };
- };
- Error::UnableToGetData {
- source: err,
- container: self.container_name.clone(),
- path: location.to_string(),
- }
- })?;
+ .into_stream()
+ .and_then(|chunk| chunk.data.collect())
+ .map_err(move |err| match err.kind() {
+ AzureErrorKind::HttpResponse {
+ status: StatusCode::NotFound,
+ ..
+ } => crate::Error::NotFound {
+ source: Box::new(err),
+ path: loc.to_string(),
+ },
+ _ => crate::Error::Generic {
+ source: Box::new(err),
+ store: "MicrosoftAzure",
Review Comment:
Yeah, this will be unavoidable - `GetResult::Stream` wants a `'static` lifetime on the stream, this prevents bundling a reference to `location` into the stream, as this would not be `'static`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org