You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by "davidhcoe (via GitHub)" <gi...@apache.org> on 2025/05/16 22:03:26 UTC
[PR] feat(csharp/src/Drivers/BigQuery): Add support for Http proxy [arrow-adbc]
davidhcoe opened a new pull request, #2831:
URL: https://github.com/apache/arrow-adbc/pull/2831
Adds support for calling a proxy via HTTP using the `"adbc.bigquery.client.proxy` parameter.
Does not support calling a proxy to the gRPC v1 storage endpoint due to net472 limitations.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(csharp/src/Drivers/BigQuery): Add support for Http proxy [arrow-adbc]
Posted by "davidhcoe (via GitHub)" <gi...@apache.org>.
davidhcoe commented on PR #2831:
URL: https://github.com/apache/arrow-adbc/pull/2831#issuecomment-2887756099
Proxy setup:
Windows machine A, running the test suite.
Windows machine B, running Fiddler on port 8888.
Set the proxy to the IP of machine B on port 8888.
A --> B --> BigQuery
Even when run `curl -x http://<ip_machine_b>:8888 https://www.google.com` from machine A it will route through and return data, but not when `http://<ip_machine_b>:8888` is set for the grpc_proxy, https_proxy or http_proxy environment variable.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(csharp/src/Drivers/BigQuery): Add support for Http proxy [arrow-adbc]
Posted by "davidhcoe (via GitHub)" <gi...@apache.org>.
davidhcoe commented on PR #2831:
URL: https://github.com/apache/arrow-adbc/pull/2831#issuecomment-2887753333
I have tried using the grpc_proxy, https_proxy and http_proxy environment variables, that hasn't worked.
I have tried for both net472 and net8.0:
```
private BigQueryReadClient GetNet472BigQueryReadClient(GoogleCredential? credential)
{
// this results in Grpc.Core.RpcException: 'Status(StatusCode="Unimplemented",
// Detail="Bad gRPC response. HTTP status code: 404")'
BigQueryReadClientBuilder? readClientBuilder;
if (!string.IsNullOrEmpty(this.proxyAddress))
{
HttpClientHandler proxyHandler = ProxyManager.GetProxyHttpClientHandler(this.proxyAddress!);
GrpcWebHandler grpcWebHandler = new GrpcWebHandler(GrpcWebMode.GrpcWeb, proxyHandler);
HttpClient proxyGrpcWebClient = new HttpClient(grpcWebHandler);
Grpc.Net.Client.GrpcChannelOptions options = new Grpc.Net.Client.GrpcChannelOptions()
{
HttpClient = proxyGrpcWebClient,
Credentials = credential?.ToChannelCredentials()
};
GrpcChannel proxyChannel = GrpcChannel.ForAddress("https://bigquerystorage.googleapis.com:443", options);
CallInvoker ci = proxyChannel.CreateCallInvoker();
readClientBuilder = new BigQueryReadClientBuilder()
{
CallInvoker = ci
};
}
else
{
readClientBuilder = new BigQueryReadClientBuilder();
readClientBuilder.Credential = credential;
}
BigQueryReadClient bigQueryReadClient = readClientBuilder.Build();
return bigQueryReadClient;
}
private BigQueryReadClient GetBigQueryReadClient(GoogleCredential? credential)
{
// this also results in
// RpcException: Status(StatusCode="Unimplemented", Detail="Bad gRPC response. HTTP status code: 404")
// when using .net 8
BigQueryReadClientBuilder? readClientBuilder;
if (!string.IsNullOrEmpty(this.proxyAddress))
{
HttpClientHandler proxyHandler = ProxyManager.GetProxyHttpClientHandler(this.proxyAddress!);
GrpcWebHandler grpcWebHandler = new GrpcWebHandler(GrpcWebMode.GrpcWeb, proxyHandler);
HttpClient proxyGrpcWebClient = new HttpClient(grpcWebHandler);
GrpcNetClientAdapter grpcNetClientAdapter =
GrpcNetClientAdapter.Default.WithAdditionalOptions(options =>
{
options.HttpClient = proxyGrpcWebClient;
});
readClientBuilder = new BigQueryReadClientBuilder()
{
GrpcAdapter = grpcNetClientAdapter,
};
}
else
{
readClientBuilder = new BigQueryReadClientBuilder();
}
readClientBuilder.Credential = credential;
BigQueryReadClient bigQueryReadClient = readClientBuilder.Build();
return bigQueryReadClient;
}
```
The only other option appears to be to use the storage v2 REST API, but then the data comes back as JSON and not Arrow. So, I am not sure what other options can be used for calling the storage backend with a proxy in the middle.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(csharp/src/Drivers/BigQuery): Add support for Http proxy [arrow-adbc]
Posted by "davidhcoe (via GitHub)" <gi...@apache.org>.
davidhcoe commented on PR #2831:
URL: https://github.com/apache/arrow-adbc/pull/2831#issuecomment-2888004241
Need to update the readme
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org