You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@gluten.apache.org by fe...@apache.org on 2024/05/24 21:19:40 UTC
(incubator-gluten) branch main updated: [VL] Add support for GCS retry properties (#5858)
This is an automated email from the ASF dual-hosted git repository.
felixybw pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/incubator-gluten.git
The following commit(s) were added to refs/heads/main by this push:
new 9f424a1d0 [VL] Add support for GCS retry properties (#5858)
9f424a1d0 is described below
commit 9f424a1d05180fcbd00ea9b84a8d3d8b7dcca8c0
Author: Sandino Flores <sa...@intel.com>
AuthorDate: Fri May 24 15:19:34 2024 -0600
[VL] Add support for GCS retry properties (#5858)
The Velox GCS connector was recently added a couple of properties:
HiveConfig::kGCSMaxRetryCount: Retry until a number of transient errors is detected.
HiveConfig::kGCSMaxRetryTime: Retry until a time elapses.
This properties are useful to allow a client to either fail immediately or keep retring for several minutes.
If none of them is set, then it will use the defaults set by GCS.
This change was tested manually, by entering an unnaccesible GCS bucket then verifying that the properties are honored.
---
cpp/velox/utils/ConfigExtractor.cc | 13 +++++++++++++
docs/get-started/VeloxGCS.md | 14 ++++++++++++++
2 files changed, 27 insertions(+)
diff --git a/cpp/velox/utils/ConfigExtractor.cc b/cpp/velox/utils/ConfigExtractor.cc
index 0cbba37a7..a71f14322 100644
--- a/cpp/velox/utils/ConfigExtractor.cc
+++ b/cpp/velox/utils/ConfigExtractor.cc
@@ -121,6 +121,19 @@ std::shared_ptr<facebook::velox::core::MemConfig> getHiveConfig(std::shared_ptr<
}
}
+ // https://github.com/GoogleCloudDataproc/hadoop-connectors/blob/master/gcs/CONFIGURATION.md#http-transport-configuration
+ // https://cloud.google.com/cpp/docs/reference/storage/latest/classgoogle_1_1cloud_1_1storage_1_1LimitedErrorCountRetryPolicy
+ auto gsMaxRetryCount = conf->get("spark.hadoop.fs.gs.http.max.retry");
+ if (gsMaxRetryCount.hasValue()) {
+ hiveConfMap[facebook::velox::connector::hive::HiveConfig::kGCSMaxRetryCount] = gsMaxRetryCount.value();
+ }
+
+ // https://cloud.google.com/cpp/docs/reference/storage/latest/classgoogle_1_1cloud_1_1storage_1_1LimitedTimeRetryPolicy
+ auto gsMaxRetryTime = conf->get("spark.hadoop.fs.gs.http.max.retry-time");
+ if (gsMaxRetryTime.hasValue()) {
+ hiveConfMap[facebook::velox::connector::hive::HiveConfig::kGCSMaxRetryTime] = gsMaxRetryTime.value();
+ }
+
// https://github.com/GoogleCloudDataproc/hadoop-connectors/blob/master/gcs/CONFIGURATION.md#authentication
auto gsAuthType = conf->get("spark.hadoop.fs.gs.auth.type");
if (gsAuthType.hasValue()) {
diff --git a/docs/get-started/VeloxGCS.md b/docs/get-started/VeloxGCS.md
index d87d9b960..453bee117 100644
--- a/docs/get-started/VeloxGCS.md
+++ b/docs/get-started/VeloxGCS.md
@@ -36,4 +36,18 @@ spark.hadoop.fs.gs.auth.service.account.json.keyfile // path to the json file wi
For cases when a GCS mock is used, an optional endpoint can be provided:
```sh
spark.hadoop.fs.gs.storage.root.url // url to the mock gcs service including starting with http or https
+```
+
+## Configuring GCS max retry count
+
+For cases when a transient server error is detected, GCS can be configured to keep retrying until a number of transient error is detected.
+```sh
+spark.hadoop.fs.gs.http.max.retry // number of times to keep retrying unless a non-transient error is detected
+```
+
+## Configuring GCS max retry time
+
+For cases when a transient server error is detected, GCS can be configured to keep retrying until the retry loop exceeds a prescribed duration.
+```sh
+spark.hadoop.fs.gs.http.max.retry-time // a string representing the time keep retring (10s, 1m, etc).
```
\ No newline at end of file
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org