You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@asterixdb.apache.org by AsterixDB Code Review <do...@asterix-gerrit.ics.uci.edu> on 2022/05/20 15:05:28 UTC
Change in asterixdb[master]: ASTERIXDB-3036: Cannot pass endpoint property for GCS external datasets
From Hussain Towaileb <hu...@gmail.com>:
Hussain Towaileb has uploaded this change for review. ( https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/16403 )
Change subject: ASTERIXDB-3036: Cannot pass endpoint property for GCS external datasets
......................................................................
ASTERIXDB-3036: Cannot pass endpoint property for GCS external datasets
Change-Id: I4204fdc48141a03306468a55bfc89b349e9afce1
---
M asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/util/ExternalDataUtils.java
M asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/util/google/gcs/GCSConstants.java
M asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/util/google/gcs/GCSUtils.java
3 files changed, 8 insertions(+), 1 deletion(-)
git pull ssh://asterix-gerrit.ics.uci.edu:29418/asterixdb refs/changes/03/16403/1
diff --git a/asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/util/ExternalDataUtils.java b/asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/util/ExternalDataUtils.java
index 702ef42..62dc074 100644
--- a/asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/util/ExternalDataUtils.java
+++ b/asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/util/ExternalDataUtils.java
@@ -530,7 +530,7 @@
case ExternalDataConstants.KEY_ADAPTER_NAME_AZURE_DATA_LAKE:
validateAzureDataLakeProperties(configuration, srcLoc, collector, appCtx);
break;
- case KEY_ADAPTER_NAME_GCS:
+ case ExternalDataConstants.KEY_ADAPTER_NAME_GCS:
validateProperties(configuration, srcLoc, collector);
break;
default:
diff --git a/asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/util/google/gcs/GCSConstants.java b/asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/util/google/gcs/GCSConstants.java
index 8a0be99..4a3ed8d 100644
--- a/asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/util/google/gcs/GCSConstants.java
+++ b/asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/util/google/gcs/GCSConstants.java
@@ -24,4 +24,5 @@
}
public static final String JSON_CREDENTIALS_FIELD_NAME = "jsonCredentials";
+ public static final String ENDPOINT_FIELD_NAME = "endpoint";
}
diff --git a/asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/util/google/gcs/GCSUtils.java b/asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/util/google/gcs/GCSUtils.java
index 553733f..69fd2d4 100644
--- a/asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/util/google/gcs/GCSUtils.java
+++ b/asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/util/google/gcs/GCSUtils.java
@@ -24,6 +24,7 @@
import static org.apache.asterix.external.util.ExternalDataUtils.getPrefix;
import static org.apache.asterix.external.util.ExternalDataUtils.isParquetFormat;
import static org.apache.asterix.external.util.ExternalDataUtils.validateIncludeExclude;
+import static org.apache.asterix.external.util.google.gcs.GCSConstants.ENDPOINT_FIELD_NAME;
import static org.apache.asterix.external.util.google.gcs.GCSConstants.JSON_CREDENTIALS_FIELD_NAME;
import static org.apache.hyracks.api.util.ExceptionUtils.getMessageOrToString;
@@ -63,6 +64,7 @@
*/
public static Storage buildClient(Map<String, String> configuration) throws CompilationException {
String jsonCredentials = configuration.get(JSON_CREDENTIALS_FIELD_NAME);
+ String endpoint = configuration.get(ENDPOINT_FIELD_NAME);
StorageOptions.Builder builder = StorageOptions.newBuilder();
@@ -75,6 +77,10 @@
}
}
+ if (endpoint != null) {
+ builder.setHost(endpoint);
+ }
+
return builder.build().getService();
}
--
To view, visit https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/16403
To unsubscribe, or for help writing mail filters, visit https://asterix-gerrit.ics.uci.edu/settings
Gerrit-Project: asterixdb
Gerrit-Branch: master
Gerrit-Change-Id: I4204fdc48141a03306468a55bfc89b349e9afce1
Gerrit-Change-Number: 16403
Gerrit-PatchSet: 1
Gerrit-Owner: Hussain Towaileb <hu...@gmail.com>
Gerrit-MessageType: newchange
Change in asterixdb[master]: ASTERIXDB-3036: Cannot pass endpoint property for GCS external datasets
Posted by AsterixDB Code Review <do...@asterix-gerrit.ics.uci.edu>.
From Hussain Towaileb <hu...@gmail.com>:
Hussain Towaileb has submitted this change. ( https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/16403 )
Change subject: ASTERIXDB-3036: Cannot pass endpoint property for GCS external datasets
......................................................................
ASTERIXDB-3036: Cannot pass endpoint property for GCS external datasets
Change-Id: I4204fdc48141a03306468a55bfc89b349e9afce1
Reviewed-on: https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/16403
Reviewed-by: Hussain Towaileb <hu...@gmail.com>
Reviewed-by: Dmitry Lychagin <dm...@couchbase.com>
Contrib: Jenkins <je...@fulliautomatix.ics.uci.edu>
Integration-Tests: Jenkins <je...@fulliautomatix.ics.uci.edu>
Tested-by: Jenkins <je...@fulliautomatix.ics.uci.edu>
---
M asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/util/ExternalDataUtils.java
M asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/util/google/gcs/GCSConstants.java
M asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/util/google/gcs/GCSUtils.java
3 files changed, 8 insertions(+), 1 deletion(-)
Approvals:
Dmitry Lychagin: Looks good to me, approved
Hussain Towaileb: Looks good to me, but someone else must approve
Jenkins: Verified; ; Verified
diff --git a/asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/util/ExternalDataUtils.java b/asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/util/ExternalDataUtils.java
index 702ef42..62dc074 100644
--- a/asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/util/ExternalDataUtils.java
+++ b/asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/util/ExternalDataUtils.java
@@ -530,7 +530,7 @@
case ExternalDataConstants.KEY_ADAPTER_NAME_AZURE_DATA_LAKE:
validateAzureDataLakeProperties(configuration, srcLoc, collector, appCtx);
break;
- case KEY_ADAPTER_NAME_GCS:
+ case ExternalDataConstants.KEY_ADAPTER_NAME_GCS:
validateProperties(configuration, srcLoc, collector);
break;
default:
diff --git a/asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/util/google/gcs/GCSConstants.java b/asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/util/google/gcs/GCSConstants.java
index 8a0be99..4a3ed8d 100644
--- a/asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/util/google/gcs/GCSConstants.java
+++ b/asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/util/google/gcs/GCSConstants.java
@@ -24,4 +24,5 @@
}
public static final String JSON_CREDENTIALS_FIELD_NAME = "jsonCredentials";
+ public static final String ENDPOINT_FIELD_NAME = "endpoint";
}
diff --git a/asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/util/google/gcs/GCSUtils.java b/asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/util/google/gcs/GCSUtils.java
index 553733f..69fd2d4 100644
--- a/asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/util/google/gcs/GCSUtils.java
+++ b/asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/util/google/gcs/GCSUtils.java
@@ -24,6 +24,7 @@
import static org.apache.asterix.external.util.ExternalDataUtils.getPrefix;
import static org.apache.asterix.external.util.ExternalDataUtils.isParquetFormat;
import static org.apache.asterix.external.util.ExternalDataUtils.validateIncludeExclude;
+import static org.apache.asterix.external.util.google.gcs.GCSConstants.ENDPOINT_FIELD_NAME;
import static org.apache.asterix.external.util.google.gcs.GCSConstants.JSON_CREDENTIALS_FIELD_NAME;
import static org.apache.hyracks.api.util.ExceptionUtils.getMessageOrToString;
@@ -63,6 +64,7 @@
*/
public static Storage buildClient(Map<String, String> configuration) throws CompilationException {
String jsonCredentials = configuration.get(JSON_CREDENTIALS_FIELD_NAME);
+ String endpoint = configuration.get(ENDPOINT_FIELD_NAME);
StorageOptions.Builder builder = StorageOptions.newBuilder();
@@ -75,6 +77,10 @@
}
}
+ if (endpoint != null) {
+ builder.setHost(endpoint);
+ }
+
return builder.build().getService();
}
--
To view, visit https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/16403
To unsubscribe, or for help writing mail filters, visit https://asterix-gerrit.ics.uci.edu/settings
Gerrit-Project: asterixdb
Gerrit-Branch: master
Gerrit-Change-Id: I4204fdc48141a03306468a55bfc89b349e9afce1
Gerrit-Change-Number: 16403
Gerrit-PatchSet: 2
Gerrit-Owner: Hussain Towaileb <hu...@gmail.com>
Gerrit-Reviewer: Dmitry Lychagin <dm...@couchbase.com>
Gerrit-Reviewer: Hussain Towaileb <hu...@gmail.com>
Gerrit-Reviewer: Jenkins <je...@fulliautomatix.ics.uci.edu>
Gerrit-Reviewer: Michael Blow <mb...@apache.org>
Gerrit-Reviewer: Till Westmann <ti...@couchbase.com>
Gerrit-MessageType: merged
Change in asterixdb[master]: ASTERIXDB-3036: Cannot pass endpoint property for GCS external datasets
Posted by AsterixDB Code Review <do...@asterix-gerrit.ics.uci.edu>.
From Hussain Towaileb <hu...@gmail.com>:
Hussain Towaileb has uploaded this change for review. ( https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/16403 )
Change subject: ASTERIXDB-3036: Cannot pass endpoint property for GCS external datasets
......................................................................
ASTERIXDB-3036: Cannot pass endpoint property for GCS external datasets
Change-Id: I4204fdc48141a03306468a55bfc89b349e9afce1
---
M asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/util/ExternalDataUtils.java
M asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/util/google/gcs/GCSConstants.java
M asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/util/google/gcs/GCSUtils.java
3 files changed, 8 insertions(+), 1 deletion(-)
git pull ssh://asterix-gerrit.ics.uci.edu:29418/asterixdb refs/changes/03/16403/1
diff --git a/asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/util/ExternalDataUtils.java b/asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/util/ExternalDataUtils.java
index 702ef42..62dc074 100644
--- a/asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/util/ExternalDataUtils.java
+++ b/asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/util/ExternalDataUtils.java
@@ -530,7 +530,7 @@
case ExternalDataConstants.KEY_ADAPTER_NAME_AZURE_DATA_LAKE:
validateAzureDataLakeProperties(configuration, srcLoc, collector, appCtx);
break;
- case KEY_ADAPTER_NAME_GCS:
+ case ExternalDataConstants.KEY_ADAPTER_NAME_GCS:
validateProperties(configuration, srcLoc, collector);
break;
default:
diff --git a/asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/util/google/gcs/GCSConstants.java b/asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/util/google/gcs/GCSConstants.java
index 8a0be99..4a3ed8d 100644
--- a/asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/util/google/gcs/GCSConstants.java
+++ b/asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/util/google/gcs/GCSConstants.java
@@ -24,4 +24,5 @@
}
public static final String JSON_CREDENTIALS_FIELD_NAME = "jsonCredentials";
+ public static final String ENDPOINT_FIELD_NAME = "endpoint";
}
diff --git a/asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/util/google/gcs/GCSUtils.java b/asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/util/google/gcs/GCSUtils.java
index 553733f..69fd2d4 100644
--- a/asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/util/google/gcs/GCSUtils.java
+++ b/asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/util/google/gcs/GCSUtils.java
@@ -24,6 +24,7 @@
import static org.apache.asterix.external.util.ExternalDataUtils.getPrefix;
import static org.apache.asterix.external.util.ExternalDataUtils.isParquetFormat;
import static org.apache.asterix.external.util.ExternalDataUtils.validateIncludeExclude;
+import static org.apache.asterix.external.util.google.gcs.GCSConstants.ENDPOINT_FIELD_NAME;
import static org.apache.asterix.external.util.google.gcs.GCSConstants.JSON_CREDENTIALS_FIELD_NAME;
import static org.apache.hyracks.api.util.ExceptionUtils.getMessageOrToString;
@@ -63,6 +64,7 @@
*/
public static Storage buildClient(Map<String, String> configuration) throws CompilationException {
String jsonCredentials = configuration.get(JSON_CREDENTIALS_FIELD_NAME);
+ String endpoint = configuration.get(ENDPOINT_FIELD_NAME);
StorageOptions.Builder builder = StorageOptions.newBuilder();
@@ -75,6 +77,10 @@
}
}
+ if (endpoint != null) {
+ builder.setHost(endpoint);
+ }
+
return builder.build().getService();
}
--
To view, visit https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/16403
To unsubscribe, or for help writing mail filters, visit https://asterix-gerrit.ics.uci.edu/settings
Gerrit-Project: asterixdb
Gerrit-Branch: master
Gerrit-Change-Id: I4204fdc48141a03306468a55bfc89b349e9afce1
Gerrit-Change-Number: 16403
Gerrit-PatchSet: 1
Gerrit-Owner: Hussain Towaileb <hu...@gmail.com>
Gerrit-MessageType: newchange
Change in asterixdb[master]: ASTERIXDB-3036: Cannot pass endpoint property for GCS external datasets
Posted by AsterixDB Code Review <do...@asterix-gerrit.ics.uci.edu>.
From Dmitry Lychagin <dm...@couchbase.com>:
Dmitry Lychagin has posted comments on this change. ( https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/16403 )
Change subject: ASTERIXDB-3036: Cannot pass endpoint property for GCS external datasets
......................................................................
Patch Set 1: Code-Review+2
--
To view, visit https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/16403
To unsubscribe, or for help writing mail filters, visit https://asterix-gerrit.ics.uci.edu/settings
Gerrit-Project: asterixdb
Gerrit-Branch: master
Gerrit-Change-Id: I4204fdc48141a03306468a55bfc89b349e9afce1
Gerrit-Change-Number: 16403
Gerrit-PatchSet: 1
Gerrit-Owner: Hussain Towaileb <hu...@gmail.com>
Gerrit-Reviewer: Dmitry Lychagin <dm...@couchbase.com>
Gerrit-Reviewer: Hussain Towaileb <hu...@gmail.com>
Gerrit-Reviewer: Jenkins <je...@fulliautomatix.ics.uci.edu>
Gerrit-Reviewer: Michael Blow <mb...@apache.org>
Gerrit-Reviewer: Till Westmann <ti...@couchbase.com>
Gerrit-Comment-Date: Sat, 21 May 2022 00:13:53 +0000
Gerrit-HasComments: No
Gerrit-Has-Labels: Yes
Gerrit-MessageType: comment
Change in asterixdb[master]: ASTERIXDB-3036: Cannot pass endpoint property for GCS external datasets
Posted by AsterixDB Code Review <do...@asterix-gerrit.ics.uci.edu>.
From Jenkins <je...@fulliautomatix.ics.uci.edu>:
Jenkins has posted comments on this change. ( https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/16403 )
Change subject: ASTERIXDB-3036: Cannot pass endpoint property for GCS external datasets
......................................................................
Patch Set 1: Contrib+1
Graphix Compatibility Tests Successful
https://asterix-jenkins.ics.uci.edu/job/graphix-compat/108/ : SUCCESS
--
To view, visit https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/16403
To unsubscribe, or for help writing mail filters, visit https://asterix-gerrit.ics.uci.edu/settings
Gerrit-Project: asterixdb
Gerrit-Branch: master
Gerrit-Change-Id: I4204fdc48141a03306468a55bfc89b349e9afce1
Gerrit-Change-Number: 16403
Gerrit-PatchSet: 1
Gerrit-Owner: Hussain Towaileb <hu...@gmail.com>
Gerrit-Reviewer: Hussain Towaileb <hu...@gmail.com>
Gerrit-Reviewer: Jenkins <je...@fulliautomatix.ics.uci.edu>
Gerrit-Reviewer: Michael Blow <mb...@apache.org>
Gerrit-Reviewer: Till Westmann <ti...@couchbase.com>
Gerrit-Comment-Date: Fri, 20 May 2022 15:26:15 +0000
Gerrit-HasComments: No
Gerrit-Has-Labels: Yes
Gerrit-MessageType: comment
Change in asterixdb[master]: ASTERIXDB-3036: Cannot pass endpoint property for GCS external datasets
Posted by AsterixDB Code Review <do...@asterix-gerrit.ics.uci.edu>.
From Hussain Towaileb <hu...@gmail.com>:
Hussain Towaileb has posted comments on this change. ( https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/16403 )
Change subject: ASTERIXDB-3036: Cannot pass endpoint property for GCS external datasets
......................................................................
Patch Set 1: Code-Review+1
--
To view, visit https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/16403
To unsubscribe, or for help writing mail filters, visit https://asterix-gerrit.ics.uci.edu/settings
Gerrit-Project: asterixdb
Gerrit-Branch: master
Gerrit-Change-Id: I4204fdc48141a03306468a55bfc89b349e9afce1
Gerrit-Change-Number: 16403
Gerrit-PatchSet: 1
Gerrit-Owner: Hussain Towaileb <hu...@gmail.com>
Gerrit-Reviewer: Hussain Towaileb <hu...@gmail.com>
Gerrit-Reviewer: Michael Blow <mb...@apache.org>
Gerrit-Reviewer: Till Westmann <ti...@couchbase.com>
Gerrit-CC: Jenkins <je...@fulliautomatix.ics.uci.edu>
Gerrit-Comment-Date: Fri, 20 May 2022 15:08:06 +0000
Gerrit-HasComments: No
Gerrit-Has-Labels: Yes
Gerrit-MessageType: comment
Change in asterixdb[master]: ASTERIXDB-3036: Cannot pass endpoint property for GCS external datasets
Posted by AsterixDB Code Review <do...@asterix-gerrit.ics.uci.edu>.
From Jenkins <je...@fulliautomatix.ics.uci.edu>:
Jenkins has posted comments on this change. ( https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/16403 )
Change subject: ASTERIXDB-3036: Cannot pass endpoint property for GCS external datasets
......................................................................
Patch Set 1:
Graphix Compatibility Tests Started https://asterix-jenkins.ics.uci.edu/job/graphix-compat/108/
--
To view, visit https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/16403
To unsubscribe, or for help writing mail filters, visit https://asterix-gerrit.ics.uci.edu/settings
Gerrit-Project: asterixdb
Gerrit-Branch: master
Gerrit-Change-Id: I4204fdc48141a03306468a55bfc89b349e9afce1
Gerrit-Change-Number: 16403
Gerrit-PatchSet: 1
Gerrit-Owner: Hussain Towaileb <hu...@gmail.com>
Gerrit-CC: Jenkins <je...@fulliautomatix.ics.uci.edu>
Gerrit-Comment-Date: Fri, 20 May 2022 15:07:23 +0000
Gerrit-HasComments: No
Gerrit-Has-Labels: No
Gerrit-MessageType: comment
Change in asterixdb[master]: ASTERIXDB-3036: Cannot pass endpoint property for GCS external datasets
Posted by AsterixDB Code Review <do...@asterix-gerrit.ics.uci.edu>.
From Jenkins <je...@fulliautomatix.ics.uci.edu>:
Jenkins has posted comments on this change. ( https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/16403 )
Change subject: ASTERIXDB-3036: Cannot pass endpoint property for GCS external datasets
......................................................................
Patch Set 1: Integration-Tests+1
Integration Tests Successful
https://asterix-jenkins.ics.uci.edu/job/asterix-gerrit-integration-tests/13136/ : SUCCESS
--
To view, visit https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/16403
To unsubscribe, or for help writing mail filters, visit https://asterix-gerrit.ics.uci.edu/settings
Gerrit-Project: asterixdb
Gerrit-Branch: master
Gerrit-Change-Id: I4204fdc48141a03306468a55bfc89b349e9afce1
Gerrit-Change-Number: 16403
Gerrit-PatchSet: 1
Gerrit-Owner: Hussain Towaileb <hu...@gmail.com>
Gerrit-Reviewer: Hussain Towaileb <hu...@gmail.com>
Gerrit-Reviewer: Jenkins <je...@fulliautomatix.ics.uci.edu>
Gerrit-Reviewer: Michael Blow <mb...@apache.org>
Gerrit-Reviewer: Till Westmann <ti...@couchbase.com>
Gerrit-Comment-Date: Fri, 20 May 2022 16:09:10 +0000
Gerrit-HasComments: No
Gerrit-Has-Labels: Yes
Gerrit-MessageType: comment