You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by GitBox <gi...@apache.org> on 2020/07/01 04:36:14 UTC
[GitHub] [iceberg] rdblue opened a new pull request #1149: Support namespaces in SparkCatalog
rdblue opened a new pull request #1149:
URL: https://github.com/apache/iceberg/pull/1149
This adds `SupportsNamespaces` to `SparkCatalog` and adds tests in `TestNamespaceSQL`.
This also adds a couple of new base classes for Spark tests that will be used for other SQL test suites.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on pull request #1149: Support namespaces in SparkCatalog
Posted by GitBox <gi...@apache.org>.
rdblue commented on pull request #1149:
URL: https://github.com/apache/iceberg/pull/1149#issuecomment-652698487
I ran the checks locally since our Travis queue seems backed up today. Thanks for the review, @danielcweeks!
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on a change in pull request #1149: Support namespaces in SparkCatalog
Posted by GitBox <gi...@apache.org>.
rdblue commented on a change in pull request #1149:
URL: https://github.com/apache/iceberg/pull/1149#discussion_r448653472
##########
File path: spark3/src/main/java/org/apache/iceberg/spark/SparkCatalog.java
##########
@@ -228,13 +247,141 @@ public void invalidateTable(Identifier ident) {
}
}
+ @Override
+ public Identifier[] listTables(String[] namespace) {
+ return icebergCatalog.listTables(Namespace.of(namespace)).stream()
+ .map(ident -> Identifier.of(ident.namespace().levels(), ident.name()))
+ .toArray(Identifier[]::new);
+ }
+
+ @Override
+ public String[] defaultNamespace() {
+ if (defaultNamespace != null) {
+ return defaultNamespace;
+ }
+
+ return new String[0];
+ }
+
+ @Override
+ public String[][] listNamespaces() {
+ if (asNamespaceCatalog != null) {
+ return asNamespaceCatalog.listNamespaces().stream()
+ .map(Namespace::levels)
+ .toArray(String[][]::new);
+ }
+
+ return new String[0][];
+ }
+
+ @Override
+ public String[][] listNamespaces(String[] namespace) throws NoSuchNamespaceException {
+ if (asNamespaceCatalog != null) {
+ try {
+ return asNamespaceCatalog.listNamespaces(Namespace.of(namespace)).stream()
+ .map(Namespace::levels)
+ .toArray(String[][]::new);
+ } catch (org.apache.iceberg.exceptions.NoSuchNamespaceException e) {
+ throw new NoSuchNamespaceException(namespace);
+ }
+ }
+
+ throw new NoSuchNamespaceException(namespace);
+ }
+
+ @Override
+ public Map<String, String> loadNamespaceMetadata(String[] namespace) throws NoSuchNamespaceException {
+ if (asNamespaceCatalog != null) {
+ try {
+ return asNamespaceCatalog.loadNamespaceMetadata(Namespace.of(namespace));
+ } catch (org.apache.iceberg.exceptions.NoSuchNamespaceException e) {
+ throw new NoSuchNamespaceException(namespace);
+ }
+ }
+
+ throw new NoSuchNamespaceException(namespace);
+ }
+
+ @Override
+ public void createNamespace(String[] namespace, Map<String, String> metadata) throws NamespaceAlreadyExistsException {
+ if (asNamespaceCatalog != null) {
+ try {
+ if (asNamespaceCatalog instanceof HadoopCatalog && DEFAULT_NS_KEYS.equals(metadata.keySet())) {
Review comment:
Yes, that is the expected behavior if you create a namespace with metadata properties in the Hadoop catalog because there is nowhere to store them.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue merged pull request #1149: Support namespaces in SparkCatalog
Posted by GitBox <gi...@apache.org>.
rdblue merged pull request #1149:
URL: https://github.com/apache/iceberg/pull/1149
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] danielcweeks commented on pull request #1149: Support namespaces in SparkCatalog
Posted by GitBox <gi...@apache.org>.
danielcweeks commented on pull request #1149:
URL: https://github.com/apache/iceberg/pull/1149#issuecomment-652697389
+1 (pending checks)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] danielcweeks commented on pull request #1149: Support namespaces in SparkCatalog
Posted by GitBox <gi...@apache.org>.
danielcweeks commented on pull request #1149:
URL: https://github.com/apache/iceberg/pull/1149#issuecomment-652677542
LGTM, but some checkstyle failures in CI?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] danielcweeks commented on a change in pull request #1149: Support namespaces in SparkCatalog
Posted by GitBox <gi...@apache.org>.
danielcweeks commented on a change in pull request #1149:
URL: https://github.com/apache/iceberg/pull/1149#discussion_r448646423
##########
File path: spark3/src/main/java/org/apache/iceberg/spark/SparkCatalog.java
##########
@@ -228,13 +247,141 @@ public void invalidateTable(Identifier ident) {
}
}
+ @Override
+ public Identifier[] listTables(String[] namespace) {
+ return icebergCatalog.listTables(Namespace.of(namespace)).stream()
+ .map(ident -> Identifier.of(ident.namespace().levels(), ident.name()))
+ .toArray(Identifier[]::new);
+ }
+
+ @Override
+ public String[] defaultNamespace() {
+ if (defaultNamespace != null) {
+ return defaultNamespace;
+ }
+
+ return new String[0];
+ }
+
+ @Override
+ public String[][] listNamespaces() {
+ if (asNamespaceCatalog != null) {
+ return asNamespaceCatalog.listNamespaces().stream()
+ .map(Namespace::levels)
+ .toArray(String[][]::new);
+ }
+
+ return new String[0][];
+ }
+
+ @Override
+ public String[][] listNamespaces(String[] namespace) throws NoSuchNamespaceException {
+ if (asNamespaceCatalog != null) {
+ try {
+ return asNamespaceCatalog.listNamespaces(Namespace.of(namespace)).stream()
+ .map(Namespace::levels)
+ .toArray(String[][]::new);
+ } catch (org.apache.iceberg.exceptions.NoSuchNamespaceException e) {
+ throw new NoSuchNamespaceException(namespace);
+ }
+ }
+
+ throw new NoSuchNamespaceException(namespace);
+ }
+
+ @Override
+ public Map<String, String> loadNamespaceMetadata(String[] namespace) throws NoSuchNamespaceException {
+ if (asNamespaceCatalog != null) {
+ try {
+ return asNamespaceCatalog.loadNamespaceMetadata(Namespace.of(namespace));
+ } catch (org.apache.iceberg.exceptions.NoSuchNamespaceException e) {
+ throw new NoSuchNamespaceException(namespace);
+ }
+ }
+
+ throw new NoSuchNamespaceException(namespace);
+ }
+
+ @Override
+ public void createNamespace(String[] namespace, Map<String, String> metadata) throws NamespaceAlreadyExistsException {
+ if (asNamespaceCatalog != null) {
+ try {
+ if (asNamespaceCatalog instanceof HadoopCatalog && DEFAULT_NS_KEYS.equals(metadata.keySet())) {
Review comment:
Maybe I'm missing something here, but doesn't this actually break for HadoopCatalog without the default set of keys (it'll drop through and be rejected in the `else` block)?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on pull request #1149: Support namespaces in SparkCatalog
Posted by GitBox <gi...@apache.org>.
rdblue commented on pull request #1149:
URL: https://github.com/apache/iceberg/pull/1149#issuecomment-652684932
Thanks for pointing that out. I've pushed a fix for checkstyle.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org