You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by "mingnuj (via GitHub)" <gi...@apache.org> on 2025/01/17 09:42:57 UTC
[I] Namespace names with dot(.) not supported in Iceberg REST API [iceberg]
mingnuj opened a new issue, #11990:
URL: https://github.com/apache/iceberg/issues/11990
### Query engine
REST API (JAVA)
### Question
I am using the Iceberg REST API([tabulario/iceberg-rest](https://hub.docker.com/r/tabulario/iceberg-rest)) with PostgreSQL JDBC as the backend catalog. However, I have encountered an issue where namespace names containing `.` are not supported.
I have verified that namespaces with `.` are correctly stored and retrieved in the PostgreSQL database. From my observation, the REST API seems to interpret `.` in namespace names as levels by default.
I could not find any mention in the specification that namespace names with `.` are disallowed. Moreover, when using double quotes (`"`) in the database, it is possible to use `.` in namespace names. I think it would be better to allow this in the REST API as well.
PostgreSQL Configuration:
```bash
docker run --name rest-postgres-test \
-p 5534:5432 \
-e POSTGRES_PASSWORD=postgres \
-e TZ=Asia/Seoul \
-v ./pgdata:/var/lib/postgresql/data \
-d postgres:latest
```
REST API Configuration:
```bash
docker run \
-e CATALOG_WAREHOUSE=file://warehouse \
-e CATALOG_IO__IMPL=org.apache.iceberg.hadoop.HadoopFileIO \
-e CATALOG_CATALOG__IMPL=org.apache.iceberg.jdbc.JdbcCatalog \
-e CATALOG_URI=jdbc:postgresql://{LOCAL_IP}:5534/postgres \
-e CATALOG_JDBC_USER=postgres \
-e CATALOG_JDBC_PASSWORD=postgres \
-e REST_PORT=8183 \
-v ./warehouse:/warehouse \
-p 8183:8183 \
tabulario/iceberg-rest:latest
```
Created `test@email.com` namespace and `test` table in that.
Namespace Display in PostgreSQL:
Could you confirm whether this is the intended behavior or a bug? If possible, I would like to request support for namespace names containing `.`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Namespace names with dot(.) not supported in Iceberg REST API [iceberg]
Posted by "mingnuj (via GitHub)" <gi...@apache.org>.
mingnuj commented on issue #11990:
URL: https://github.com/apache/iceberg/issues/11990#issuecomment-2598347992
> [@mingnuj](https://github.com/mingnuj) can you please share which issue you ran into? Any stack traces and error messages will help
Thank you for your response. I'm sorry for not clarifying where the issue occurs.
The problem arises when retrieving the namespace.
When I connect to the database and use the `listNamespaces` operation for get namespace names. The full namespace name is `test@email.com`, but only `test@email` is returned. As a result, I cannot access the tables within this namespace.
Is there any way to resolve this issue?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Namespace names with dot(.) not supported by JDBC catalog when listing namespaces [iceberg]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #11990:
URL: https://github.com/apache/iceberg/issues/11990#issuecomment-3684619840
This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occurs. To permanently prevent this issue from being considered stale, add the label 'not-stale', but commenting on the issue is preferred when possible.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Namespace names with dot(.) not supported by JDBC catalog when listing namespaces [iceberg]
Posted by "nastra (via GitHub)" <gi...@apache.org>.
nastra commented on issue #11990:
URL: https://github.com/apache/iceberg/issues/11990#issuecomment-2630793937
Re-opening this since it hasn't been fixed in Iceberg itself
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Namespace names with dot(.) not supported by JDBC catalog when listing namespaces [iceberg]
Posted by "mingnuj (via GitHub)" <gi...@apache.org>.
mingnuj commented on issue #11990:
URL: https://github.com/apache/iceberg/issues/11990#issuecomment-2630158964
I gave it some more thought, and I was wondering if you could explain why the function that splits based on dots is located in `JdbcUtil.java`.
In my opinion, since namespaces should be split at dots regardless of whether they are used in JDBC or FILE, it seems more appropriate to handle this at a higher level.
I would really appreciate it if someone could clarify this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Namespace names with dot(.) not supported by JDBC catalog when listing namespaces [iceberg]
Posted by "patricklucas (via GitHub)" <gi...@apache.org>.
patricklucas commented on issue #11990:
URL: https://github.com/apache/iceberg/issues/11990#issuecomment-3004231324
I also ran into this. I created a namespace like `Namespace.of("foo", "bar.baz")` expecting that the namespace "levels" would be treated opaquely like with other catalogs. However, after a round trip to the `iceberg_tables` table, this namespace is now interpreted as `Namespace.of("foo", "bar", "baz")`.
In the `RESTCatalog`, for instance, when namespace are included in URLs, each level gets URL-encoded and then the levels are joined by `%2f` (i.e. `/`), so dots would be preserved. That said, I'm not sure what the behavior is if if a namespace level includes a slash.
Nevertheless, Iceberg should take a clear stance on namespace handling that is consistent across catalog implementations. The most explicit approach would be to say that namespace levels may contain any Unicode character, and then catalog implementations always act accordingly, encoding and decoding as necessary. This would require each implementation to choose how to represent them, and "double encoding" might be required, for instance to solve the potential slashes issue in `RESTCatalog`.
Example: encoding namespace `Namespace.of("foo/bar", "baz")` for inclusion in a URL, becomes `foo%252Fbar%2Fbaz`:
```
In [1]: import urllib
In [2]: level1 = urllib.parse.quote_plus('foo/bar')
In [3]: level1
Out[3]: 'foo%2Fbar'
In [4]: level2 = urllib.parse.quote_plus('baz')
In [5]: level2
Out[5]: 'baz'
In [6]: encoded = urllib.parse.quote_plus(level1 + '/' + level2)
In [7]: encoded
Out[7]: 'foo%252Fbar%2Fbaz'
In [8]: [urllib.parse.unquote(level) for level in urllib.parse.unquote(encoded).split('/')]
Out[8]: ['foo/bar', 'baz']
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Namespace names with dot(.) not supported by JDBC catalog when listing namespaces [iceberg]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #11990:
URL: https://github.com/apache/iceberg/issues/11990#issuecomment-3712557974
This issue has been closed because it has not received any activity in the last 14 days since being marked as 'stale'
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Namespace names with dot(.) not supported by JDBC catalog when listing namespaces [iceberg]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed issue #11990: Namespace names with dot(.) not supported by JDBC catalog when listing namespaces
URL: https://github.com/apache/iceberg/issues/11990
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Namespace names with dot(.) not supported by JDBC catalog when listing namespaces [iceberg]
Posted by "mingnuj (via GitHub)" <gi...@apache.org>.
mingnuj commented on issue #11990:
URL: https://github.com/apache/iceberg/issues/11990#issuecomment-2630077016
I am attaching the solution I personally implemented based on the provided guidance, as someone else might encounter the same issue.
I wanted the namespace to allow dots not only in tests but also in actual operation. Based on the test added in #11991 , I introduced a new flag, jdbc.supports-names-with-dot.
https://github.com/apache/iceberg/blob/472ec6cbdcaa24773ef5c64c9444ae933805381d/core/src/main/java/org/apache/iceberg/jdbc/JdbcUtil.java#L43-L44
```java
static final String INIT_CATALOG_TABLES_PROPERTY =
JdbcCatalog.PROPERTY_PREFIX + "init-catalog-tables";
+ // property to control if namespace can contain dots
+ static final String SUPPORTS_NAMES_WITH_DOT = JdbcCatalog.PROPERTY_PREFIX + "supports-names-with-dot";
```
The flag, withDot field is defined in JdbcCatalog and is initialized in the initialize function, where the flag is set and stored as a boolean value.
at here,
https://github.com/apache/iceberg/blob/472ec6cbdcaa24773ef5c64c9444ae933805381d/core/src/main/java/org/apache/iceberg/jdbc/JdbcCatalog.java#L109
add these lines
```java
boolean withDot = PropertyUtil.propertyAsBoolean(properties, JdbcUtil.SUPPORTS_NAMES_WITH_DOT, false);
this.withDot = withDot;
```
Following @nastra's guidance, modified the `stringToNamespace` function. I sincerely appreciate it.
https://github.com/apache/iceberg/blob/472ec6cbdcaa24773ef5c64c9444ae933805381d/core/src/main/java/org/apache/iceberg/jdbc/JdbcUtil.java#L501-L508
like this
```java
static Namespace stringToNamespace(String namespace, boolean withDot) {
Preconditions.checkArgument(namespace != null, "Invalid namespace %s", namespace);
if (namespace.isEmpty()) {
return Namespace.empty();
}
if (withDot) {
return Namespace.of(namespace);
} else {
return Namespace.of(Iterables.toArray(SPLITTER_DOT.split(namespace), String.class));
}
}
```
The rest was simply adjusting the function parameters accordingly.
After modifying the relevant parts, I built and ran the project separately, and it worked as expected.
Thank you all for your help.
I will now close this issue.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Namespace names with dot(.) not supported by JDBC catalog when listing namespaces [iceberg]
Posted by "mingnuj (via GitHub)" <gi...@apache.org>.
mingnuj closed issue #11990: Namespace names with dot(.) not supported by JDBC catalog when listing namespaces
URL: https://github.com/apache/iceberg/issues/11990
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Namespace names with dot(.) not supported in Iceberg REST API [iceberg]
Posted by "nastra (via GitHub)" <gi...@apache.org>.
nastra commented on issue #11990:
URL: https://github.com/apache/iceberg/issues/11990#issuecomment-2598314036
@mingnuj can you please share which issue you ran into? Any stack traces and error messages will help
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Namespace names with dot(.) not supported in Iceberg REST API [iceberg]
Posted by "nastra (via GitHub)" <gi...@apache.org>.
nastra commented on issue #11990:
URL: https://github.com/apache/iceberg/issues/11990#issuecomment-2598374019
I checked and it turns out that this isn't an issue with the REST catalog but rather with the JDBC catalog itself. The problematic piece of code is here: https://github.com/apache/iceberg/blob/472ec6cbdcaa24773ef5c64c9444ae933805381d/core/src/main/java/org/apache/iceberg/jdbc/JdbcUtil.java#L507
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org