You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@polaris.apache.org by "ajantha-bhat (via GitHub)" <gi...@apache.org> on 2025/10/03 14:46:16 UTC
[I] Cannot register Iceberg tables created by HMS. [polaris]
ajantha-bhat opened a new issue, #2756:
URL: https://github.com/apache/polaris/issues/2756
### Describe the bug
Register table with table metadata file created by HMS fails.
Metadata path: `file:/var/folders/dz/m6q699tj3wzglncnrckg77yc0000gn/T/junit-476754053339093856/sourceCatalogWarehouse/bar.db/tbl3/metadata/00000-6dd90eeb-97f1-4582-bd6a-237c32cc0473.metadata.json`
Allowed location: `file:/var/folders/dz/m6q699tj3wzglncnrckg77yc0000gn/T/junit-9672202773330345462/sourceCatalogWarehouse/`
Error: Invalid locations '[file:/var/folders/dz/m6q699tj3wzglncnrckg77yc0000gn/T/junit-3374837708835005156/sourceCatalogWarehouse/bar.db/tbl3]' for identifier 'bar.tbl3': file:/var/folders/dz/m6q699tj3wzglncnrckg77yc0000gn/T/junit-3374837708835005156/sourceCatalogWarehouse/bar.db/tbl3 is not in the list of allowed locations: [file:///var/folders/dz/m6q699tj3wzglncnrckg77yc0000gn/T/junit-3374837708835005156/sourceCatalogWarehouse/bar]
Stacktrace:
```
cause = {ForbiddenException@11379} "org.apache.iceberg.exceptions.ForbiddenException: Invalid locations '[file:/var/folders/dz/m6q699tj3wzglncnrckg77yc0000gn/T/junit-12580969953422585520/sourceCatalogWarehouse/bar.db/tbl3]' for identifier 'bar.tbl3': file:/var/folders/dz/m6q699tj3wzglncnrckg77yc0000gn/T/junit-12580969953422585520/sourceCatalogWarehouse/bar.db/tbl3 is not in the list of allowed locations: [file:///var/folders/dz/m6q699tj3wzglncnrckg77yc0000gn/T/junit-12580969953422585520/sourceCatalogWarehouse/bar]"
backtrace = {Object[7]@11563}
detailMessage = "Invalid locations '[file:/var/folders/dz/m6q699tj3wzglncnrckg77yc0000gn/T/junit-12580969953422585520/sourceCatalogWarehouse/bar.db/tbl3]' for identifier 'bar.tbl3': file:/var/folders/dz/m6q699tj3wzglncnrckg77yc0000gn/T/junit-12580969953422585520/sourceCatalogWarehouse/bar.db/tbl3 is not in the list of allowed locations: [file:///var/folders/dz/m6q699tj3wzglncnrckg77yc0000gn/T/junit-12580969953422585520/sourceCatalogWarehouse/bar]"
cause = {ForbiddenException@11379} "org.apache.iceberg.exceptions.ForbiddenException: Invalid locations '[file:/var/folders/dz/m6q699tj3wzglncnrckg77yc0000gn/T/junit-12580969953422585520/sourceCatalogWarehouse/bar.db/tbl3]' for identifier 'bar.tbl3': file:/var/folders/dz/m6q699tj3wzglncnrckg77yc0000gn/T/junit-12580969953422585520/sourceCatalogWarehouse/bar.db/tbl3 is not in the list of allowed locations: [file:///var/folders/dz/m6q699tj3wzglncnrckg77yc0000gn/T/junit-12580969953422585520/sourceCatalogWarehouse/bar]"
stackTrace = {StackTraceElement[70]@11565}
0 = {StackTraceElement@11413} "org.apache.polaris.core.storage.LocationRestrictions.lambda$validateLocations$0(LocationRestrictions.java:114)"
1 = {StackTraceElement@11414} "java.base/java.util.HashMap$Values.forEach(HashMap.java:1073)"
2 = {StackTraceElement@11415} "org.apache.polaris.core.storage.LocationRestrictions.lambda$validateLocations$1(LocationRestrictions.java:109)"
3 = {StackTraceElement@11416} "java.base/java.util.HashMap$Values.forEach(HashMap.java:1073)"
4 = {StackTraceElement@11417} "org.apache.polaris.core.storage.LocationRestrictions.validateLocations(LocationRestrictions.java:105)"
5 = {StackTraceElement@11418} "org.apache.polaris.core.storage.LocationRestrictions.validate(LocationRestrictions.java:89)"
6 = {StackTraceElement@11419} "org.apache.polaris.service.catalog.iceberg.IcebergCatalog.lambda$validateLocationsForTableLike$6(IcebergCatalog.java:1011)"
7 = {StackTraceElement@11420} "java.base/java.util.Optional.ifPresentOrElse(Optional.java:196)"
8 = {StackTraceElement@11421} "org.apache.polaris.service.catalog.iceberg.IcebergCatalog.validateLocationsForTableLike(IcebergCatalog.java:1010)"
9 = {StackTraceElement@11422} "org.apache.polaris.service.catalog.iceberg.IcebergCatalog$BasePolarisTableOperations.doCommit(IcebergCatalog.java:1438)"
10 = {StackTraceElement@11423} "org.apache.polaris.service.catalog.iceberg.IcebergCatalog$BasePolarisTableOperations.commit(IcebergCatalog.java:1311)"
11 = {StackTraceElement@11424} "org.apache.polaris.service.catalog.iceberg.IcebergCatalog.registerTable(IcebergCatalog.java:324)"
12 = {StackTraceElement@11425} "org.apache.polaris.service.catalog.iceberg.CatalogHandlerUtils.registerTable(CatalogHandlerUtils.java:351)"
13 = {StackTraceElement@11426} "org.apache.polaris.service.catalog.iceberg.CatalogHandlerUtils_ClientProxy.registerTable(Unknown Source)"
14 = {StackTraceElement@11427} "org.apache.polaris.service.catalog.iceberg.IcebergCatalogHandler.registerTable(IcebergCatalogHandler.java:542)"
15 = {StackTraceElement@11428} "org.apache.polaris.service.catalog.iceberg.IcebergCatalogAdapter.lambda$registerTable$13(IcebergCatalogAdapter.java:516)"
16 = {StackTraceElement@11429} "org.apache.polaris.service.catalog.iceberg.IcebergCatalogAdapter.withCatalog(IcebergCatalogAdapter.java:194)"
17 = {StackTraceElement@11430} "org.apache.polaris.service.catalog.iceberg.IcebergCatalogAdapter.registerTable(IcebergCatalogAdapter.java:512)"
18 = {StackTraceElement@11431} "org.apache.polaris.service.catalog.iceberg.IcebergCatalogAdapter_Subclass.registerTable$$superforward(Unknown Source)"
19 = {StackTraceElement@11432} "org.apache.polaris.service.catalog.iceberg.IcebergRestCatalogEventServiceDelegator_jERUcZHTfCJqY_ybaNsfhH2VA0g_Delegate_Subclass.registerTable(Unknown Source)"
20 = {StackTraceElement@11433} "org.apache.polaris.service.catalog.iceberg.IcebergRestCatalogEventServiceDelegator.registerTable(IcebergRestCatalogEventServiceDelegator.java:172)"
21 = {StackTraceElement@11434} "org.apache.polaris.service.catalog.iceberg.IcebergCatalogAdapter_Subclass.registerTable(Unknown Source)"
22 = {StackTraceElement@11435} "org.apache.polaris.service.catalog.iceberg.IcebergCatalogAdapter_ClientProxy.registerTable(Unknown Source)"
23 = {StackTraceElement@11436} "org.apache.polaris.service.catalog.api.IcebergRestCatalogApi.registerTable(IcebergRestCatalogApi.java:672)"
24 = {StackTraceElement@11437} "org.apache.polaris.service.catalog.api.IcebergRestCatalogApi_Subclass.registerTable$$superforward(Unknown Source)"
```
### To Reproduce
_No response_
### Actual Behavior
_No response_
### Expected Behavior
_No response_
### Additional context
_No response_
### System information
_No response_
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Cannot register Iceberg tables created by HMS. [polaris]
Posted by "ajantha-bhat (via GitHub)" <gi...@apache.org>.
ajantha-bhat commented on issue #2756:
URL: https://github.com/apache/polaris/issues/2756#issuecomment-3370023372
Since HMS adds the suffix ".db" to the namespace folder, this check is failing.
> You can disable the check if you'd like and understand the implications of doing so.
Whats the configuration to disable it? I am thinking in terms of end user. How does HMS users can register their table to Apache polaris?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Cannot register Iceberg tables created by HMS. [polaris]
Posted by "ajantha-bhat (via GitHub)" <gi...@apache.org>.
ajantha-bhat commented on issue #2756:
URL: https://github.com/apache/polaris/issues/2756#issuecomment-3374892412
> I'm not sure exactly what you mean by this, but Polaris does not allow you to create a table outside of its parent namespace by default.
Iceberg has a register table API. Where you can register the existing metadata files to a table.
So, I am trying a usecase where users are moving from HMS to Polaris using register table API with their existing metadata file. When trying to register that table metadata file to the polaris. I got above error.
> There were some changes to these configs recently, but I think ALLOW_EXTERNAL_TABLE_LOCATION is meant to allow you to create a table outside of its parent namespace if, for some reason, you need to do so.
Thanks. Let me check setting `ALLOW_EXTERNAL_TABLE_LOCATION` property and try it out.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Cannot register Iceberg tables created by HMS. [polaris]
Posted by "ajantha-bhat (via GitHub)" <gi...@apache.org>.
ajantha-bhat commented on issue #2756:
URL: https://github.com/apache/polaris/issues/2756#issuecomment-3374901633
So, I see two options as per above discussions.
Manually add the namespace location in `allowedLocations` of `storageConfigInfo` during catalog creation. But If I have too many namespaces and tables from HMS, I don't think this is efficient as I need to add each one here.
Other option is to just enable `ALLOW_EXTERNAL_TABLE_LOCATION`.
I will try out both and share the results here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Cannot register Iceberg tables created by HMS. [polaris]
Posted by "ajantha-bhat (via GitHub)" <gi...@apache.org>.
ajantha-bhat commented on issue #2756:
URL: https://github.com/apache/polaris/issues/2756#issuecomment-3366025676
The Iceberg table (bar.tbl3) created by HMS is in a single folder "bar.tbl3" polaris is considers namespace as a separate entity. So, it might be expecting a folder with "bar" which is having subfolder "tbl3".
Code assumption that namespace entity folder always exists maybe wrong?
Exception is from `IcebergCatalog.validateLocationsForTableLike` which is based on `PolarisResolvedPathWrapper`.
I do have a repo to validate the fix. Testcase will be uploaded to polaris tools repo soon under `ITHiveToPolarisCLIMigrationTest`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Cannot register Iceberg tables created by HMS. [polaris]
Posted by "flyrain (via GitHub)" <gi...@apache.org>.
flyrain commented on issue #2756:
URL: https://github.com/apache/polaris/issues/2756#issuecomment-3374114344
Hi @ajantha-bhat , users could set the namespace location to resolve this issue. Custom namespace location is supported by Polaris.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Cannot register Iceberg tables created by HMS. [polaris]
Posted by "ajantha-bhat (via GitHub)" <gi...@apache.org>.
ajantha-bhat closed issue #2756: Cannot register Iceberg tables created by HMS.
URL: https://github.com/apache/polaris/issues/2756
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Cannot register Iceberg tables created by HMS. [polaris]
Posted by "eric-maynard (via GitHub)" <gi...@apache.org>.
eric-maynard closed issue #2756: Cannot register Iceberg tables created by HMS.
URL: https://github.com/apache/polaris/issues/2756
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Cannot register Iceberg tables created by HMS. [polaris]
Posted by "eric-maynard (via GitHub)" <gi...@apache.org>.
eric-maynard commented on issue #2756:
URL: https://github.com/apache/polaris/issues/2756#issuecomment-3374402841
> How does HMS users can register their table to Apache polaris?
I'm not sure exactly what you mean by this, but Polaris does not allow you to create a table outside of its parent namespace by default.
As others have noted, you can set the location of the namespace in Polaris to match the actual physical location of the namespace.
There were some changes to these configs recently, but I think `ALLOW_EXTERNAL_TABLE_LOCATION` is meant to allow you to create a table outside of its parent namespace if, for some reason, you need to do so.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Cannot register Iceberg tables created by HMS. [polaris]
Posted by "eric-maynard (via GitHub)" <gi...@apache.org>.
eric-maynard commented on issue #2756:
URL: https://github.com/apache/polaris/issues/2756#issuecomment-3369201912
This is not a bug but expected behavior; tables are expected to reside inside the parent namespace location. You can disable the check if you'd like and understand the implications of doing so.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Cannot register Iceberg tables created by HMS. [polaris]
Posted by "dimas-b (via GitHub)" <gi...@apache.org>.
dimas-b commented on issue #2756:
URL: https://github.com/apache/polaris/issues/2756#issuecomment-3374304756
@ajantha-bhat : If the expected location discoverable in this case, then it's a matter of setting the `location` property on the namespace in Polaris.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Cannot register Iceberg tables created by HMS. [polaris]
Posted by "ajantha-bhat (via GitHub)" <gi...@apache.org>.
ajantha-bhat commented on issue #2756:
URL: https://github.com/apache/polaris/issues/2756#issuecomment-3379627915
> There were some changes to these configs recently, but I think ALLOW_EXTERNAL_TABLE_LOCATION is meant to allow you to create a table outside of its parent namespace if, for some reason, you need to do so.
Ok. I experimented with the configs. `ALLOW_EXTERNAL_TABLE_LOCATION` is not the one.
But setting `ALLOW_UNSTRUCTURED_TABLE_LOCATION` helped in bypassing that validation and table can be registered.
I have updated the steps [here](https://github.com/apache/polaris-tools/pull/35/files#diff-5db20a834389437e758a23b4dcdd323866a2dc1989d0d4a0204eaf93e7254383R221-R223)
Closing the issue as we can enable the configuration to fix this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org