You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by "Dzeri96 (via GitHub)" <gi...@apache.org> on 2025/10/27 09:06:14 UTC
[I] `spark.sql.catalog.spark_catalog.default-namespace` has no effect with SparkSessionCatalog [iceberg]
Dzeri96 opened a new issue, #14424:
URL: https://github.com/apache/iceberg/issues/14424
### Apache Iceberg version
1.9.1
### Query engine
Spark
### Please describe the bug 🐞
As discussed on [Slack](https://apache-iceberg.slack.com/archives/C025PH0G1D4/p1761315026299809), the default database/namespace in the spark session catalog implementation is hard-coded to `default`, meaning that `spark.sql.catalog.spark_catalog.default-namespace` has no effect.
As a workaround, I've figured out that `spark.sql.catalog.spark_catalog.defaultDatabase` works, however we should either fix the misleading documentation, or see if we can get the config to work.
### Willingness to contribute
- [ ] I can contribute a fix for this bug independently
- [ ] I would be willing to contribute a fix for this bug with guidance from the Iceberg community
- [ ] I cannot contribute a fix for this bug at this time
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] `spark.sql.catalog.spark_catalog.default-namespace` has no effect with SparkSessionCatalog [iceberg]
Posted by "manuzhang (via GitHub)" <gi...@apache.org>.
manuzhang commented on issue #14424:
URL: https://github.com/apache/iceberg/issues/14424#issuecomment-3471558164
`spark.sql.catalog.spark_catalog.defaultDatabase` is a Spark config whose default value is `default`. I think default namespace from `SparkSessionCatalog` is aligning with it. Please open a PR to document it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] `spark.sql.catalog.spark_catalog.default-namespace` has no effect with SparkSessionCatalog [iceberg]
Posted by "Dzeri96 (via GitHub)" <gi...@apache.org>.
Dzeri96 commented on issue #14424:
URL: https://github.com/apache/iceberg/issues/14424#issuecomment-3675689775
I just found the time to go a bit deeper with this issue and I have some thoughts.
First of all, my assumption on why the default namespace is hard-coded in the `SparkSessionCatalog` is because the delegating catalog is instantiated before it by Spark, so as far as I know, you cannot affect its creation with the ´default-namespace´ config key like you can when your are creating your own `SparkCatalog`.
With this said, I think the `defaultNamespace()` method can just call the delegating catalog. If you change the default DB with `spark_catalog.defaultDatabase`, it will give you the name of this DB, which is semantically equivalent to a namespace. This is something that I would definitely fix if it's ok with you.
What I'm not sure about, is if we want to use `catalog.defaultDatabse` as a fallback when configuring `SparkCatalog`. I think it makes sense because it's a standard spark config key.
Going another level, maybe we can go the other way, and configure the `SparkSessionCatalog` with `default-namespace`. I guess we would need to hook into the spark startup somehow. This behavior is described by the current Iceberg docs.
Let me know what you think is the best way to proceed. Based on that, I'll change the code and the docs.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] `spark.sql.catalog.spark_catalog.default-namespace` has no effect with SparkSessionCatalog [iceberg]
Posted by "manuzhang (via GitHub)" <gi...@apache.org>.
manuzhang commented on issue #14424:
URL: https://github.com/apache/iceberg/issues/14424#issuecomment-3677386885
@Dzeri96 It will be best to demonstrate your idea with PRs and examples.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] `spark.sql.catalog.spark_catalog.default-namespace` has no effect with SparkSessionCatalog [iceberg]
Posted by "Dzeri96 (via GitHub)" <gi...@apache.org>.
Dzeri96 commented on issue #14424:
URL: https://github.com/apache/iceberg/issues/14424#issuecomment-4852262637
@manuzhang Sorry it took so long to open the PR, but after some consideration, I think it makes sense to deprecate the `default-namespace` config. You will find the changes quite small. If they get accepted, the PR from @wombatu-kun won't be needed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] `spark.sql.catalog.spark_catalog.default-namespace` has no effect with SparkSessionCatalog [iceberg]
Posted by "wombatu-kun (via GitHub)" <gi...@apache.org>.
wombatu-kun commented on issue #14424:
URL: https://github.com/apache/iceberg/issues/14424#issuecomment-4562573065
Picking this up. I prototyped the natural code fix in `SparkSessionCatalog` (parse `default-namespace` in `initialize()`, return it from `defaultNamespace()`, mirroring what `SparkCatalog` already does). A unit test confirmed the method then honored the option, but an end-to-end test failed:
```java
spark.conf().set("spark.sql.catalog.spark_catalog.default-namespace", "my_ns");
spark.sql("CREATE NAMESPACE IF NOT EXISTS my_ns");
spark.sessionState().catalogManager().reset();
spark.sql("USE spark_catalog");
// Expected: ["my_ns"]
// Actual: ["default"]
assertThat(spark.sessionState().catalogManager().currentNamespace())
.containsExactly("my_ns");
```
Tracing why: Spark's `V2SessionCatalog.defaultNamespace()` also hard-codes `["default"]`, and for `spark_catalog` the current namespace flows through V1 `SessionCatalog.currentDatabase`, which is not influenced by the V2 catalog's `defaultNamespace()`. The `spark.sql.catalog.spark_catalog.defaultDatabase` workaround works precisely because it feeds into V1 initialization directly. Propagating `default-namespace` into V1 from our `initialize()` would mean issuing side effects on Spark's session catalog, which feels out of scope for a documentation gap.
Given that, I opened #16591 with a docs-only change pointing users to `defaultDatabase`. Happy to revisit a code fix if maintainers prefer that path - would just want guidance on whether V1 side effects from `initialize()` are acceptable.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org