You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@polaris.apache.org by "MonkeyCanCode (via GitHub)" <gi...@apache.org> on 2026/08/09 05:11:27 UTC
[PR] Fix spark plugin getting started example [polaris]
MonkeyCanCode opened a new pull request, #5259:
URL: https://github.com/apache/polaris/pull/5259
<!--
๐ Describe what changes you're proposing, especially breaking or user-facing changes.
๐ See https://github.com/apache/polaris/blob/main/CONTRIBUTING.md for more.
-->
There are two issues as of today for spark plugin getting started examples:
1. Missing spec dir in Dockerfile
2. Using custom location which is no longer allowed by default
For missing spec dir in Dockerfile, we will get the following error (during image build):
```
> [8/8] RUN python3 -m venv /home/spark/venv && . /home/spark/venv/bin/activate && pip install -r /tmp/requirements.txt && cd client/python && uv lock && uv sync --active --all-extras && uv pip install -r /tmp/notebook_requirements.txt:
3.528 Collecting uv>=0.9.0
3.636 Downloading uv-0.12.3-py3-none-manylinux_2_28_aarch64.whl (21.3 MB)
4.045 โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ 21.3/21.3 MB 47.6 MB/s eta 0:00:00
4.072 Installing collected packages: uv
4.292 Successfully installed uv-0.12.3
4.382 warning: Ignoring existing virtual environment linked to non-existent Python interpreter: .venv/bin/python3 -> python
4.468 Using CPython 3.10.12 interpreter at: /usr/bin/python3
4.476 Resolved 64 packages in 2ms
4.483 Resolved 64 packages in 0.38ms
4.491 Building apache-polaris @ file:///home/spark/client/python
4.599 Downloading pygments (1.2MiB)
4.601 Downloading virtualenv (5.7MiB)
4.612 Downloading ast-serialize (1.2MiB)
4.612 Downloading botocore (14.3MiB)
4.612 Downloading pyroaring (1.9MiB)
4.616 Downloading zstandard (4.8MiB)
4.616 Downloading mypy (13.6MiB)
4.616 Downloading openapi-generator-cli (25.9MiB)
4.639 Downloading pydantic-core (1.9MiB)
4.878 Downloaded ast-serialize
4.881 Downloaded pygments
4.962 Downloaded pyroaring
4.966 Downloaded pydantic-core
5.163 Downloaded zstandard
5.167 Downloaded virtualenv
5.622 Downloaded botocore
5.728 Downloaded mypy
6.167 Downloaded openapi-generator-cli
6.426 ร Failed to build `apache-polaris @ file:///home/spark/client/python`
6.426 โโโถ The build backend returned an error
6.426 โฐโโถ Call to `hatchling.build.build_editable` failed (exit status: 1)
6.426
6.426 [stderr]
6.426 Preparing spec directory...
6.426 Fatal: spec directory is missing and the source to copy it from was
6.426 not found.
6.426 Traceback (most recent call last):
6.426 File "<string>", line 11, in <module>
6.426 File
6.426 "/home/spark/.cache/uv/builds-v0/.tmpZQLq3T/lib/python3.10/site-packages/hatchling/build.py",
6.426 line 83, in build_editable
6.426 return os.path.basename(next(builder.build(directory=wheel_directory,
6.426 versions=["editable"])))
6.426 File
6.426 "/home/spark/.cache/uv/builds-v0/.tmpZQLq3T/lib/python3.10/site-packages/hatchling/builders/plugin/interface.py",
6.426 line 149, in build
6.426 build_hook.initialize(version, build_data)
6.426 File "/home/spark/client/python/hatch_build.py", line 26, in
6.426 initialize
6.426 subprocess.check_call([sys.executable, "generate_clients.py"])
6.426 File "/usr/lib/python3.10/subprocess.py", line 369, in check_call
6.426 raise CalledProcessError(retcode, cmd)
6.426 subprocess.CalledProcessError: Command
6.426 '['/home/spark/.cache/uv/builds-v0/.tmpZQLq3T/bin/python',
6.426 'generate_clients.py']' returned non-zero exit status 1.
6.426
6.426
6.426 hint: Build failures usually indicate a problem with the package or the build environment
```
For using custom location which is no longer allowed by default, we will get the following error (within notebook):
```
---------------------------------------------------------------------------
Py4JJavaError Traceback (most recent call last)
Cell In[10], line 1
----> 1 spark.sql("CREATE NAMESPACE IF NOT EXISTS DELTA_NS")
2 # spark.sql("CREATE NAMESPACE IF NOT EXISTS DELTA_NS.PUBLIC")
3 # spark.sql("SHOW NAMESPACES IN DELTA_NS").show()
File /opt/spark/python/pyspark/sql/session.py:1631, in SparkSession.sql(self, sqlQuery, args, **kwargs)
1627 assert self._jvm is not None
1628 litArgs = self._jvm.PythonUtils.toArray(
1629 [_to_java_column(lit(v)) for v in (args or [])]
1630 )
-> 1631 return DataFrame(self._jsparkSession.sql(sqlQuery, litArgs), self)
1632 finally:
1633 if len(kwargs) > 0:
File /opt/spark/python/lib/py4j-0.10.9.7-src.zip/py4j/java_gateway.py:1322, in JavaMember.__call__(self, *args)
1316 command = proto.CALL_COMMAND_NAME +\
1317 self.command_header +\
1318 args_command +\
1319 proto.END_COMMAND_PART
1321 answer = self.gateway_client.send_command(command)
-> 1322 return_value = get_return_value(
1323 answer, self.gateway_client, self.target_id, self.name)
1325 for temp_arg in temp_args:
1326 if hasattr(temp_arg, "_detach"):
File /opt/spark/python/pyspark/errors/exceptions/captured.py:179, in capture_sql_exception.<locals>.deco(*a, **kw)
177 def deco(*a: Any, **kw: Any) -> Any:
178 try:
--> 179 return f(*a, **kw)
180 except Py4JJavaError as e:
181 converted = convert_exception(e.java_exception)
File /opt/spark/python/lib/py4j-0.10.9.7-src.zip/py4j/protocol.py:326, in get_return_value(answer, gateway_client, target_id, name)
324 value = OUTPUT_CONVERTER[type](answer[2:], gateway_client)
325 if answer[1] == REFERENCE_TYPE:
--> 326 raise Py4JJavaError(
327 "An error occurred while calling {0}{1}{2}.\n".
328 format(target_id, ".", name), value)
329 else:
330 raise Py4JError(
331 "An error occurred while calling {0}{1}{2}. Trace:\n{3}\n".
332 format(target_id, ".", name, value))
Py4JJavaError: An error occurred while calling o56.sql.
: org.apache.iceberg.exceptions.BadRequestException: Malformed request: Namespace DELTA_NS has a custom location, which is not enabled. Expected a location in: [file:///tmp/DELTA_NS/]. Got location: file:///tmp/polaris/DELTA_NS/]
at org.apache.iceberg.rest.ErrorHandlers$NamespaceErrorHandler.accept(ErrorHandlers.java:276)
at org.apache.iceberg.rest.ErrorHandlers$NamespaceErrorHandler.accept(ErrorHandlers.java:266)
at org.apache.iceberg.rest.HTTPClient.throwFailure(HTTPClient.java:242)
at org.apache.iceberg.rest.HTTPClient.execute(HTTPClient.java:347)
at org.apache.iceberg.rest.HTTPClient.execute(HTTPClient.java:299)
at org.apache.iceberg.rest.BaseHTTPClient.post(BaseHTTPClient.java:112)
at org.apache.iceberg.rest.RESTClient.post(RESTClient.java:150)
at org.apache.iceberg.rest.RESTSessionCatalog.createNamespace(RESTSessionCatalog.java:745)
at org.apache.iceberg.catalog.BaseSessionCatalog$AsCatalog.createNamespace(BaseSessionCatalog.java:134)
at org.apache.iceberg.rest.RESTCatalog.createNamespace(RESTCatalog.java:251)
at org.apache.iceberg.spark.SparkCatalog.createNamespace(SparkCatalog.java:498)
at org.apache.polaris.spark.BasePolarisSparkCatalog.createNamespace(BasePolarisSparkCatalog.java:288)
at org.apache.spark.sql.execution.datasources.v2.CreateNamespaceExec.run(CreateNamespaceExec.scala:47)
at org.apache.spark.sql.execution.datasources.v2.V2CommandExec.result$lzycompute(V2CommandExec.scala:43)
at org.apache.spark.sql.execution.datasources.v2.V2CommandExec.result(V2CommandExec.scala:43)
at org.apache.spark.sql.execution.datasources.v2.V2CommandExec.executeCollect(V2CommandExec.scala:49)
at org.apache.spark.sql.execution.QueryExecution$$anonfun$eagerlyExecuteCommands$1.$anonfun$applyOrElse$1(QueryExecution.scala:107)
at org.apache.spark.sql.execution.SQLExecution$.$anonfun$withNewExecutionId$6(SQLExecution.scala:125)
at org.apache.spark.sql.execution.SQLExecution$.withSQLConfPropagated(SQLExecution.scala:201)
at org.apache.spark.sql.execution.SQLExecution$.$anonfun$withNewExecutionId$1(SQLExecution.scala:108)
at org.apache.spark.sql.SparkSession.withActive(SparkSession.scala:900)
at org.apache.spark.sql.execution.SQLExecution$.withNewExecutionId(SQLExecution.scala:66)
at org.apache.spark.sql.execution.QueryExecution$$anonfun$eagerlyExecuteCommands$1.applyOrElse(QueryExecution.scala:107)
at org.apache.spark.sql.execution.QueryExecution$$anonfun$eagerlyExecuteCommands$1.applyOrElse(QueryExecution.scala:98)
at org.apache.spark.sql.catalyst.trees.TreeNode.$anonfun$transformDownWithPruning$1(TreeNode.scala:461)
at org.apache.spark.sql.catalyst.trees.CurrentOrigin$.withOrigin(origin.scala:76)
at org.apache.spark.sql.catalyst.trees.TreeNode.transformDownWithPruning(TreeNode.scala:461)
at org.apache.spark.sql.catalyst.plans.logical.LogicalPlan.org$apache$spark$sql$catalyst$plans$logical$AnalysisHelper$$super$transformDownWithPruning(LogicalPlan.scala:32)
at org.apache.spark.sql.catalyst.plans.logical.AnalysisHelper.transformDownWithPruning(AnalysisHelper.scala:267)
at org.apache.spark.sql.catalyst.plans.logical.AnalysisHelper.transformDownWithPruning$(AnalysisHelper.scala:263)
at org.apache.spark.sql.catalyst.plans.logical.LogicalPlan.transformDownWithPruning(LogicalPlan.scala:32)
at org.apache.spark.sql.catalyst.plans.logical.LogicalPlan.transformDownWithPruning(LogicalPlan.scala:32)
at org.apache.spark.sql.catalyst.trees.TreeNode.transformDown(TreeNode.scala:437)
at org.apache.spark.sql.execution.QueryExecution.eagerlyExecuteCommands(QueryExecution.scala:98)
at org.apache.spark.sql.execution.QueryExecution.commandExecuted$lzycompute(QueryExecution.scala:85)
at org.apache.spark.sql.execution.QueryExecution.commandExecuted(QueryExecution.scala:83)
at org.apache.spark.sql.Dataset.<init>(Dataset.scala:220)
at org.apache.spark.sql.Dataset$.$anonfun$ofRows$2(Dataset.scala:100)
at org.apache.spark.sql.SparkSession.withActive(SparkSession.scala:900)
at org.apache.spark.sql.Dataset$.ofRows(Dataset.scala:97)
at org.apache.spark.sql.SparkSession.$anonfun$sql$1(SparkSession.scala:638)
at org.apache.spark.sql.SparkSession.withActive(SparkSession.scala:900)
at org.apache.spark.sql.SparkSession.sql(SparkSession.scala:629)
at org.apache.spark.sql.SparkSession.sql(SparkSession.scala:659)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:569)
at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:244)
at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:374)
at py4j.Gateway.invoke(Gateway.java:282)
at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132)
at py4j.commands.CallCommand.execute(CallCommand.java:79)
at py4j.ClientServerConnection.waitForCommands(ClientServerConnection.java:182)
at py4j.ClientServerConnection.run(ClientServerConnection.java:106)
at java.base/java.lang.Thread.run(Thread.java:840)
```
## Checklist
- [ ] ๐ก๏ธ Don't disclose security issues! (contact security@apache.org)
- [ ] ๐ Clearly explained why the changes are needed, or linked related issues: Fixes #
- [ ] ๐งช Added/updated tests with good coverage, or manually tested (and explained how)
- [ ] ๐ก Added comments for complex logic
- [ ] ๐งพ Updated `CHANGELOG.md` (if needed)
- [ ] ๐ Updated documentation in `site/content/in-dev/unreleased` (if needed)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Fix spark plugin getting started example [polaris]
Posted by "MonkeyCanCode (via GitHub)" <gi...@apache.org>.
MonkeyCanCode merged PR #5259:
URL: https://github.com/apache/polaris/pull/5259
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Fix spark plugin getting started example [polaris]
Posted by "MonkeyCanCode (via GitHub)" <gi...@apache.org>.
MonkeyCanCode commented on PR #5259:
URL: https://github.com/apache/polaris/pull/5259#issuecomment-5262425509
> +1 thanks for the fix, @MonkeyCanCode ! A side note: can we have a CI for getting started examples? A daily CI should be good enough. Otherwise, they can be broken without any developer noticed.
Yes. I was thinking about the same. Created https://github.com/apache/polaris/issues/5278 as follow up.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org