You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by "jaceklaskowski (via GitHub)" <gi...@apache.org> on 2025/10/26 16:08:03 UTC
[PR] [SDP] Allow full-refresh-all with dp.create_sink (and no flows) [spark]
jaceklaskowski opened a new pull request, #52735:
URL: https://github.com/apache/spark/pull/52735
### What changes were proposed in this pull request?
Executing `spark-pipelines run --full-refresh-all` with `dp.create_sink` and no flows fails with the following `NoSuchElementException`:
```
java.util.NoSuchElementException: key not found: `my_sink`
at scala.collection.immutable.BitmapIndexedMapNode.apply(HashMap.scala:674)
at scala.collection.immutable.HashMap.apply(HashMap.scala:132)
at org.apache.spark.sql.pipelines.graph.State$.$anonfun$findElementsToReset$5(State.scala:66)
at scala.collection.immutable.List.flatMap(List.scala:283)
at scala.collection.immutable.List.flatMap(List.scala:79)
at org.apache.spark.sql.pipelines.graph.State$.findElementsToReset(State.scala:66)
at org.apache.spark.sql.pipelines.graph.State$.reset(State.scala:75)
at org.apache.spark.sql.pipelines.graph.PipelineExecution.startPipeline(PipelineExecution.scala:50)
at org.apache.spark.sql.pipelines.graph.PipelineExecution.runPipeline(PipelineExecution.scala:67)
at org.apache.spark.sql.connect.pipelines.PipelinesHandler$.$anonfun$startRun$1(PipelinesHandler.scala:382)
at org.apache.spark.sql.connect.pipelines.PipelinesHandler$.$anonfun$startRun$1$adapted(PipelinesHandler.scala:344)
at scala.util.Using$.resource(Using.scala:296)
at org.apache.spark.sql.connect.pipelines.PipelinesHandler$.startRun(PipelinesHandler.scala:344)
at org.apache.spark.sql.connect.pipelines.PipelinesHandler$.handlePipelinesCommand(PipelinesHandler.scala:118)
at org.apache.spark.sql.connect.planner.SparkConnectPlanner.handlePipelineCommand(SparkConnectPlanner.scala:2898)
at org.apache.spark.sql.connect.planner.SparkConnectPlanner.process(SparkConnectPlanner.scala:2868)
at org.apache.spark.sql.connect.execution.SparkConnectPlanExecution.handlePlan(SparkConnectPlanExecution.scala:95)
at org.apache.spark.sql.connect.execution.ExecuteThreadRunner.$anonfun$executeInternal$1(ExecuteThreadRunner.scala:224)
at org.apache.spark.sql.connect.execution.ExecuteThreadRunner.$anonfun$executeInternal$1$adapted(ExecuteThreadRunner.scala:196)
at org.apache.spark.sql.connect.service.SessionHolder.$anonfun$withSession$2(SessionHolder.scala:394)
at org.apache.spark.sql.SparkSession.withActive(SparkSession.scala:804)
```
`stream_transformations/dp_create_sink.py`:
```py
from pyspark import pipelines as dp
dp.create_sink(name="my_sink", format="delta")
```
### Why are the changes needed?
It should be acceptable to have a `create_sink` alone.
### Does this PR introduce _any_ user-facing change?
Yes. No exception.
### How was this patch tested?
Manual execution. Happy to contribute a test if guided.
### Was this patch authored or co-authored using generative AI tooling?
No
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-54549][SDP] Allow full-refresh-all with dp.create_sink (and no flows) [spark]
Posted by "jaceklaskowski (via GitHub)" <gi...@apache.org>.
jaceklaskowski commented on PR #52735:
URL: https://github.com/apache/spark/pull/52735#issuecomment-3598020570
Closing the PR as I can no longer reproduce the issue.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SDP] Allow full-refresh-all with dp.create_sink (and no flows) [spark]
Posted by "jaceklaskowski (via GitHub)" <gi...@apache.org>.
jaceklaskowski commented on PR #52735:
URL: https://github.com/apache/spark/pull/52735#issuecomment-3448661547
/cc @sryza @jackywang-db
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SDP] Allow full-refresh-all with dp.create_sink (and no flows) [spark]
Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun commented on PR #52735:
URL: https://github.com/apache/spark/pull/52735#issuecomment-3493546033
> @dongjoon-hyun Why should we file a JIRA issue since it's about an unreleased code (master)? Just curious what I'm supposed to file an issue about?
Is' for traceability, @jaceklaskowski . In the community, we use JIRA ID instead of GitHash pointing to your contribution.
```
$ git log --oneline -n3 --format=%s
[SPARK-54176][GEO][PYTHON] Introduce Geography and Geometry data types to PySpark Connect
[SPARK-54156][PROTOBUF] Classify errors for ProtobufOptions casting failure
[SPARK-54180][SQL] Override the toString of BinaryFileFormat
```
It's also useful when we do `[FOLLOWUP]`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SDP] Allow full-refresh-all with dp.create_sink (and no flows) [spark]
Posted by "jaceklaskowski (via GitHub)" <gi...@apache.org>.
jaceklaskowski commented on PR #52735:
URL: https://github.com/apache/spark/pull/52735#issuecomment-3493525295
@dongjoon-hyun Why should we file a JIRA issue since it's about an unreleased code (master)? Just curious what I'm supposed to file an issue about?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SDP] Allow full-refresh-all with dp.create_sink (and no flows) [spark]
Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun commented on PR #52735:
URL: https://github.com/apache/spark/pull/52735#issuecomment-3493556083
To be clear, all patches are already a part of Apache Spark 4.2.0, @jaceklaskowski . `master` branch is not a dangling personal feature branch.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-54549][SDP] Allow full-refresh-all with dp.create_sink (and no flows) [spark]
Posted by "jaceklaskowski (via GitHub)" <gi...@apache.org>.
jaceklaskowski closed pull request #52735: [SPARK-54549][SDP] Allow full-refresh-all with dp.create_sink (and no flows)
URL: https://github.com/apache/spark/pull/52735
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org