You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by "dongjoon-hyun (via GitHub)" <gi...@apache.org> on 2026/08/03 22:04:06 UTC
[PR] [SPARK-58528] Support cache artifact upload in `SparkConnectClient` [spark-connect-swift]
dongjoon-hyun opened a new pull request, #472:
URL: https://github.com/apache/spark-connect-swift/pull/472
### What changes were proposed in this pull request?
This PR aims to support cache artifact upload in `SparkConnectClient` by adding a new file, `SparkConnectClient+Artifact.swift`, with the following internal APIs.
- `artifactExists`: checks whether an artifact exists in the server-side session via the `ArtifactStatus` RPC.
- `cacheArtifact`: caches the given data as a `cache/<sha256>` artifact and returns its SHA-256 hash. The upload is skipped if the artifact already exists. Data up to 32KiB is uploaded as a single-chunk batch, and larger data as a chunked artifact stream, like PySpark's `ArtifactManager.cache_artifact`.
The existing `addArtifact` is moved to the new file to group all artifact operations together.
### Why are the changes needed?
This is an infrastructure for supporting large `createDataFrame` data via `CachedLocalRelation`.
### Does this PR introduce _any_ user-facing change?
No, these are internal APIs.
### How was this patch tested?
Pass the CIs with a new test suite, `SparkConnectClientArtifactTests`.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Fable 5
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-58528] Support cache artifact upload in `SparkConnectClient` [spark-connect-swift]
Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun closed pull request #472: [SPARK-58528] Support cache artifact upload in `SparkConnectClient`
URL: https://github.com/apache/spark-connect-swift/pull/472
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-58528] Support cache artifact upload in `SparkConnectClient` [spark-connect-swift]
Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun commented on PR #472:
URL: https://github.com/apache/spark-connect-swift/pull/472#issuecomment-5172346385
Thank you, @viirya !
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-58528] Support cache artifact upload in `SparkConnectClient` [spark-connect-swift]
Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun commented on PR #472:
URL: https://github.com/apache/spark-connect-swift/pull/472#issuecomment-5172352473
Merged to main
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org