You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by "HyukjinKwon (via GitHub)" <gi...@apache.org> on 2024/02/07 04:46:16 UTC
[PR] [DO-NOT-MERGE] Decouple PySpark core API to pyspark.core package [spark]
HyukjinKwon opened a new pull request, #45053:
URL: https://github.com/apache/spark/pull/45053
### What changes were proposed in this pull request?
TBD
### Why are the changes needed?
TBD
### Does this PR introduce _any_ user-facing change?
No.
### How was this patch tested?
Existing test cases
### Was this patch authored or co-authored using generative AI tooling?
No.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-47683][PYTHON][BUILD] Decouple PySpark core API to pyspark.core package [spark]
Posted by "HyukjinKwon (via GitHub)" <gi...@apache.org>.
HyukjinKwon commented on PR #45053:
URL: https://github.com/apache/spark/pull/45053#issuecomment-2036015694
Merged to master.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-47683][PYTHON][BUILD] Decouple PySpark core API to pyspark.core package [spark]
Posted by "HyukjinKwon (via GitHub)" <gi...@apache.org>.
HyukjinKwon commented on code in PR #45053:
URL: https://github.com/apache/spark/pull/45053#discussion_r1550657673
##########
python/pyspark/util.py:
##########
@@ -503,11 +534,236 @@ def start(self) -> None:
return super(InheritableThread, self).start()
+class PythonEvalType:
Review Comment:
This isn't an API. `PandasUDFType` is the API.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-47683][PYTHON][BUILD] Decouple PySpark core API to pyspark.core package [spark]
Posted by "HyukjinKwon (via GitHub)" <gi...@apache.org>.
HyukjinKwon closed pull request #45053: [SPARK-47683][PYTHON][BUILD] Decouple PySpark core API to pyspark.core package
URL: https://github.com/apache/spark/pull/45053
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-47683][PYTHON][BUILD] Decouple PySpark core API to pyspark.core package [spark]
Posted by "allisonwang-db (via GitHub)" <gi...@apache.org>.
allisonwang-db commented on code in PR #45053:
URL: https://github.com/apache/spark/pull/45053#discussion_r1550550357
##########
python/pyspark/util.py:
##########
@@ -503,11 +534,236 @@ def start(self) -> None:
return super(InheritableThread, self).start()
+class PythonEvalType:
Review Comment:
Is this a breaking change? I.e the old import path will now throw an exception
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-47683][PYTHON][BUILD] Decouple PySpark core API to pyspark.core package [spark]
Posted by "HyukjinKwon (via GitHub)" <gi...@apache.org>.
HyukjinKwon commented on PR #45053:
URL: https://github.com/apache/spark/pull/45053#issuecomment-2035851943
I restored the references for our internal API. Explicitly private attributes starting `_` are not restored.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-47683][PYTHON][BUILD] Decouple PySpark core API to pyspark.core package [spark]
Posted by "ueshin (via GitHub)" <gi...@apache.org>.
ueshin commented on code in PR #45053:
URL: https://github.com/apache/spark/pull/45053#discussion_r1550363925
##########
python/pyspark/core/broadcast.py:
##########
Review Comment:
We may want to keep the original files of the files moved to `core` with importing all for `pyspark` package for backward compatibility?
- e.g., `pyspark/broadcast.py`
```py
from pyspark.util import is_remote_only
if not is_remote_only():
from pyspark.core.broadcast import *
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-47683][PYTHON][BUILD] Decouple PySpark core API to pyspark.core package [spark]
Posted by "HyukjinKwon (via GitHub)" <gi...@apache.org>.
HyukjinKwon commented on code in PR #45053:
URL: https://github.com/apache/spark/pull/45053#discussion_r1550656525
##########
python/pyspark/core/broadcast.py:
##########
Review Comment:
Those files are actually not meant to be used by end users .. but thinking about it, probably it's safer to keep them. Let me take a quick look.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-47683][PYTHON][BUILD] Decouple PySpark core API to pyspark.core package [spark]
Posted by "HyukjinKwon (via GitHub)" <gi...@apache.org>.
HyukjinKwon commented on PR #45053:
URL: https://github.com/apache/spark/pull/45053#issuecomment-2033351917
cc @zhengruifeng @grundprinzip @ueshin @hvanhovell @itholic @WeichenXu123 @mengxr @allisonwang-db @xinrong-meng @gatorsmile @cloud-fan This is ready for a look (before merging, should wait one more day for the SPIP to pass though)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org