You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by "leewyang (via GitHub)" <gi...@apache.org> on 2023/04/26 20:54:24 UTC
[GitHub] [spark] leewyang opened a new pull request, #40967: predict_batch_udf with scalar input fails with batch size of one
leewyang opened a new pull request, #40967:
URL: https://github.com/apache/spark/pull/40967
### What changes were proposed in this pull request?
This is a followup to #39817 to handle another error condition when the input batch is a single scalar value (where the previous fix focused on a single scalar value output).
### Why are the changes needed?
Using `predict_batch_udf` fails when the input batch size is one.
```
import numpy as np
from pyspark.ml.functions import predict_batch_udf
from pyspark.sql.types import DoubleType
df = spark.createDataFrame([[1.0],[2.0]], schema=["a"])
def make_predict_fn():
def predict(inputs):
return inputs
return predict
identity = predict_batch_udf(make_predict_fn, return_type=DoubleType(), batch_size=1)
preds = df.withColumn("preds", identity("a")).show()
```
fails with:
```
File "/.../spark/python/pyspark/worker.py", line 869, in main
process()
File "/.../spark/python/pyspark/worker.py", line 861, in process
serializer.dump_stream(out_iter, outfile)
File "/.../spark/python/pyspark/sql/pandas/serializers.py", line 354, in dump_stream
return ArrowStreamSerializer.dump_stream(self, init_stream_yield_batches(), stream)
File "/.../spark/python/pyspark/sql/pandas/serializers.py", line 86, in dump_stream
for batch in iterator:
File "/.../spark/python/pyspark/sql/pandas/serializers.py", line 347, in init_stream_yield_batches
for series in iterator:
File "/.../spark/python/pyspark/worker.py", line 555, in func
for result_batch, result_type in result_iter:
File "/.../spark/python/pyspark/ml/functions.py", line 818, in predict
yield _validate_and_transform_prediction_result(
File "/.../spark/python/pyspark/ml/functions.py", line 339, in _validate_and_transform_prediction_result
if len(preds_array) != num_input_rows:
TypeError: len() of unsized object
```
After the fix:
```
+---+-----+
| a|preds|
+---+-----+
|1.0| 1.0|
|2.0| 2.0|
+---+-----+
```
### Does this PR introduce _any_ user-facing change?
This fixes a bug in the feature that was released in Spark 3.4.0.
### How was this patch tested?
Unit test was added.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] zhengruifeng closed pull request #40967: [SPARK-43298][PYTHON][ML] predict_batch_udf with scalar input fails with batch size of one
Posted by "zhengruifeng (via GitHub)" <gi...@apache.org>.
zhengruifeng closed pull request #40967: [SPARK-43298][PYTHON][ML] predict_batch_udf with scalar input fails with batch size of one
URL: https://github.com/apache/spark/pull/40967
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on pull request #40967: [SPARK-43298][PYTHON][ML] predict_batch_udf with scalar input fails with batch size of one
Posted by "HyukjinKwon (via GitHub)" <gi...@apache.org>.
HyukjinKwon commented on PR #40967:
URL: https://github.com/apache/spark/pull/40967#issuecomment-1526176743
oops, I missed that the linter failed. Reverting, and reopening the PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] zhengruifeng commented on pull request #40967: [SPARK-43298][PYTHON][ML] predict_batch_udf with scalar input fails with batch size of one
Posted by "zhengruifeng (via GitHub)" <gi...@apache.org>.
zhengruifeng commented on PR #40967:
URL: https://github.com/apache/spark/pull/40967#issuecomment-1526819976
merged to master again
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on pull request #40967: [SPARK-43298][PYTHON][ML] predict_batch_udf with scalar input fails with batch size of one
Posted by "HyukjinKwon (via GitHub)" <gi...@apache.org>.
HyukjinKwon commented on PR #40967:
URL: https://github.com/apache/spark/pull/40967#issuecomment-1524585405
Mind taking a look at https://github.com/apache/spark/pull/40967/checks?check_run_id=13051337101?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] leewyang commented on pull request #40967: predict_batch_udf with scalar input fails with batch size of one
Posted by "leewyang (via GitHub)" <gi...@apache.org>.
leewyang commented on PR #40967:
URL: https://github.com/apache/spark/pull/40967#issuecomment-1524033540
cc @HyukjinKwon @mengxr @WeichenXu123
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] leewyang commented on pull request #40967: [SPARK-43298][PYTHON][ML] predict_batch_udf with scalar input fails with batch size of one
Posted by "leewyang (via GitHub)" <gi...@apache.org>.
leewyang commented on PR #40967:
URL: https://github.com/apache/spark/pull/40967#issuecomment-1525887988
Hmm, looks like I might be stuck with [this issue](https://github.com/docker/build-push-action/issues/687), any ideas? I've configured my "actions permissions" to match [this comment](https://github.com/docker/build-push-action/issues/687#issuecomment-1238980158), but I'm still getting this error:
```
Error: buildx failed with: ERROR: failed to solve: failed to push ghcr.io/leewyang/apache-spark-ci-image:master-4815979720: failed commit on ref "manifest-sha256:80eca005ea656d063e07f9059619043cd701e0c0d17029523fc167e4915405b4": unexpected status: 403 Forbidden
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] leewyang commented on pull request #40967: [SPARK-43298][PYTHON][ML] predict_batch_udf with scalar input fails with batch size of one
Posted by "leewyang (via GitHub)" <gi...@apache.org>.
leewyang commented on PR #40967:
URL: https://github.com/apache/spark/pull/40967#issuecomment-1525941689
Nm, got past this issue... waiting on the rest of the build results (but already failed the [Kubernetes Integration Test](https://github.com/leewyang/spark/actions/runs/4815979720/jobs/8587955821)).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon closed pull request #40967: [SPARK-43298][PYTHON][ML] predict_batch_udf with scalar input fails with batch size of one
Posted by "HyukjinKwon (via GitHub)" <gi...@apache.org>.
HyukjinKwon closed pull request #40967: [SPARK-43298][PYTHON][ML] predict_batch_udf with scalar input fails with batch size of one
URL: https://github.com/apache/spark/pull/40967
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on pull request #40967: [SPARK-43298][PYTHON][ML] predict_batch_udf with scalar input fails with batch size of one
Posted by "HyukjinKwon (via GitHub)" <gi...@apache.org>.
HyukjinKwon commented on PR #40967:
URL: https://github.com/apache/spark/pull/40967#issuecomment-1526142286
Merged to master.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org