You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by "leewyang (via GitHub)" <gi...@apache.org> on 2023/04/26 20:54:24 UTC

[GitHub] [spark] leewyang opened a new pull request, #40967: predict_batch_udf with scalar input fails with batch size of one

leewyang opened a new pull request, #40967:
URL: https://github.com/apache/spark/pull/40967

   ### What changes were proposed in this pull request?
   
   This is a followup to #39817 to handle another error condition when the input batch is a single scalar value (where the previous fix focused on a single scalar value output).
   
   ### Why are the changes needed?
   Using `predict_batch_udf` fails when the input batch size is one.
   ```
   import numpy as np
   from pyspark.ml.functions import predict_batch_udf
   from pyspark.sql.types import DoubleType
   
   df = spark.createDataFrame([[1.0],[2.0]], schema=["a"])
   
   def make_predict_fn():
       def predict(inputs):
           return inputs
   
       return predict
   
   identity = predict_batch_udf(make_predict_fn, return_type=DoubleType(), batch_size=1)
   preds = df.withColumn("preds", identity("a")).show()
   ```
   fails with:
   ```
     File "/.../spark/python/pyspark/worker.py", line 869, in main
       process()
     File "/.../spark/python/pyspark/worker.py", line 861, in process
       serializer.dump_stream(out_iter, outfile)
     File "/.../spark/python/pyspark/sql/pandas/serializers.py", line 354, in dump_stream
       return ArrowStreamSerializer.dump_stream(self, init_stream_yield_batches(), stream)
     File "/.../spark/python/pyspark/sql/pandas/serializers.py", line 86, in dump_stream
       for batch in iterator:
     File "/.../spark/python/pyspark/sql/pandas/serializers.py", line 347, in init_stream_yield_batches
       for series in iterator:
     File "/.../spark/python/pyspark/worker.py", line 555, in func
       for result_batch, result_type in result_iter:
     File "/.../spark/python/pyspark/ml/functions.py", line 818, in predict
       yield _validate_and_transform_prediction_result(
     File "/.../spark/python/pyspark/ml/functions.py", line 339, in _validate_and_transform_prediction_result
       if len(preds_array) != num_input_rows:
   TypeError: len() of unsized object
   ```
   
   After the fix:
   ```
   +---+-----+
   |  a|preds|
   +---+-----+
   |1.0|  1.0|
   |2.0|  2.0|
   +---+-----+
   ```
   
   ### Does this PR introduce _any_ user-facing change?
   This fixes a bug in the feature that was released in Spark 3.4.0.
   
   ### How was this patch tested?
   Unit test was added.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] zhengruifeng closed pull request #40967: [SPARK-43298][PYTHON][ML] predict_batch_udf with scalar input fails with batch size of one

Posted by "zhengruifeng (via GitHub)" <gi...@apache.org>.
zhengruifeng closed pull request #40967: [SPARK-43298][PYTHON][ML] predict_batch_udf with scalar input fails with batch size of one
URL: https://github.com/apache/spark/pull/40967


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] HyukjinKwon commented on pull request #40967: [SPARK-43298][PYTHON][ML] predict_batch_udf with scalar input fails with batch size of one

Posted by "HyukjinKwon (via GitHub)" <gi...@apache.org>.
HyukjinKwon commented on PR #40967:
URL: https://github.com/apache/spark/pull/40967#issuecomment-1526176743

   oops, I missed that the linter failed. Reverting, and reopening the PR.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] zhengruifeng commented on pull request #40967: [SPARK-43298][PYTHON][ML] predict_batch_udf with scalar input fails with batch size of one

Posted by "zhengruifeng (via GitHub)" <gi...@apache.org>.
zhengruifeng commented on PR #40967:
URL: https://github.com/apache/spark/pull/40967#issuecomment-1526819976

   merged to master again


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] HyukjinKwon commented on pull request #40967: [SPARK-43298][PYTHON][ML] predict_batch_udf with scalar input fails with batch size of one

Posted by "HyukjinKwon (via GitHub)" <gi...@apache.org>.
HyukjinKwon commented on PR #40967:
URL: https://github.com/apache/spark/pull/40967#issuecomment-1524585405

   Mind taking a look at https://github.com/apache/spark/pull/40967/checks?check_run_id=13051337101?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] leewyang commented on pull request #40967: predict_batch_udf with scalar input fails with batch size of one

Posted by "leewyang (via GitHub)" <gi...@apache.org>.
leewyang commented on PR #40967:
URL: https://github.com/apache/spark/pull/40967#issuecomment-1524033540

   cc @HyukjinKwon @mengxr @WeichenXu123 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] leewyang commented on pull request #40967: [SPARK-43298][PYTHON][ML] predict_batch_udf with scalar input fails with batch size of one

Posted by "leewyang (via GitHub)" <gi...@apache.org>.
leewyang commented on PR #40967:
URL: https://github.com/apache/spark/pull/40967#issuecomment-1525887988

   Hmm, looks like I might be stuck with [this issue](https://github.com/docker/build-push-action/issues/687), any ideas?  I've configured my "actions permissions" to match [this comment](https://github.com/docker/build-push-action/issues/687#issuecomment-1238980158), but I'm still getting this error:
   ```
   Error: buildx failed with: ERROR: failed to solve: failed to push ghcr.io/leewyang/apache-spark-ci-image:master-4815979720: failed commit on ref "manifest-sha256:80eca005ea656d063e07f9059619043cd701e0c0d17029523fc167e4915405b4": unexpected status: 403 Forbidden
   ```


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] leewyang commented on pull request #40967: [SPARK-43298][PYTHON][ML] predict_batch_udf with scalar input fails with batch size of one

Posted by "leewyang (via GitHub)" <gi...@apache.org>.
leewyang commented on PR #40967:
URL: https://github.com/apache/spark/pull/40967#issuecomment-1525941689

   Nm, got past this issue... waiting on the rest of the build results (but already failed the [Kubernetes Integration Test](https://github.com/leewyang/spark/actions/runs/4815979720/jobs/8587955821)).


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] HyukjinKwon closed pull request #40967: [SPARK-43298][PYTHON][ML] predict_batch_udf with scalar input fails with batch size of one

Posted by "HyukjinKwon (via GitHub)" <gi...@apache.org>.
HyukjinKwon closed pull request #40967: [SPARK-43298][PYTHON][ML] predict_batch_udf with scalar input fails with batch size of one
URL: https://github.com/apache/spark/pull/40967


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] HyukjinKwon commented on pull request #40967: [SPARK-43298][PYTHON][ML] predict_batch_udf with scalar input fails with batch size of one

Posted by "HyukjinKwon (via GitHub)" <gi...@apache.org>.
HyukjinKwon commented on PR #40967:
URL: https://github.com/apache/spark/pull/40967#issuecomment-1526142286

   Merged to master.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org