You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2023/01/09 13:05:59 UTC

[GitHub] [spark] HyukjinKwon opened a new pull request, #39467: [SPARK-41950][PS][BUILD] Pin mlflow to 2.0.X

HyukjinKwon opened a new pull request, #39467:
URL: https://github.com/apache/spark/pull/39467

   ### What changes were proposed in this pull request?
   
   After MLflow 2.1 release (https://github.com/mlflow/mlflow/releases), the unittest in pandas API on Spark is broken.
   
   ```python
   from mlflow.tracking import MlflowClient, set_tracking_uri
   from sklearn.linear_model import LinearRegression
   import mlflow.sklearn
   from tempfile import mkdtemp
   d = mkdtemp("pandas_on_spark_mlflow")
   set_tracking_uri("file:%s"%d)
   client = MlflowClient()
   exp_id = mlflow.create_experiment("my_experiment")
   exp = mlflow.set_experiment("my_experiment")
   train = pd.DataFrame({"x1": np.arange(8), "x2": np.arange(8)**2,
                         "y": np.log(2 + np.arange(8))})
   train_x = train[["x1", "x2"]]
   train_y = train[["y"]]
   with mlflow.start_run():
       lr = LinearRegression()
       lr.fit(train_x, train_y)
       mlflow.sklearn.log_model(lr, "model")
   from pyspark.pandas.mlflow import load_model
   run_info = client.search_runs(exp_id)[-1].info
   model = load_model("runs:/{run_id}/model".format(run_id=run_info.run_id))
   prediction_df = ps.DataFrame({"x1": [2.0], "x2": [4.0]})
   prediction_df["prediction"] = model.predict(prediction_df)
   print(prediction_df)
   ```
   
   https://github.com/apache/spark/blob/06ec98b0d6a51e0c3ffec70e78d86d577b0e7a72/python/pyspark/pandas/mlflow.py#L134-L202
   
   ```
   File "/__w/spark/spark/python/pyspark/pandas/mlflow.py", line 172, in pyspark.pandas.mlflow.load_model
   Failed example:
       prediction_df
   Exception raised:
       Traceback (most recent call last):
         File "/usr/lib/python3.9/doctest.py", line 1336, in __run
           exec(compile(example.source, filename, "single",
         File "<doctest pyspark.pandas.mlflow.load_model[18]>", line 1, in <module>
           prediction_df
         File "/__w/spark/spark/python/pyspark/pandas/frame.py", line 13322, in __repr__
           pdf = cast("DataFrame", self._get_or_create_repr_pandas_cache(max_display_count))
         File "/__w/spark/spark/python/pyspark/pandas/frame.py", line 13313, in _get_or_create_repr_pandas_cache
           self, "_repr_pandas_cache", {n: self.head(n + 1)._to_internal_pandas()}
         File "/__w/spark/spark/python/pyspark/pandas/frame.py", line 13308, in _to_internal_pandas
           return self._internal.to_pandas_frame
         File "/__w/spark/spark/python/pyspark/pandas/utils.py", line 588, in wrapped_lazy_property
           setattr(self, attr_name, fn(self))
         File "/__w/spark/spark/python/pyspark/pandas/internal.py", line 1056, in to_pandas_frame
           pdf = sdf.toPandas()
         File "/__w/spark/spark/python/pyspark/sql/pandas/conversion.py", line 208, in toPandas
           pdf = pd.DataFrame.from_records(self.collect(), columns=self.columns)
         File "/__w/spark/spark/python/pyspark/sql/dataframe.py", line 1197, in collect
           sock_info = self._jdf.collectToPython()
         File "/__w/spark/spark/python/lib/py4j-0.10.9.7-src.zip/py4j/java_gateway.py", line 1322, in __call__
           return_value = get_return_value(
         File "/__w/spark/spark/python/pyspark/sql/utils.py", line 209, in deco
           raise converted from None
       pyspark.sql.utils.PythonException: 
         An exception was thrown from the Python worker. Please see the stack trace below.
       Traceback (most recent call last):
         File "/__w/spark/spark/python/lib/pyspark.zip/pyspark/worker.py", line 829, in main
           process()
         File "/__w/spark/spark/python/lib/pyspark.zip/pyspark/worker.py", line 821, in process
           serializer.dump_stream(out_iter, outfile)
         File "/__w/spark/spark/python/lib/pyspark.zip/pyspark/sql/pandas/serializers.py", line 345, in dump_stream
           return ArrowStreamSerializer.dump_stream(self, init_stream_yield_batches(), stream)
         File "/__w/spark/spark/python/lib/pyspark.zip/pyspark/sql/pandas/serializers.py", line 86, in dump_stream
           for batch in iterator:
         File "/__w/spark/spark/python/lib/pyspark.zip/pyspark/sql/pandas/serializers.py", line 338, in init_stream_yield_batches
           for series in iterator:
         File "/__w/spark/spark/python/lib/pyspark.zip/pyspark/worker.py", line 519, in func
           for result_batch, result_type in result_iter:
         File "/usr/local/lib/python3.9/dist-packages/mlflow/pyfunc/__init__.py", line 1253, in udf
           yield _predict_row_batch(batch_predict_fn, row_batch_args)
         File "/usr/local/lib/python3.9/dist-packages/mlflow/pyfunc/__init__.py", line 1057, in _predict_row_batch
           result = predict_fn(pdf)
         File "/usr/local/lib/python3.9/dist-packages/mlflow/pyfunc/__init__.py", line 1237, in batch_predict_fn
           return loaded_model.predict(pdf)
         File "/usr/local/lib/python3.9/dist-packages/mlflow/pyfunc/__init__.py", line 413, in predict
           return self._predict_fn(data)
         File "/usr/local/lib/python3.9/dist-packages/sklearn/linear_model/_base.py", line 355, in predict
           return self._decision_function(X)
         File "/usr/local/lib/python3.9/dist-packages/sklearn/linear_model/_base.py", line 338, in _decision_function
           X = self._validate_data(X, accept_sparse=["csr", "csc", "coo"], reset=False)
         File "/usr/local/lib/python3.9/dist-packages/sklearn/base.py", line 518, in _validate_data
           self._check_feature_names(X, reset=reset)
         File "/usr/local/lib/python3.9/dist-packages/sklearn/base.py", line 451, in _check_feature_names
           raise ValueError(message)
       ValueError: The feature names should match those that were passed during fit.
       Feature names unseen at fit time:
       - 0
       - 1
       Feature names seen at fit time, yet now missing:
       - x1
       - x2
   ```
   
   https://github.com/apache/spark/actions/runs/3871715040/jobs/6600578830
   
   ### Why are the changes needed?
   
   To recover the broken test.
   
   ### Does this PR introduce _any_ user-facing change?
   
   No, test-only (for now).
   Maybe a regression but from MLflow.
   
   ### How was this patch tested?
   
   CI in this PR should test it out.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] dongjoon-hyun commented on pull request #39467: [SPARK-41950][PS][BUILD] Pin mlflow to 2.0.X

Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on PR #39467:
URL: https://github.com/apache/spark/pull/39467#issuecomment-1376169584

   Python CIs seems to fail still. Could you re-trigger the failed ones?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] HyukjinKwon commented on pull request #39467: [SPARK-41950][PS][BUILD] Pin mlflow to 2.0.X

Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on PR #39467:
URL: https://github.com/apache/spark/pull/39467#issuecomment-1375596732

   cc @WeichenXu123 and @harupy FYI


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] HyukjinKwon commented on a diff in pull request #39467: [SPARK-41950][PS][BUILD] Pin mlflow to 2.0.X

Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on code in PR #39467:
URL: https://github.com/apache/spark/pull/39467#discussion_r1065239028


##########
dev/infra/Dockerfile:
##########
@@ -65,7 +65,7 @@ RUN Rscript -e "devtools::install_version('roxygen2', version='7.2.0', repos='ht
 ENV R_LIBS_SITE "/usr/local/lib/R/site-library:${R_LIBS_SITE}:/usr/lib/R/library"
 
 RUN pypy3 -m pip install numpy 'pandas<=1.5.2' scipy coverage matplotlib
-RUN python3.9 -m pip install numpy pyarrow 'pandas<=1.5.2' scipy unittest-xml-reporting plotly>=4.8 sklearn 'mlflow>=1.0' coverage matplotlib openpyxl 'memory-profiler==0.60.0'
+RUN python3.9 -m pip install numpy pyarrow 'pandas<=1.5.2' scipy unittest-xml-reporting plotly>=4.8 sklearn 'mlflow>=1.0' coverage matplotlib openpyxl 'memory-profiler==0.60.0' 'scikit-learn~=1.1.3'

Review Comment:
   ```suggestion
   RUN python3.9 -m pip install numpy pyarrow 'pandas<=1.5.2' scipy unittest-xml-reporting plotly>=4.8 sklearn 'mlflow>=1.0' coverage matplotlib openpyxl 'memory-profiler==0.60.0' 'scikit-learn==1.1.*'
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] harupy commented on pull request #39467: [SPARK-41950][PS][BUILD] Pin scikit-learn to 1.1.X

Posted by GitBox <gi...@apache.org>.
harupy commented on PR #39467:
URL: https://github.com/apache/spark/pull/39467#issuecomment-1376761925

   @HyukjinKwon Is it possible to add a name to each `pd.Series` that `iterator`?
   
   
   ```python
       # MLflow's pandas_udf looks like this:
   
       @pandas_udf(result_type)
       def udf(
           iterator: Iterator[Tuple[Union[pandas.Series, pandas.DataFrame], ...]]
       ) -> Iterator[result_type_hint]:
           # iterator yields a pd.Series
           ...
   ```


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] HyukjinKwon commented on a diff in pull request #39467: [SPARK-41950][PS][BUILD] Pin mlflow to 2.0.X

Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on code in PR #39467:
URL: https://github.com/apache/spark/pull/39467#discussion_r1065238847


##########
dev/infra/Dockerfile:
##########
@@ -65,7 +65,7 @@ RUN Rscript -e "devtools::install_version('roxygen2', version='7.2.0', repos='ht
 ENV R_LIBS_SITE "/usr/local/lib/R/site-library:${R_LIBS_SITE}:/usr/lib/R/library"
 
 RUN pypy3 -m pip install numpy 'pandas<=1.5.2' scipy coverage matplotlib
-RUN python3.9 -m pip install numpy pyarrow 'pandas<=1.5.2' scipy unittest-xml-reporting plotly>=4.8 sklearn 'mlflow>=1.0' coverage matplotlib openpyxl 'memory-profiler==0.60.0'
+RUN python3.9 -m pip install numpy pyarrow 'pandas<=1.5.2' scipy unittest-xml-reporting plotly>=4.8 sklearn 'mlflow~=2.0.0' coverage matplotlib openpyxl 'memory-profiler==0.60.0'

Review Comment:
   ```suggestion
   RUN python3.9 -m pip install numpy pyarrow 'pandas<=1.5.2' scipy unittest-xml-reporting plotly>=4.8 sklearn 'mlflow>=1.0' coverage matplotlib openpyxl 'memory-profiler==0.60.0' 'scikit-learn~=1.1.3'
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] HyukjinKwon commented on pull request #39467: [SPARK-41950][PS][BUILD] Pin scikit-learn to 1.1.X

Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on PR #39467:
URL: https://github.com/apache/spark/pull/39467#issuecomment-1376672653

   All tests passed.
   
   Merged to master.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] dongjoon-hyun commented on pull request #39467: [SPARK-41950][PS][BUILD] Pin mlflow to 2.0.X

Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on PR #39467:
URL: https://github.com/apache/spark/pull/39467#issuecomment-1376209043

   cc @xinrong-meng since she is the release manager for Apache Spark 3.4.0.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] itholic commented on pull request #39467: [SPARK-41950][PS][BUILD] Pin scikit-learn to 1.1.X

Posted by GitBox <gi...@apache.org>.
itholic commented on PR #39467:
URL: https://github.com/apache/spark/pull/39467#issuecomment-1376687585

   Thanks!!!


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] HyukjinKwon closed pull request #39467: [SPARK-41950][PS][BUILD] Pin scikit-learn to 1.1.X

Posted by GitBox <gi...@apache.org>.
HyukjinKwon closed pull request #39467: [SPARK-41950][PS][BUILD] Pin scikit-learn to 1.1.X
URL: https://github.com/apache/spark/pull/39467


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] harupy commented on pull request #39467: [SPARK-41950][PS][BUILD] Pin scikit-learn to 1.1.X

Posted by GitBox <gi...@apache.org>.
harupy commented on PR #39467:
URL: https://github.com/apache/spark/pull/39467#issuecomment-1376753692

   I was able to reproduce the issue.
   
   The root cause is this line:
   https://github.com/mlflow/mlflow/blob/d2ae53af1af2b7908d294749996fc30866af3bd2/mlflow/pyfunc/__init__.py#L1055
   
   ```
           if pdf is None:
               print(args)
               args = list(args)
               if input_schema is None:
                   names = [str(i) for i in range(len(args))]
               else:
                   names = input_schema.input_names()
                   if len(args) > len(names):
                       args = args[: len(names)]
                   if len(args) < len(names):
                       raise MlflowException(
                           "Model input is missing columns. Expected {} input columns {},"
                           " but the model received only {} unnamed input columns"
                           " (Since the columns were passed unnamed they are expected to be in"
                           " the order specified by the schema).".format(len(names), names, len(args))
                       )
               pdf = pandas.DataFrame(data={names[i]: x for i, x in enumerate(args)}, columns=names)  👈👈👈 this line
               print(pdf)
   ```
   
   This line creates a dataframe with columns that look like `["0", "1", ...]` unless `input_schema` is specified.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] HyukjinKwon commented on pull request #39467: [SPARK-41950][PS][BUILD] Pin mlflow to 2.0.X

Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on PR #39467:
URL: https://github.com/apache/spark/pull/39467#issuecomment-1375601216

   See also https://github.com/apache/spark/actions/runs/3869770926/jobs/6596152018#step:9:56 and https://github.com/apache/spark/actions/runs/3872798855/jobs/6603071801#step:9:57 for rationale of pinning the version


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org