You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2022/11/11 03:32:13 UTC
[GitHub] [spark] xinrong-meng opened a new pull request, #38611: [SPARK-41107] Install memory-profiler in the CI
xinrong-meng opened a new pull request, #38611:
URL: https://github.com/apache/spark/pull/38611
### What changes were proposed in this pull request?
Install [memory-profiler](https://pypi.org/project/memory-profiler/) in CI in order to enable memory profiling tests.
### Why are the changes needed?
That's a sub-task of [SPARK-40281](https://issues.apache.org/jira/browse/SPARK-40281) Memory Profiler on Executors.
PySpark memory profiler depends on memory-profiler. The PR proposes to install memory-profiler in the CI to enable related tests.
### Does this PR introduce _any_ user-facing change?
No.
### How was this patch tested?
Existing tests.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] Yikun closed pull request #38611: [SPARK-41107][PYTHON][INFRA][TESTS] Install memory-profiler in the CI
Posted by GitBox <gi...@apache.org>.
Yikun closed pull request #38611: [SPARK-41107][PYTHON][INFRA][TESTS] Install memory-profiler in the CI
URL: https://github.com/apache/spark/pull/38611
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on pull request #38611: [SPARK-41107][PYTHON][INFRA][TESTS] Install memory-profiler in the CI
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on PR #38611:
URL: https://github.com/apache/spark/pull/38611#issuecomment-1319432329
I think mlflow got upgraded together for some reasons.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #38611: [SPARK-41107][PYTHON][INFRA][TEST] Install memory-profiler in the CI
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on PR #38611:
URL: https://github.com/apache/spark/pull/38611#issuecomment-1312240061
Can one of the admins verify this patch?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on pull request #38611: [SPARK-41107][PYTHON][INFRA][TESTS] Install memory-profiler in the CI
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on PR #38611:
URL: https://github.com/apache/spark/pull/38611#issuecomment-1319430302
The test failure this time seems different:
```
**********************************************************************
File "/__w/spark/spark/python/pyspark/pandas/mlflow.py", line 168, in pyspark.pandas.mlflow.load_model
Failed example:
run_info = client.list_run_infos(exp_id)[-1]
Exception raised:
Traceback (most recent call last):
File "/usr/lib/python3.9/doctest.py", line 1336, in __run
exec(compile(example.source, filename, "single",
File "<doctest pyspark.pandas.mlflow.load_model[14]>", line 1, in <module>
run_info = client.list_run_infos(exp_id)[-1]
AttributeError: 'MlflowClient' object has no attribute 'list_run_infos'
**********************************************************************
File "/__w/spark/spark/python/pyspark/pandas/mlflow.py", line 169, in pyspark.pandas.mlflow.load_model
Failed example:
model = load_model("runs:/{run_id}/model".format(run_id=run_info.run_uuid))
Exception raised:
Traceback (most recent call last):
File "/usr/lib/python3.9/doctest.py", line 1336, in __run
exec(compile(example.source, filename, "single",
File "<doctest pyspark.pandas.mlflow.load_model[15]>", line 1, in <module>
model = load_model("runs:/{run_id}/model".format(run_id=run_info.run_uuid))
NameError: name 'run_info' is not defined
**********************************************************************
File "/__w/spark/spark/python/pyspark/pandas/mlflow.py", line 171, in pyspark.pandas.mlflow.load_model
Failed example:
prediction_df["prediction"] = model.predict(prediction_df)
Exception raised:
Traceback (most recent call last):
File "/usr/lib/python3.9/doctest.py", line 1336, in __run
exec(compile(example.source, filename, "single",
File "<doctest pyspark.pandas.mlflow.load_model[17]>", line 1, in <module>
prediction_df["prediction"] = model.predict(prediction_df)
NameError: name 'model' is not defined
**********************************************************************
File "/__w/spark/spark/python/pyspark/pandas/mlflow.py", line 172, in pyspark.pandas.mlflow.load_model
Failed example:
prediction_df
Expected:
x1 x2 prediction
0 2.0 4.0 1.355551
Got:
x1 x2
0 2.0 4.0
**********************************************************************
File "/__w/spark/spark/python/pyspark/pandas/mlflow.py", line 178, in pyspark.pandas.mlflow.load_model
Failed example:
model.predict(prediction_df[["x1", "x2"]].to_pandas())
Exception raised:
Traceback (most recent call last):
File "/usr/lib/python3.9/doctest.py", line 1336, in __run
exec(compile(example.source, filename, "single",
File "<doctest pyspark.pandas.mlflow.load_model[19]>", line 1, in <module>
model.predict(prediction_df[["x1", "x2"]].to_pandas())
NameError: name 'model' is not defined
**********************************************************************
File "/__w/spark/spark/python/pyspark/pandas/mlflow.py", line 189, in pyspark.pandas.mlflow.load_model
Failed example:
y = model.predict(features)
Exception raised:
Traceback (most recent call last):
File "/usr/lib/python3.9/doctest.py", line 1336, in __run
exec(compile(example.source, filename, "single",
File "<doctest pyspark.pandas.mlflow.load_model[22]>", line 1, in <module>
y = model.predict(features)
NameError: name 'model' is not defined
**********************************************************************
File "/__w/spark/spark/python/pyspark/pandas/mlflow.py", line 198, in pyspark.pandas.mlflow.load_model
Failed example:
features['y'] = y
Exception raised:
Traceback (most recent call last):
File "/usr/lib/python3.9/doctest.py", line 1336, in __run
exec(compile(example.source, filename, "single",
File "<doctest pyspark.pandas.mlflow.load_model[25]>", line 1, in <module>
features['y'] = y
NameError: name 'y' is not defined
**********************************************************************
File "/__w/spark/spark/python/pyspark/pandas/mlflow.py", line 200, in pyspark.pandas.mlflow.load_model
Failed example:
everything
Expected:
x1 x2 z y
0 2.0 3.0 -1 1.376932
Got:
x1 x2 z
0 2.0 3.0 -1
**********************************************************************
8 of 26 in pyspark.pandas.mlflow.load_model
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on a diff in pull request #38611: [SPARK-41107][PYTHON][INFRA][TEST] Install memory-profiler in the CI
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on code in PR #38611:
URL: https://github.com/apache/spark/pull/38611#discussion_r1022222629
##########
dev/infra/Dockerfile:
##########
@@ -32,7 +32,7 @@ RUN $APT_INSTALL software-properties-common git libxml2-dev pkg-config curl wget
RUN update-alternatives --set java /usr/lib/jvm/java-8-openjdk-amd64/jre/bin/java
RUN curl -sS https://bootstrap.pypa.io/get-pip.py | python3.9
-RUN python3.9 -m pip install numpy pyarrow 'pandas<=1.5.1' scipy unittest-xml-reporting plotly>=4.8 sklearn 'mlflow>=1.0' coverage matplotlib openpyxl
+RUN python3.9 -m pip install numpy pyarrow 'pandas<=1.5.1' scipy unittest-xml-reporting plotly>=4.8 sklearn 'mlflow>=1.0' coverage matplotlib openpyxl 'memory-profiler==0.60.0'
RUN add-apt-repository ppa:pypy/ppa
Review Comment:
```
> [ 8/26] RUN add-apt-repository ppa:pypy/ppa:
#14 523.3 Cannot add PPA: 'ppa:~pypy/ubuntu/ppa'.
#14 523.3 ERROR: '~pypy' user or team does not exist.
------
Dockerfile:37
--------------------
35 | RUN python3.9 -m pip install numpy pyarrow 'pandas<=1.5.1' scipy unittest-xml-reporting plotly>=4.8 sklearn 'mlflow>=1.0' coverage matplotlib openpyxl 'memory-profiler==0.60.0'
36 |
37 | >>> RUN add-apt-repository ppa:pypy/ppa
38 | RUN apt update
39 | RUN $APT_INSTALL gfortran libopenblas-dev liblapack-dev
```
@Yikun do you have an idea why this fails?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] xinrong-meng commented on a diff in pull request #38611: [SPARK-41107][PYTHON][INFRA][TEST] Install memory-profiler in the CI
Posted by GitBox <gi...@apache.org>.
xinrong-meng commented on code in PR #38611:
URL: https://github.com/apache/spark/pull/38611#discussion_r1020600278
##########
dev/infra/Dockerfile:
##########
@@ -32,7 +32,7 @@ RUN $APT_INSTALL software-properties-common git libxml2-dev pkg-config curl wget
RUN update-alternatives --set java /usr/lib/jvm/java-8-openjdk-amd64/jre/bin/java
RUN curl -sS https://bootstrap.pypa.io/get-pip.py | python3.9
-RUN python3.9 -m pip install numpy pyarrow 'pandas<=1.5.1' scipy unittest-xml-reporting plotly>=4.8 sklearn 'mlflow>=1.0' coverage matplotlib openpyxl
+RUN python3.9 -m pip install numpy pyarrow 'pandas<=1.5.1' scipy unittest-xml-reporting plotly>=4.8 sklearn 'mlflow>=1.0' coverage matplotlib openpyxl memory-profiler
Review Comment:
Good point, modified.
##########
python/pyspark/testing/utils.py:
##########
@@ -28,6 +28,7 @@
have_scipy = False
have_numpy = False
+have_memory_profiler = False
Review Comment:
Sounds good.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] xinrong-meng commented on pull request #38611: [SPARK-41107][PYTHON][INFRA][TEST] Install memory-profiler in the CI
Posted by GitBox <gi...@apache.org>.
xinrong-meng commented on PR #38611:
URL: https://github.com/apache/spark/pull/38611#issuecomment-1312958146
Failed as below. Irrelevant to the PR. Retriggered tests.
```
[error] (unsafe / Test / compileIncremental) Compilation failed
[error] Total time: 799 s (13:19), completed Nov 12, 2022 12:06:53 AM
[error] running /__w/spark/spark/build/sbt -Phadoop-3 -Phadoop-cloud -Pyarn -Pkinesis-asl -Pmesos -Pdocker-integration-tests -Pspark-ganglia-lgpl -Phive -Pkubernetes -Phive-thriftserver Test/package streaming-kinesis-asl-assembly/assembly connect/assembly ; received return code 1
Error: Process completed with exit code 16.
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] Yikun commented on a diff in pull request #38611: [SPARK-41107][PYTHON][INFRA][TEST] Install memory-profiler in the CI
Posted by GitBox <gi...@apache.org>.
Yikun commented on code in PR #38611:
URL: https://github.com/apache/spark/pull/38611#discussion_r1022254122
##########
dev/infra/Dockerfile:
##########
@@ -32,7 +32,7 @@ RUN $APT_INSTALL software-properties-common git libxml2-dev pkg-config curl wget
RUN update-alternatives --set java /usr/lib/jvm/java-8-openjdk-amd64/jre/bin/java
RUN curl -sS https://bootstrap.pypa.io/get-pip.py | python3.9
-RUN python3.9 -m pip install numpy pyarrow 'pandas<=1.5.1' scipy unittest-xml-reporting plotly>=4.8 sklearn 'mlflow>=1.0' coverage matplotlib openpyxl
+RUN python3.9 -m pip install numpy pyarrow 'pandas<=1.5.1' scipy unittest-xml-reporting plotly>=4.8 sklearn 'mlflow>=1.0' coverage matplotlib openpyxl 'memory-profiler==0.60.0'
RUN add-apt-repository ppa:pypy/ppa
Review Comment:
This should be the flaky error when adding apt repository (I haven't saw this error before), I think retry can help.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] ueshin commented on a diff in pull request #38611: [SPARK-41107][PYTHON][INFRA][TEST] Install memory-profiler in the CI
Posted by GitBox <gi...@apache.org>.
ueshin commented on code in PR #38611:
URL: https://github.com/apache/spark/pull/38611#discussion_r1020481025
##########
dev/infra/Dockerfile:
##########
@@ -32,7 +32,7 @@ RUN $APT_INSTALL software-properties-common git libxml2-dev pkg-config curl wget
RUN update-alternatives --set java /usr/lib/jvm/java-8-openjdk-amd64/jre/bin/java
RUN curl -sS https://bootstrap.pypa.io/get-pip.py | python3.9
-RUN python3.9 -m pip install numpy pyarrow 'pandas<=1.5.1' scipy unittest-xml-reporting plotly>=4.8 sklearn 'mlflow>=1.0' coverage matplotlib openpyxl
+RUN python3.9 -m pip install numpy pyarrow 'pandas<=1.5.1' scipy unittest-xml-reporting plotly>=4.8 sklearn 'mlflow>=1.0' coverage matplotlib openpyxl memory-profiler
Review Comment:
Shall we freeze the version of `memory-profiler`, just in case?
##########
python/pyspark/testing/utils.py:
##########
@@ -28,6 +28,7 @@
have_scipy = False
have_numpy = False
+have_memory_profiler = False
Review Comment:
I guess we can reuse `pyspark.profiler.has_memory_profiler`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] xinrong-meng commented on pull request #38611: [SPARK-41107][PYTHON][INFRA][TESTS] Install memory-profiler in the CI
Posted by GitBox <gi...@apache.org>.
xinrong-meng commented on PR #38611:
URL: https://github.com/apache/spark/pull/38611#issuecomment-1319423102
Do you happen to know if it's normal to fail so many times? @Yikun
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] xinrong-meng commented on a diff in pull request #38611: [SPARK-41107][PYTHON][INFRA][TESTS] Install memory-profiler in the CI
Posted by GitBox <gi...@apache.org>.
xinrong-meng commented on code in PR #38611:
URL: https://github.com/apache/spark/pull/38611#discussion_r1023163181
##########
dev/infra/Dockerfile:
##########
@@ -32,7 +32,7 @@ RUN $APT_INSTALL software-properties-common git libxml2-dev pkg-config curl wget
RUN update-alternatives --set java /usr/lib/jvm/java-8-openjdk-amd64/jre/bin/java
RUN curl -sS https://bootstrap.pypa.io/get-pip.py | python3.9
-RUN python3.9 -m pip install numpy pyarrow 'pandas<=1.5.1' scipy unittest-xml-reporting plotly>=4.8 sklearn 'mlflow>=1.0' coverage matplotlib openpyxl
+RUN python3.9 -m pip install numpy pyarrow 'pandas<=1.5.1' scipy unittest-xml-reporting plotly>=4.8 sklearn 'mlflow>=1.0' coverage matplotlib openpyxl 'memory-profiler==0.60.0'
RUN add-apt-repository ppa:pypy/ppa
Review Comment:
Thanks! I'll retry.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] Yikun commented on pull request #38611: [SPARK-41107][PYTHON][INFRA][TESTS] Install memory-profiler in the CI
Posted by GitBox <gi...@apache.org>.
Yikun commented on PR #38611:
URL: https://github.com/apache/spark/pull/38611#issuecomment-1319457396
Yes, new version mlflow had some breaking change, you could first install memory-profile in the end of dockerfile(like connect).
I will find sometime today to fix the doctest for new mlflow, in a separate PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] ueshin commented on a diff in pull request #38611: [SPARK-41107][PYTHON][INFRA][TEST] Install memory-profiler in the CI
Posted by GitBox <gi...@apache.org>.
ueshin commented on code in PR #38611:
URL: https://github.com/apache/spark/pull/38611#discussion_r1020481025
##########
dev/infra/Dockerfile:
##########
@@ -32,7 +32,7 @@ RUN $APT_INSTALL software-properties-common git libxml2-dev pkg-config curl wget
RUN update-alternatives --set java /usr/lib/jvm/java-8-openjdk-amd64/jre/bin/java
RUN curl -sS https://bootstrap.pypa.io/get-pip.py | python3.9
-RUN python3.9 -m pip install numpy pyarrow 'pandas<=1.5.1' scipy unittest-xml-reporting plotly>=4.8 sklearn 'mlflow>=1.0' coverage matplotlib openpyxl
+RUN python3.9 -m pip install numpy pyarrow 'pandas<=1.5.1' scipy unittest-xml-reporting plotly>=4.8 sklearn 'mlflow>=1.0' coverage matplotlib openpyxl memory-profiler
Review Comment:
Shall we fix the version of `memory-profiler`, just in case?
##########
python/pyspark/testing/utils.py:
##########
@@ -28,6 +28,7 @@
have_scipy = False
have_numpy = False
+have_memory_profiler = False
Review Comment:
Can't we reuse `pyspark.profiler.has_memory_profiler`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org