You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2020/06/10 07:35:36 UTC

[GitHub] [spark] HyukjinKwon opened a new pull request #28777: [SPARK-31915][SQL][PYTHON] Resolve the grouping column properly per the case sensitivity in grouped and cogrouped pandas UDFs

HyukjinKwon opened a new pull request #28777:
URL: https://github.com/apache/spark/pull/28777


   ### What changes were proposed in this pull request?
   
   This is another approach to fix the issue. See the previous try https://github.com/apache/spark/pull/28745. It was too invasive so I took more conservative approach.
   
   This PR proposes to resolve grouping attributes separately first so it can be properly referred when `FlatMapGroupsInPandas` and `FlatMapCoGroupsInPandas` are resolved without ambiguity.
   
   Previously, 
   
   ```python
   from pyspark.sql.functions import *
   df = spark.createDataFrame([[1, 1]], ["column", "Score"])
   @pandas_udf("column integer, Score float", PandasUDFType.GROUPED_MAP)
   def my_pandas_udf(pdf):
       return pdf.assign(Score=0.5)
   
   df.groupby('COLUMN').apply(my_pandas_udf).show()
   ```
   
   was failed as below:
   
   ```
   pyspark.sql.utils.AnalysisException: "Reference 'COLUMN' is ambiguous, could be: COLUMN, COLUMN.;"
   ```
   because the unresolved `COLUMN` in `FlatMapGroupsInPandas` doesn't know which reference to take from the child projection.
   
   After this fix, it resolves the child projection first with grouping keys and pass, to `FlatMapGroupsInPandas`, the attribute as a grouping key from the child projection that is positionally selected.
   
   ### Why are the changes needed?
   
   To resolve grouping keys correctly.
   
   ### Does this PR introduce _any_ user-facing change?
   
   Yes,
   
   
   
   ```python
   from pyspark.sql.functions import *
   df = spark.createDataFrame([[1, 1]], ["column", "Score"])
   @pandas_udf("column integer, Score float", PandasUDFType.GROUPED_MAP)
   def my_pandas_udf(pdf):
       return pdf.assign(Score=0.5)
   
   df.groupby('COLUMN').apply(my_pandas_udf).show()
   ```
   
   ```python
   df1 = spark.createDataFrame([(1, 1)], ("column", "value"))
   df2 = spark.createDataFrame([(1, 1)], ("column", "value"))
   
   df1.groupby("COLUMN").cogroup(
       df2.groupby("COLUMN")
   ).applyInPandas(lambda r, l: r + l, df1.schema).show()
   ```
   
   Before:
   
   ```
   pyspark.sql.utils.AnalysisException: Reference 'COLUMN' is ambiguous, could be: COLUMN, COLUMN.;
   ```
   
   ```
   pyspark.sql.utils.AnalysisException: cannot resolve '`COLUMN`' given input columns: [COLUMN, COLUMN, value, value];;
   'FlatMapCoGroupsInPandas ['COLUMN], ['COLUMN], <lambda>(column#9L, value#10L, column#13L, value#14L), [column#22L, value#23L]
   :- Project [COLUMN#9L, column#9L, value#10L]
   :  +- LogicalRDD [column#9L, value#10L], false
   +- Project [COLUMN#13L, column#13L, value#14L]
      +- LogicalRDD [column#13L, value#14L], false
   ```
   
   
   After:
   
   ```
   +------+-----+
   |column|Score|
   +------+-----+
   |     1|  0.5|
   +------+-----+
   ```
   
   ```
   +------+-----+
   |column|value|
   +------+-----+
   |     2|    2|
   +------+-----+
   ```
   
   ### How was this patch tested?
   
   Unittests were added and manually tested.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA removed a comment on pull request #28777: [SPARK-31915][SQL][PYTHON] Resolve the grouping column properly per the case sensitivity in grouped and cogrouped pandas UDFs

Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #28777:
URL: https://github.com/apache/spark/pull/28777#issuecomment-641794736


   **[Test build #123737 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/123737/testReport)** for PR 28777 at commit [`c656ddb`](https://github.com/apache/spark/commit/c656ddb03d56a59a02c6b6c38a731475625a38fc).


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] HyukjinKwon commented on pull request #28777: [SPARK-31915][SQL][PYTHON] Resolve the grouping column properly per the case sensitivity in grouped and cogrouped pandas UDFs

Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on pull request #28777:
URL: https://github.com/apache/spark/pull/28777#issuecomment-642383298


   Actually, I believe this is a pretty conservative fix bug fix .. let me port this back to 3.0 as well... if requested, I don't mind porting back to 2.4 as well.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on pull request #28777: [SPARK-31915][SQL][PYTHON] Resolve the grouping column properly per the case sensitivity in grouped and cogrouped pandas UDFs

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #28777:
URL: https://github.com/apache/spark/pull/28777#issuecomment-641938913






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on pull request #28777: [SPARK-31915][SQL][PYTHON] Resolve the grouping column properly per the case sensitivity in grouped and cogrouped pandas UDFs

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #28777:
URL: https://github.com/apache/spark/pull/28777#issuecomment-641948089






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] HyukjinKwon commented on pull request #28777: [SPARK-31915][SQL][PYTHON] Resolve the grouping column properly per the case sensitivity in grouped and cogrouped pandas UDFs

Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on pull request #28777:
URL: https://github.com/apache/spark/pull/28777#issuecomment-642390534


   Merged to branch-3.0 as well!


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on pull request #28777: [SPARK-31915][SQL][PYTHON] Resolve the grouping column properly per the case sensitivity in grouped and cogrouped pandas UDFs

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #28777:
URL: https://github.com/apache/spark/pull/28777#issuecomment-641948089






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #28777: [SPARK-31915][SQL][PYTHON] Resolve the grouping column properly per the case sensitivity in grouped and cogrouped pandas UDFs

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #28777:
URL: https://github.com/apache/spark/pull/28777#issuecomment-641947442


   **[Test build #123759 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/123759/testReport)** for PR 28777 at commit [`c656ddb`](https://github.com/apache/spark/commit/c656ddb03d56a59a02c6b6c38a731475625a38fc).


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on pull request #28777: [SPARK-31915][SQL][PYTHON] Resolve the grouping column properly per the case sensitivity in grouped and cogrouped pandas UDFs

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #28777:
URL: https://github.com/apache/spark/pull/28777#issuecomment-641938913


   Merged build finished. Test FAILed.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] BryanCutler commented on pull request #28777: [SPARK-31915][SQL][PYTHON] Resolve the grouping column properly per the case sensitivity in grouped and cogrouped pandas UDFs

Posted by GitBox <gi...@apache.org>.
BryanCutler commented on pull request #28777:
URL: https://github.com/apache/spark/pull/28777#issuecomment-642309232


   merged to master, thanks @HyukjinKwon !


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on pull request #28777: [SPARK-31915][SQL][PYTHON] Resolve the grouping column properly per the case sensitivity in grouped and cogrouped pandas UDFs

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #28777:
URL: https://github.com/apache/spark/pull/28777#issuecomment-642166640






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] HyukjinKwon commented on pull request #28777: [SPARK-31915][SQL][PYTHON] Resolve the grouping column properly per the case sensitivity in grouped and cogrouped pandas UDFs

Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on pull request #28777:
URL: https://github.com/apache/spark/pull/28777#issuecomment-641945440


   retest this please


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA removed a comment on pull request #28777: [SPARK-31915][SQL][PYTHON] Resolve the grouping column properly per the case sensitivity in grouped and cogrouped pandas UDFs

Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #28777:
URL: https://github.com/apache/spark/pull/28777#issuecomment-641947442


   **[Test build #123759 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/123759/testReport)** for PR 28777 at commit [`c656ddb`](https://github.com/apache/spark/commit/c656ddb03d56a59a02c6b6c38a731475625a38fc).


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] HyukjinKwon commented on pull request #28777: [SPARK-31915][SQL][PYTHON] Resolve the grouping column properly per the case sensitivity in grouped and cogrouped pandas UDFs

Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on pull request #28777:
URL: https://github.com/apache/spark/pull/28777#issuecomment-642334873


   Thanks guys!


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on pull request #28777: [SPARK-31915][SQL][PYTHON] Resolve the grouping column properly per the case sensitivity in grouped and cogrouped pandas UDFs

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #28777:
URL: https://github.com/apache/spark/pull/28777#issuecomment-641795537






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #28777: [SPARK-31915][SQL][PYTHON] Resolve the grouping column properly per the case sensitivity in grouped and cogrouped pandas UDFs

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #28777:
URL: https://github.com/apache/spark/pull/28777#issuecomment-641937966


   **[Test build #123737 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/123737/testReport)** for PR 28777 at commit [`c656ddb`](https://github.com/apache/spark/commit/c656ddb03d56a59a02c6b6c38a731475625a38fc).
    * This patch **fails Spark unit tests**.
    * This patch merges cleanly.
    * This patch adds no public classes.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #28777: [SPARK-31915][SQL][PYTHON] Resolve the grouping column properly per the case sensitivity in grouped and cogrouped pandas UDFs

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #28777:
URL: https://github.com/apache/spark/pull/28777#issuecomment-642165555


   **[Test build #123759 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/123759/testReport)** for PR 28777 at commit [`c656ddb`](https://github.com/apache/spark/commit/c656ddb03d56a59a02c6b6c38a731475625a38fc).
    * This patch passes all tests.
    * This patch merges cleanly.
    * This patch adds no public classes.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on pull request #28777: [SPARK-31915][SQL][PYTHON] Resolve the grouping column properly per the case sensitivity in grouped and cogrouped pandas UDFs

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #28777:
URL: https://github.com/apache/spark/pull/28777#issuecomment-641938922


   Test FAILed.
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/123737/
   Test FAILed.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #28777: [SPARK-31915][SQL][PYTHON] Resolve the grouping column properly per the case sensitivity in grouped and cogrouped pandas UDFs

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #28777:
URL: https://github.com/apache/spark/pull/28777#issuecomment-641794736


   **[Test build #123737 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/123737/testReport)** for PR 28777 at commit [`c656ddb`](https://github.com/apache/spark/commit/c656ddb03d56a59a02c6b6c38a731475625a38fc).


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on pull request #28777: [SPARK-31915][SQL][PYTHON] Resolve the grouping column properly per the case sensitivity in grouped and cogrouped pandas UDFs

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #28777:
URL: https://github.com/apache/spark/pull/28777#issuecomment-642166640






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] BryanCutler closed pull request #28777: [SPARK-31915][SQL][PYTHON] Resolve the grouping column properly per the case sensitivity in grouped and cogrouped pandas UDFs

Posted by GitBox <gi...@apache.org>.
BryanCutler closed pull request #28777:
URL: https://github.com/apache/spark/pull/28777


   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on pull request #28777: [SPARK-31915][SQL][PYTHON] Resolve the grouping column properly per the case sensitivity in grouped and cogrouped pandas UDFs

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #28777:
URL: https://github.com/apache/spark/pull/28777#issuecomment-641795537






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org