Re: [PR] [SPARK-49383][SQL][PYTHON][CONNECT] Support Transpose DataFrame API [spark]

["xinrong-meng (via GitHub)" <gi...@apache.org>]

xinrong-meng commented on PR #47884:
URL: https://github.com/apache/spark/pull/47884#issuecomment-2323642874

   I verified that `transpose` works on Spark Connect Scala Client as
   ```
   scala> spark.range(2) 
   res0: org.apache.spark.sql.Dataset[java.lang.Long] = [id: bigint]
   
   scala> spark.range(2).show() 
   +---+
   | id|
   +---+
   |  0|
   |  1|
   +---+
   
   
   scala> spark.range(2).transpose().show() 
   +---+---+---+
   |key|  0|  1|
   +---+---+---+
   +---+---+---+
   
   scala> spark.range(2).transpose().explain(true) 
   == Parsed Logical Plan ==
   'UnresolvedTranspose id#38: bigint
   +- Range (0, 2, step=1, splits=Some(10))
   
   == Analyzed Logical Plan ==
   key: string, 0: string, 1: string
   Transpose [key#40, 0#41, 1#42], false
   
   == Optimized Logical Plan ==
   LocalRelation <empty>, [key#40, 0#41, 1#42]
   
   == Physical Plan ==
   LocalTableScan <empty>, [key#40, 0#41, 1#42]
   
   scala> sc 
   cmd3.sc:1: not found: value sc
   val res3 = sc
              ^
   Compilation Failed
   ```
   
   However, it failed test suite `org.apache.spark.sql.connect.ProtoToParsedPlanTestSuite` by 
   
   ```
   [info] - transpose_no_index_colomn *** FAILED *** (6 milliseconds)
   [info]  org.apache.spark.SparkException: [INTERNAL_ERROR] Found the unresolved operator: 'UnresolvedTranspose id#431: bigint SQLSTATE: XX000
   ```
   
   My guess is that the way we launch a SparkSession in the test suite doesn't mimic exactly how we launch bin/spark-shell. Can we skip the test suite in the PR and fix the test in a follow-up?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org