[GitHub] [spark] mazeboard commented on issue #24299: [SPARK-27388][SQL] expression encoder for objects defined by properties

[GitBox <gi...@apache.org>]

mazeboard commented on issue #24299: [SPARK-27388][SQL] expression encoder for objects defined by properties
URL: https://github.com/apache/spark/pull/24299#issuecomment-480845183
 
 
   1. JavaBean do not support Avro fixed types, because the fixed type has one
   property and is named `bytes`; javaBean only accepts properties fixed with
   set/get
   
   2. I believe that the current implementation has a bug, indeed
   
   Line 136
   in sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/JavaTypeInference.scala
   should be:
   val properties = getJavaBeanReadableAndWritableProperties(other) and not
   val properties = getJavaBeanReadableProperties(other) 3. toDS, toDF, always
   uses the expression encoder I corrected the bug mentioned above and tested
   it, by using Encoders.bean, and I had three issues: 1. the ds.map fails
   with no encoder found because it is using the ScalaReflection to match for
   an encoder 2. the Encoders.bean fails for java enums (assertion fails, not
   a StructType since an enum is saved as String) 3. and for Avro fixed types
   are not supported because the property in fixed types is not prefixed by
   set/get regards Taoufik
   
   On Mon, Apr 8, 2019 at 3:47 PM Sean Owen <no...@github.com> wrote:
   
   > Overall: isn't this what the Encoder for Java Beans is for and already
   > supports? I am not sure it obvious to me this should go in ScalaReflection.
   > Separately I don't think it's worth further supporting JavaBean
   > properties. That's pretty old and rare, I think.
   >
   > —
   > You are receiving this because you authored the thread.
   > Reply to this email directly, view it on GitHub
   > <https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_apache_spark_pull_24299-23issuecomment-2D480838251&d=DwMCaQ&c=oYSInPoxeTs4m10ilnGu9Q&r=tJmPZfODPMHyXvtcDPDLh8Y6NivZyWO2Jcs2xoRat90&m=1XuL-1AY_mS1qCc4oZ3D2tNdNu8ktGbSIZBkjEfoOFU&s=03jB8i0A-2N65hOzloj_HlIMEx8b4_TWOuCjXoFMwbg&e=>,
   > or mute the thread
   > <https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_notifications_unsubscribe-2Dauth_AG82a0A2aSqET6khk5U0VMPd4vP6-2D5EJks5ve0hjgaJpZM4cdqpS&d=DwMCaQ&c=oYSInPoxeTs4m10ilnGu9Q&r=tJmPZfODPMHyXvtcDPDLh8Y6NivZyWO2Jcs2xoRat90&m=1XuL-1AY_mS1qCc4oZ3D2tNdNu8ktGbSIZBkjEfoOFU&s=UeN37Pqqn63me2GLDt3OxMo7wZ-0G5_MOFDttZIyKQg&e=>
   > .
   >
   
   -- 
   This e-mail and any attachment are confidential and intended solely for the 
   use of the individual to whom it is addressed. If you are not the intended 
   recipient, please telephone or email the sender and delete this message and 
   any attachment from your system. Unauthorized publication, use, 
   dissemination, forwarding, printing or copying of this e-mail and its 
   associated attachments is strictly prohibited.
   
   
   
   http://disclaimer.carrefour.com/ <http://disclaimer.carrefour.com/>
   
   
   Let's 
   respect the environment together. Only print this message if necessary.
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org