[GitHub] [spark] mickjermsurawong-stripe commented on pull request #27153: [SPARK-20384][SQL] Support value class in schema of Dataset (without breaking existing current projection)

[GitBox <gi...@apache.org>]

mickjermsurawong-stripe commented on pull request #27153:
URL: https://github.com/apache/spark/pull/27153#issuecomment-873105068


   Hey @eejbyfeldt  sorry i realized that the internal fork we addressed this issue is in a different class of our custom encoder, so i didn't to integrate it with the standard encoder here.
   
   The underlying issue we addressed is exactly the allocation you mentioned: From doc on value class: https://docs.scala-lang.org/overviews/core/value-classes.html Given: `class Wrapper(val underlying: Int) extends AnyVal`,
   
   - "The type at compile time is Wrapper, but at runtime, the representation is an Int"
   This implies that when our struct has a field of AnyVal case class, our generated code
   should support the underlying type during runtime execution.
   
   - The Wrapper "must be instantiated... when a value class is used as a type argument".
   This implies that scala.Tuple[Wrapper, ...], Seq[Wrapper], Map[String, Wrapper], Option[Wrapper] will still contain Wrapper as-is in during runtime instead of Int.
   
   Hope this could help. It's a long weekend here, so I will make sure I'll get to work on this. But please feel free to pursue independently as well.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org