You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Deng Ziming (Jira)" <ji...@apache.org> on 2024/12/09 02:44:00 UTC
[jira] [Resolved] (SPARK-45236) ToPrettyString will treat java encoder or kryo encoder as BinaryType
[ https://issues.apache.org/jira/browse/SPARK-45236?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Deng Ziming resolved SPARK-45236.
---------------------------------
Resolution: Fixed
This seems has been fixed unintentionally.
{code:java}
// spark sql
import org.apache.spark.sql.catalyst.encoders.AgnosticEncoders.{BoxedIntEncoder, MapEncoder}
implicit val im: ExpressionEncoder[java.util.Map[Int, Int]] = ExpressionEncoder(MapEncoder(ClassTag(classOf[java.util.Map[Int, Int]]), BoxedIntEncoder, BoxedIntEncoder, false)) val ds = spark.createDataset(Seq(Collections.singletonMap(1,1)))
ds.show()
// +--------+
// | value|
// +--------+
// |{1 -> 1}|
// +--------+{code}
> ToPrettyString will treat java encoder or kryo encoder as BinaryType
> --------------------------------------------------------------------
>
> Key: SPARK-45236
> URL: https://issues.apache.org/jira/browse/SPARK-45236
> Project: Spark
> Issue Type: Improvement
> Components: SQL
> Affects Versions: 3.5.0
> Reporter: Deng Ziming
> Priority: Minor
>
> {code:java}
> // spark sql
> // java encoder or kryo encoder
> implicit val im = Encoders.kryo[java.util.Map[Int, Int]].asInstanceOf[ExpressionEncoder[util.Map[Int, Int]]]
> val ds = spark.createDataset(Seq(Collections.singletonMap(1,1)))
> ds.show()
> // +--------------------+
> // | value|
> // +--------------------+
> // |[01 00 6A 61 76 6...|
> // +--------------------+
> ds.collect().foreach(println(_))
> // {1=1}
> {code}
> ds.collect().foreach(println) is more readable than ds.show(), not sure is the designed intentionally.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org