You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Giambattista Bloisi (Jira)" <ji...@apache.org> on 2023/08/02 13:58:00 UTC
[jira] [Updated] (SPARK-44634) Encoders.bean does no longer support nested beans with type arguments
[ https://issues.apache.org/jira/browse/SPARK-44634?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Giambattista Bloisi updated SPARK-44634:
----------------------------------------
Description:
Hi,
while upgrading a project from spark 2.4.0 to 3.4.1 version, I have encountered the same problem described in [java - Encoders.bean attempts to check the validity of a return type considering its generic type and not its concrete class, with Spark 3.4.0 - Stack Overflow|https://stackoverflow.com/questions/76045255/encoders-bean-attempts-to-check-the-validity-of-a-return-type-considering-its-ge].
Put it short, starting from Spark 3.4.x Encoders.bean throws an exception when the passed class contains a field whose type is a nested bean with type arguments:
{code:java}
class A<T> {
T value;
// value getter and setter
}
class B {
A<String> stringHolder;
// stringHolder getter and setter
}
Encoders.bean(B.class); // throws "SparkUnsupportedOperationException: [ENCODER_NOT_FOUND]..."{code}
It looks like this is a regression introduced with [link SPARK-42093 SQL Move JavaTypeInference to AgnosticEncoders|https://github.com/apache/spark/commit/18672003513d5a4aa610b6b94dbbc15c33185d3#diff-1191737b908340a2f4c22b71b1c40ebaa0da9d8b40c958089c346a3bda26943b] while getting rid of TypeToken, that somehow managed that case.
I'm going to submit a PR to re-enable this functionality.
was:
Hi,
while upgrading a project from spark 2.4.0 to 3.4.1 version, I have encountered the same problem described in [java - Encoders.bean attempts to check the validity of a return type considering its generic type and not its concrete class, with Spark 3.4.0 - Stack Overflow|https://stackoverflow.com/questions/76045255/encoders-bean-attempts-to-check-the-validity-of-a-return-type-considering-its-ge].
Put it short, starting from Spark 3.4.x Encoders.bean throws an exception when the passed class contains a field whose type is a nested bean with type arguments:
{code:java}
class A<T> {
T value;
// value getter and setter
}
class B {
A<String> stringHolder;
// stringHolder getter and setter
}
Encoders.bean(B.class); // throws "SparkUnsupportedOperationException: [ENCODER_NOT_FOUND]..."{code}
It looks like this is a regression introduced with [[SPARK-42093][SQL] Move JavaTypeInference to AgnosticEncoders - apache/spark@1867200 (github.com) |https://github.com/apache/spark/commit/18672003513d5a4aa610b6b94dbbc15c33185d3a#diff-1191737b908340a2f4c22b71b1c40ebaa0da9d8b40c958089c346a3bda26943b]while getting rid of TypeToken, that somehow managed that case.
I'm going to submit a PR to re-enable this functionality.
> Encoders.bean does no longer support nested beans with type arguments
> ---------------------------------------------------------------------
>
> Key: SPARK-44634
> URL: https://issues.apache.org/jira/browse/SPARK-44634
> Project: Spark
> Issue Type: Bug
> Components: SQL
> Affects Versions: 3.4.1, 3.5.0, 4.0.0
> Reporter: Giambattista Bloisi
> Priority: Major
>
> Hi,
> while upgrading a project from spark 2.4.0 to 3.4.1 version, I have encountered the same problem described in [java - Encoders.bean attempts to check the validity of a return type considering its generic type and not its concrete class, with Spark 3.4.0 - Stack Overflow|https://stackoverflow.com/questions/76045255/encoders-bean-attempts-to-check-the-validity-of-a-return-type-considering-its-ge].
> Put it short, starting from Spark 3.4.x Encoders.bean throws an exception when the passed class contains a field whose type is a nested bean with type arguments:
>
> {code:java}
> class A<T> {
> T value;
> // value getter and setter
> }
> class B {
> A<String> stringHolder;
> // stringHolder getter and setter
> }
> Encoders.bean(B.class); // throws "SparkUnsupportedOperationException: [ENCODER_NOT_FOUND]..."{code}
>
>
> It looks like this is a regression introduced with [link SPARK-42093 SQL Move JavaTypeInference to AgnosticEncoders|https://github.com/apache/spark/commit/18672003513d5a4aa610b6b94dbbc15c33185d3#diff-1191737b908340a2f4c22b71b1c40ebaa0da9d8b40c958089c346a3bda26943b] while getting rid of TypeToken, that somehow managed that case.
> I'm going to submit a PR to re-enable this functionality.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org