You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Marc Le Bihan (Jira)" <ji...@apache.org> on 2023/11/12 10:56:00 UTC
[jira] [Comment Edited] (SPARK-45311) Encoder fails on many "NoSuchElementException: None.get" since 3.4.x, search for an encoder for a generic type, and since 3.5.x isn't "an expression encoder"
[ https://issues.apache.org/jira/browse/SPARK-45311?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17785275#comment-17785275 ]
Marc Le Bihan edited comment on SPARK-45311 at 11/12/23 10:55 AM:
------------------------------------------------------------------
The problem *java.util.NoSuchElementException: None.get* happens in `{color:#0033b3} {color}{color:#000000}JavaBeanEncoder{color}({color:#000000}tag{color}, {color:#000000}fields{color})` case of
`deserializerFor` method in `ScalaReflexion`
(sorry for the indentation that I had to change due to a bad copy-paste)
{code:java}
case JavaBeanEncoder(tag, fields) =>
val setters = fields.map { f =>
val newTypePath = walkedTypePath.recordField(
f.enc.clsTag.runtimeClass.getName,
f.name)
val setter = expressionWithNullSafety(
deserializerFor(
f.enc,
addToPath(path, f.name, f.enc.dataType, newTypePath),
newTypePath),
nullable = f.nullable,
newTypePath)
f.writeMethod.get -> setter {color}}
{code}
`f.writeMethod` is valued with `None` if it happens that the method observed, a `getSomething()` or `isSomething()` doesn't have a correspondant setter method `setSomething(...)`.
I cannot tell if it's the new wished behavior for Spark after the `3.4.x` or if it's a regression.
But until `3.3.x`, it was possible to have class having `isSomething()` methods, in order to do some checkings, without the need of having a setter.
—
The workaround is to rename a method not having a setter to `something()` only, without `get` or `is` prefix.
Then Spark will skip it.
was (Author: mlebihan):
The problem *java.util.NoSuchElementException: None.get* happens in `{color:#0033b3} {color}{color:#000000}JavaBeanEncoder{color}({color:#000000}tag{color}, {color:#000000}fields{color})` case of
`deserializerFor` method in `ScalaReflexion`
(sorry for the indentation that I had to change due to a bad copy-paste)
```scala
{color:#0033b3}case {color}{color:#000000}JavaBeanEncoder{color}({color:#000000}tag{color}, {color:#000000}fields{color}) =>
{color:#0033b3}val {color}{color:#000000}setters {color}= {color:#000000}fields{color}.map { f =>
{color:#0033b3}val {color}{color:#000000}newTypePath {color}= walkedTypePath.recordField(
f.enc.clsTag.runtimeClass.getName,
f.name)
{color:#0033b3} val {color}{color:#000000}setter {color}= expressionWithNullSafety(
deserializerFor(
f.enc,
addToPath(path, f.name, f.enc.dataType, {color:#000000}newTypePath{color}),
{color:#000000}newTypePath{color}),
nullable = f.nullable,
{color:#000000}newTypePath{color})
f.writeMethod.get -> {color:#000000}setter{color} \{color}}
```
`f.writeMethod` is valued with `None` if it happens that the method observed, a `getSomething()` or `isSomething()` doesn't have a correspondant setter method `setSomething(...)`.
I cannot tell if it's the new wished behavior for Spark after the `3.4.x` or if it's a regression.
But until `3.3.x`, it was possible to have class having `isSomething()` methods, in order to do some checkings, without the need of having a setter.
—
The workaround is to rename a method not having a setter to `something()` only, without `get` or `is` prefix.
Then Spark will skip it.
> Encoder fails on many "NoSuchElementException: None.get" since 3.4.x, search for an encoder for a generic type, and since 3.5.x isn't "an expression encoder"
> -------------------------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: SPARK-45311
> URL: https://issues.apache.org/jira/browse/SPARK-45311
> Project: Spark
> Issue Type: Bug
> Components: Spark Core
> Affects Versions: 3.4.0, 3.4.1, 3.5.0
> Environment: Debian 12
> Java 17
> Underlying Spring-Boot 2.7.14
> Reporter: Marc Le Bihan
> Priority: Major
>
> If you find it convenient, you might clone the [https://gitlab.com/territoirevif/minimal-tests-spark-issue] project (that does many operations around cities, local authorities and accounting with open data) where I've extracted from my work what's necessary to make a set of 35 tests that run correctly with Spark 3.3.x, and show the troubles encountered with 3.4.x and 3.5.x.
>
> It is working well with Spark 3.2.x, 3.3.x. But as soon as I selec{*}t Spark 3.4.x{*}, where the encoder seems to have deeply changed, the encoder fails with two problems:
>
> *1)* It throws *java.util.NoSuchElementException: None.get* messages everywhere.
> Asking over the Internet, I wasn't alone facing this problem. Reading it, you'll see that I've attempted a debug but my Scala skills are low.
> [https://stackoverflow.com/questions/76036349/encoders-bean-doesnt-work-anymore-on-a-java-pojo-with-spark-3-4-0]
> {color:#172b4d}by the way, if possible, the encoder and decoder functions should forward a parameter as soon as the name of the field being handled is known, and then all the long of their process, so that when the encoder is at any point where it has to throw an exception, it knows the field it is handling in its specific call and can send a message like:{color}
> {color:#00875a}_java.util.NoSuchElementException: None.get when encoding [the method or field it was targeting]_{color}
>
> *2)* *Not found an encoder of the type RS to Spark SQL internal representation.* Consider to change the input type to one of supported at (...)
> Or : Not found an encoder of the type *OMI_ID* to Spark SQL internal representation (...)
>
> where *RS* and *OMI_ID* are generic types.
> This is strange.
> [https://stackoverflow.com/questions/76045255/encoders-bean-attempts-to-check-the-validity-of-a-return-type-considering-its-ge]
>
> *3)* When I switch to the *Spark 3.5.0* version, the same problems remain, but another add itself to the list:
> "{*}Only expression encoders are supported for now{*}" on what was accepted and working before.
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org