You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@flink.apache.org by Nathan Yu <Nu...@twosigma.com> on 2021/08/13 00:38:32 UTC
Bug with PojoSerializer? java.lang.IllegalArgumentException: Can not
set final double field Event.rating to java.lang.Integer
* Using local environment: StreamExecutionEnvironment.createLocalEnvironment()
* Event is a POJO class, with int, double, enum, and String fields
* Unfortunately it's hard for me to reproduce in a small example, as it seems to occur after 10e6+ events.
* Using flink-core-1.12.4
Stack:
Caused by: java.lang.IllegalArgumentException: Can not set final double field com.twosigma.research.options.optticks.core.types.Event.askPrice to java.lang.Integer
at java.base/jdk.internal.reflect.UnsafeFieldAccessorImpl.throwSetIllegalArgumentException(UnsafeFieldAccessorImpl.java:167)
at java.base/jdk.internal.reflect.UnsafeFieldAccessorImpl.throwSetIllegalArgumentException(UnsafeFieldAccessorImpl.java:171)
at java.base/jdk.internal.reflect.UnsafeFieldAccessorImpl.ensureObj(UnsafeFieldAccessorImpl.java:58)
at java.base/jdk.internal.reflect.UnsafeQualifiedDoubleFieldAccessorImpl.set(UnsafeQualifiedDoubleFieldAccessorImpl.java:77)
at java.base/java.lang.reflect.Field.set(Field.java:780)
at org.apache.flink.api.java.typeutils.runtime.PojoSerializer.initializeFields(PojoSerializer.java:205)
at org.apache.flink.api.java.typeutils.runtime.PojoSerializer.deserialize(PojoSerializer.java:388)
at org.apache.flink.api.java.typeutils.runtime.PojoSerializer.deserialize(PojoSerializer.java:409)
at org.apache.flink.streaming.runtime.streamrecord.StreamElementSerializer.deserialize(StreamElementSerializer.java:191)
at org.apache.flink.streaming.runtime.streamrecord.StreamElementSerializer.deserialize(StreamElementSerializer.java:46)
at org.apache.flink.runtime.plugable.NonReusingDeserializationDelegate.read(NonReusingDeserializationDelegate.java:53)
Re: Bug with PojoSerializer? java.lang.IllegalArgumentException: Can
not set final double field Event.rating to java.lang.Integer
Posted by JING ZHANG <be...@gmail.com>.
Hi Yu,
The type of field
`com.twosigma.research.options.optticks.core.types.Event.askPrice` in the
original POJO class is Double, right?
If it's right, then next step we should find out why source.readUTF() is
`java.lang.Integer` instead of Double.
Nathan Yu <Nu...@twosigma.com> 于2021年8月13日周五 下午9:05写道：
> When the exception is thrown in PojoSerializer for my Event class,
>
> yoinks
>
> String subclassName = source.readUTF(); //
> subclassName is “java.lang.Integer”
>
> try {
>
> actualSubclass = Class.forName(subclassName, true, cl);
>
> } catch (ClassNotFoundException e) {
>
> throw new RuntimeException("Cannot instantiate class.", e);
>
> }
>
> subclassSerializer = getSubclassSerializer(actualSubclass);
>
> target = (T)
> subclassSerializer.createInstance(); // target is an
> Integer instead of my Event class??
>
> // also initialize fields for which the subclass serializer is
> not responsible
>
> initializeFields(target);
>
>
>
>
>
>
>
> *From:* Nathan Yu <Nu...@twosigma.com>
> *Sent:* Friday, August 13, 2021 8:49 AM
> *To:* JING ZHANG <be...@gmail.com>; Schwalbe Matthias <
> Matthias.Schwalbe@viseca.ch>
> *Cc:* user@flink.apache.org
> *Subject:* RE: Bug with PojoSerializer?
> java.lang.IllegalArgumentException: Can not set final double field
> Event.rating to java.lang.Integer
>
>
>
> Does flink provide any hooks for objects before/after they are
> serialized/deserialized?
>
RE: Bug with PojoSerializer? java.lang.IllegalArgumentException: Can
not set final double field Event.rating to java.lang.Integer
Posted by Nathan Yu <Nu...@twosigma.com>.
When the exception is thrown in PojoSerializer for my Event class,
yoinks
String subclassName = source.readUTF(); // subclassName is “java.lang.Integer”
try {
actualSubclass = Class.forName(subclassName, true, cl);
} catch (ClassNotFoundException e) {
throw new RuntimeException("Cannot instantiate class.", e);
}
subclassSerializer = getSubclassSerializer(actualSubclass);
target = (T) subclassSerializer.createInstance(); // target is an Integer instead of my Event class??
// also initialize fields for which the subclass serializer is not responsible
initializeFields(target);
From: Nathan Yu <Nu...@twosigma.com>
Sent: Friday, August 13, 2021 8:49 AM
To: JING ZHANG <be...@gmail.com>; Schwalbe Matthias <Ma...@viseca.ch>
Cc: user@flink.apache.org
Subject: RE: Bug with PojoSerializer? java.lang.IllegalArgumentException: Can not set final double field Event.rating to java.lang.Integer
Does flink provide any hooks for objects before/after they are serialized/deserialized?
RE: Bug with PojoSerializer? java.lang.IllegalArgumentException: Can
not set final double field Event.rating to java.lang.Integer
Posted by Nathan Yu <Nu...@twosigma.com>.
Does flink provide any hooks for objects before/after they are serialized/deserialized?
Re: Bug with PojoSerializer? java.lang.IllegalArgumentException: Can
not set final double field Event.rating to java.lang.Integer
Posted by JING ZHANG <be...@gmail.com>.
Hi Yu,
Thias provides a nice method to debug the issue. Big +1.
Please try the way, feel free get back for further discussion.
Best,
JING ZHANG
Schwalbe Matthias <Ma...@viseca.ch> 于2021年8月13日周五 下午3:22写道：
> Good Morning Nathan,
>
>
>
> The exception stack does not give enough information yet to come to a
> solution, the way I would continue is this:
>
> - Given that you run in a local environment probably means that you
> could run your job in a debugger and
> - Place an exception break point for java.lang.IllegalArgumentException
> - Once after the 10^6 events you trap into the problem walk up the
> call stack to
> org.apache.flink.api.java.typeutils.runtime.PojoSerializer#initializeFields
> and
> - Try to find out why this PojoSerializer instance has some int
> serializer instead of a double serializer in the respective
> field-serializer index:
>
>
>
> protected void initializeFields(T t) {
> for (int i = 0; i < numFields; i++) {
> if (fields[i] != null) {
> try {
> fields[i].set(t, fieldSerializers[i].createInstance()); //it
> happens all here
> } catch (IllegalAccessException e) {
> throw new RuntimeException("Cannot initialize fields.",
> e);
> }
> }
> }
> }
>
>
>
> I hope you come closer to a solution if you poke a little around.
>
>
>
> Feel free to get back to me for further clarifications.
>
>
>
> Thias
>
>
>
>
>
> *From:* Nathan Yu <Nu...@twosigma.com>
> *Sent:* Freitag, 13. August 2021 05:27
> *To:* JING ZHANG <be...@gmail.com>
> *Cc:* user@flink.apache.org
> *Subject:* RE: Bug with PojoSerializer?
> java.lang.IllegalArgumentException: Can not set final double field
> Event.rating to java.lang.Integer
>
>
>
> Yea, it goes through many events from this input before the exception is
> thrown. I don’t know how the input schema can change though, as the input
> is always producing objects from the same class.
>
>
>
>
>
> *From:* JING ZHANG <be...@gmail.com>
> *Sent:* Thursday, August 12, 2021 10:22 PM
> *To:* Nathan Yu <Nu...@twosigma.com>
> *Cc:* user@flink.apache.org
> *Subject:* Re: Bug with PojoSerializer?
> java.lang.IllegalArgumentException: Can not set final double field
> Event.rating to java.lang.Integer
>
>
>
> Hi Yu,
>
> The exception is thrown after processing some input data instead of at the
> beginning of the input, right?
>
> Is there any possible that input schema has updated?
>
>
>
> Nathan Yu <Nu...@twosigma.com> 于2021年8月13日周五 上午8:38写道：
>
> · Using local environment:
> StreamExecutionEnvironment.createLocalEnvironment()
>
> · Event is a POJO class, with int, double, enum, and String fields
>
> · Unfortunately it’s hard for me to reproduce in a small example,
> as it seems to occur after 10e6+ events.
>
> · Using flink-core-1.12.4
>
>
>
> Stack:
>
> Caused by: java.lang.IllegalArgumentException: Can not set final double
> field com.twosigma.research.options.optticks.core.types.Event.askPrice to
> java.lang.Integer
>
> at
> java.base/jdk.internal.reflect.UnsafeFieldAccessorImpl.throwSetIllegalArgumentException(UnsafeFieldAccessorImpl.java:167)
>
> at
> java.base/jdk.internal.reflect.UnsafeFieldAccessorImpl.throwSetIllegalArgumentException(UnsafeFieldAccessorImpl.java:171)
>
> at
> java.base/jdk.internal.reflect.UnsafeFieldAccessorImpl.ensureObj(UnsafeFieldAccessorImpl.java:58)
>
> at
> java.base/jdk.internal.reflect.UnsafeQualifiedDoubleFieldAccessorImpl.set(UnsafeQualifiedDoubleFieldAccessorImpl.java:77)
>
> at java.base/java.lang.reflect.Field.set(Field.java:780)
>
> at
> org.apache.flink.api.java.typeutils.runtime.PojoSerializer.initializeFields(PojoSerializer.java:205)
>
> at
> org.apache.flink.api.java.typeutils.runtime.PojoSerializer.deserialize(PojoSerializer.java:388)
>
> at
> org.apache.flink.api.java.typeutils.runtime.PojoSerializer.deserialize(PojoSerializer.java:409)
>
> at
> org.apache.flink.streaming.runtime.streamrecord.StreamElementSerializer.deserialize(StreamElementSerializer.java:191)
>
> at
> org.apache.flink.streaming.runtime.streamrecord.StreamElementSerializer.deserialize(StreamElementSerializer.java:46)
>
> at
> org.apache.flink.runtime.plugable.NonReusingDeserializationDelegate.read(NonReusingDeserializationDelegate.java:53)
>
> Diese Nachricht ist ausschliesslich für den Adressaten bestimmt und
> beinhaltet unter Umständen vertrauliche Mitteilungen. Da die
> Vertraulichkeit von e-Mail-Nachrichten nicht gewährleistet werden kann,
> übernehmen wir keine Haftung für die Gewährung der Vertraulichkeit und
> Unversehrtheit dieser Mitteilung. Bei irrtümlicher Zustellung bitten wir
> Sie um Benachrichtigung per e-Mail und um Löschung dieser Nachricht sowie
> eventueller Anhänge. Jegliche unberechtigte Verwendung oder Verbreitung
> dieser Informationen ist streng verboten.
>
> This message is intended only for the named recipient and may contain
> confidential or privileged information. As the confidentiality of email
> communication cannot be guaranteed, we do not accept any responsibility for
> the confidentiality and the intactness of this message. If you have
> received it in error, please advise the sender by return e-mail and delete
> this message and any attachments. Any unauthorised use or dissemination of
> this information is strictly prohibited.
>
RE: Bug with PojoSerializer? java.lang.IllegalArgumentException: Can
not set final double field Event.rating to java.lang.Integer
Posted by Schwalbe Matthias <Ma...@viseca.ch>.
Good Morning Nathan,
The exception stack does not give enough information yet to come to a solution, the way I would continue is this:
* Given that you run in a local environment probably means that you could run your job in a debugger and
* Place an exception break point for java.lang.IllegalArgumentException
* Once after the 10^6 events you trap into the problem walk up the call stack to org.apache.flink.api.java.typeutils.runtime.PojoSerializer#initializeFields and
* Try to find out why this PojoSerializer instance has some int serializer instead of a double serializer in the respective field-serializer index:
protected void initializeFields(T t) {
for (int i = 0; i < numFields; i++) {
if (fields[i] != null) {
try {
fields[i].set(t, fieldSerializers[i].createInstance()); //it happens all here
} catch (IllegalAccessException e) {
throw new RuntimeException("Cannot initialize fields.", e);
}
}
}
}
I hope you come closer to a solution if you poke a little around.
Feel free to get back to me for further clarifications.
Thias
From: Nathan Yu <Nu...@twosigma.com>
Sent: Freitag, 13. August 2021 05:27
To: JING ZHANG <be...@gmail.com>
Cc: user@flink.apache.org
Subject: RE: Bug with PojoSerializer? java.lang.IllegalArgumentException: Can not set final double field Event.rating to java.lang.Integer
Yea, it goes through many events from this input before the exception is thrown. I don’t know how the input schema can change though, as the input is always producing objects from the same class.
From: JING ZHANG <be...@gmail.com>>
Sent: Thursday, August 12, 2021 10:22 PM
To: Nathan Yu <Nu...@twosigma.com>>
Cc: user@flink.apache.org<ma...@flink.apache.org>
Subject: Re: Bug with PojoSerializer? java.lang.IllegalArgumentException: Can not set final double field Event.rating to java.lang.Integer
Hi Yu,
The exception is thrown after processing some input data instead of at the beginning of the input, right?
Is there any possible that input schema has updated?
Nathan Yu <Nu...@twosigma.com>> 于2021年8月13日周五 上午8:38写道：
• Using local environment: StreamExecutionEnvironment.createLocalEnvironment()
• Event is a POJO class, with int, double, enum, and String fields
• Unfortunately it’s hard for me to reproduce in a small example, as it seems to occur after 10e6+ events.
• Using flink-core-1.12.4
Stack:
Caused by: java.lang.IllegalArgumentException: Can not set final double field com.twosigma.research.options.optticks.core.types.Event.askPrice to java.lang.Integer
at java.base/jdk.internal.reflect.UnsafeFieldAccessorImpl.throwSetIllegalArgumentException(UnsafeFieldAccessorImpl.java:167)
at java.base/jdk.internal.reflect.UnsafeFieldAccessorImpl.throwSetIllegalArgumentException(UnsafeFieldAccessorImpl.java:171)
at java.base/jdk.internal.reflect.UnsafeFieldAccessorImpl.ensureObj(UnsafeFieldAccessorImpl.java:58)
at java.base/jdk.internal.reflect.UnsafeQualifiedDoubleFieldAccessorImpl.set(UnsafeQualifiedDoubleFieldAccessorImpl.java:77)
at java.base/java.lang.reflect.Field.set(Field.java:780)
at org.apache.flink.api.java.typeutils.runtime.PojoSerializer.initializeFields(PojoSerializer.java:205)
at org.apache.flink.api.java.typeutils.runtime.PojoSerializer.deserialize(PojoSerializer.java:388)
at org.apache.flink.api.java.typeutils.runtime.PojoSerializer.deserialize(PojoSerializer.java:409)
at org.apache.flink.streaming.runtime.streamrecord.StreamElementSerializer.deserialize(StreamElementSerializer.java:191)
at org.apache.flink.streaming.runtime.streamrecord.StreamElementSerializer.deserialize(StreamElementSerializer.java:46)
at org.apache.flink.runtime.plugable.NonReusingDeserializationDelegate.read(NonReusingDeserializationDelegate.java:53)
Diese Nachricht ist ausschliesslich für den Adressaten bestimmt und beinhaltet unter Umständen vertrauliche Mitteilungen. Da die Vertraulichkeit von e-Mail-Nachrichten nicht gewährleistet werden kann, übernehmen wir keine Haftung für die Gewährung der Vertraulichkeit und Unversehrtheit dieser Mitteilung. Bei irrtümlicher Zustellung bitten wir Sie um Benachrichtigung per e-Mail und um Löschung dieser Nachricht sowie eventueller Anhänge. Jegliche unberechtigte Verwendung oder Verbreitung dieser Informationen ist streng verboten.
This message is intended only for the named recipient and may contain confidential or privileged information. As the confidentiality of email communication cannot be guaranteed, we do not accept any responsibility for the confidentiality and the intactness of this message. If you have received it in error, please advise the sender by return e-mail and delete this message and any attachments. Any unauthorised use or dissemination of this information is strictly prohibited.
RE: Bug with PojoSerializer? java.lang.IllegalArgumentException: Can
not set final double field Event.rating to java.lang.Integer
Posted by Nathan Yu <Nu...@twosigma.com>.
Yea, it goes through many events from this input before the exception is thrown. I don’t know how the input schema can change though, as the input is always producing objects from the same class.
From: JING ZHANG <be...@gmail.com>
Sent: Thursday, August 12, 2021 10:22 PM
To: Nathan Yu <Nu...@twosigma.com>
Cc: user@flink.apache.org
Subject: Re: Bug with PojoSerializer? java.lang.IllegalArgumentException: Can not set final double field Event.rating to java.lang.Integer
Hi Yu,
The exception is thrown after processing some input data instead of at the beginning of the input, right?
Is there any possible that input schema has updated?
Nathan Yu <Nu...@twosigma.com>> 于2021年8月13日周五 上午8:38写道：
• Using local environment: StreamExecutionEnvironment.createLocalEnvironment()
• Event is a POJO class, with int, double, enum, and String fields
• Unfortunately it’s hard for me to reproduce in a small example, as it seems to occur after 10e6+ events.
• Using flink-core-1.12.4
Stack:
Caused by: java.lang.IllegalArgumentException: Can not set final double field com.twosigma.research.options.optticks.core.types.Event.askPrice to java.lang.Integer
at java.base/jdk.internal.reflect.UnsafeFieldAccessorImpl.throwSetIllegalArgumentException(UnsafeFieldAccessorImpl.java:167)
at java.base/jdk.internal.reflect.UnsafeFieldAccessorImpl.throwSetIllegalArgumentException(UnsafeFieldAccessorImpl.java:171)
at java.base/jdk.internal.reflect.UnsafeFieldAccessorImpl.ensureObj(UnsafeFieldAccessorImpl.java:58)
at java.base/jdk.internal.reflect.UnsafeQualifiedDoubleFieldAccessorImpl.set(UnsafeQualifiedDoubleFieldAccessorImpl.java:77)
at java.base/java.lang.reflect.Field.set(Field.java:780)
at org.apache.flink.api.java.typeutils.runtime.PojoSerializer.initializeFields(PojoSerializer.java:205)
at org.apache.flink.api.java.typeutils.runtime.PojoSerializer.deserialize(PojoSerializer.java:388)
at org.apache.flink.api.java.typeutils.runtime.PojoSerializer.deserialize(PojoSerializer.java:409)
at org.apache.flink.streaming.runtime.streamrecord.StreamElementSerializer.deserialize(StreamElementSerializer.java:191)
at org.apache.flink.streaming.runtime.streamrecord.StreamElementSerializer.deserialize(StreamElementSerializer.java:46)
at org.apache.flink.runtime.plugable.NonReusingDeserializationDelegate.read(NonReusingDeserializationDelegate.java:53)
Re: Bug with PojoSerializer? java.lang.IllegalArgumentException: Can
not set final double field Event.rating to java.lang.Integer
Posted by JING ZHANG <be...@gmail.com>.
Hi Yu,
The exception is thrown after processing some input data instead of at the
beginning of the input, right?
Is there any possible that input schema has updated?
Nathan Yu <Nu...@twosigma.com> 于2021年8月13日周五 上午8:38写道：
> · Using local environment:
> StreamExecutionEnvironment.createLocalEnvironment()
>
> · Event is a POJO class, with int, double, enum, and String fields
>
> · Unfortunately it’s hard for me to reproduce in a small example,
> as it seems to occur after 10e6+ events.
>
> · Using flink-core-1.12.4
>
>
>
> Stack:
>
> Caused by: java.lang.IllegalArgumentException: Can not set final double
> field com.twosigma.research.options.optticks.core.types.Event.askPrice to
> java.lang.Integer
>
> at
> java.base/jdk.internal.reflect.UnsafeFieldAccessorImpl.throwSetIllegalArgumentException(UnsafeFieldAccessorImpl.java:167)
>
> at
> java.base/jdk.internal.reflect.UnsafeFieldAccessorImpl.throwSetIllegalArgumentException(UnsafeFieldAccessorImpl.java:171)
>
> at
> java.base/jdk.internal.reflect.UnsafeFieldAccessorImpl.ensureObj(UnsafeFieldAccessorImpl.java:58)
>
> at
> java.base/jdk.internal.reflect.UnsafeQualifiedDoubleFieldAccessorImpl.set(UnsafeQualifiedDoubleFieldAccessorImpl.java:77)
>
> at java.base/java.lang.reflect.Field.set(Field.java:780)
>
> at
> org.apache.flink.api.java.typeutils.runtime.PojoSerializer.initializeFields(PojoSerializer.java:205)
>
> at
> org.apache.flink.api.java.typeutils.runtime.PojoSerializer.deserialize(PojoSerializer.java:388)
>
> at
> org.apache.flink.api.java.typeutils.runtime.PojoSerializer.deserialize(PojoSerializer.java:409)
>
> at
> org.apache.flink.streaming.runtime.streamrecord.StreamElementSerializer.deserialize(StreamElementSerializer.java:191)
>
> at
> org.apache.flink.streaming.runtime.streamrecord.StreamElementSerializer.deserialize(StreamElementSerializer.java:46)
>
> at
> org.apache.flink.runtime.plugable.NonReusingDeserializationDelegate.read(NonReusingDeserializationDelegate.java:53)
>