You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@avro.apache.org by "ASF GitHub Bot (Jira)" <ji...@apache.org> on 2022/06/13 08:11:00 UTC
[jira] [Work logged] (AVRO-3531) GenericDatumReader in multithread lead to infinite loop cause misused of IdentityHashMap
[ https://issues.apache.org/jira/browse/AVRO-3531?focusedWorklogId=780735&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-780735 ]
ASF GitHub Bot logged work on AVRO-3531:
----------------------------------------
Author: ASF GitHub Bot
Created on: 13/Jun/22 08:10
Start Date: 13/Jun/22 08:10
Worklog Time Spent: 10m
Work Description: clesaec opened a new pull request, #1719:
URL: https://github.com/apache/avro/pull/1719
Make sure you have checked _all_ steps below.
### Jira
- [x] My PR addresses the following [Avro Jira 3531](https://issues.apache.org/jira/browse/AVRO-3531) issues
### Tests
- [x] My PR adds the following unit tests: TestGenericDatumReader
Issue Time Tracking
-------------------
Worklog Id: (was: 780735)
Remaining Estimate: 0h
Time Spent: 10m
> GenericDatumReader in multithread lead to infinite loop cause misused of IdentityHashMap
> ----------------------------------------------------------------------------------------
>
> Key: AVRO-3531
> URL: https://issues.apache.org/jira/browse/AVRO-3531
> Project: Apache Avro
> Issue Type: Bug
> Components: java
> Affects Versions: 1.11.0
> Reporter: tansion
> Priority: Critical
> Time Spent: 10m
> Remaining Estimate: 0h
>
> Hi,
> I am working on a java project that uses Kafka with Avro serialization/deserialization in an messaging platform.
> In production enrionment, we meet a serious issue on the deserialization processs. The GenericDatumReader process some how get into a infinite loop status, and it is happened accationally.
> When the issue happens, The thread stack is like this:
>
> {code:java}
> "DmqFixedRateConsumer-Thread-17" #453 daemon prio=5 os_prio=0 tid=0x00007f2ae1832800 nid=0xef49 runnable [0x00007f2a743fc000]
> java.lang.Thread.State: RUNNABLE
> at java.util.IdentityHashMap.get(IdentityHashMap.java:337)
> at org.apache.avro.generic.GenericDatumReader.getStringClass(GenericDatumReader.java:503)
> at org.apache.avro.generic.GenericDatumReader.readString(GenericDatumReader.java:454)
> at org.apache.avro.generic.GenericDatumReader.readWithoutConversion(GenericDatumReader.java:191)
> at org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:160)
> at org.apache.avro.generic.GenericDatumReader.readWithoutConversion(GenericDatumReader.java:187)
> at org.apache.avro.reflect.ReflectDatumReader.readField(ReflectDatumReader.java:291)
> at org.apache.avro.generic.GenericDatumReader.readRecord(GenericDatumReader.java:247)
> at org.apache.avro.specific.SpecificDatumReader.readRecord(SpecificDatumReader.java:123)
> at org.apache.avro.generic.GenericDatumReader.readWithoutConversion(GenericDatumReader.java:179)
> at org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:160)
> at org.apache.avro.generic.GenericDatumReader.readWithoutConversion(GenericDatumReader.java:187)
> at org.apache.avro.reflect.ReflectDatumReader.readField(ReflectDatumReader.java:291)
> at org.apache.avro.generic.GenericDatumReader.readRecord(GenericDatumReader.java:247)
> at org.apache.avro.specific.SpecificDatumReader.readRecord(SpecificDatumReader.java:123)
> at org.apache.avro.generic.GenericDatumReader.readWithoutConversion(GenericDatumReader.java:179)
> at org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:160)
> at org.apache.avro.generic.GenericDatumReader.readWithoutConversion(GenericDatumReader.java:187)
> at org.apache.avro.reflect.ReflectDatumReader.readField(ReflectDatumReader.java:291)
> at org.apache.avro.generic.GenericDatumReader.readRecord(GenericDatumReader.java:247)
> at org.apache.avro.specific.SpecificDatumReader.readRecord(SpecificDatumReader.java:123)
> at org.apache.avro.generic.GenericDatumReader.readWithoutConversion(GenericDatumReader.java:179)
> at org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:160)
> at org.apache.avro.generic.GenericDatumReader.readWithoutConversion(GenericDatumReader.java:187)
> at org.apache.avro.reflect.ReflectDatumReader.readField(ReflectDatumReader.java:291)
> at org.apache.avro.generic.GenericDatumReader.readRecord(GenericDatumReader.java:247)
> at org.apache.avro.specific.SpecificDatumReader.readRecord(SpecificDatumReader.java:123)
> at org.apache.avro.generic.GenericDatumReader.readWithoutConversion(GenericDatumReader.java:179)
> at org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:160)
> at org.apache.avro.generic.GenericDatumReader.readWithoutConversion(GenericDatumReader.java:187)
> at org.apache.avro.reflect.ReflectDatumReader.readField(ReflectDatumReader.java:291)
> at org.apache.avro.generic.GenericDatumReader.readRecord(GenericDatumReader.java:247)
> at org.apache.avro.specific.SpecificDatumReader.readRecord(SpecificDatumReader.java:123)
> at org.apache.avro.generic.GenericDatumReader.readWithoutConversion(GenericDatumReader.java:179)
> at org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:160)
> at org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:153)
> at com.xxx.xxx.xxx.xxx.xxx.XXX.deserialize(XXX.java:252)
> at com.xxx.xxx.xxx.xxx.xxx.ZZZ.deserialize(ZZZ.java:216)
> at com.xxx.xxx.xxx.xxx.xxx.SSS.processMessage(SSS.java:152)
> at com.xxx.xxx.xxx.xxx.xxx.SSS.loopProcess(SSS.java:127)
> at com.xxx.xxx.xxx.xxx.xxx.SSS$$Lambda$172/367082698.run(Unknown Source)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748) {code}
> We create 30 threads, and all the threads are the same as above! They all get stuck in the IdentityHashMap.get() method.
>
> Accroding to this mail [1.7.6 Slow Deserialization|https://www.mail-archive.com/user@avro.apache.org/msg02902.html], the Reader is thread-safe, But actually, it seems not.
> Why?
> org.apache.avro.generic.GenericDatumReader#getStringClass
>
> {code:java}
> /**
> * Called to read strings. Subclasses may override to use a different string
> * representation. By default, this calls {@link #readString(Object,Decoder)}.
> */
> protected Object readString(Object old, Schema expected, Decoder in) throws IOException {
> Class stringClass = getStringClass(expected);
> if (stringClass == String.class) {
> return in.readString();
> }
> if (stringClass == CharSequence.class) {
> return readString(old, in);
> }
> return newInstanceFromString(stringClass, in.readString());
> }
> private Map<Schema, Class> stringClassCache = new IdentityHashMap<>();
> private Class getStringClass(Schema s) {
> Class c = stringClassCache.get(s);
> if (c == null) {
> c = findStringClass(s);
> stringClassCache.put(s, c);
> }
> return c;
> }
> {code}
> The IdentityHashMap is not thread-safe, which is addressed by javadoc clearly! Like Hashmap infinite loop issue in multithread using, same issue happen to IdentityHashMap,too.
> My question is: Can the class GenericDatumReader fix this issue and act like real thread-safe? Or we need to avoid use the single instance of GenericDatumReader in multithread?
> Thanks a lot,
> Xtsong.
>
--
This message was sent by Atlassian Jira
(v8.20.7#820007)