You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@calcite.apache.org by "Abhishek Singh Chouhan (Jira)" <ji...@apache.org> on 2023/05/05 20:57:00 UTC
[jira] [Commented] (CALCITE-5690) ConcurrentModificationException during validation of table with DynamicRecordType
[ https://issues.apache.org/jira/browse/CALCITE-5690?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17720002#comment-17720002 ]
Abhishek Singh Chouhan commented on CALCITE-5690:
-------------------------------------------------
[~julianhyde] Any thoughts on what the right thing to do here is? We're caching DynamicRecordTypeImpl which is mutable. We could look into not caching this or making the fields list object in RelDataTypeHolder something like a CopyOnWriteArraylist
> ConcurrentModificationException during validation of table with DynamicRecordType
> ---------------------------------------------------------------------------------
>
> Key: CALCITE-5690
> URL: https://issues.apache.org/jira/browse/CALCITE-5690
> Project: Calcite
> Issue Type: Bug
> Affects Versions: 1.32.0
> Reporter: Abhishek Singh Chouhan
> Priority: Major
>
> When multiple threads are doing validation on a Table with DynamicRecordTypeImpl, we run into a ConcurrentModificationException. One of the instances where this happens is when two validations are happening in parallel
> Thread1 -
> SELECT DS_GET_QUANTILE(DS_QUANTILES_SKETCH(COL1), 0.9) FROM TABLE1 GROUP BY COL1, COL2, COL3
> Thread2 - SELECT DS_GET_QUANTILE(DS_QUANTILES_SKETCH(COL1), 0.9) FROM TABLE1 GROUP BY COL1, COL2
> While expanding the groupby in thread1 we intern the type at multiple places, eg. DelegatingScope#fullyQualify -> SqlValidatorScope#rowType -> RelDataTypeFactoryImpl#createTypeWithNullability -> canonize
> During the validation in thread1 we cache DynamicRecordTypeImpl with Col1 and Col2 as the fields.
> The race condition happens when thread2 gets a hold of this(abovementioned) cached DynamicRecordTypeImpl during its validation and is iterating over the fields eg during SqlValidatorImpl.validateGroupItem and thread 1 then goes and inserts Col3 field in the holder.
> {code:java}
> java.util.ConcurrentModificationException
> at java.base/java.util.ArrayList$Itr.checkForComodification(ArrayList.java:1043)
> at java.base/java.util.ArrayList$Itr.next(ArrayList.java:997)
> at org.apache.calcite.rel.type.RelDataTypeHolder.getFieldOrInsert(RelDataTypeHolder.java:56)
> at org.apache.calcite.rel.type.DynamicRecordTypeImpl.getField(DynamicRecordTypeImpl.java:59)
> at org.apache.calcite.sql.validate.SqlNameMatchers$BaseMatcher.field(SqlNameMatchers.java:126)
> at org.apache.calcite.sql.validate.SqlValidatorImpl$DeriveTypeVisitor.visit(SqlValidatorImpl.java:6455)
> at org.apache.calcite.sql.validate.SqlValidatorImpl$DeriveTypeVisitor.visit(SqlValidatorImpl.java:6360)
> at org.apache.calcite.sql.SqlIdentifier.accept(SqlIdentifier.java:324)
> at org.apache.calcite.sql.validate.SqlValidatorImpl.deriveTypeImpl(SqlValidatorImpl.java:1869)
> at org.apache.calcite.sql.validate.SqlValidatorImpl.deriveType(SqlValidatorImpl.java:1854)
> at org.apache.calcite.sql.validate.SqlValidatorImpl.validateGroupItem(SqlValidatorImpl.java:4393)
> at org.apache.calcite.sql.validate.SqlValidatorImpl.validateGroupClause(SqlValidatorImpl.java:4366)
> at org.apache.calcite.sql.validate.SqlValidatorImpl.validateSelect(SqlValidatorImpl.java:3701)
> at org.apache.calcite.sql.validate.SelectNamespace.validateImpl(SelectNamespace.java:64)
> at org.apache.calcite.sql.validate.AbstractNamespace.validate(AbstractNamespace.java:89)
> at org.apache.calcite.sql.validate.SqlValidatorImpl.validateNamespace(SqlValidatorImpl.java:1107)
> at org.apache.calcite.sql.validate.SqlValidatorImpl.validateQuery(SqlValidatorImpl.java:1078)
> at org.apache.calcite.sql.SqlSelect.validate(SqlSelect.java:248)
> at org.apache.calcite.sql.validate.SqlValidatorImpl.validateScopedExpression(SqlValidatorImpl.java:1053)
> at org.apache.calcite.sql.validate.SqlValidatorImpl.validate(SqlValidatorImpl.java:759) {code}
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)