You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@calcite.apache.org by Josh Elser <jo...@gmail.com> on 2016/01/12 17:49:19 UTC

Seeming test deadlock between JdbcTest and JdbcFrontLinqBackTest

I'm seeing the occasional "deadlock" on some tests in calcite-core. The 
top of the stacktrace seems to get stuck in the following:

"pool-1-thread-1" prio=5 tid=0x00007fd50611e800 nid=0x5103 runnable 
[0x0000000115b81000]
    java.lang.Thread.State: RUNNABLE
	at java.util.HashMap.getEntry(HashMap.java:465)
	at java.util.HashMap.get(HashMap.java:417)
	at 
org.apache.calcite.rel.metadata.ReflectiveRelMetadataProvider.apply(ReflectiveRelMetadataProvider.java:289)
	at 
org.apache.calcite.rel.metadata.ReflectiveRelMetadataProvider.apply(ReflectiveRelMetadataProvider.java:278)
	at 
org.apache.calcite.rel.metadata.ChainedRelMetadataProvider.apply(ChainedRelMetadataProvider.java:63)
	at 
org.apache.calcite.rel.metadata.ChainedRelMetadataProvider.apply(ChainedRelMetadataProvider.java:63)
	at 
org.apache.calcite.rel.metadata.ChainedRelMetadataProvider.apply(ChainedRelMetadataProvider.java:63)
	at 
org.apache.calcite.rel.metadata.MetadataFactoryImpl$2.load(MetadataFactoryImpl.java:57)
	at 
org.apache.calcite.rel.metadata.MetadataFactoryImpl$2.load(MetadataFactoryImpl.java:54)
	at 
com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3599)
	at 
com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2379)
	at 
com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2342)
	- locked <0x00000007e1287590> (a 
com.google.common.cache.LocalCache$StrongEntry)
	at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2257)
	at com.google.common.cache.LocalCache.get(LocalCache.java:4000)
	at com.google.common.cache.LocalCache.getOrLoad(LocalCache.java:4004)
	at 
com.google.common.cache.LocalCache$LocalLoadingCache.get(LocalCache.java:4874)
	at 
org.apache.calcite.rel.metadata.MetadataFactoryImpl.query(MetadataFactoryImpl.java:71)
	at 
org.apache.calcite.rel.AbstractRelNode.metadata(AbstractRelNode.java:296)
	at 
org.apache.calcite.rel.metadata.RelMetadataQuery.getNonCumulativeCost(RelMetadataQuery.java:140)

or

"pool-1-thread-6" prio=5 tid=0x00007fd50684c000 nid=0x5d03 runnable 
[0x0000000116193000]
    java.lang.Thread.State: RUNNABLE
	at java.util.HashMap.put(HashMap.java:494)
	at 
org.apache.calcite.rel.metadata.ReflectiveRelMetadataProvider.apply(ReflectiveRelMetadataProvider.java:292)
	at 
org.apache.calcite.rel.metadata.ReflectiveRelMetadataProvider.apply(ReflectiveRelMetadataProvider.java:278)
	at 
org.apache.calcite.rel.metadata.ChainedRelMetadataProvider.apply(ChainedRelMetadataProvider.java:63)
	at 
org.apache.calcite.rel.metadata.ChainedRelMetadataProvider.apply(ChainedRelMetadataProvider.java:63)
	at 
org.apache.calcite.rel.metadata.MetadataFactoryImpl$2.load(MetadataFactoryImpl.java:57)
	at 
org.apache.calcite.rel.metadata.MetadataFactoryImpl$2.load(MetadataFactoryImpl.java:54)
	at 
com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3599)
	at 
com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2379)
	at 
com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2342)
	- locked <0x00000007a0fa41a0> (a 
com.google.common.cache.LocalCache$StrongEntry)
	at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2257)
	at com.google.common.cache.LocalCache.get(LocalCache.java:4000)
	at com.google.common.cache.LocalCache.getOrLoad(LocalCache.java:4004)
	at 
com.google.common.cache.LocalCache$LocalLoadingCache.get(LocalCache.java:4874)
	at 
org.apache.calcite.rel.metadata.MetadataFactoryImpl.query(MetadataFactoryImpl.java:71)
	at 
org.apache.calcite.rel.AbstractRelNode.metadata(AbstractRelNode.java:296)
	at 
org.apache.calcite.rel.metadata.RelMetadataQuery.getNonCumulativeCost(RelMetadataQuery.java:140)

Before I go digging in, does this ring any bells to anyone? It's 
currently hanging here for over 5 minutes, but there is still heap 
activity happening. Just took a dump with jvisualvm too. Will poke around.

- Josh

Re: Seeming test deadlock between JdbcTest and JdbcFrontLinqBackTest

Posted by Julian Hyde <jh...@apache.org>.
Oops, I meant to say https://issues.apache.org/jira/browse/CALCITE-1053 <https://issues.apache.org/jira/browse/CALCITE-1053>.

> On Jan 14, 2016, at 12:50 PM, Julian Hyde <jh...@apache.org> wrote:
> 
> Thanks for the fix, Vladimir.
> 
> FYI, Josh had logged a JIRA case, https://issues.apache.org/jira/browse/CALCITE-1054 <https://issues.apache.org/jira/browse/CALCITE-1054>. I changed your commit comment to reflect that.
> 
> Julian
> 
>> On Jan 13, 2016, at 12:37 AM, Vladimir Sitnikov <sitnikov.vladimir@gmail.com <ma...@gmail.com>> wrote:
>> 
>> I'll fixed the problem here:
>> https://github.com/apache/calcite/commit/fe91d46761f460927b21ef334fee5e686b54e397 <https://github.com/apache/calcite/commit/fe91d46761f460927b21ef334fee5e686b54e397>
>> 
>> Vladimir
>

Re: Seeming test deadlock between JdbcTest and JdbcFrontLinqBackTest

Posted by Julian Hyde <jh...@apache.org>.
Thanks for the fix, Vladimir.

FYI, Josh had logged a JIRA case, https://issues.apache.org/jira/browse/CALCITE-1054 <https://issues.apache.org/jira/browse/CALCITE-1054>. I changed your commit comment to reflect that.

Julian

> On Jan 13, 2016, at 12:37 AM, Vladimir Sitnikov <si...@gmail.com> wrote:
> 
> I'll fixed the problem here:
> https://github.com/apache/calcite/commit/fe91d46761f460927b21ef334fee5e686b54e397
> 
> Vladimir

Re: Seeming test deadlock between JdbcTest and JdbcFrontLinqBackTest

Posted by Vladimir Sitnikov <si...@gmail.com>.
I'll fixed the problem here:
https://github.com/apache/calcite/commit/fe91d46761f460927b21ef334fee5e686b54e397

Vladimir

Re: Seeming test deadlock between JdbcTest and JdbcFrontLinqBackTest

Posted by Josh Elser <jo...@gmail.com>.
Will do.

Julian Hyde wrote:
> I fixed a deadlock issue a few months back (CALCITE-874) but it's
> entirely possible I missed something, and I do see intermittent hangs
> in Travis-CI. Josh, can you please log a JIRA case with the stack?
> It's not critical for this release.
>
> On Tue, Jan 12, 2016 at 11:05 AM, Josh Elser<jo...@gmail.com>  wrote:
>> Thanks for the pointer, Vladimir. I haven't dug into this corner of the code
>> yet, so I wasn't really sure what the guarantees on concurrency were meant
>> to be.
>>
>> I plan to focus on the other issues from yesterday, but will try to swing
>> around to this after. Anyone else who has cycles/motivation should feel free
>> to look :)
>>
>>
>> Vladimir Sitnikov wrote:
>>> As far as I can see, the problem is "non-properly-synchronized"
>>> HashMap in ReflectiveRelMetadataProvider.
>>>
>>> The fix would probably be to using ConcurrentHashMap there, however it
>>> is better to doublecheck if there are more high-level data races
>>> there.
>>>
>>> Vladimir

Re: Seeming test deadlock between JdbcTest and JdbcFrontLinqBackTest

Posted by Julian Hyde <jh...@apache.org>.
I fixed a deadlock issue a few months back (CALCITE-874) but it's
entirely possible I missed something, and I do see intermittent hangs
in Travis-CI. Josh, can you please log a JIRA case with the stack?
It's not critical for this release.

On Tue, Jan 12, 2016 at 11:05 AM, Josh Elser <jo...@gmail.com> wrote:
> Thanks for the pointer, Vladimir. I haven't dug into this corner of the code
> yet, so I wasn't really sure what the guarantees on concurrency were meant
> to be.
>
> I plan to focus on the other issues from yesterday, but will try to swing
> around to this after. Anyone else who has cycles/motivation should feel free
> to look :)
>
>
> Vladimir Sitnikov wrote:
>>
>> As far as I can see, the problem is "non-properly-synchronized"
>> HashMap in ReflectiveRelMetadataProvider.
>>
>> The fix would probably be to using ConcurrentHashMap there, however it
>> is better to doublecheck if there are more high-level data races
>> there.
>>
>> Vladimir

Re: Seeming test deadlock between JdbcTest and JdbcFrontLinqBackTest

Posted by Josh Elser <jo...@gmail.com>.
Thanks for the pointer, Vladimir. I haven't dug into this corner of the 
code yet, so I wasn't really sure what the guarantees on concurrency 
were meant to be.

I plan to focus on the other issues from yesterday, but will try to 
swing around to this after. Anyone else who has cycles/motivation should 
feel free to look :)

Vladimir Sitnikov wrote:
> As far as I can see, the problem is "non-properly-synchronized"
> HashMap in ReflectiveRelMetadataProvider.
>
> The fix would probably be to using ConcurrentHashMap there, however it
> is better to doublecheck if there are more high-level data races
> there.
>
> Vladimir

Re: Seeming test deadlock between JdbcTest and JdbcFrontLinqBackTest

Posted by Vladimir Sitnikov <si...@gmail.com>.
As far as I can see, the problem is "non-properly-synchronized"
HashMap in ReflectiveRelMetadataProvider.

The fix would probably be to using ConcurrentHashMap there, however it
is better to doublecheck if there are more high-level data races
there.

Vladimir