You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@groovy.apache.org by jwagenleitner <gi...@git.apache.org> on 2017/02/05 07:05:10 UTC
[GitHub] groovy pull request #489: GROOVY-8067: Possible deadlock when creating new C...
GitHub user jwagenleitner opened a pull request:
https://github.com/apache/groovy/pull/489
GROOVY-8067: Possible deadlock when creating new ClassInfo entries in the cache
As suggested in PR #484 removed the locking on the `ManagedLinkedList` by creating a new `ManagedConcurrentLinkedQueue`.
Also added a `stress` subproject for tests that employ many threads, need GC, or just in general try to break things and take a long time. These require a special property to be set in order to run, otherwise they will be skipped. I tried to work it out in the `performance` subproject, but that seems to be very specialized for the compiler tests. Open to suggestions on a better way to handle these types of tests.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/jwagenleitner/groovy groovy8067-mclq
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/groovy/pull/489.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #489
----
commit bb2464a919a3655f36707fa72fb30080c92a7288
Author: John Wagenleitner <jw...@apache.org>
Date: 2017-02-05T06:13:26Z
GROOVY-8067: Possible deadlock when creating new ClassInfo entries in the cache
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] groovy pull request #489: GROOVY-8067: Possible deadlock when creating new C...
Posted by jwagenleitner <gi...@git.apache.org>.
Github user jwagenleitner commented on a diff in the pull request:
https://github.com/apache/groovy/pull/489#discussion_r100162813
--- Diff: src/main/org/codehaus/groovy/util/ManagedConcurrentLinkedQueue.java ---
@@ -0,0 +1,187 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.codehaus.groovy.util;
+
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.Iterator;
+import java.util.List;
+import java.util.NoSuchElementException;
+import java.util.concurrent.ConcurrentLinkedQueue;
+
+/**
+ * A queue that stores the values wrapped in a Reference, the type of which is
+ * determined by the provided {@link ReferenceBundle}. Values stored in this queue
+ * that are put on the {@code ReferenceQueue} will be removed from the list when
+ * reference processing for the {@code ReferenceQueue} is done.
+ *
+ * This queue is backed by a {@link ConcurrentLinkedQueue} and is thread safe. The
+ * iterator will only return non-null values (reachable) and is based on the
+ * "weakly consistent" iterator of the underlying {@link ConcurrentLinkedQueue}.
+ *
+ * @param <T> the type of values to store
+ */
+public class ManagedConcurrentLinkedQueue<T> implements Iterable<T> {
+
+ private final ReferenceBundle bundle;
+ private final ConcurrentLinkedQueue<Element<T>> queue;
+
+ /**
+ * Creates an empty ManagedConcurrentLinkedQueue that will use the given
+ * {@code ReferenceBundle} to store values as the given Reference
+ * type.
+ *
+ * @param bundle used to create the appropriate Reference type
+ * for the values stored
+ */
+ public ManagedConcurrentLinkedQueue(ReferenceBundle bundle) {
+ this.bundle = bundle;
+ this.queue = new ConcurrentLinkedQueue<Element<T>>();
+ }
+
+ /**
+ * Adds the specified value to the queue.
+ *
+ * @param value the value to add
+ */
+ public void add(T value) {
+ Element<T> e = new Element<T>(value);
+ queue.offer(e);
+ }
+
+ /**
+ * Returns {@code true} if this queue contains no elements.
+ * <p>
+ * This method does not check the elements to verify they contain
+ * non-null reference values.
+ */
+ public boolean isEmpty() {
+ return queue.isEmpty();
+ }
+
+ /**
+ * Returns an array containing all values from this queue in the sequence they
+ * were added.
+ *
+ * @param tArray the array to populate if big enough, else a new array with
+ * the same runtime type
+ * @return an array containing all non-null values in this queue
+ */
+ public T[] toArray(T[] tArray) {
+ return values().toArray(tArray);
+ }
+
+ /**
+ * Returns an unmodifiable list containing all values from this queue in the
+ * sequence they were added.
+ */
+ public List<T> values() {
+ Iterator<T> itr = iterator();
+ if (!itr.hasNext()) {
+ return Collections.emptyList();
+ }
+ List<T> result = new ArrayList<T>(100);
+ result.add(itr.next());
+ while (itr.hasNext()) {
+ result.add(itr.next());
+ }
+ return Collections.unmodifiableList(result);
+ }
+
+ /**
+ * Returns an iterator over all non-null values in this queue. The values should be
+ * returned in the order they were added.
+ */
+ @Override
+ public Iterator<T> iterator() {
+ return new Iter(queue.iterator());
+ }
+
+ private final class Element<V> extends ManagedReference<V> {
+
+ Element(V value) {
+ super(bundle, value);
+ }
+
+ @Override
+ public void finalizeReference() {
+ queue.remove(this);
+ super.finalizeReference();
+ }
+
+ }
+
+ private final class Iter implements Iterator<T> {
--- End diff --
think this can be made a static nested class
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] groovy pull request #489: GROOVY-8067: Possible deadlock when creating new C...
Posted by jwagenleitner <gi...@git.apache.org>.
Github user jwagenleitner commented on a diff in the pull request:
https://github.com/apache/groovy/pull/489#discussion_r99482851
--- Diff: src/main/org/codehaus/groovy/reflection/ClassInfo.java ---
@@ -186,30 +185,20 @@ public void setStrongMetaClass(MetaClass answer) {
MetaClass strongRef = strongMetaClass;
if (strongRef instanceof ExpandoMetaClass) {
- ((ExpandoMetaClass)strongRef).inRegistry = false;
- synchronized(modifiedExpandos){
- for (Iterator<ClassInfo> it = modifiedExpandos.iterator(); it.hasNext(); ) {
- ClassInfo info = it.next();
- if(info == this){
- it.remove();
- }
+ ((ExpandoMetaClass)strongRef).inRegistry = false;
+ for (Iterator<ClassInfo> itr = modifiedExpandos.iterator(); itr.hasNext(); ) {
+ ClassInfo info = itr.next();
+ if(info == this) {
+ itr.remove();
+ }
}
- }
}
strongMetaClass = answer;
if (answer instanceof ExpandoMetaClass) {
- ((ExpandoMetaClass)answer).inRegistry = true;
- synchronized(modifiedExpandos){
- for (Iterator<ClassInfo> it = modifiedExpandos.iterator(); it.hasNext(); ) {
- ClassInfo info = it.next();
- if(info == this){
- it.remove();
- }
- }
- modifiedExpandos.add(this);
- }
--- End diff --
Removed because it appeared to be a duplicate of the logic above. Only if the previous previous `strongMetaClass` value was an expando would it have been added to the map, so the iteration/removal above should have taken care to remove this `ClassInfo` from the `modifiedExpandos` and there's no reason to iterate again.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] groovy pull request #489: GROOVY-8067: Possible deadlock when creating new C...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/groovy/pull/489
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] groovy pull request #489: GROOVY-8067: Possible deadlock when creating new C...
Posted by jwagenleitner <gi...@git.apache.org>.
Github user jwagenleitner commented on a diff in the pull request:
https://github.com/apache/groovy/pull/489#discussion_r100162935
--- Diff: subprojects/stress/src/test/java/org/codehaus/groovy/reflection/ClassInfoDeadlockStressTest.java ---
@@ -0,0 +1,140 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.codehaus.groovy.reflection;
+
+import groovy.lang.GroovyClassLoader;
+
+import java.util.concurrent.CountDownLatch;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicInteger;
+
+import org.apache.groovy.stress.util.GCUtils;
+import org.junit.Test;
+import static org.junit.Assert.*;
+
+/**
+ * Tests for deadlocks in the ClassInfo caching.
+ *
+ */
+public class ClassInfoDeadlockStressTest {
+
+ private static final int DEADLOCK_TRIES = 8;
+ private static final int THREAD_COUNT = 8;
+
+ private final CountDownLatch startLatch = new CountDownLatch(1);
+ private final CountDownLatch completeLatch = new CountDownLatch(THREAD_COUNT);
+ private final GroovyClassLoader gcl = new GroovyClassLoader();
+ private final AtomicInteger counter = new AtomicInteger();
+
+ /**
+ * We first generate a large number of ClassInfo instances for classes
+ * that are no longer reachable. Then queue up threads to all request
+ * ClassInfo instances for new classes simultaneously to ensure that
+ * clearing the old references wont deadlock the creation of new
+ * instances.
+ * <p>
+ * GROOVY-8067
+ */
+ @Test
+ public void testDeadlock() throws Exception {
+ for (int i = 1; i <= DEADLOCK_TRIES; i++) {
+ System.out.println("Test Number: " + i);
+ generateGarbage();
+ GCUtils.gc();
+ attemptDeadlock(null);
+ }
+ }
+
+ @Test
+ public void testRequestsForSameClassInfo() throws Exception {
+ Class<?> newClass = createRandomClass();
+ for (int i = 1; i <= DEADLOCK_TRIES; i++) {
+ System.out.println("Test Number: " + i);
+ generateGarbage();
+ GCUtils.gc();
+ attemptDeadlock(newClass);
+ }
+ ClassInfo newClassInfo = ClassInfo.getClassInfo(newClass);
+ for (ClassInfo ci : ClassInfo.getAllClassInfo()) {
+ if (ci.getTheClass() == newClass && ci != newClassInfo) {
+ fail("Found multiple ClassInfo instances for class");
+ }
+ }
+ }
+
+ private void attemptDeadlock(final Class<?> cls) throws Exception {
+ for (int i = 0; i < THREAD_COUNT; i++) {
+ Runnable runnable = new Runnable() {
+ @Override
+ public void run() {
+ Class<?> newClass = (cls == null) ? createRandomClass() : cls;
+ try {
+ startLatch.await();
+ } catch (InterruptedException ie) {
--- End diff --
can use `ThreadUtils.awaite()`
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---