You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@ignite.apache.org by "Alexey Zinoviev (Jira)" <ji...@apache.org> on 2020/10/06 10:46:00 UTC
[jira] [Updated] (IGNITE-13533) [ML] Tutorial examples runs more
than 300000ms
[ https://issues.apache.org/jira/browse/IGNITE-13533?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Alexey Zinoviev updated IGNITE-13533:
-------------------------------------
Description:
Test has been timed out [test=testExample, timeout=300000]
Seems like we have a race condition in Genetic Parallel Hyper-parameter tuning
{code:java}
[12:22:10] : [Step 4/5] Thread [name="test-runner-#31311%ml.TutorialStepByStepExampleSelfTest%", id=32007, state=RUNNABLE, blockCnt=1982, waitCnt=91727][12:22:10] : [Step 4/5] Thread [name="test-runner-#31311%ml.TutorialStepByStepExampleSelfTest%", id=32007, state=RUNNABLE, blockCnt=1982, waitCnt=91727][12:22:10] : [Step 4/5] at java.lang.System.identityHashCode(Native Method)[12:22:10] : [Step 4/5] at java.io.ObjectOutputStream$HandleTable.hash(ObjectOutputStream.java:2360)[12:22:10] : [Step 4/5] at java.io.ObjectOutputStream$HandleTable.lookup(ObjectOutputStream.java:2293)[12:22:10] : [Step 4/5] at java.io.ObjectOutputStream$ReplaceTable.lookup(ObjectOutputStream.java:2399)[12:22:10] : [Step 4/5] at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1113)[12:22:10] : [Step 4/5] at java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1548)[12:22:10] : [Step 4/5] at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1509)[12:22:10] : [Step 4/5] at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1432)[12:22:10] : [Step 4/5] at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1178)[12:22:10] : [Step 4/5] at java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:348)[12:22:10] : [Step 4/5] at o.a.i.marshaller.jdk.JdkMarshaller.marshal0(JdkMarshaller.java:97)[12:22:10] : [Step 4/5] at o.a.i.marshaller.jdk.JdkMarshaller.marshal0(JdkMarshaller.java:109)[12:22:10] : [Step 4/5] at o.a.i.marshaller.AbstractNodeNameAwareMarshaller.marshal(AbstractNodeNameAwareMarshaller.java:56)[12:22:10] : [Step 4/5] at o.a.i.i.util.IgniteUtils.marshal(IgniteUtils.java:10505)[12:22:10] : [Step 4/5] at o.a.i.i.processors.cache.GridCacheProcessor$7.applyx(GridCacheProcessor.java:4952)[12:22:10] : [Step 4/5] at o.a.i.i.processors.cache.GridCacheProcessor$7.applyx(GridCacheProcessor.java:4933)[12:22:10] : [Step 4/5] at o.a.i.i.processors.cache.GridCacheProcessor.withBinaryContext(GridCacheProcessor.java:4978)[12:22:10] : [Step 4/5] at o.a.i.i.processors.cache.GridCacheProcessor.cloneCheckSerializable(GridCacheProcessor.java:4933)[12:22:10] : [Step 4/5] at o.a.i.i.processors.cache.GridCacheProcessor.prepareCacheChangeRequest(GridCacheProcessor.java:5036)[12:22:10] : [Step 4/5] at o.a.i.i.processors.cache.GridCacheProcessor.lambda$dynamicStartCache$26(GridCacheProcessor.java:3472)[12:22:10] : [Step 4/5] at o.a.i.i.processors.cache.GridCacheProcessor$$Lambda$722/1638695311.apply(Unknown Source)[12:22:10] : [Step 4/5] at o.a.i.i.processors.cache.GridCacheProcessor.dynamicStartCache(GridCacheProcessor.java:3503)[12:22:10] : [Step 4/5] at o.a.i.i.processors.cache.GridCacheProcessor.dynamicStartCache(GridCacheProcessor.java:3408)[12:22:10] : [Step 4/5] at o.a.i.i.IgniteKernal.createCache(IgniteKernal.java:3191)[12:22:10] : [Step 4/5] at o.a.i.ml.dataset.impl.cache.CacheBasedDatasetBuilder.build(CacheBasedDatasetBuilder.java:151)[12:22:10] : [Step 4/5] at o.a.i.ml.dataset.impl.cache.CacheBasedDatasetBuilder.build(CacheBasedDatasetBuilder.java:43)[12:22:10] : [Step 4/5] at o.a.i.ml.selection.scoring.evaluator.Evaluator.evaluate(Evaluator.java:429)[12:22:10] : [Step 4/5] at o.a.i.ml.selection.cv.AbstractCrossValidation.score(AbstractCrossValidation.java:350)[12:22:10] : [Step 4/5] at o.a.i.ml.selection.cv.CrossValidation.scoreOnIgnite(CrossValidation.java:79)[12:22:10] : [Step 4/5] at o.a.i.ml.selection.cv.CrossValidation.scoreByFolds(CrossValidation.java:53)[12:22:10] : [Step 4/5] at o.a.i.ml.selection.cv.AbstractCrossValidation.calculateScoresForFixedParamSet(AbstractCrossValidation.java:294)[12:22:10] : [Step 4/5] at o.a.i.ml.selection.cv.AbstractCrossValidation.lambda$scoreEvolutionAlgorithmSearchHyperparameterOptimization$0(AbstractCrossValidation.java:142)[12:22:10] : [Step 4/5] at o.a.i.ml.selection.cv.AbstractCrossValidation$$Lambda$1731/1372309000.apply(Unknown Source)[12:22:10] : [Step 4/5] at o.a.i.ml.util.genetic.Population.calculateFitnessForChromosome(Population.java:58)[12:22:10] : [Step 4/5] at o.a.i.ml.util.genetic.GeneticAlgorithm.run(GeneticAlgorithm.java:118)[12:22:10] : [Step 4/5] at o.a.i.ml.selection.cv.AbstractCrossValidation.scoreEvolutionAlgorithmSearchHyperparameterOptimization(AbstractCrossValidation.java:180)[12:22:10] : [Step 4/5] at o.a.i.ml.selection.cv.AbstractCrossValidation.tuneHyperParameters(AbstractCrossValidation.java:119)[12:22:10] : [Step 4/5] at o.a.i.examples.ml.tutorial.hyperparametertuning.Step_16_Genetic_Programming_Search.main(Step_16_Genetic_Programming_Search.java:146)[12:22:10] : [Step 4/5] at o.a.i.examples.ml.tutorial.TutorialStepByStepExample.main(TutorialStepByStepExample.java:53)[12:22:10] : [Step 4/5] at o.a.i.examples.ml.TutorialStepByStepExampleSelfTest.testExample(TutorialStepByStepExampleSelfTest.java)[12:22:10] : [Step 4/5] at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)[12:22:10] : [Step 4/5] at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)[12:22:10] : [Step 4/5] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)[12:22:10] : [Step 4/5] at java.lang.reflect.Method.invoke(Method.java:498)[12:22:10] : [Step 4/5] at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)[12:22:10] : [Step 4/5] at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)[12:22:10] : [Step 4/5] at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)[12:22:10] : [Step 4/5] at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)[12:22:10] : [Step 4/5] at o.a.i.testframework.junits.GridAbstractTest$7.run(GridAbstractTest.java:2373)[12:22:10] : [Step 4/5] at java.lang.Thread.run(Thread.java:748)[12:22:10] : [Step 4/5] [12:22:10] : [Step 4/5] Locked synchronizers:[12:22:10] : [Step 4/5] java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync@693089ab[12:22:10] : [Step 4/5] java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync@6d3f3553[12:22:10] : [Step 4/5] java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync@43b97de1[12:22:10] : [Step 4/5] java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync@9d91005[12:22:10] : [Step 4/5] java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync@64c9bd44[12:22:10] : [Step 4/5] java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync@70c88923[12:22:10] : [Step 4/5] java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync@7896b4ae[12:22:10] : [Step 4/5] java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync@390eca81[12:22:10] : [Step 4/5] java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync@10047379[12:22:10] : [Step 4/5] java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync@243161cb
{code}
was:
Test has been timed out [test=testExample, timeout=300000]
Seems like we have a race condition in Genetic Parallel Hyper-parameter tuning
> [ML] Tutorial examples runs more than 300000ms
> ----------------------------------------------
>
> Key: IGNITE-13533
> URL: https://issues.apache.org/jira/browse/IGNITE-13533
> Project: Ignite
> Issue Type: Bug
> Reporter: Alexey Zinoviev
> Assignee: Alexey Zinoviev
> Priority: Major
>
> Test has been timed out [test=testExample, timeout=300000]
> Seems like we have a race condition in Genetic Parallel Hyper-parameter tuning
>
> {code:java}
> [12:22:10] : [Step 4/5] Thread [name="test-runner-#31311%ml.TutorialStepByStepExampleSelfTest%", id=32007, state=RUNNABLE, blockCnt=1982, waitCnt=91727][12:22:10] : [Step 4/5] Thread [name="test-runner-#31311%ml.TutorialStepByStepExampleSelfTest%", id=32007, state=RUNNABLE, blockCnt=1982, waitCnt=91727][12:22:10] : [Step 4/5] at java.lang.System.identityHashCode(Native Method)[12:22:10] : [Step 4/5] at java.io.ObjectOutputStream$HandleTable.hash(ObjectOutputStream.java:2360)[12:22:10] : [Step 4/5] at java.io.ObjectOutputStream$HandleTable.lookup(ObjectOutputStream.java:2293)[12:22:10] : [Step 4/5] at java.io.ObjectOutputStream$ReplaceTable.lookup(ObjectOutputStream.java:2399)[12:22:10] : [Step 4/5] at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1113)[12:22:10] : [Step 4/5] at java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1548)[12:22:10] : [Step 4/5] at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1509)[12:22:10] : [Step 4/5] at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1432)[12:22:10] : [Step 4/5] at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1178)[12:22:10] : [Step 4/5] at java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:348)[12:22:10] : [Step 4/5] at o.a.i.marshaller.jdk.JdkMarshaller.marshal0(JdkMarshaller.java:97)[12:22:10] : [Step 4/5] at o.a.i.marshaller.jdk.JdkMarshaller.marshal0(JdkMarshaller.java:109)[12:22:10] : [Step 4/5] at o.a.i.marshaller.AbstractNodeNameAwareMarshaller.marshal(AbstractNodeNameAwareMarshaller.java:56)[12:22:10] : [Step 4/5] at o.a.i.i.util.IgniteUtils.marshal(IgniteUtils.java:10505)[12:22:10] : [Step 4/5] at o.a.i.i.processors.cache.GridCacheProcessor$7.applyx(GridCacheProcessor.java:4952)[12:22:10] : [Step 4/5] at o.a.i.i.processors.cache.GridCacheProcessor$7.applyx(GridCacheProcessor.java:4933)[12:22:10] : [Step 4/5] at o.a.i.i.processors.cache.GridCacheProcessor.withBinaryContext(GridCacheProcessor.java:4978)[12:22:10] : [Step 4/5] at o.a.i.i.processors.cache.GridCacheProcessor.cloneCheckSerializable(GridCacheProcessor.java:4933)[12:22:10] : [Step 4/5] at o.a.i.i.processors.cache.GridCacheProcessor.prepareCacheChangeRequest(GridCacheProcessor.java:5036)[12:22:10] : [Step 4/5] at o.a.i.i.processors.cache.GridCacheProcessor.lambda$dynamicStartCache$26(GridCacheProcessor.java:3472)[12:22:10] : [Step 4/5] at o.a.i.i.processors.cache.GridCacheProcessor$$Lambda$722/1638695311.apply(Unknown Source)[12:22:10] : [Step 4/5] at o.a.i.i.processors.cache.GridCacheProcessor.dynamicStartCache(GridCacheProcessor.java:3503)[12:22:10] : [Step 4/5] at o.a.i.i.processors.cache.GridCacheProcessor.dynamicStartCache(GridCacheProcessor.java:3408)[12:22:10] : [Step 4/5] at o.a.i.i.IgniteKernal.createCache(IgniteKernal.java:3191)[12:22:10] : [Step 4/5] at o.a.i.ml.dataset.impl.cache.CacheBasedDatasetBuilder.build(CacheBasedDatasetBuilder.java:151)[12:22:10] : [Step 4/5] at o.a.i.ml.dataset.impl.cache.CacheBasedDatasetBuilder.build(CacheBasedDatasetBuilder.java:43)[12:22:10] : [Step 4/5] at o.a.i.ml.selection.scoring.evaluator.Evaluator.evaluate(Evaluator.java:429)[12:22:10] : [Step 4/5] at o.a.i.ml.selection.cv.AbstractCrossValidation.score(AbstractCrossValidation.java:350)[12:22:10] : [Step 4/5] at o.a.i.ml.selection.cv.CrossValidation.scoreOnIgnite(CrossValidation.java:79)[12:22:10] : [Step 4/5] at o.a.i.ml.selection.cv.CrossValidation.scoreByFolds(CrossValidation.java:53)[12:22:10] : [Step 4/5] at o.a.i.ml.selection.cv.AbstractCrossValidation.calculateScoresForFixedParamSet(AbstractCrossValidation.java:294)[12:22:10] : [Step 4/5] at o.a.i.ml.selection.cv.AbstractCrossValidation.lambda$scoreEvolutionAlgorithmSearchHyperparameterOptimization$0(AbstractCrossValidation.java:142)[12:22:10] : [Step 4/5] at o.a.i.ml.selection.cv.AbstractCrossValidation$$Lambda$1731/1372309000.apply(Unknown Source)[12:22:10] : [Step 4/5] at o.a.i.ml.util.genetic.Population.calculateFitnessForChromosome(Population.java:58)[12:22:10] : [Step 4/5] at o.a.i.ml.util.genetic.GeneticAlgorithm.run(GeneticAlgorithm.java:118)[12:22:10] : [Step 4/5] at o.a.i.ml.selection.cv.AbstractCrossValidation.scoreEvolutionAlgorithmSearchHyperparameterOptimization(AbstractCrossValidation.java:180)[12:22:10] : [Step 4/5] at o.a.i.ml.selection.cv.AbstractCrossValidation.tuneHyperParameters(AbstractCrossValidation.java:119)[12:22:10] : [Step 4/5] at o.a.i.examples.ml.tutorial.hyperparametertuning.Step_16_Genetic_Programming_Search.main(Step_16_Genetic_Programming_Search.java:146)[12:22:10] : [Step 4/5] at o.a.i.examples.ml.tutorial.TutorialStepByStepExample.main(TutorialStepByStepExample.java:53)[12:22:10] : [Step 4/5] at o.a.i.examples.ml.TutorialStepByStepExampleSelfTest.testExample(TutorialStepByStepExampleSelfTest.java)[12:22:10] : [Step 4/5] at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)[12:22:10] : [Step 4/5] at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)[12:22:10] : [Step 4/5] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)[12:22:10] : [Step 4/5] at java.lang.reflect.Method.invoke(Method.java:498)[12:22:10] : [Step 4/5] at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)[12:22:10] : [Step 4/5] at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)[12:22:10] : [Step 4/5] at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)[12:22:10] : [Step 4/5] at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)[12:22:10] : [Step 4/5] at o.a.i.testframework.junits.GridAbstractTest$7.run(GridAbstractTest.java:2373)[12:22:10] : [Step 4/5] at java.lang.Thread.run(Thread.java:748)[12:22:10] : [Step 4/5] [12:22:10] : [Step 4/5] Locked synchronizers:[12:22:10] : [Step 4/5] java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync@693089ab[12:22:10] : [Step 4/5] java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync@6d3f3553[12:22:10] : [Step 4/5] java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync@43b97de1[12:22:10] : [Step 4/5] java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync@9d91005[12:22:10] : [Step 4/5] java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync@64c9bd44[12:22:10] : [Step 4/5] java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync@70c88923[12:22:10] : [Step 4/5] java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync@7896b4ae[12:22:10] : [Step 4/5] java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync@390eca81[12:22:10] : [Step 4/5] java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync@10047379[12:22:10] : [Step 4/5] java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync@243161cb
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)