You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@crunch.apache.org by "Surbhi Mungre (JIRA)" <ji...@apache.org> on 2015/01/29 22:23:35 UTC
[jira] [Created] (CRUNCH-494) Unable to union large number of
PCollections
Surbhi Mungre created CRUNCH-494:
------------------------------------
Summary: Unable to union large number of PCollections
Key: CRUNCH-494
URL: https://issues.apache.org/jira/browse/CRUNCH-494
Project: Crunch
Issue Type: Bug
Components: Core
Affects Versions: 0.8.3
Reporter: Surbhi Mungre
Assignee: Josh Wills
Priority: Minor
If you try to union large number of PCollections(~5K), then Crunch throws StackOverflowError exception.
{noformat}
java.lang.StackOverflowError
at com.google.common.collect.AbstractIndexedListIterator.<init>(AbstractIndexedListIterator.java:68)
at com.google.common.collect.AbstractIndexedListIterator.<init>(AbstractIndexedListIterator.java:54)
at com.google.common.collect.Iterators$12.<init>(Iterators.java:1072)
at com.google.common.collect.Iterators.forArray(Iterators.java:1072)
at com.google.common.collect.RegularImmutableList.iterator(RegularImmutableList.java:68)
at com.google.common.collect.RegularImmutableList.iterator(RegularImmutableList.java:31)
at org.apache.crunch.impl.dist.collect.PCollectionImpl.getTargetDependencies(PCollectionImpl.java:291)
at org.apache.crunch.impl.dist.collect.PCollectionImpl.getTargetDependencies(PCollectionImpl.java:292)
at org.apache.crunch.impl.dist.collect.PCollectionImpl.getTargetDependencies(PCollectionImpl.java:292)
at org.apache.crunch.impl.dist.collect.PCollectionImpl.getTargetDependencies(PCollectionImpl.java:292)
at org.apache.crunch.impl.dist.collect.PCollectionImpl.getTargetDependencies(PCollectionImpl.java:292)
at org.apache.crunch.impl.dist.collect.PCollectionImpl.getTargetDependencies(PCollectionImpl.java:292)
at org.apache.crunch.impl.dist.collect.PCollectionImpl.getTargetDependencies(PCollectionImpl.java:292)
at org.apache.crunch.impl.dist.collect.PCollectionImpl.getTargetDependencies(PCollectionImpl.java:292)
at org.apache.crunch.impl.dist.collect.PCollectionImpl.getTargetDependencies(PCollectionImpl.java:292)
at org.apache.crunch.impl.dist.collect.PCollectionImpl.getTargetDependencies(PCollectionImpl.java:292)
at org.apache.crunch.impl.dist.collect.PCollectionImpl.getTargetDependencies(PCollectionImpl.java:292)
at org.apache.crunch.impl.dist.collect.PCollectionImpl.getTargetDependencies(PCollectionImpl.java:292)
at org.apache.crunch.impl.dist.collect.PCollectionImpl.getTargetDependencies(PCollectionImpl.java:292)
at org.apache.crunch.impl.dist.collect.PCollectionImpl.getTargetDependencies(PCollectionImpl.java:292)
at org.apache.crunch.impl.dist.collect.PCollectionImpl.getTargetDependencies(PCollectionImpl.java:292)
at org.apache.crunch.impl.dist.collect.PCollectionImpl.getTargetDependencies(PCollectionImpl.java:292)
at org.apache.crunch.impl.dist.collect.PCollectionImpl.getTargetDependencies(PCollectionImpl.java:292)
at org.apache.crunch.impl.dist.collect.PCollectionImpl.getTargetDependencies(PCollectionImpl.java:292)
at org.apache.crunch.impl.dist.collect.PCollectionImpl.getTargetDependencies(PCollectionImpl.java:292)
at org.apache.crunch.impl.dist.collect.PCollectionImpl.getTargetDependencies(PCollectionImpl.java:292)
at org.apache.crunch.impl.dist.collect.PCollectionImpl.getTargetDependencies(PCollectionImpl.java:292)
at org.apache.crunch.impl.dist.collect.PCollectionImpl.getTargetDependencies(PCollectionImpl.java:292)
at org.apache.crunch.impl.dist.collect.PCollectionImpl.getTargetDependencies(PCollectionImpl.java:292)
at org.apache.crunch.impl.dist.collect.PCollectionImpl.getTargetDependencies(PCollectionImpl.java:292)
at org.apache.crunch.impl.dist.collect.PCollectionImpl.getTargetDependencies(PCollectionImpl.java:292)
at org.apache.crunch.impl.dist.collect.PCollectionImpl.getTargetDependencies(PCollectionImpl.java:292)
{noformat}
Here is a simple test which can reproduce the issue.
https://gist.github.com/anonymous/22f08511604341d0ffda
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)