You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by Joe Wass <jw...@crossref.org> on 2015/05/23 16:24:35 UTC

Is anyone using Amazon EC2? (second attempt!)

I used Spark on EC2 a while ago, but recent revisions seem to have broken
the functionality.

Is anyone actually using Spark on EC2 at the moment?

The bug in question is:

https://issues.apache.org/jira/browse/SPARK-5008

It makes it impossible to use persistent HDFS without a workround on each
slave node.

No-one seems to be interested in the bug, so I wonder if other people
aren't actually having this problem. If this is the case, any suggestions?

Joe

Re: Is anyone using Amazon EC2? (second attempt!)

Posted by Sanjay Subramanian <sa...@yahoo.com.INVALID>.
I use spark on EC2 but it's a CDH 5.3.3 distribution (starving developer version) installed thru Cloudera Manager. Spark is configured to run on Yarn. 

Regards
Sanjay

Sent from my iPhone

> On May 29, 2015, at 6:16 PM, roni <ro...@gmail.com> wrote:
> 
> Hi ,
> Any update on this? 
> I am not sure if the issue I am seeing is related ..
> I have 8 slaves and when I created the cluster I specified ebs volume with 100G.
> I see on Ec2 8 volumes created and each attached to the corresponding slave.
> But when I try to copy data on it , it complains that 
> /root/ephemeral-hdfs/bin/hadoop fs -cp /intersection hdfs://ec2-54-149-112-136.us-west-2.compute.amazonaws.com:9010/
> 
> 2015-05-28 23:40:35,447 WARN  hdfs.DFSClient (DFSOutputStream.java:run(562)) - DataStreamer Exception
> 
> org.apache.hadoop.ipc.RemoteException(java.io.IOException): File /intersection/kmer150/commonGoodKmers/_temporary/_attempt_201504010056_0004_m_000428_3948/part-00428._COPYING_ could only be replicated to 0 nodes instead of minReplication (=1).  There are 1 datanode(s) running and no node(s) are excluded in this operation.
> 
> 
> 
> It shows only 1 datanode , but for ephermal-hdfs it shows 8 datanodes.
> 
> Any thoughts?
> 
> Thanks
> 
> _R
> 
> 
>> On Sat, May 23, 2015 at 7:24 AM, Joe Wass <jw...@crossref.org> wrote:
>> I used Spark on EC2 a while ago, but recent revisions seem to have broken the functionality.
>> 
>> Is anyone actually using Spark on EC2 at the moment?
>> 
>> The bug in question is:
>> 
>> https://issues.apache.org/jira/browse/SPARK-5008
>> 
>> It makes it impossible to use persistent HDFS without a workround on each slave node.
>> 
>> No-one seems to be interested in the bug, so I wonder if other people aren't actually having this problem. If this is the case, any suggestions? 
>> 
>> Joe
>

Re: Is anyone using Amazon EC2? (second attempt!)

Posted by roni <ro...@gmail.com>.
Hi ,
Any update on this?
I am not sure if the issue I am seeing is related ..
I have 8 slaves and when I created the cluster I specified ebs volume with
100G.
I see on Ec2 8 volumes created and each attached to the corresponding slave.
But when I try to copy data on it , it complains that

/root/ephemeral-hdfs/bin/hadoop fs -cp /intersection hdfs://
ec2-54-149-112-136.us-west-2.compute.amazonaws.com:9010/

2015-05-28 23:40:35,447 WARN  hdfs.DFSClient
(DFSOutputStream.java:run(562)) - DataStreamer Exception

org.apache.hadoop.ipc.RemoteException(java.io.IOException): File
/intersection/kmer150/commonGoodKmers/_temporary/_attempt_201504010056_0004_m_000428_3948/part-00428._COPYING_
could only be replicated to 0 nodes instead of minReplication (=1).  There
are 1 datanode(s) running and no node(s) are excluded in this operation.


It shows only 1 datanode , but for ephermal-hdfs it shows 8 datanodes.

Any thoughts?

Thanks

_R

On Sat, May 23, 2015 at 7:24 AM, Joe Wass <jw...@crossref.org> wrote:

> I used Spark on EC2 a while ago, but recent revisions seem to have broken
> the functionality.
>
> Is anyone actually using Spark on EC2 at the moment?
>
> The bug in question is:
>
> https://issues.apache.org/jira/browse/SPARK-5008
>
> It makes it impossible to use persistent HDFS without a workround on each
> slave node.
>
> No-one seems to be interested in the bug, so I wonder if other people
> aren't actually having this problem. If this is the case, any suggestions?
>
> Joe
>