You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by Sameer Tilak <ss...@live.com> on 2014/08/27 20:42:28 UTC

Amplab: big-data-benchmark

Hi All,
I am planning to run amplab benchmark suite to evaluate the performance of our cluster. I looked at: https://amplab.cs.berkeley.edu/benchmark/ and it mentions about data avallability at:
s3n://big-data-benchmark/pavlo/[text|text-deflate|sequence|sequence-snappy]/[suffix]where /tiny/, /1node/ and /5nodes/ are options for suffix. However, I am not able to doanload these datasets directly. Here is what I see. I read that they can be used directly by doing : sc.textFile(s3:/....). However, I wanted to make sure that my understanding is correct. Here is what I see at http://s3.amazonaws.com/big-data-benchmark/
I do not see anything for sequence or text-deflate.
I see sequence-snappy dataset:
<Contents><Key>pavlo/sequence-snappy/5nodes/crawl/000738_0</Key><LastModified>2013-05-27T21:26:40.000Z</LastModified><ETag>"a978d18721d5a533d38a88f558461644"</ETag><Size>42958735</Size><StorageClass>STANDARD</StorageClass></Contents>
For text, I get the following error:
<Error><Code>NoSuchKey</Code><Message>The specified key does not exist.</Message><Key>pavlo/text/1node/crawl</Key><RequestId>166D239D38399526</RequestId><HostId>4Bg8BHomWqJ6BXOkx/3fQZhN5Uw1TtCn01uQzm+1qYffx2s/oPV+9sGoAWV2thCI</HostId></Error>

Please let me know if there is a way to readily download the dataset and view it. 		 	   		  

RE: Amplab: big-data-benchmark

Posted by Sameer Tilak <ss...@live.com>.
Hi Burak,Thanks,  I will then start benchmarking the cluster.

> Date: Wed, 27 Aug 2014 11:52:05 -0700
> From: byavuz@stanford.edu
> To: sstilak@live.com
> CC: user@spark.apache.org
> Subject: Re: Amplab: big-data-benchmark
> 
> Hi Sameer,
> 
> I've faced this issue before. They don't show up on http://s3.amazonaws.com/big-data-benchmark/. But you can directly use: 
> `sc.textFile("s3n://big-data-benchmark/pavlo/text/tiny/crawl")`
> The gotcha is that you also need to supply which dataset you want: crawl, uservisits, or rankings in lower case after the format and size you want them in.
> They should be there.
> 
> Best,
> Burak
> 
> ----- Original Message -----
> From: "Sameer Tilak" <ss...@live.com>
> To: user@spark.apache.org
> Sent: Wednesday, August 27, 2014 11:42:28 AM
> Subject: Amplab: big-data-benchmark
> 
> Hi All,
> I am planning to run amplab benchmark suite to evaluate the performance of our cluster. I looked at: https://amplab.cs.berkeley.edu/benchmark/ and it mentions about data avallability at:
> s3n://big-data-benchmark/pavlo/[text|text-deflate|sequence|sequence-snappy]/[suffix]where /tiny/, /1node/ and /5nodes/ are options for suffix. However, I am not able to doanload these datasets directly. Here is what I see. I read that they can be used directly by doing : sc.textFile(s3:/....). However, I wanted to make sure that my understanding is correct. Here is what I see at http://s3.amazonaws.com/big-data-benchmark/
> I do not see anything for sequence or text-deflate.
> I see sequence-snappy dataset:
> <Contents><Key>pavlo/sequence-snappy/5nodes/crawl/000738_0</Key><LastModified>2013-05-27T21:26:40.000Z</LastModified><ETag>"a978d18721d5a533d38a88f558461644"</ETag><Size>42958735</Size><StorageClass>STANDARD</StorageClass></Contents>
> For text, I get the following error:
> <Error><Code>NoSuchKey</Code><Message>The specified key does not exist.</Message><Key>pavlo/text/1node/crawl</Key><RequestId>166D239D38399526</RequestId><HostId>4Bg8BHomWqJ6BXOkx/3fQZhN5Uw1TtCn01uQzm+1qYffx2s/oPV+9sGoAWV2thCI</HostId></Error>
> 
> Please let me know if there is a way to readily download the dataset and view it. 		 	   		  
> 
> 
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
> For additional commands, e-mail: user-help@spark.apache.org
> 
 		 	   		  

Re: Amplab: big-data-benchmark

Posted by Burak Yavuz <by...@stanford.edu>.
Hi Sameer,

I've faced this issue before. They don't show up on http://s3.amazonaws.com/big-data-benchmark/. But you can directly use: 
`sc.textFile("s3n://big-data-benchmark/pavlo/text/tiny/crawl")`
The gotcha is that you also need to supply which dataset you want: crawl, uservisits, or rankings in lower case after the format and size you want them in.
They should be there.

Best,
Burak

----- Original Message -----
From: "Sameer Tilak" <ss...@live.com>
To: user@spark.apache.org
Sent: Wednesday, August 27, 2014 11:42:28 AM
Subject: Amplab: big-data-benchmark

Hi All,
I am planning to run amplab benchmark suite to evaluate the performance of our cluster. I looked at: https://amplab.cs.berkeley.edu/benchmark/ and it mentions about data avallability at:
s3n://big-data-benchmark/pavlo/[text|text-deflate|sequence|sequence-snappy]/[suffix]where /tiny/, /1node/ and /5nodes/ are options for suffix. However, I am not able to doanload these datasets directly. Here is what I see. I read that they can be used directly by doing : sc.textFile(s3:/....). However, I wanted to make sure that my understanding is correct. Here is what I see at http://s3.amazonaws.com/big-data-benchmark/
I do not see anything for sequence or text-deflate.
I see sequence-snappy dataset:
<Contents><Key>pavlo/sequence-snappy/5nodes/crawl/000738_0</Key><LastModified>2013-05-27T21:26:40.000Z</LastModified><ETag>"a978d18721d5a533d38a88f558461644"</ETag><Size>42958735</Size><StorageClass>STANDARD</StorageClass></Contents>
For text, I get the following error:
<Error><Code>NoSuchKey</Code><Message>The specified key does not exist.</Message><Key>pavlo/text/1node/crawl</Key><RequestId>166D239D38399526</RequestId><HostId>4Bg8BHomWqJ6BXOkx/3fQZhN5Uw1TtCn01uQzm+1qYffx2s/oPV+9sGoAWV2thCI</HostId></Error>

Please let me know if there is a way to readily download the dataset and view it. 		 	   		  


---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org