You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@spark.apache.org by Raymond Honderdors <Ra...@sizmek.com> on 2016/03/31 11:00:51 UTC

Question Create External table location S3

Hi,

I pulled the latest version

git pull git://github.com/apache/spark.git

Compiled:
mvn -Pyarn -Phadoop-2.6 -Dhadoop.version=2.6.0 -Phive -Phive-thriftserver -DskipTests clean package


now I am getting the following error:
Error: org.apache.spark.sql.execution.QueryExecutionException: FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask. MetaException(message:Got exception: java.io.IOException No FileSystem for scheme: s3n) (state=,code=0)

Did anyone else experience this?


Raymond Honderdors
Team Lead Analytics BI
Business Intelligence Developer
raymond.honderdors@sizmek.com<ma...@sizmek.com>
T +972.7325.3569
Herzliya


[Read More]<http://feeds.feedburner.com/~r/sizmek-blog/~6/1>

[http://www.sizmek.com/Sizmek.png]<http://www.sizmek.com/>

Re: Question Create External table location S3

Posted by Raymond Honderdors <Ra...@sizmek.com>.
Thanks for the insites
Ill try to add it

Sent from Outlook Mobile<https://aka.ms/blhgte>



On Thu, Mar 31, 2016 at 4:39 AM -0700, "Steve Loughran" <st...@hortonworks.com>> wrote:


On 31 Mar 2016, at 10:00, Raymond Honderdors <Ra...@sizmek.com>> wrote:

Hi,

I pulled the latest version

git pull git://github.com/apache/spark.git

Compiled:
mvn -Pyarn -Phadoop-2.6 -Dhadoop.version=2.6.0 -Phive -Phive-thriftserver -DskipTests clean package


now I am getting the following error:
Error: org.apache.spark.sql.execution.QueryExecutionException: FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask. MetaException(message:Got exception: java.io.IOException No FileSystem for scheme: s3n) (state=,code=0)

Did anyone else experience this?


Raymond Honderdors
Team Lead Analytics BI
Business Intelligence Developer
raymond.honderdors@sizmek.com<ma...@sizmek.com>
T +972.7325.3569
Herzliya


[Read More]<http://feeds.feedburner.com/~r/sizmek-blog/~6/1>

[http://www.sizmek.com/Sizmek.png]<http://www.sizmek.com/>


You are going to need to add hadoop-aws.JAR to your classpath, along with amazon's aws-java-sdk v 1.7.4 on your CP

I am actually working on a PR to add the hadoop-aws, openstack and (hadoop 2.7+) azure JARs to spark-assembly, but you'll still need to add the relevant amazon SDK (which has proven brittle across versions):

https://github.com/apache/spark/pull/12004

It's not ready yet; once I've got the 2.7 profile working with tests for openstack and azure, *and documentation on use and testing* then you'll be able to play with.

Re: Question Create External table location S3

Posted by Steve Loughran <st...@hortonworks.com>.
On 31 Mar 2016, at 10:00, Raymond Honderdors <Ra...@sizmek.com>> wrote:

Hi,

I pulled the latest version

git pull git://github.com/apache/spark.git

Compiled:
mvn -Pyarn -Phadoop-2.6 -Dhadoop.version=2.6.0 -Phive -Phive-thriftserver -DskipTests clean package


now I am getting the following error:
Error: org.apache.spark.sql.execution.QueryExecutionException: FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask. MetaException(message:Got exception: java.io.IOException No FileSystem for scheme: s3n) (state=,code=0)

Did anyone else experience this?


Raymond Honderdors
Team Lead Analytics BI
Business Intelligence Developer
raymond.honderdors@sizmek.com<ma...@sizmek.com>
T +972.7325.3569
Herzliya


[Read More]<http://feeds.feedburner.com/~r/sizmek-blog/~6/1>

[http://www.sizmek.com/Sizmek.png]<http://www.sizmek.com/>


You are going to need to add hadoop-aws.JAR to your classpath, along with amazon's aws-java-sdk v 1.7.4 on your CP

I am actually working on a PR to add the hadoop-aws, openstack and (hadoop 2.7+) azure JARs to spark-assembly, but you'll still need to add the relevant amazon SDK (which has proven brittle across versions):

https://github.com/apache/spark/pull/12004

It's not ready yet; once I've got the 2.7 profile working with tests for openstack and azure, *and documentation on use and testing* then you'll be able to play with.