You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by Shuai Zheng <sz...@gmail.com> on 2015/03/16 19:46:57 UTC

sqlContext.parquetFile doesn't work with s3n in version 1.3.0

Hi All,

 

I just upgrade the system to use version 1.3.0, but then the
sqlContext.parquetFile doesn't work with s3n. I have test the same code with
1.2.1 and it works.

 

A simple test running in spark-shell:

 

val parquetFile = sqlContext.parquetFile("""s3n://****/test/2.parq """)

java.lang.IllegalArgumentException: Wrong FS: s3n://****/test/2.parq,
expected: file:/// <file:///\\> 

 

And same test work with spark-shell under 1.2.1

 

Regards,

 

Shuai

Re: sqlContext.parquetFile doesn't work with s3n in version 1.3.0

Posted by Michael Armbrust <mi...@databricks.com>.
We will be including this fix in Spark 1.3.1 which we hope to make in the
next week or so.

On Mon, Mar 16, 2015 at 12:01 PM, Shuai Zheng <sz...@gmail.com> wrote:

> I see, but this is really a… big issue. anyway for me to work around? I
> try to set the fs.default.name = s3n, but looks like it doesn’t work.
>
>
>
> I must upgrade to 1.3.0 because I face the package incompatible issue in
> 1.2.1, and if I must patch something, I rather go with latest version.
>
>
>
> Regards,
>
>
>
> Shuai
>
>
>
> *From:* Kelly, Jonathan [mailto:jonathak@amazon.com]
> *Sent:* Monday, March 16, 2015 2:54 PM
> *To:* Shuai Zheng; user@spark.apache.org
> *Subject:* Re: sqlContext.parquetFile doesn't work with s3n in version
> 1.3.0
>
>
>
> See https://issues.apache.org/jira/browse/SPARK-6351
>
>
>
> ~ Jonathan
>
>
>
> *From: *Shuai Zheng <sz...@gmail.com>
> *Date: *Monday, March 16, 2015 at 11:46 AM
> *To: *"user@spark.apache.org" <us...@spark.apache.org>
> *Subject: *sqlContext.parquetFile doesn't work with s3n in version 1.3.0
>
>
>
> Hi All,
>
>
>
> I just upgrade the system to use version 1.3.0, but then the
> sqlContext.parquetFile doesn’t work with s3n. I have test the same code
> with 1.2.1 and it works.
>
>
>
> A simple test running in spark-shell:
>
>
>
> val parquetFile = sqlContext.parquetFile("""s3n://****/test/2.parq """)
>
> java.lang.IllegalArgumentException: Wrong FS: s3n://****/test/2.parq,
> expected: file:///
>
>
>
> And same test work with spark-shell under 1.2.1
>
>
>
> Regards,
>
>
>
> Shuai
>

RE: sqlContext.parquetFile doesn't work with s3n in version 1.3.0

Posted by Shuai Zheng <sz...@gmail.com>.
I see, but this is really a. big issue. anyway for me to work around? I try
to set the fs.default.name = s3n, but looks like it doesn't work. 

 

I must upgrade to 1.3.0 because I face the package incompatible issue in
1.2.1, and if I must patch something, I rather go with latest version.

 

Regards,

 

Shuai

 

From: Kelly, Jonathan [mailto:jonathak@amazon.com] 
Sent: Monday, March 16, 2015 2:54 PM
To: Shuai Zheng; user@spark.apache.org
Subject: Re: sqlContext.parquetFile doesn't work with s3n in version 1.3.0

 

See https://issues.apache.org/jira/browse/SPARK-6351

 

~ Jonathan

 

From: Shuai Zheng <sz...@gmail.com>
Date: Monday, March 16, 2015 at 11:46 AM
To: "user@spark.apache.org" <us...@spark.apache.org>
Subject: sqlContext.parquetFile doesn't work with s3n in version 1.3.0

 

Hi All,

 

I just upgrade the system to use version 1.3.0, but then the
sqlContext.parquetFile doesn't work with s3n. I have test the same code with
1.2.1 and it works.

 

A simple test running in spark-shell:

 

val parquetFile = sqlContext.parquetFile("""s3n://****/test/2.parq """)

java.lang.IllegalArgumentException: Wrong FS: s3n://****/test/2.parq,
expected: file:/// <file:///\\> 

 

And same test work with spark-shell under 1.2.1

 

Regards,

 

Shuai

Re: sqlContext.parquetFile doesn't work with s3n in version 1.3.0

Posted by "Kelly, Jonathan" <jo...@amazon.com>.
See https://issues.apache.org/jira/browse/SPARK-6351

~ Jonathan

From: Shuai Zheng <sz...@gmail.com>>
Date: Monday, March 16, 2015 at 11:46 AM
To: "user@spark.apache.org<ma...@spark.apache.org>" <us...@spark.apache.org>>
Subject: sqlContext.parquetFile doesn't work with s3n in version 1.3.0

Hi All,

I just upgrade the system to use version 1.3.0, but then the sqlContext.parquetFile doesn’t work with s3n. I have test the same code with 1.2.1 and it works.

A simple test running in spark-shell:

val parquetFile = sqlContext.parquetFile("""s3n://****/test/2.parq """)
java.lang.IllegalArgumentException: Wrong FS: s3n://****/test/2.parq, expected: file:///<file:///\\>

And same test work with spark-shell under 1.2.1

Regards,

Shuai