You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Sungwon (Jira)" <ji...@apache.org> on 2021/09/15 15:08:00 UTC
[jira] [Updated] (SPARK-36766) Spark SQL DDL does not recognize
fs.s3.impl implied filesystem in LOCATION tag
[ https://issues.apache.org/jira/browse/SPARK-36766?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Sungwon updated SPARK-36766:
----------------------------
Description:
CREATE TABLE with the LOCATION tag via SPARK SQL DDL does not seem to recognize the fileSystem implied by setting fs.s3.impl.
The environment setting is set to:
|spark.hadoop.fs.s3.impl|org.apache.hadoop.fs.s3a.S3AFileSystem|
The following spark.sql ddl raises an exception:
spark.sql("""CREATE TABLE local.db.table (
col1 string,
col2 string,
)
LOCATION 's3://bucket/prefix'
""")
Exception:
Py4JJavaError: An error occurred while calling o86.sql. : org.apache.iceberg.exceptions.CommitStateUnknownException: Got exception: org.apache.hadoop.fs.UnsupportedFileSystemException No FileSystem for scheme "s3"
This is inconsistent with other spark functions such as spark.read.parquet("s3://bucket/prefix") which succeeds with the same configuration
was:
CREATE TABLE with the LOCATION tag via SPARK SQL DDL does not seem to recognize the fileSystem implied by setting fs.s3.impl.
The environment setting is set to:
|spark.hadoop.fs.s3.impl|org.apache.hadoop.fs.s3a.S3AFileSystem|
The following spark.sql ddl raises an exception:
spark.sql("""CREATE TABLE loca.db.table (
col1 string,
col2 string,
)
USING iceberg
LOCATION 's3://bucket/prefix'
""")
Exception:
Py4JJavaError: An error occurred while calling o86.sql. : org.apache.iceberg.exceptions.CommitStateUnknownException: Got exception: org.apache.hadoop.fs.UnsupportedFileSystemException No FileSystem for scheme "s3"
This is inconsistent with other spark functions such as spark.read.parquet("s3://bucket/prefix") which succeeds with the same configuration
> Spark SQL DDL does not recognize fs.s3.impl implied filesystem in LOCATION tag
> ------------------------------------------------------------------------------
>
> Key: SPARK-36766
> URL: https://issues.apache.org/jira/browse/SPARK-36766
> Project: Spark
> Issue Type: Bug
> Components: SQL
> Affects Versions: 3.0.2
> Reporter: Sungwon
> Priority: Major
>
> CREATE TABLE with the LOCATION tag via SPARK SQL DDL does not seem to recognize the fileSystem implied by setting fs.s3.impl.
> The environment setting is set to:
> |spark.hadoop.fs.s3.impl|org.apache.hadoop.fs.s3a.S3AFileSystem|
> The following spark.sql ddl raises an exception:
> spark.sql("""CREATE TABLE local.db.table (
> col1 string,
> col2 string,
> )
> LOCATION 's3://bucket/prefix'
> """)
> Exception:
> Py4JJavaError: An error occurred while calling o86.sql. : org.apache.iceberg.exceptions.CommitStateUnknownException: Got exception: org.apache.hadoop.fs.UnsupportedFileSystemException No FileSystem for scheme "s3"
> This is inconsistent with other spark functions such as spark.read.parquet("s3://bucket/prefix") which succeeds with the same configuration
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org