[jira] [Created] (SPARK-28092) Spark cannot load files with COLON(:) char if not specified full path

["Ladislav Jech (JIRA)" <ji...@apache.org>]

Ladislav Jech created SPARK-28092:
-------------------------------------

             Summary: Spark cannot load files with COLON(:) char if not specified full path
                 Key: SPARK-28092
                 URL: https://issues.apache.org/jira/browse/SPARK-28092
             Project: Spark
          Issue Type: Bug
          Components: Spark Core
    Affects Versions: 2.4.3
         Environment: Cloudera 6.2

Spark latest parcel (I think 2.4.3)
            Reporter: Ladislav Jech


Scenario:

I have CSV files in S3 bucket like this:

s3a://bucket/prefix/myfile_2019:04:05.csv

s3a://bucket/prefix/myfile_2019:04:06.csv

Now when I try to load files with something like:
df = spark.read.load("s3://bucket/prefix/*", format="csv", sep=":", inferSchema="true", header="true")
 
It fails on error about URI (sorry don't have here exact exception), but when I list all files from S3 and provide path like array:
df = spark.read.load(path=["s3://bucket/prefix/myfile_2019:04:05.csv","s3://bucket/prefix/myfile_2019:04:05.csv"], format="csv", sep=":", inferSchema="true", header="true")
 
It works, the reason is COLON character in the name of files as per my observations.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org