[jira] [Created] (SPARK-20592) Alter table concatenate is not working as expected.

["Guru Prabhakar Reddy Marthala (JIRA)" <ji...@apache.org>]

Guru Prabhakar Reddy Marthala created SPARK-20592:
-----------------------------------------------------

             Summary: Alter table concatenate is not working as expected.
                 Key: SPARK-20592
                 URL: https://issues.apache.org/jira/browse/SPARK-20592
             Project: Spark
          Issue Type: Bug
          Components: PySpark
    Affects Versions: 2.1.0
            Reporter: Guru Prabhakar Reddy Marthala


Created a table using CTAS from csv to parquet.Parquet table generated numerous small files.tried alter table concatenate but it's not working as expected.

spark.sql("CREATE TABLE flight.flight_data(year INT,   month INT,   day INT,   day_of_week INT,   dep_time INT,   crs_dep_time INT,   arr_time INT,   crs_arr_time INT,   unique_carrier STRING,   flight_num INT,   tail_num STRING,   actual_elapsed_time INT,   crs_elapsed_time INT,   air_time INT,   arr_delay INT,   dep_delay INT,   origin STRING,   dest STRING,   distance INT,   taxi_in INT,   taxi_out INT,   cancelled INT,   cancellation_code STRING,   diverted INT,   carrier_delay STRING,   weather_delay STRING,   nas_delay STRING,   security_delay STRING,   late_aircraft_delay STRING) ROW FORMAT DELIMITED FIELDS TERMINATED BY ',' stored as textfile")

spark.sql("load data local INPATH 'i:/2008/2008.csv' INTO TABLE flight.flight_data")

spark.sql("create table flight.flight_data_pq stored as parquet as select * from flight.flight_data")

spark.sql("create table flight.flight_data_orc stored as orc as select * from flight.flight_data")

pyspark.sql.utils.ParseException: u'\nOperation not allowed: alter table concatenate(line 1, pos 0)\n\n== SQL ==\nalter table flight_data.flight_data_pq concatenate\n^^^\n'

Tried on both orc and parquet format.It's not working.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org