You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Josh Rosen (JIRA)" <ji...@apache.org> on 2015/09/16 00:51:45 UTC
[jira] [Resolved] (SPARK-5624) Can't find new column
[ https://issues.apache.org/jira/browse/SPARK-5624?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Josh Rosen resolved SPARK-5624.
-------------------------------
Resolution: Fixed
Resolving as "Fixed" per claim that this works in a newer release.
> Can't find new column
> ----------------------
>
> Key: SPARK-5624
> URL: https://issues.apache.org/jira/browse/SPARK-5624
> Project: Spark
> Issue Type: Bug
> Components: SQL
> Affects Versions: 1.1.1
> Reporter: Alex Liu
> Priority: Minor
>
> The following test fails
> {code}
> 0: jdbc:hive2://localhost:10000> DROP TABLE IF EXISTS alter_test_table;
> +---------+
> | Result |
> +---------+
> +---------+
> No rows selected (0.175 seconds)
> 0: jdbc:hive2://localhost:10000> DROP TABLE IF EXISTS alter_test_table_ctas;
> +---------+
> | Result |
> +---------+
> +---------+
> No rows selected (0.155 seconds)
> 0: jdbc:hive2://localhost:10000> DROP TABLE IF EXISTS alter_test_table_renamed;
> +---------+
> | Result |
> +---------+
> +---------+
> No rows selected (0.162 seconds)
> 0: jdbc:hive2://localhost:10000> CREATE TABLE alter_test_table (foo INT, bar STRING) COMMENT 'table to test DDL ops' PARTITIONED BY (ds STRING) STORED AS TEXTFILE;
> +---------+
> | result |
> +---------+
> +---------+
> No rows selected (0.247 seconds)
> 0: jdbc:hive2://localhost:10000> LOAD DATA LOCAL INPATH '/Users/alex/project/automaton/resources/tests/data/files/kv1.txt' OVERWRITE INTO TABLE alter_test_table PARTITION (ds='2008-08-08');
> +---------+
> | result |
> +---------+
> +---------+
> No rows selected (0.367 seconds)
> 0: jdbc:hive2://localhost:10000> CREATE TABLE alter_test_table_ctas as SELECT * FROM alter_test_table;
> +------+------+-----+
> | foo | bar | ds |
> +------+------+-----+
> +------+------+-----+
> No rows selected (0.641 seconds)
> 0: jdbc:hive2://localhost:10000> ALTER TABLE alter_test_table ADD COLUMNS (new_col1 INT);
> +---------+
> | result |
> +---------+
> +---------+
> No rows selected (0.226 seconds)
> 0: jdbc:hive2://localhost:10000> INSERT OVERWRITE TABLE alter_test_table PARTITION (ds='2008-08-15') SELECT foo, bar, 333333333 FROM alter_test_table_ctas WHERE ds='2008-08-08';
> +------+------+------+
> | foo | bar | c_2 |
> +------+------+------+
> +------+------+------+
> No rows selected (0.522 seconds)
> 0: jdbc:hive2://localhost:10000> select * from alter_test_table ;
> Error: org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 35.0 failed 4 times, most recent failure: Lost task 0.3 in stage 35.0 (TID 66, 127.0.0.1): java.lang.RuntimeException: cannot find field new_col1 from [0:foo, 1:bar]
> org.apache.hadoop.hive.serde2.objectinspector.ObjectInspectorUtils.getStandardStructFieldRef(ObjectInspectorUtils.java:367)
> org.apache.hadoop.hive.serde2.lazy.objectinspector.LazySimpleStructObjectInspector.getStructFieldRef(LazySimpleStructObjectInspector.java:168)
> org.apache.spark.sql.hive.HadoopTableReader$$anonfun$9.apply(TableReader.scala:275)
> org.apache.spark.sql.hive.HadoopTableReader$$anonfun$9.apply(TableReader.scala:275)
> scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244)
> scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244)
> scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)
> scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:47)
> scala.collection.TraversableLike$class.map(TraversableLike.scala:244)
> scala.collection.AbstractTraversable.map(Traversable.scala:105)
> org.apache.spark.sql.hive.HadoopTableReader$.fillObject(TableReader.scala:275)
> org.apache.spark.sql.hive.HadoopTableReader$$anonfun$3$$anonfun$apply$1.apply(TableReader.scala:193)
> org.apache.spark.sql.hive.HadoopTableReader$$anonfun$3$$anonfun$apply$1.apply(TableReader.scala:187)
> org.apache.spark.rdd.RDD$$anonfun$13.apply(RDD.scala:596)
> org.apache.spark.rdd.RDD$$anonfun$13.apply(RDD.scala:596)
> org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
> org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:262)
> org.apache.spark.rdd.RDD.iterator(RDD.scala:229)
> org.apache.spark.rdd.UnionRDD.compute(UnionRDD.scala:87)
> org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:262)
> org.apache.spark.rdd.RDD.iterator(RDD.scala:229)
> org.apache.spark.rdd.MappedRDD.compute(MappedRDD.scala:31)
> org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:262)
> org.apache.spark.rdd.RDD.iterator(RDD.scala:229)
> org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:62)
> org.apache.spark.scheduler.Task.run(Task.scala:54)
> org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:177)
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> java.lang.Thread.run(Thread.java:745)
> Driver stacktrace: (state=,code=0)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org