You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by centerqi hu <ce...@gmail.com> on 2014/09/02 09:35:14 UTC
Unsupported language features in query
hql("""CREATE TABLE tmp_adclick_gm_all ROW FORMAT DELIMITED FIELDS
TERMINATED BY ',' LINES TERMINATED BY '\n' as SELECT SUM(uv) as uv,
round(SUM(cost),2) as total, round(SUM(cost)/SUM(uv),2) FROM
tmp_adclick_sellplat """)
14/09/02 15:32:28 INFO ParseDriver: Parse Completed
java.lang.RuntimeException:
Unsupported language features in query: CREATE TABLE
tmp_adclick_gm_all ROW FORMAT DELIMITED FIELDS TERMINATED BY ',' LINES
TERMINATED BY 'abc' as SELECT SUM(uv) as uv, round(SUM(cost),2) as
total, round(SUM(cost)/SUM(uv),2) FROM tmp_adclick_sellplat
TOK_CREATETABLE
TOK_TABNAME
tmp_adclick_gm_all
TOK_LIKETABLE
TOK_TABLEROWFORMAT
TOK_SERDEPROPS
TOK_TABLEROWFORMATFIELD
','
TOK_TABLEROWFORMATLINES
'abc'
TOK_QUERY
TOK_FROM
TOK_TABREF
TOK_TABNAME
tmp_adclick_sellplat
TOK_INSERT
TOK_DESTINATION
TOK_DIR
TOK_TMP_FILE
TOK_SELECT
TOK_SELEXPR
TOK_FUNCTION
SUM
TOK_TABLE_OR_COL
uv
uv
TOK_SELEXPR
TOK_FUNCTION
round
TOK_FUNCTION
SUM
TOK_TABLE_OR_COL
cost
2
total
TOK_SELEXPR
TOK_FUNCTION
round
/
TOK_FUNCTION
SUM
TOK_TABLE_OR_COL
cost
TOK_FUNCTION
SUM
TOK_TABLE_OR_COL
uv
2
at scala.sys.package$.error(package.scala:27)
at org.apache.spark.sql.hive.HiveQl$.parseSql(HiveQl.scala:255)
at org.apache.spark.sql.hive.HiveContext.hiveql(HiveContext.scala:75)
at org.apache.spark.sql.hive.HiveContext.hql(HiveContext.scala:78)
at $iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC.<init>(<console>:22)
at $iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC.<init>(<console>:27)
at $iwC$$iwC$$iwC$$iwC$$iwC$$iwC.<init>(<console>:29)
at $iwC$$iwC$$iwC$$iwC$$iwC.<init>(<console>:31)
at $iwC$$iwC$$iwC$$iwC.<init>(<console>:33)
at $iwC$$iwC$$iwC.<init>(<console>:35)
at $iwC$$iwC.<init>(<console>:37)
at $iwC.<init>(<console>:39)
--
centerqi@gmail.com|齐忠
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org
Re: Unsupported language features in query
Posted by centerqi hu <ce...@gmail.com>.
Thanks Chen Hao.
This bug have been submited.
https://issues.apache.org/jira/browse/SPARK-3343
2014-09-02 15:55 GMT+08:00 Cheng, Hao <ha...@intel.com>:
> I am afraid no, but you can report that in Jira (
> https://issues.apache.org/jira/browse/SPARK) if you meet the missing
> functionalities in SparkSQL.
>
> SparkSQL aims to support all of the Hive functionalities (at least most of
> it) for HQL dialect.
>
> -----Original Message-----
> From: centerqi hu [mailto:centerqi@gmail.com]
> Sent: Tuesday, September 02, 2014 3:46 PM
> To: Cheng, Hao
> Cc: user@spark.apache.org
> Subject: Re: Unsupported language features in query
>
> Thanks Cheng Hao
>
> Have a way of obtaining spark support hive statement list?
>
> Thanks
>
> 2014-09-02 15:39 GMT+08:00 Cheng, Hao <ha...@intel.com>:
> > Currently SparkSQL doesn’t support the row format/serde in CTAS. The
> work around is create the table first.
> >
> > -----Original Message-----
> > From: centerqi hu [mailto:centerqi@gmail.com]
> > Sent: Tuesday, September 02, 2014 3:35 PM
> > To: user@spark.apache.org
> > Subject: Unsupported language features in query
> >
> > hql("""CREATE TABLE tmp_adclick_gm_all ROW FORMAT DELIMITED FIELDS
> TERMINATED BY ',' LINES TERMINATED BY '\n' as SELECT SUM(uv) as uv,
> > round(SUM(cost),2) as total, round(SUM(cost)/SUM(uv),2) FROM
> tmp_adclick_sellplat """)
> >
> >
> > 14/09/02 15:32:28 INFO ParseDriver: Parse Completed
> >
> > java.lang.RuntimeException:
> >
> > Unsupported language features in query: CREATE TABLE tmp_adclick_gm_all
> ROW FORMAT DELIMITED FIELDS TERMINATED BY ',' LINES TERMINATED BY 'abc' as
> SELECT SUM(uv) as uv, round(SUM(cost),2) as total,
> round(SUM(cost)/SUM(uv),2) FROM tmp_adclick_sellplat
> >
> > TOK_CREATETABLE
> >
> > TOK_TABNAME
> >
> > tmp_adclick_gm_all
> >
> > TOK_LIKETABLE
> >
> > TOK_TABLEROWFORMAT
> >
> > TOK_SERDEPROPS
> >
> > TOK_TABLEROWFORMATFIELD
> >
> > ','
> >
> > TOK_TABLEROWFORMATLINES
> >
> > 'abc'
> >
> > TOK_QUERY
> >
> > TOK_FROM
> >
> > TOK_TABREF
> >
> > TOK_TABNAME
> >
> > tmp_adclick_sellplat
> >
> > TOK_INSERT
> >
> > TOK_DESTINATION
> >
> > TOK_DIR
> >
> > TOK_TMP_FILE
> >
> > TOK_SELECT
> >
> > TOK_SELEXPR
> >
> > TOK_FUNCTION
> >
> > SUM
> >
> > TOK_TABLE_OR_COL
> >
> > uv
> >
> > uv
> >
> > TOK_SELEXPR
> >
> > TOK_FUNCTION
> >
> > round
> >
> > TOK_FUNCTION
> >
> > SUM
> >
> > TOK_TABLE_OR_COL
> >
> > cost
> >
> > 2
> >
> > total
> >
> > TOK_SELEXPR
> >
> > TOK_FUNCTION
> >
> > round
> >
> > /
> >
> > TOK_FUNCTION
> >
> > SUM
> >
> > TOK_TABLE_OR_COL
> >
> > cost
> >
> > TOK_FUNCTION
> >
> > SUM
> >
> > TOK_TABLE_OR_COL
> >
> > uv
> >
> > 2
> >
> >
> >
> >
> > at scala.sys.package$.error(package.scala:27)
> >
> > at org.apache.spark.sql.hive.HiveQl$.parseSql(HiveQl.scala:255)
> >
> > at org.apache.spark.sql.hive.HiveContext.hiveql(HiveContext.scala:75)
> >
> > at org.apache.spark.sql.hive.HiveContext.hql(HiveContext.scala:78)
> >
> > at $iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC.<init>(<console>:22)
> >
> > at $iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC.<init>(<console>:27)
> >
> > at $iwC$$iwC$$iwC$$iwC$$iwC$$iwC.<init>(<console>:29)
> >
> > at $iwC$$iwC$$iwC$$iwC$$iwC.<init>(<console>:31)
> >
> > at $iwC$$iwC$$iwC$$iwC.<init>(<console>:33)
> >
> > at $iwC$$iwC$$iwC.<init>(<console>:35)
> >
> > at $iwC$$iwC.<init>(<console>:37)
> >
> > at $iwC.<init>(<console>:39)
> >
> >
> > --
> > centerqi@gmail.com|齐忠
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: user-unsubscribe@spark.apache.org For
> additional commands, e-mail: user-help@spark.apache.org
> >
>
> --
> centerqi@gmail.com|齐忠
>
--
centerqi@gmail.com|齐忠
RE: Unsupported language features in query
Posted by "Cheng, Hao" <ha...@intel.com>.
I am afraid no, but you can report that in Jira (https://issues.apache.org/jira/browse/SPARK) if you meet the missing functionalities in SparkSQL.
SparkSQL aims to support all of the Hive functionalities (at least most of it) for HQL dialect.
-----Original Message-----
From: centerqi hu [mailto:centerqi@gmail.com]
Sent: Tuesday, September 02, 2014 3:46 PM
To: Cheng, Hao
Cc: user@spark.apache.org
Subject: Re: Unsupported language features in query
Thanks Cheng Hao
Have a way of obtaining spark support hive statement list?
Thanks
2014-09-02 15:39 GMT+08:00 Cheng, Hao <ha...@intel.com>:
> Currently SparkSQL doesn’t support the row format/serde in CTAS. The work around is create the table first.
>
> -----Original Message-----
> From: centerqi hu [mailto:centerqi@gmail.com]
> Sent: Tuesday, September 02, 2014 3:35 PM
> To: user@spark.apache.org
> Subject: Unsupported language features in query
>
> hql("""CREATE TABLE tmp_adclick_gm_all ROW FORMAT DELIMITED FIELDS TERMINATED BY ',' LINES TERMINATED BY '\n' as SELECT SUM(uv) as uv,
> round(SUM(cost),2) as total, round(SUM(cost)/SUM(uv),2) FROM tmp_adclick_sellplat """)
>
>
> 14/09/02 15:32:28 INFO ParseDriver: Parse Completed
>
> java.lang.RuntimeException:
>
> Unsupported language features in query: CREATE TABLE tmp_adclick_gm_all ROW FORMAT DELIMITED FIELDS TERMINATED BY ',' LINES TERMINATED BY 'abc' as SELECT SUM(uv) as uv, round(SUM(cost),2) as total, round(SUM(cost)/SUM(uv),2) FROM tmp_adclick_sellplat
>
> TOK_CREATETABLE
>
> TOK_TABNAME
>
> tmp_adclick_gm_all
>
> TOK_LIKETABLE
>
> TOK_TABLEROWFORMAT
>
> TOK_SERDEPROPS
>
> TOK_TABLEROWFORMATFIELD
>
> ','
>
> TOK_TABLEROWFORMATLINES
>
> 'abc'
>
> TOK_QUERY
>
> TOK_FROM
>
> TOK_TABREF
>
> TOK_TABNAME
>
> tmp_adclick_sellplat
>
> TOK_INSERT
>
> TOK_DESTINATION
>
> TOK_DIR
>
> TOK_TMP_FILE
>
> TOK_SELECT
>
> TOK_SELEXPR
>
> TOK_FUNCTION
>
> SUM
>
> TOK_TABLE_OR_COL
>
> uv
>
> uv
>
> TOK_SELEXPR
>
> TOK_FUNCTION
>
> round
>
> TOK_FUNCTION
>
> SUM
>
> TOK_TABLE_OR_COL
>
> cost
>
> 2
>
> total
>
> TOK_SELEXPR
>
> TOK_FUNCTION
>
> round
>
> /
>
> TOK_FUNCTION
>
> SUM
>
> TOK_TABLE_OR_COL
>
> cost
>
> TOK_FUNCTION
>
> SUM
>
> TOK_TABLE_OR_COL
>
> uv
>
> 2
>
>
>
>
> at scala.sys.package$.error(package.scala:27)
>
> at org.apache.spark.sql.hive.HiveQl$.parseSql(HiveQl.scala:255)
>
> at org.apache.spark.sql.hive.HiveContext.hiveql(HiveContext.scala:75)
>
> at org.apache.spark.sql.hive.HiveContext.hql(HiveContext.scala:78)
>
> at $iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC.<init>(<console>:22)
>
> at $iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC.<init>(<console>:27)
>
> at $iwC$$iwC$$iwC$$iwC$$iwC$$iwC.<init>(<console>:29)
>
> at $iwC$$iwC$$iwC$$iwC$$iwC.<init>(<console>:31)
>
> at $iwC$$iwC$$iwC$$iwC.<init>(<console>:33)
>
> at $iwC$$iwC$$iwC.<init>(<console>:35)
>
> at $iwC$$iwC.<init>(<console>:37)
>
> at $iwC.<init>(<console>:39)
>
>
> --
> centerqi@gmail.com|齐忠
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org For additional commands, e-mail: user-help@spark.apache.org
>
--
centerqi@gmail.com|齐忠
Re: Unsupported language features in query
Posted by centerqi hu <ce...@gmail.com>.
Thanks Cheng Hao
Have a way of obtaining spark support hive statement list?
Thanks
2014-09-02 15:39 GMT+08:00 Cheng, Hao <ha...@intel.com>:
> Currently SparkSQL doesn’t support the row format/serde in CTAS. The work around is create the table first.
>
> -----Original Message-----
> From: centerqi hu [mailto:centerqi@gmail.com]
> Sent: Tuesday, September 02, 2014 3:35 PM
> To: user@spark.apache.org
> Subject: Unsupported language features in query
>
> hql("""CREATE TABLE tmp_adclick_gm_all ROW FORMAT DELIMITED FIELDS TERMINATED BY ',' LINES TERMINATED BY '\n' as SELECT SUM(uv) as uv,
> round(SUM(cost),2) as total, round(SUM(cost)/SUM(uv),2) FROM tmp_adclick_sellplat """)
>
>
> 14/09/02 15:32:28 INFO ParseDriver: Parse Completed
>
> java.lang.RuntimeException:
>
> Unsupported language features in query: CREATE TABLE tmp_adclick_gm_all ROW FORMAT DELIMITED FIELDS TERMINATED BY ',' LINES TERMINATED BY 'abc' as SELECT SUM(uv) as uv, round(SUM(cost),2) as total, round(SUM(cost)/SUM(uv),2) FROM tmp_adclick_sellplat
>
> TOK_CREATETABLE
>
> TOK_TABNAME
>
> tmp_adclick_gm_all
>
> TOK_LIKETABLE
>
> TOK_TABLEROWFORMAT
>
> TOK_SERDEPROPS
>
> TOK_TABLEROWFORMATFIELD
>
> ','
>
> TOK_TABLEROWFORMATLINES
>
> 'abc'
>
> TOK_QUERY
>
> TOK_FROM
>
> TOK_TABREF
>
> TOK_TABNAME
>
> tmp_adclick_sellplat
>
> TOK_INSERT
>
> TOK_DESTINATION
>
> TOK_DIR
>
> TOK_TMP_FILE
>
> TOK_SELECT
>
> TOK_SELEXPR
>
> TOK_FUNCTION
>
> SUM
>
> TOK_TABLE_OR_COL
>
> uv
>
> uv
>
> TOK_SELEXPR
>
> TOK_FUNCTION
>
> round
>
> TOK_FUNCTION
>
> SUM
>
> TOK_TABLE_OR_COL
>
> cost
>
> 2
>
> total
>
> TOK_SELEXPR
>
> TOK_FUNCTION
>
> round
>
> /
>
> TOK_FUNCTION
>
> SUM
>
> TOK_TABLE_OR_COL
>
> cost
>
> TOK_FUNCTION
>
> SUM
>
> TOK_TABLE_OR_COL
>
> uv
>
> 2
>
>
>
>
> at scala.sys.package$.error(package.scala:27)
>
> at org.apache.spark.sql.hive.HiveQl$.parseSql(HiveQl.scala:255)
>
> at org.apache.spark.sql.hive.HiveContext.hiveql(HiveContext.scala:75)
>
> at org.apache.spark.sql.hive.HiveContext.hql(HiveContext.scala:78)
>
> at $iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC.<init>(<console>:22)
>
> at $iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC.<init>(<console>:27)
>
> at $iwC$$iwC$$iwC$$iwC$$iwC$$iwC.<init>(<console>:29)
>
> at $iwC$$iwC$$iwC$$iwC$$iwC.<init>(<console>:31)
>
> at $iwC$$iwC$$iwC$$iwC.<init>(<console>:33)
>
> at $iwC$$iwC$$iwC.<init>(<console>:35)
>
> at $iwC$$iwC.<init>(<console>:37)
>
> at $iwC.<init>(<console>:39)
>
>
> --
> centerqi@gmail.com|齐忠
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org For additional commands, e-mail: user-help@spark.apache.org
>
--
centerqi@gmail.com|齐忠
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org
RE: Unsupported language features in query
Posted by "Cheng, Hao" <ha...@intel.com>.
Currently SparkSQL doesn’t support the row format/serde in CTAS. The work around is create the table first.
-----Original Message-----
From: centerqi hu [mailto:centerqi@gmail.com]
Sent: Tuesday, September 02, 2014 3:35 PM
To: user@spark.apache.org
Subject: Unsupported language features in query
hql("""CREATE TABLE tmp_adclick_gm_all ROW FORMAT DELIMITED FIELDS TERMINATED BY ',' LINES TERMINATED BY '\n' as SELECT SUM(uv) as uv,
round(SUM(cost),2) as total, round(SUM(cost)/SUM(uv),2) FROM tmp_adclick_sellplat """)
14/09/02 15:32:28 INFO ParseDriver: Parse Completed
java.lang.RuntimeException:
Unsupported language features in query: CREATE TABLE tmp_adclick_gm_all ROW FORMAT DELIMITED FIELDS TERMINATED BY ',' LINES TERMINATED BY 'abc' as SELECT SUM(uv) as uv, round(SUM(cost),2) as total, round(SUM(cost)/SUM(uv),2) FROM tmp_adclick_sellplat
TOK_CREATETABLE
TOK_TABNAME
tmp_adclick_gm_all
TOK_LIKETABLE
TOK_TABLEROWFORMAT
TOK_SERDEPROPS
TOK_TABLEROWFORMATFIELD
','
TOK_TABLEROWFORMATLINES
'abc'
TOK_QUERY
TOK_FROM
TOK_TABREF
TOK_TABNAME
tmp_adclick_sellplat
TOK_INSERT
TOK_DESTINATION
TOK_DIR
TOK_TMP_FILE
TOK_SELECT
TOK_SELEXPR
TOK_FUNCTION
SUM
TOK_TABLE_OR_COL
uv
uv
TOK_SELEXPR
TOK_FUNCTION
round
TOK_FUNCTION
SUM
TOK_TABLE_OR_COL
cost
2
total
TOK_SELEXPR
TOK_FUNCTION
round
/
TOK_FUNCTION
SUM
TOK_TABLE_OR_COL
cost
TOK_FUNCTION
SUM
TOK_TABLE_OR_COL
uv
2
at scala.sys.package$.error(package.scala:27)
at org.apache.spark.sql.hive.HiveQl$.parseSql(HiveQl.scala:255)
at org.apache.spark.sql.hive.HiveContext.hiveql(HiveContext.scala:75)
at org.apache.spark.sql.hive.HiveContext.hql(HiveContext.scala:78)
at $iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC.<init>(<console>:22)
at $iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC.<init>(<console>:27)
at $iwC$$iwC$$iwC$$iwC$$iwC$$iwC.<init>(<console>:29)
at $iwC$$iwC$$iwC$$iwC$$iwC.<init>(<console>:31)
at $iwC$$iwC$$iwC$$iwC.<init>(<console>:33)
at $iwC$$iwC$$iwC.<init>(<console>:35)
at $iwC$$iwC.<init>(<console>:37)
at $iwC.<init>(<console>:39)
--
centerqi@gmail.com|齐忠
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org For additional commands, e-mail: user-help@spark.apache.org