You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by Yusuf Can Gürkan <yu...@useinsider.com> on 2015/08/19 17:44:56 UTC
SQLContext Create Table Problem
Hello,
I’m trying to create a table with sqlContext.sql method as below:
val sc = new SparkContext()
val sqlContext = new SQLContext(sc)
import sqlContext.implicits._
sqlContext.sql(s"""
create table if not exists landing (
date string,
referrer string
)
partitioned by (partnerid string,dt string)
row format delimited fields terminated by '\t' lines terminated by '\n'
STORED AS TEXTFILE LOCATION 's3n://...'
"”")
It gives error on spark-submit:
Exception in thread "main" java.lang.RuntimeException: [2.1] failure: ``with'' expected but identifier create found
create external table if not exists landing (
^
at scala.sys.package$.error(package.scala:27)
at org.apache.spark.sql.catalyst.AbstractSparkSQLParser.parse(AbstractSparkSQLParser.scala:36)
at org.apache.spark.sql.catalyst.DefaultParserDialect.parse(ParserDialect.scala:67)
What can be the reason??
Re: SQLContext Create Table Problem
Posted by Yusuf Can Gürkan <yu...@useinsider.com>.
Someone has any idea about this subject?
> On 20 Aug 2015, at 00:47, Yusuf Can Gürkan <yu...@useinsider.com> wrote:
>
> Hey,
>
> This is my spark-env:
>
> # Add Hadoop libraries to Spark classpath
> SPARK_CLASSPATH="${SPARK_CLASSPATH}:${HADOOP_HOME}/*:${HADOOP_HOME}/../hadoop-hdfs/*:${HADOOP_HOME}/../hadoop-mapreduce/*:${HADOOP_HOME}/../hadoop-yarn/*"
> LD_LIBRARY_PATH="${LD_LIBRARY_PATH}:${HADOOP_HOME}/lib/native"
>
> # Add Hive conf to Spark classpath
> SPARK_CLASSPATH="${SPARK_CLASSPATH}:${HIVE_CONF_DIR}"
>
> # Add Hadoop LZO library to Spark classpath and native library to library path
> SPARK_CLASSPATH="${SPARK_CLASSPATH}:${HADOOP_HOME}/../hadoop-lzo/lib/*"
> LD_LIBRARY_PATH="${LD_LIBRARY_PATH}:${HADOOP_HOME}/../hadoop-lzo/lib/native"
>
> # Add EMRFS libraries to Spark classpath
> SPARK_CLASSPATH="${SPARK_CLASSPATH}:/usr/share/aws/emr/emrfs/conf:/usr/share/aws/emr/emrfs/lib/*:/usr/share/aws/emr/emrfs/auxlib/*"
>
> export SPARK_CLASSPATH
> export LD_LIBRARY_PATH
>
> I think hive is on class path, should i do another thing?
>
>> On 19 Aug 2015, at 20:12, Eric Walker <eric@node.io <ma...@node.io>> wrote:
>>
>> Hi,
>>
>> On EMR, take a look at the "SPARK_CLASSPATH" variable in /home/hadoop/spark/conf/spark-env.sh. You may need to add a jar file to it.
>>
>> Regards,
>> Eric
>>
>>
>> On Wed, Aug 19, 2015 at 11:03 AM, Yusuf Can Gürkan <yusuf@useinsider.com <ma...@useinsider.com>> wrote:
>> Hey Yin,
>>
>> Thanks for answer. I thought that this could be problem but i can not create HiveContext because i can not import org.apache.spark.sql.hive.HiveContext. It does not see this package.
>>
>> I read that i should build spark with -PHive but i’m running on Amazon EMR 1.4.1 and on spark-shell i can import hive package but can not do the same on spark-submit. Do you have any idea why? Because if it’s related to build with -PHive, how can i import it in spark-shell?
>>
>>> On 19 Aug 2015, at 18:59, Yin Huai <yhuai@databricks.com <ma...@databricks.com>> wrote:
>>>
>>> Can you try to use HiveContext instead of SQLContext? Your query is trying to create a table and persist the metadata of the table in metastore, which is only supported by HiveContext.
>>>
>>> On Wed, Aug 19, 2015 at 8:44 AM, Yusuf Can Gürkan <yusuf@useinsider.com <ma...@useinsider.com>> wrote:
>>> Hello,
>>>
>>> I’m trying to create a table with sqlContext.sql method as below:
>>>
>>> val sc = new SparkContext()
>>> val sqlContext = new SQLContext(sc)
>>>
>>> import sqlContext.implicits._
>>>
>>> sqlContext.sql(s"""
>>> create table if not exists landing (
>>> date string,
>>> referrer string
>>> )
>>> partitioned by (partnerid string,dt string)
>>> row format delimited fields terminated by '\t' lines terminated by '\n'
>>> STORED AS TEXTFILE LOCATION 's3n://...' <>
>>> "”")
>>>
>>>
>>> It gives error on spark-submit:
>>>
>>> Exception in thread "main" java.lang.RuntimeException: [2.1] failure: ``with'' expected but identifier create found
>>>
>>> create external table if not exists landing (
>>>
>>> ^
>>> at scala.sys.package$.error(package.scala:27)
>>> at org.apache.spark.sql.catalyst.AbstractSparkSQLParser.parse(AbstractSparkSQLParser.scala:36)
>>> at org.apache.spark.sql.catalyst.DefaultParserDialect.parse(ParserDialect.scala:67)
>>>
>>>
>>>
>>> What can be the reason??
>>>
>>
>>
>
Re: SQLContext Create Table Problem
Posted by Yusuf Can Gürkan <yu...@useinsider.com>.
Hey,
This is my spark-env:
# Add Hadoop libraries to Spark classpath
SPARK_CLASSPATH="${SPARK_CLASSPATH}:${HADOOP_HOME}/*:${HADOOP_HOME}/../hadoop-hdfs/*:${HADOOP_HOME}/../hadoop-mapreduce/*:${HADOOP_HOME}/../hadoop-yarn/*"
LD_LIBRARY_PATH="${LD_LIBRARY_PATH}:${HADOOP_HOME}/lib/native"
# Add Hive conf to Spark classpath
SPARK_CLASSPATH="${SPARK_CLASSPATH}:${HIVE_CONF_DIR}"
# Add Hadoop LZO library to Spark classpath and native library to library path
SPARK_CLASSPATH="${SPARK_CLASSPATH}:${HADOOP_HOME}/../hadoop-lzo/lib/*"
LD_LIBRARY_PATH="${LD_LIBRARY_PATH}:${HADOOP_HOME}/../hadoop-lzo/lib/native"
# Add EMRFS libraries to Spark classpath
SPARK_CLASSPATH="${SPARK_CLASSPATH}:/usr/share/aws/emr/emrfs/conf:/usr/share/aws/emr/emrfs/lib/*:/usr/share/aws/emr/emrfs/auxlib/*"
export SPARK_CLASSPATH
export LD_LIBRARY_PATH
I think hive is on class path, should i do another thing?
> On 19 Aug 2015, at 20:12, Eric Walker <er...@node.io> wrote:
>
> Hi,
>
> On EMR, take a look at the "SPARK_CLASSPATH" variable in /home/hadoop/spark/conf/spark-env.sh. You may need to add a jar file to it.
>
> Regards,
> Eric
>
>
> On Wed, Aug 19, 2015 at 11:03 AM, Yusuf Can Gürkan <yusuf@useinsider.com <ma...@useinsider.com>> wrote:
> Hey Yin,
>
> Thanks for answer. I thought that this could be problem but i can not create HiveContext because i can not import org.apache.spark.sql.hive.HiveContext. It does not see this package.
>
> I read that i should build spark with -PHive but i’m running on Amazon EMR 1.4.1 and on spark-shell i can import hive package but can not do the same on spark-submit. Do you have any idea why? Because if it’s related to build with -PHive, how can i import it in spark-shell?
>
>> On 19 Aug 2015, at 18:59, Yin Huai <yhuai@databricks.com <ma...@databricks.com>> wrote:
>>
>> Can you try to use HiveContext instead of SQLContext? Your query is trying to create a table and persist the metadata of the table in metastore, which is only supported by HiveContext.
>>
>> On Wed, Aug 19, 2015 at 8:44 AM, Yusuf Can Gürkan <yusuf@useinsider.com <ma...@useinsider.com>> wrote:
>> Hello,
>>
>> I’m trying to create a table with sqlContext.sql method as below:
>>
>> val sc = new SparkContext()
>> val sqlContext = new SQLContext(sc)
>>
>> import sqlContext.implicits._
>>
>> sqlContext.sql(s"""
>> create table if not exists landing (
>> date string,
>> referrer string
>> )
>> partitioned by (partnerid string,dt string)
>> row format delimited fields terminated by '\t' lines terminated by '\n'
>> STORED AS TEXTFILE LOCATION 's3n://...' <>
>> "”")
>>
>>
>> It gives error on spark-submit:
>>
>> Exception in thread "main" java.lang.RuntimeException: [2.1] failure: ``with'' expected but identifier create found
>>
>> create external table if not exists landing (
>>
>> ^
>> at scala.sys.package$.error(package.scala:27)
>> at org.apache.spark.sql.catalyst.AbstractSparkSQLParser.parse(AbstractSparkSQLParser.scala:36)
>> at org.apache.spark.sql.catalyst.DefaultParserDialect.parse(ParserDialect.scala:67)
>>
>>
>>
>> What can be the reason??
>>
>
>
Re: SQLContext Create Table Problem
Posted by Yusuf Can Gürkan <yu...@useinsider.com>.
Hey Yin,
Thanks for answer. I thought that this could be problem but i can not create HiveContext because i can not import org.apache.spark.sql.hive.HiveContext. It does not see this package.
I read that i should build spark with -PHive but i’m running on Amazon EMR 1.4.1 and on spark-shell i can import hive package but can not do the same on spark-submit. Do you have any idea why? Because if it’s related to build with -PHive, how can i import it in spark-shell?
> On 19 Aug 2015, at 18:59, Yin Huai <yh...@databricks.com> wrote:
>
> Can you try to use HiveContext instead of SQLContext? Your query is trying to create a table and persist the metadata of the table in metastore, which is only supported by HiveContext.
>
> On Wed, Aug 19, 2015 at 8:44 AM, Yusuf Can Gürkan <yusuf@useinsider.com <ma...@useinsider.com>> wrote:
> Hello,
>
> I’m trying to create a table with sqlContext.sql method as below:
>
> val sc = new SparkContext()
> val sqlContext = new SQLContext(sc)
>
> import sqlContext.implicits._
>
> sqlContext.sql(s"""
> create table if not exists landing (
> date string,
> referrer string
> )
> partitioned by (partnerid string,dt string)
> row format delimited fields terminated by '\t' lines terminated by '\n'
> STORED AS TEXTFILE LOCATION 's3n://...' <>
> "”")
>
>
> It gives error on spark-submit:
>
> Exception in thread "main" java.lang.RuntimeException: [2.1] failure: ``with'' expected but identifier create found
>
> create external table if not exists landing (
>
> ^
> at scala.sys.package$.error(package.scala:27)
> at org.apache.spark.sql.catalyst.AbstractSparkSQLParser.parse(AbstractSparkSQLParser.scala:36)
> at org.apache.spark.sql.catalyst.DefaultParserDialect.parse(ParserDialect.scala:67)
>
>
>
> What can be the reason??
>
Re: SQLContext Create Table Problem
Posted by Yin Huai <yh...@databricks.com>.
Can you try to use HiveContext instead of SQLContext? Your query is trying
to create a table and persist the metadata of the table in metastore, which
is only supported by HiveContext.
On Wed, Aug 19, 2015 at 8:44 AM, Yusuf Can Gürkan <yu...@useinsider.com>
wrote:
> Hello,
>
> I’m trying to create a table with sqlContext.sql method as below:
>
> *val sc = new SparkContext()*
> *val sqlContext = new SQLContext(sc)*
>
> *import sqlContext.implicits._*
>
> *sqlContext.sql(s"""*
> *create table if not exists landing (*
> *date string,*
> *referrer string*
> *)*
> *partitioned by (partnerid string,dt string)*
> *row format delimited fields terminated by '\t' lines terminated by '\n'*
> *STORED AS TEXTFILE LOCATION 's3n://...'*
> * "”")*
>
>
> It gives error on spark-submit:
>
> *Exception in thread "main" java.lang.RuntimeException: [2.1] failure:
> ``with'' expected but identifier create found*
>
> *create external table if not exists landing (*
>
> *^*
> * at scala.sys.package$.error(package.scala:27)*
> * at
> org.apache.spark.sql.catalyst.AbstractSparkSQLParser.parse(AbstractSparkSQLParser.scala:36)*
> * at
> org.apache.spark.sql.catalyst.DefaultParserDialect.parse(ParserDialect.scala:67)*
>
>
>
> What can be the reason??
>