You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by Jack Yang <ji...@uow.edu.au> on 2015/07/21 06:31:08 UTC
standalone to connect mysql
Hi there,
I would like to use spark to access the data in mysql. So firstly I tried to run the program using:
spark-submit --class "sparkwithscala.SqlApp" --driver-class-path /home/lib/mysql-connector-java-5.1.34.jar --master local[4] /home/myjar.jar
that returns me the correct results. Then I tried the standalone version using:
spark-submit --class "sparkwithscala.SqlApp" --driver-class-path /home/lib/mysql-connector-java-5.1.34.jar --master spark://hadoop1:7077 /home/myjar.jar
(the mysql-connector-java-5.1.34.jar i have them on all worker nodes.)
and the error is:
Exception in thread "main" org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 0.0 failed 4 times, most recent failure: Lost task 0.3 in stage 0.0 (TID 3, 192.168.157.129): java.sql.SQLException: No suitable driver found for jdbc:mysql://hadoop1:3306/sparkMysqlDB?user=root&password=root
I also found the similar problem before in https://jira.talendforge.org/browse/TBD-2244.
Is this a bug to be fixed later? Or do I miss anything?
Best regards,
Jack
Re: standalone to connect mysql
Posted by Jack Yang <ji...@uow.edu.au>.
I maybe find the answer from the sqlparser.scala file.
Looks like the syntax spark used for insert is different from what we normally used for MySQL.
I hope if someone can confirm this. Also I will appreciate if there is a SQL reference list available.
Sent from my iPhone
On 21 Jul 2015, at 9:21 pm, "Jack Yang" <ji...@uow.edu.au>> wrote:
No. I did not use hiveContext at this stage.
I am talking the embedded SQL syntax for pure spark sql.
Thanks, mate.
On 21 Jul 2015, at 6:13 pm, "Terry Hole" <hu...@gmail.com>> wrote:
Jack,
You can refer the hive sql syntax if you use HiveContext: https://cwiki.apache.org/confluence/display/Hive/LanguageManual+DML
Thanks!
-Terry
That works! Thanks.
Can I ask you one further question?
How did spark sql support insertion?
That is say, if I did:
sqlContext.sql("insert into newStu values ("10","a",1)
the error is:
failure: ``table'' expected but identifier newStu found
insert into newStu values ('10', aa, 1)
but if I did:
sqlContext.sql(s"insert into Table newStu select * from otherStu")
that works.
Is there any document addressing that?
Best regards,
Jack
From: Terry Hole [mailto:hujie.eagle@gmail.com<ma...@gmail.com>]
Sent: Tuesday, 21 July 2015 4:17 PM
To: Jack Yang; user@spark.apache.org<ma...@spark.apache.org>
Subject: Re: standalone to connect mysql
Maybe you can try: spark-submit --class "sparkwithscala.SqlApp" --jars /home/lib/mysql-connector-java-5.1.34.jar --master spark://hadoop1:7077 /home/myjar.jar
Thanks!
-Terry
Hi there,
I would like to use spark to access the data in mysql. So firstly I tried to run the program using:
spark-submit --class "sparkwithscala.SqlApp" --driver-class-path /home/lib/mysql-connector-java-5.1.34.jar --master local[4] /home/myjar.jar
that returns me the correct results. Then I tried the standalone version using:
spark-submit --class "sparkwithscala.SqlApp" --driver-class-path /home/lib/mysql-connector-java-5.1.34.jar --master spark://hadoop1:7077 /home/myjar.jar
(the mysql-connector-java-5.1.34.jar i have them on all worker nodes.)
and the error is:
Exception in thread "main" org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 0.0 failed 4 times, most recent failure: Lost task 0.3 in stage 0.0 (TID 3, 192.168.157.129): java.sql.SQLException: No suitable driver found for jdbc:mysql://hadoop1:3306/sparkMysqlDB?user=root&password=root
I also found the similar problem before in https://jira.talendforge.org/browse/TBD-2244.
Is this a bug to be fixed later? Or do I miss anything?
Best regards,
Jack
Re: standalone to connect mysql
Posted by Jack Yang <ji...@uow.edu.au>.
No. I did not use hiveContext at this stage.
I am talking the embedded SQL syntax for pure spark sql.
Thanks, mate.
On 21 Jul 2015, at 6:13 pm, "Terry Hole" <hu...@gmail.com>> wrote:
Jack,
You can refer the hive sql syntax if you use HiveContext: https://cwiki.apache.org/confluence/display/Hive/LanguageManual+DML
Thanks!
-Terry
That works! Thanks.
Can I ask you one further question?
How did spark sql support insertion?
That is say, if I did:
sqlContext.sql("insert into newStu values ("10","a",1)
the error is:
failure: ``table'' expected but identifier newStu found
insert into newStu values ('10', aa, 1)
but if I did:
sqlContext.sql(s"insert into Table newStu select * from otherStu")
that works.
Is there any document addressing that?
Best regards,
Jack
From: Terry Hole [mailto:hujie.eagle@gmail.com<ma...@gmail.com>]
Sent: Tuesday, 21 July 2015 4:17 PM
To: Jack Yang; user@spark.apache.org<ma...@spark.apache.org>
Subject: Re: standalone to connect mysql
Maybe you can try: spark-submit --class "sparkwithscala.SqlApp" --jars /home/lib/mysql-connector-java-5.1.34.jar --master spark://hadoop1:7077 /home/myjar.jar
Thanks!
-Terry
Hi there,
I would like to use spark to access the data in mysql. So firstly I tried to run the program using:
spark-submit --class "sparkwithscala.SqlApp" --driver-class-path /home/lib/mysql-connector-java-5.1.34.jar --master local[4] /home/myjar.jar
that returns me the correct results. Then I tried the standalone version using:
spark-submit --class "sparkwithscala.SqlApp" --driver-class-path /home/lib/mysql-connector-java-5.1.34.jar --master spark://hadoop1:7077 /home/myjar.jar
(the mysql-connector-java-5.1.34.jar i have them on all worker nodes.)
and the error is:
Exception in thread "main" org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 0.0 failed 4 times, most recent failure: Lost task 0.3 in stage 0.0 (TID 3, 192.168.157.129): java.sql.SQLException: No suitable driver found for jdbc:mysql://hadoop1:3306/sparkMysqlDB?user=root&password=root
I also found the similar problem before in https://jira.talendforge.org/browse/TBD-2244.
Is this a bug to be fixed later? Or do I miss anything?
Best regards,
Jack
Re: standalone to connect mysql
Posted by Terry Hole <hu...@gmail.com>.
Jack,
You can refer the hive sql syntax if you use HiveContext:
https://cwiki.apache.org/confluence/display/Hive/LanguageManual+DML
Thanks!
-Terry
That works! Thanks.
>
Can I ask you one further question？
>
> How did spark sql support insertion?
>
>
>
> That is say, if I did:
>
> sqlContext.sql("insert into newStu values (“10”,”a”,1)
>
>
>
> the error is:
>
> failure: ``table'' expected but identifier newStu found
>
> insert into newStu values ('10', aa, 1)
>
>
>
> but if I did:
>
> sqlContext.sql(s"insert into Table newStu select * from otherStu")
>
> that works.
>
>
>
> Is there any document addressing that?
>
>
>
>
>
> Best regards,
>
> Jack
>
>
>
>
>
> *From:* Terry Hole [mailto:hujie.eagle@gmail.com]
> *Sent:* Tuesday, 21 July 2015 4:17 PM
> *To:* Jack Yang; user@spark.apache.org
> *Subject:* Re: standalone to connect mysql
>
>
>
> Maybe you can try: spark-submit --class "sparkwithscala.SqlApp"
> --jars /home/lib/mysql-connector-java-5.1.34.jar --master spark://hadoop1:7077
> /home/myjar.jar
>
>
>
> Thanks!
>
> -Terry
>
> Hi there,
>
>
>
> I would like to use spark to access the data in mysql. So firstly I tried
> to run the program using:
>
> spark-submit --class "sparkwithscala.SqlApp" --driver-class-path
> /home/lib/mysql-connector-java-5.1.34.jar --master local[4] /home/myjar.jar
>
>
>
> that returns me the correct results. Then I tried the standalone version
> using:
>
> spark-submit --class "sparkwithscala.SqlApp" --driver-class-path
> /home/lib/mysql-connector-java-5.1.34.jar --master spark://hadoop1:7077
> /home/myjar.jar
>
> (the mysql-connector-java-5.1.34.jar i have them on all worker nodes.)
>
> and the error is:
>
>
>
> Exception in thread "main" org.apache.spark.SparkException: Job aborted
> due to stage failure: Task 0 in stage 0.0 failed 4 times, most recent
> failure: Lost task 0.3 in stage 0.0 (TID 3, 192.168.157.129):
> java.sql.SQLException: No suitable driver found for
> jdbc:mysql://hadoop1:3306/sparkMysqlDB?user=root&password=root
>
>
>
> I also found the similar problem before in
> https://jira.talendforge.org/browse/TBD-2244.
>
>
>
> Is this a bug to be fixed later? Or do I miss anything?
>
>
>
>
>
>
>
> Best regards,
>
> Jack
>
>
>
>
RE: standalone to connect mysql
Posted by Jack Yang <ji...@uow.edu.au>.
That works! Thanks.
Can I ask you one further question？
How did spark sql support insertion?
That is say, if I did:
sqlContext.sql("insert into newStu values (“10”,”a”,1)
the error is:
failure: ``table'' expected but identifier newStu found
insert into newStu values ('10', aa, 1)
but if I did:
sqlContext.sql(s"insert into Table newStu select * from otherStu")
that works.
Is there any document addressing that?
Best regards,
Jack
From: Terry Hole [mailto:hujie.eagle@gmail.com]
Sent: Tuesday, 21 July 2015 4:17 PM
To: Jack Yang; user@spark.apache.org
Subject: Re: standalone to connect mysql
Maybe you can try: spark-submit --class "sparkwithscala.SqlApp" --jars /home/lib/mysql-connector-java-5.1.34.jar --master spark://hadoop1:7077 /home/myjar.jar
Thanks!
-Terry
Hi there,
I would like to use spark to access the data in mysql. So firstly I tried to run the program using:
spark-submit --class "sparkwithscala.SqlApp" --driver-class-path /home/lib/mysql-connector-java-5.1.34.jar --master local[4] /home/myjar.jar
that returns me the correct results. Then I tried the standalone version using:
spark-submit --class "sparkwithscala.SqlApp" --driver-class-path /home/lib/mysql-connector-java-5.1.34.jar --master spark://hadoop1:7077 /home/myjar.jar
(the mysql-connector-java-5.1.34.jar i have them on all worker nodes.)
and the error is:
Exception in thread "main" org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 0.0 failed 4 times, most recent failure: Lost task 0.3 in stage 0.0 (TID 3, 192.168.157.129): java.sql.SQLException: No suitable driver found for jdbc:mysql://hadoop1:3306/sparkMysqlDB?user=root&password=root
I also found the similar problem before in https://jira.talendforge.org/browse/TBD-2244.
Is this a bug to be fixed later? Or do I miss anything?
Best regards,
Jack
Re: standalone to connect mysql
Posted by Terry Hole <hu...@gmail.com>.
Maybe you can try: spark-submit --class "sparkwithscala.SqlApp" --jars
/home/lib/mysql-connector-java-5.1.34.jar --master spark://hadoop1:7077
/home/myjar.jar
Thanks!
-Terry
> Hi there,
>
>
>
> I would like to use spark to access the data in mysql. So firstly I tried
> to run the program using:
>
> spark-submit --class "sparkwithscala.SqlApp" --driver-class-path
> /home/lib/mysql-connector-java-5.1.34.jar --master local[4] /home/myjar.jar
>
>
>
> that returns me the correct results. Then I tried the standalone version
> using:
>
> spark-submit --class "sparkwithscala.SqlApp" --driver-class-path
> /home/lib/mysql-connector-java-5.1.34.jar --master spark://hadoop1:7077
> /home/myjar.jar
>
> (the mysql-connector-java-5.1.34.jar i have them on all worker nodes.)
>
> and the error is:
>
>
>
> Exception in thread "main" org.apache.spark.SparkException: Job aborted
> due to stage failure: Task 0 in stage 0.0 failed 4 times, most recent
> failure: Lost task 0.3 in stage 0.0 (TID 3, 192.168.157.129):
> java.sql.SQLException: No suitable driver found for
> jdbc:mysql://hadoop1:3306/sparkMysqlDB?user=root&password=root
>
>
>
> I also found the similar problem before in
> https://jira.talendforge.org/browse/TBD-2244.
>
>
>
> Is this a bug to be fixed later? Or do I miss anything?
>
>
>
>
>
>
>
> Best regards,
>
> Jack
>
>
>