You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by Hafiz Mujadid <ha...@gmail.com> on 2015/05/01 22:56:23 UTC
empty jdbc RDD in spark
Hi all!
I am trying to read hana database using spark jdbc RDD
here is my code
def readFromHana() {
val conf = new SparkConf()
conf.setAppName("test").setMaster("local")
val sc = new SparkContext(conf)
val rdd = new JdbcRDD(sc, () => {
Class.forName("com.sap.db.jdbc.Driver").newInstance()
DriverManager.getConnection("jdbc:sap://54.69.200.113:30015/?currentschema=LIVE2",
"mujadid", "786Xyz123")
},
"SELECT * FROM MEMBERS LIMIT ? OFFSET ?",
0, 100, 1,
(r: ResultSet) => convert(r) )
println(rdd.count());
sc.stop()
}
def convert(rs: ResultSet):String={
val rsmd = rs.getMetaData()
val numberOfColumns = rsmd.getColumnCount()
var i = 1
val row=new StringBuilder
while (i <= numberOfColumns) {
row.append( rs.getString(i)+",")
i += 1
}
row.toString()
}
The resultant count is 0
Any suggestion?
Thanks
--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/empty-jdbc-RDD-in-spark-tp22736.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org
Re: empty jdbc RDD in spark
Posted by Cody Koeninger <co...@koeninger.org>.
https://spark.apache.org/docs/latest/api/scala/index.html#org.apache.spark.rdd.JdbcRDD
The arguments are sql string, lower bound, upper bound, number of
partitions.
Your call SELECT * FROM MEMBERS LIMIT ? OFFSET ?, 0, 100, 1
would thus be run as
SELECT * FROM MEMBERS LIMIT 0 OFFSET 100
Naturally limit 0 will yield 0 results.
JdbcRDD is designed to be used with multiple partitions, with some kind of
numeric index.
Try something more like
SELECT * FROM MEMBERS WHERE ID <= ? AND ID < ?, 0, howeverManyRowsYouHave, 8
On Fri, May 1, 2015 at 3:56 PM, Hafiz Mujadid <ha...@gmail.com>
wrote:
> Hi all!
> I am trying to read hana database using spark jdbc RDD
> here is my code
> def readFromHana() {
> val conf = new SparkConf()
> conf.setAppName("test").setMaster("local")
> val sc = new SparkContext(conf)
> val rdd = new JdbcRDD(sc, () => {
> Class.forName("com.sap.db.jdbc.Driver").newInstance()
>
> DriverManager.getConnection("jdbc:sap://
> 54.69.200.113:30015/?currentschema=LIVE2",
> "mujadid", "786Xyz123")
> },
> "SELECT * FROM MEMBERS LIMIT ? OFFSET ?",
> 0, 100, 1,
> (r: ResultSet) => convert(r) )
> println(rdd.count());
> sc.stop()
> }
> def convert(rs: ResultSet):String={
> val rsmd = rs.getMetaData()
> val numberOfColumns = rsmd.getColumnCount()
> var i = 1
> val row=new StringBuilder
> while (i <= numberOfColumns) {
> row.append( rs.getString(i)+",")
> i += 1
> }
> row.toString()
> }
>
> The resultant count is 0
>
> Any suggestion?
>
> Thanks
>
>
>
> --
> View this message in context:
> http://apache-spark-user-list.1001560.n3.nabble.com/empty-jdbc-RDD-in-spark-tp22736.html
> Sent from the Apache Spark User List mailing list archive at Nabble.com.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
> For additional commands, e-mail: user-help@spark.apache.org
>
>
Re: empty jdbc RDD in spark
Posted by Ted Yu <yu...@gmail.com>.
bq. "SELECT * FROM MEMBERS LIMIT ? OFFSET ?",
Have you tried dropping limit and offset clause from the above query ?
Cheers
On Fri, May 1, 2015 at 1:56 PM, Hafiz Mujadid <ha...@gmail.com>
wrote:
> Hi all!
> I am trying to read hana database using spark jdbc RDD
> here is my code
> def readFromHana() {
> val conf = new SparkConf()
> conf.setAppName("test").setMaster("local")
> val sc = new SparkContext(conf)
> val rdd = new JdbcRDD(sc, () => {
> Class.forName("com.sap.db.jdbc.Driver").newInstance()
>
> DriverManager.getConnection("jdbc:sap://
> 54.69.200.113:30015/?currentschema=LIVE2",
> "mujadid", "786Xyz123")
> },
> "SELECT * FROM MEMBERS LIMIT ? OFFSET ?",
> 0, 100, 1,
> (r: ResultSet) => convert(r) )
> println(rdd.count());
> sc.stop()
> }
> def convert(rs: ResultSet):String={
> val rsmd = rs.getMetaData()
> val numberOfColumns = rsmd.getColumnCount()
> var i = 1
> val row=new StringBuilder
> while (i <= numberOfColumns) {
> row.append( rs.getString(i)+",")
> i += 1
> }
> row.toString()
> }
>
> The resultant count is 0
>
> Any suggestion?
>
> Thanks
>
>
>
> --
> View this message in context:
> http://apache-spark-user-list.1001560.n3.nabble.com/empty-jdbc-RDD-in-spark-tp22736.html
> Sent from the Apache Spark User List mailing list archive at Nabble.com.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
> For additional commands, e-mail: user-help@spark.apache.org
>
>