You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Gary Liu (Jira)" <ji...@apache.org> on 2021/12/10 19:49:00 UTC
[jira] [Updated] (SPARK-37608) Can anyone help me to compile a scala file
[ https://issues.apache.org/jira/browse/SPARK-37608?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Gary Liu updated SPARK-37608:
-----------------------------
Description:
I am working with spark SQL, trying to read data from SAS using JDBC. I have to use customized JdbcDialect to handle special format of the SAS data. Here is the code:
{code:java}
import org.apache.spark.sql.jdbc.JdbcDialectobject SasDialect extends JdbcDialect {
override def canHandle(url: String): Boolean = url.startsWith("jdbc:sharenet")
override def quoteIdentifier(colName: String): String = "\"" + colName + "\"n"
}
{code}
It worked well in Scala, but my major language is Python, so I need to compile this into jar file, so I can call it from python, like
{code:python}
from py4j.java_gateway import java_import
gw = spark.sparkContext._gateway
java_import(gw.jvm, "com.me.SasJDBCDialect")
gw.jvm.org.apache.spark.sql.jdbc.JdbcDialects.registerDialect(
gw.jvm.com.me.MyJDBCDialect()){code}
But I am a newbie of Scala, and don't know how to make it a jar file so it can be call as "com.me.MyJDBCDialect". Can anyone here do me a favour to generate a jar file for me?
Thanks!
was:
I am working with spark SQL, trying to read data from SAS using JDBC. I have to use customized JdbcDialect to handle special format of the SAS data. Here is the code:
{code:java}
import org.apache.spark.sql.jdbc.JdbcDialectobject SasDialect extends JdbcDialect { override def canHandle(url: String): Boolean = url.startsWith("jdbc:sharenet") override def quoteIdentifier(colName: String): String = "\"" + colName + "\"n" }
{code}
It worked well in Scala, but my major language is Python, so I need to compile this into jar file, so I can call it from python, like
{code:python}
from py4j.java_gateway import java_import
gw = spark.sparkContext._gateway
java_import(gw.jvm, "com.me.MyJDBCDialect")
gw.jvm.org.apache.spark.sql.jdbc.JdbcDialects.registerDialect(
gw.jvm.com.me.MyJDBCDialect()){code}
But I am a newbie of Scala, and don't know how to make it a jar file so it can be call as "com.me.MyJDBCDialect". Can anyone here do me a favour to generate a jar file for me?
Thanks!
> Can anyone help me to compile a scala file
> ------------------------------------------
>
> Key: SPARK-37608
> URL: https://issues.apache.org/jira/browse/SPARK-37608
> Project: Spark
> Issue Type: Question
> Components: SQL
> Affects Versions: 2.2.3
> Environment: I am currently using spark 2.3.2.3.1.5.0-152 and Scala 2.11.12
> Reporter: Gary Liu
> Priority: Major
>
> I am working with spark SQL, trying to read data from SAS using JDBC. I have to use customized JdbcDialect to handle special format of the SAS data. Here is the code:
> {code:java}
> import org.apache.spark.sql.jdbc.JdbcDialectobject SasDialect extends JdbcDialect {
> override def canHandle(url: String): Boolean = url.startsWith("jdbc:sharenet")
> override def quoteIdentifier(colName: String): String = "\"" + colName + "\"n"
> }
> {code}
> It worked well in Scala, but my major language is Python, so I need to compile this into jar file, so I can call it from python, like
> {code:python}
> from py4j.java_gateway import java_import
> gw = spark.sparkContext._gateway
> java_import(gw.jvm, "com.me.SasJDBCDialect")
> gw.jvm.org.apache.spark.sql.jdbc.JdbcDialects.registerDialect(
> gw.jvm.com.me.MyJDBCDialect()){code}
> But I am a newbie of Scala, and don't know how to make it a jar file so it can be call as "com.me.MyJDBCDialect". Can anyone here do me a favour to generate a jar file for me?
>
> Thanks!
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org