[jira] [Updated] (SPARK-3991) Not Serializable , Nullpinter Exceptions in SQL server mode

["eblaas (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/SPARK-3991?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

eblaas updated SPARK-3991:
--------------------------
    Attachment: not_serializable_exception.patch

> Not Serializable , Nullpinter Exceptions in SQL server mode
> -----------------------------------------------------------
>
>                 Key: SPARK-3991
>                 URL: https://issues.apache.org/jira/browse/SPARK-3991
>             Project: Spark
>          Issue Type: Bug
>          Components: SQL
>    Affects Versions: 1.1.0
>            Reporter: eblaas
>            Priority: Blocker
>         Attachments: not_serializable_exception.patch
>
>
> I'm working on connecting Mondrian with Spark SQL via JDBC. Good news, it works but there are some bugs to fix.
> I customized the HiveThriftServer2 class to load, transform and register tables (ETL) with the HiveContext. Data tables are generated from Cassandra and from a relational database.
> * 1 st problem : 
> hiveContext.registerRDDAsTable(treeSchema,"tree") , does not register the table in hive metastore ("show tables;" via JDBC does not list the table, but I can query it e.g. select * from tree) dirty workaround create a table with same name and schema, this was necessary because mondrian validates table existence 
> hiveContext.sql("CREATE TABLE tree (dp_id BIGINT, h1 STRING, h2 STRING, h3 STRING)")
> * 2 nd problem :
> mondrian creates complex joins, witch results in Serialization Exceptions
> 2 classes in hibeUdfs.scala have to be serializable
> - DeferredObjectAdapter and HiveGenericUdaf
> * 3 td  problem
> Nullpointer Exception in InMemoryRelation
> 42: override lazy val statistics =  Statistics(sizeInBytes = child.sqlContext.defaultSizeInBytes)
> the sqlContext in child was null, quick fix set default value from SparkContext
> override lazy val statistics = Statistics(sizeInBytes = 10000)
> I'm not sure how to fix this bugs but with the patch file it works at least. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org