Re: Spark sql query the hive external table mapped from phoenix always throw out Class org.apache.phoenix.hive.PhoenixSerDe not found exception

[Josh Elser <el...@apache.org>]

(Moving this over to the user list as that's the appropriate list for 
this question)

Do you get an error? We can't help you with only a "it didn't work" :)

I'd suggest that you try to narrow down the scope of the problem: is it 
unique to Hive external tables? Can you use a different Hive 
StorageHandler successfully (e.g. the HBaseStorageHandler)?

Finally, as you're using HDP, please also consider using their customer 
support.

On 7/11/19 2:18 AM, 马士成 wrote:
> Hello All,
> 
> In Apache Phoenix homepage,  It shows two additional functions: Apache 
> Spark Integration and Phoenix Storage Handler for Apache Hive,
> 
> According the guidance, I can query phoenix table from beeline-cli, I 
> can load phoenix table as dataframe using Spark-sql.
> 
> So my question is :
> 
> Does Phoenix support spark-sql query the hive external table mapped from 
> Phoenix ?
> 
> I am working on hdp3.0 ( Phoenix 5.0 Hbase 2.0, Hive 3.1.0 ,Spark2.3.1 
>   )  and facing the issue as subject mentioned.
> 
> I tried to solve this problem but failed, I found some similar questions 
> on internet but the answers didn’t work for me.
> 
> My submit command :
> 
>    spark-submit test3.py --jars \
> 
>    /usr/hdp/current/phoenix-client/lib/phoenix-hive-5.0.0.3.0.0.0-1634.jar\
> 
>    ,/usr/hdp/current/phoenix-client/lib/hadoop-mapreduce-client-core.jar\
> 
>    ,/usr/hdp/current/phoenix-client/lib/phoenix-core-5.0.0.3.0.0.0-1634.jar\
> 
>    
> ,/usr/hdp/current/phoenix-client/lib/phoenix-spark-5.0.0.3.0.0.0-1634.jar\
> 
>    ,/usr/hdp/current/hive-client/lib/hive-metastore-3.1.0.3.0.0.0-1634.jar\
> 
>    ,/usr/hdp/current/hive-client/lib/hive-common-3.1.0.3.0.0.0-1634.jar\
> 
>    ,/usr/hdp/current/hive-client/lib/hbase-client-2.0.0.3.0.0.0-1634.jar\
> 
>    ,/usr/hdp/current/hive-client/lib/hbase-mapreduce-2.0.0.3.0.0.0-1634.jar\
> 
>    ,/usr/hdp/current/hive-client/lib/hive-serde-3.1.0.3.0.0.0-1634.jar\
> 
>    ,/usr/hdp/current/hive-client/lib/hive-shims-3.1.0.3.0.0.0-1634.jar
> 
> Log attached and Demo code as below:
> 
>    from pyspark.sql import SparkSession
> 
>    if __name__ == '__main__':
> 
>        spark = SparkSession.builder \
> 
>            .appName("test") \
> 
>            .enableHiveSupport() \
> 
>            .getOrCreate()
> 
>        df= spark.sql("select count(*) from ajmide_dw.part_device")
> 
>        df.show()
> 
> Similar Issues:
> 
> https://community.hortonworks.com/questions/140097/facing-issue-from-spark-sql.html
> 
> https://stackoverflow.com/questions/51501044/unable-to-access-hive-external-tables-from-spark-shell
> 
> Any comment or suggestion is appreciated!
> 
> Thanks,
> 
> Shi-Cheng, Ma
>