You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@sedona.apache.org by "Jean-Denis Giguère (Jira)" <ji...@apache.org> on 2022/01/26 00:52:00 UTC
[jira] [Commented] (SEDONA-79) Pure SQL does not work with Thrift Server
[ https://issues.apache.org/jira/browse/SEDONA-79?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17482163#comment-17482163 ]
Jean-Denis Giguère commented on SEDONA-79:
------------------------------------------
Based on a propositon of Paweł Kociński @Imbruced on [https://gitter.im/apache/sedona?,] I find an alternative way to create the thrift server. The reference proposed by Paweł is [https://geoatscale.com/article?id=b5b1fd23-87b5-4369-8089-3ea1c371a829] (redaction in progress).
This is the scala code I used:
{code:java}
import org.apache.sedona.sql.utils.SedonaSQLRegistrator
import org.apache.sedona.viz.core.Serde.SedonaVizKryoRegistrator
import org.apache.spark.serializer.KryoSerializer
import org.apache.spark.sql.SparkSession
import org.apache.spark.sql.hive.thriftserver.HiveThriftServer2
object SedonaThrift {
def main(args: Array[String]): Unit = {
val sparkSession: SparkSession =
SparkSession
.builder()
.master("local[2]")
.config("spark.sql.hive.thriftServer.singleSession", "true")
.config("hive.server2.thrift.bind.host", "0.0.0.0")
.config("hive.server2.thrift.port", "10001")
.config("spark.serializer", classOf[KryoSerializer].getName)
.config("spark.kryo.registrator", classOf[SedonaVizKryoRegistrator].getName)
.appName("SedonaThrift")
.enableHiveSupport()
.getOrCreate()
SedonaSQLRegistrator.registerAll(sparkSession)
HiveThriftServer2.startWithContext(sparkSession.sqlContext)
}
}
{code}
> Pure SQL does not work with Thrift Server
> -----------------------------------------
>
> Key: SEDONA-79
> URL: https://issues.apache.org/jira/browse/SEDONA-79
> Project: Apache Sedona
> Issue Type: Bug
> Reporter: Jean-Denis Giguère
> Priority: Minor
> Labels: sql
>
> h2. Current situation
> When I add Sedona SQL support to the Thrift Server on a local Spark Cluster, a {{scala.ScalaReflectionException}} associated with the class {{org.locationtech.jts.geom.Geometry}} is triggered.
> {code:bash}
> /opt/spark/bin/spark-submit --class org.apache.spark.sql.hive.thriftserver.HiveThriftServer2 \
> --master $SPARK_MASTER \
> --hiveconf hive.server2.thrift.port=10001 --hiveconf hive.server2.thrift.bind.host=0.0.0.0 \
> --packages org.apache.sedona:sedona-python-adapter-3.0_2.12:1.1.1-incubating,org.apache.sedona:sedona-viz-3.0_2.12:1.1.1-incubating,org.datasyslab:geotools-wrapper:geotools-24.0 \
> --conf spark.serializer=org.apache.spark.serializer.KryoSerializer \
> --conf spark.kryo.registrator=org.apache.sedona.viz.core.Serde.SedonaVizKryoRegistrator \
> --conf spark.sql.extensions=org.apache.sedona.viz.sql.SedonaVizExtensions,org.apache.sedona.sql.SedonaSqlExtensions
> {code}
> The error is not triggered when I launch the server, but when I attempt to connect to it with the Apache Hive JDBC driver.
> This error is not present when I use {{spark-shell}} with the same parameter.
> h2. Expected situation
> I would expect to be able to start the thrift server to expose sedona SQL on a standalone cluster.
> h2. Logs
> {noformat}
> docker-compose up spark-thrift
> WARNING: The Docker Engine you're using is running in swarm mode.
> Compose does not use swarm mode to deploy services to multiple nodes in a swarm. All containers will be scheduled on the current node.
> To deploy your application across the swarm, use `docker stack deploy`.
> spark-dbt_spark-master_1 is up-to-date
> spark-dbt_spark-worker-b_1 is up-to-date
> spark-dbt_spark-worker-a_1 is up-to-date
> Starting spark-dbt_spark-thrift_1 ... done
> Attaching to spark-dbt_spark-thrift_1
> spark-thrift_1 | WARNING: An illegal reflective access operation has occurred
> spark-thrift_1 | WARNING: Illegal reflective access by org.apache.spark.unsafe.Platform (file:/opt/spark/jars/spark-unsafe_2.12-3.0.2.jar) to constructor java.nio.DirectByteBuffer(long,int)
> spark-thrift_1 | WARNING: Please consider reporting this to the maintainers of org.apache.spark.unsafe.Platform
> spark-thrift_1 | WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
> spark-thrift_1 | WARNING: All illegal access operations will be denied in a future release
> spark-thrift_1 | Ivy Default Cache set to: /root/.ivy2/cache
> spark-thrift_1 | The jars for the packages stored in: /root/.ivy2/jars
> spark-thrift_1 | :: loading settings :: url = jar:file:/opt/spark/jars/ivy-2.4.0.jar!/org/apache/ivy/core/settings/ivysettings.xml
> spark-thrift_1 | org.apache.sedona#sedona-python-adapter-3.0_2.12 added as a dependency
> spark-thrift_1 | org.apache.sedona#sedona-viz-3.0_2.12 added as a dependency
> spark-thrift_1 | org.datasyslab#geotools-wrapper added as a dependency
> spark-thrift_1 | :: resolving dependencies :: org.apache.spark#spark-submit-parent-56248f53-61ad-4c72-bd1f-c13739ab99c0;1.0
> spark-thrift_1 | confs: [default]
> spark-thrift_1 | found org.apache.sedona#sedona-python-adapter-3.0_2.12;1.1.1-incubating in central
> spark-thrift_1 | found org.locationtech.jts#jts-core;1.18.0 in central
> spark-thrift_1 | found org.wololo#jts2geojson;0.16.1 in central
> spark-thrift_1 | found com.fasterxml.jackson.core#jackson-databind;2.12.2 in central
> spark-thrift_1 | found com.fasterxml.jackson.core#jackson-annotations;2.12.2 in central
> spark-thrift_1 | found com.fasterxml.jackson.core#jackson-core;2.12.2 in central
> spark-thrift_1 | found org.apache.sedona#sedona-core-3.0_2.12;1.1.1-incubating in central
> spark-thrift_1 | found org.scala-lang.modules#scala-collection-compat_2.12;2.5.0 in central
> spark-thrift_1 | found org.apache.sedona#sedona-sql-3.0_2.12;1.1.1-incubating in central
> spark-thrift_1 | found org.apache.sedona#sedona-viz-3.0_2.12;1.1.1-incubating in central
> spark-thrift_1 | found org.beryx#awt-color-factory;1.0.0 in central
> spark-thrift_1 | found org.datasyslab#geotools-wrapper;geotools-24.0 in central
> spark-thrift_1 | :: resolution report :: resolve 533ms :: artifacts dl 15ms
> spark-thrift_1 | :: modules in use:
> spark-thrift_1 | com.fasterxml.jackson.core#jackson-annotations;2.12.2 from central in [default]
> spark-thrift_1 | com.fasterxml.jackson.core#jackson-core;2.12.2 from central in [default]
> spark-thrift_1 | com.fasterxml.jackson.core#jackson-databind;2.12.2 from central in [default]
> spark-thrift_1 | org.apache.sedona#sedona-core-3.0_2.12;1.1.1-incubating from central in [default]
> spark-thrift_1 | org.apache.sedona#sedona-python-adapter-3.0_2.12;1.1.1-incubating from central in [default]
> spark-thrift_1 | org.apache.sedona#sedona-sql-3.0_2.12;1.1.1-incubating from central in [default]
> spark-thrift_1 | org.apache.sedona#sedona-viz-3.0_2.12;1.1.1-incubating from central in [default]
> spark-thrift_1 | org.beryx#awt-color-factory;1.0.0 from central in [default]
> spark-thrift_1 | org.datasyslab#geotools-wrapper;geotools-24.0 from central in [default]
> spark-thrift_1 | org.locationtech.jts#jts-core;1.18.0 from central in [default]
> spark-thrift_1 | org.scala-lang.modules#scala-collection-compat_2.12;2.5.0 from central in [default]
> spark-thrift_1 | org.wololo#jts2geojson;0.16.1 from central in [default]
> spark-thrift_1 | :: evicted modules:
> spark-thrift_1 | org.locationtech.jts#jts-core;1.18.1 by [org.locationtech.jts#jts-core;1.18.0] in [default]
> spark-thrift_1 | ---------------------------------------------------------------------
> spark-thrift_1 | | | modules || artifacts |
> spark-thrift_1 | | conf | number| search|dwnlded|evicted|| number|dwnlded|
> spark-thrift_1 | ---------------------------------------------------------------------
> spark-thrift_1 | | default | 13 | 0 | 0 | 1 || 12 | 0 |
> spark-thrift_1 | ---------------------------------------------------------------------
> spark-thrift_1 | :: retrieving :: org.apache.spark#spark-submit-parent-56248f53-61ad-4c72-bd1f-c13739ab99c0
> spark-thrift_1 | confs: [default]
> spark-thrift_1 | 0 artifacts copied, 12 already retrieved (0kB/13ms)
> spark-thrift_1 | 22/01/24 20:55:06 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
> spark-thrift_1 | Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties
> spark-thrift_1 | 22/01/24 20:55:06 INFO HiveThriftServer2: Started daemon with process name: 7@9d3fb159061a
> spark-thrift_1 | 22/01/24 20:55:06 INFO SignalUtils: Registered signal handler for TERM
> spark-thrift_1 | 22/01/24 20:55:06 INFO SignalUtils: Registered signal handler for HUP
> spark-thrift_1 | 22/01/24 20:55:06 INFO SignalUtils: Registered signal handler for INT
> spark-thrift_1 | 22/01/24 20:55:06 INFO HiveThriftServer2: Starting SparkContext
> spark-thrift_1 | 22/01/24 20:55:06 INFO HiveConf: Found configuration file null
> spark-thrift_1 | 22/01/24 20:55:07 INFO SparkContext: Running Spark version 3.0.2
> spark-thrift_1 | 22/01/24 20:55:07 INFO ResourceUtils: ==============================================================
> spark-thrift_1 | 22/01/24 20:55:07 INFO ResourceUtils: Resources for spark.driver:
> spark-thrift_1 |
> spark-thrift_1 | 22/01/24 20:55:07 INFO ResourceUtils: ==============================================================
> spark-thrift_1 | 22/01/24 20:55:07 INFO SparkContext: Submitted application: SparkSQL::172.28.0.5
> spark-thrift_1 | 22/01/24 20:55:07 INFO SecurityManager: Changing view acls to: root
> spark-thrift_1 | 22/01/24 20:55:07 INFO SecurityManager: Changing modify acls to: root
> spark-thrift_1 | 22/01/24 20:55:07 INFO SecurityManager: Changing view acls groups to:
> spark-thrift_1 | 22/01/24 20:55:07 INFO SecurityManager: Changing modify acls groups to:
> spark-thrift_1 | 22/01/24 20:55:07 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(root); groups with view permissions: Set(); users with modify permissions: Set(root); groups with modify permissions: Set()
> spark-thrift_1 | 22/01/24 20:55:07 INFO Utils: Successfully started service 'sparkDriver' on port 34759.
> spark-thrift_1 | 22/01/24 20:55:07 INFO SparkEnv: Registering MapOutputTracker
> spark-thrift_1 | 22/01/24 20:55:07 INFO SparkEnv: Registering BlockManagerMaster
> spark-thrift_1 | 22/01/24 20:55:07 INFO BlockManagerMasterEndpoint: Using org.apache.spark.storage.DefaultTopologyMapper for getting topology information
> spark-thrift_1 | 22/01/24 20:55:07 INFO BlockManagerMasterEndpoint: BlockManagerMasterEndpoint up
> spark-thrift_1 | 22/01/24 20:55:07 INFO SparkEnv: Registering BlockManagerMasterHeartbeat
> spark-thrift_1 | 22/01/24 20:55:07 INFO DiskBlockManager: Created local directory at /tmp/blockmgr-2f049248-da56-420b-b1ec-2c5fe1e924fc
> spark-thrift_1 | 22/01/24 20:55:07 INFO MemoryStore: MemoryStore started with capacity 434.4 MiB
> spark-thrift_1 | 22/01/24 20:55:07 INFO SparkEnv: Registering OutputCommitCoordinator
> spark-thrift_1 | 22/01/24 20:55:07 INFO Utils: Successfully started service 'SparkUI' on port 4040.
> spark-thrift_1 | 22/01/24 20:55:07 INFO SparkUI: Bound SparkUI to spark-thrift, and started at http://9d3fb159061a:4040
> spark-thrift_1 | 22/01/24 20:55:07 INFO SparkContext: Added JAR file:///root/.ivy2/jars/org.apache.sedona_sedona-python-adapter-3.0_2.12-1.1.1-incubating.jar at spark://9d3fb159061a:34759/jars/org.apache.sedona_sedona-python-adapter-3.0_2.12-1.1.1-incubating.jar with timestamp 1643057707087
> spark-thrift_1 | 22/01/24 20:55:07 INFO SparkContext: Added JAR file:///root/.ivy2/jars/org.apache.sedona_sedona-viz-3.0_2.12-1.1.1-incubating.jar at spark://9d3fb159061a:34759/jars/org.apache.sedona_sedona-viz-3.0_2.12-1.1.1-incubating.jar with timestamp 1643057707087
> spark-thrift_1 | 22/01/24 20:55:07 INFO SparkContext: Added JAR file:///root/.ivy2/jars/org.datasyslab_geotools-wrapper-geotools-24.0.jar at spark://9d3fb159061a:34759/jars/org.datasyslab_geotools-wrapper-geotools-24.0.jar with timestamp 1643057707087
> spark-thrift_1 | 22/01/24 20:55:07 INFO SparkContext: Added JAR file:///root/.ivy2/jars/org.locationtech.jts_jts-core-1.18.0.jar at spark://9d3fb159061a:34759/jars/org.locationtech.jts_jts-core-1.18.0.jar with timestamp 1643057707087
> spark-thrift_1 | 22/01/24 20:55:07 INFO SparkContext: Added JAR file:///root/.ivy2/jars/org.wololo_jts2geojson-0.16.1.jar at spark://9d3fb159061a:34759/jars/org.wololo_jts2geojson-0.16.1.jar with timestamp 1643057707087
> spark-thrift_1 | 22/01/24 20:55:07 INFO SparkContext: Added JAR file:///root/.ivy2/jars/org.apache.sedona_sedona-core-3.0_2.12-1.1.1-incubating.jar at spark://9d3fb159061a:34759/jars/org.apache.sedona_sedona-core-3.0_2.12-1.1.1-incubating.jar with timestamp 1643057707087
> spark-thrift_1 | 22/01/24 20:55:07 INFO SparkContext: Added JAR file:///root/.ivy2/jars/org.apache.sedona_sedona-sql-3.0_2.12-1.1.1-incubating.jar at spark://9d3fb159061a:34759/jars/org.apache.sedona_sedona-sql-3.0_2.12-1.1.1-incubating.jar with timestamp 1643057707087
> spark-thrift_1 | 22/01/24 20:55:07 INFO SparkContext: Added JAR file:///root/.ivy2/jars/org.scala-lang.modules_scala-collection-compat_2.12-2.5.0.jar at spark://9d3fb159061a:34759/jars/org.scala-lang.modules_scala-collection-compat_2.12-2.5.0.jar with timestamp 1643057707087
> spark-thrift_1 | 22/01/24 20:55:07 INFO SparkContext: Added JAR file:///root/.ivy2/jars/com.fasterxml.jackson.core_jackson-databind-2.12.2.jar at spark://9d3fb159061a:34759/jars/com.fasterxml.jackson.core_jackson-databind-2.12.2.jar with timestamp 1643057707087
> spark-thrift_1 | 22/01/24 20:55:07 INFO SparkContext: Added JAR file:///root/.ivy2/jars/com.fasterxml.jackson.core_jackson-annotations-2.12.2.jar at spark://9d3fb159061a:34759/jars/com.fasterxml.jackson.core_jackson-annotations-2.12.2.jar with timestamp 1643057707087
> spark-thrift_1 | 22/01/24 20:55:07 INFO SparkContext: Added JAR file:///root/.ivy2/jars/com.fasterxml.jackson.core_jackson-core-2.12.2.jar at spark://9d3fb159061a:34759/jars/com.fasterxml.jackson.core_jackson-core-2.12.2.jar with timestamp 1643057707087
> spark-thrift_1 | 22/01/24 20:55:07 INFO SparkContext: Added JAR file:///root/.ivy2/jars/org.beryx_awt-color-factory-1.0.0.jar at spark://9d3fb159061a:34759/jars/org.beryx_awt-color-factory-1.0.0.jar with timestamp 1643057707087
> spark-thrift_1 | 22/01/24 20:55:07 WARN SparkContext: Please ensure that the number of slots available on your executors is limited by the number of cores to task cpus and not another custom resource. If cores is not the limiting resource then dynamic allocation will not work properly!
> spark-thrift_1 | 22/01/24 20:55:07 INFO StandaloneAppClient$ClientEndpoint: Connecting to master spark://spark-master:7077...
> spark-thrift_1 | 22/01/24 20:55:07 INFO TransportClientFactory: Successfully created connection to spark-master/172.28.0.2:7077 after 24 ms (0 ms spent in bootstraps)
> spark-thrift_1 | 22/01/24 20:55:08 INFO StandaloneSchedulerBackend: Connected to Spark cluster with app ID app-20220124205508-0004
> spark-thrift_1 | 22/01/24 20:55:08 INFO StandaloneAppClient$ClientEndpoint: Executor added: app-20220124205508-0004/0 on worker-20220124201503-172.28.0.3-7000 (172.28.0.3:7000) with 2 core(s)
> spark-thrift_1 | 22/01/24 20:55:08 INFO StandaloneSchedulerBackend: Granted executor ID app-20220124205508-0004/0 on hostPort 172.28.0.3:7000 with 2 core(s), 1024.0 MiB RAM
> spark-thrift_1 | 22/01/24 20:55:08 INFO StandaloneAppClient$ClientEndpoint: Executor added: app-20220124205508-0004/1 on worker-20220124201513-172.28.0.4-7000 (172.28.0.4:7000) with 2 core(s)
> spark-thrift_1 | 22/01/24 20:55:08 INFO StandaloneSchedulerBackend: Granted executor ID app-20220124205508-0004/1 on hostPort 172.28.0.4:7000 with 2 core(s), 1024.0 MiB RAM
> spark-thrift_1 | 22/01/24 20:55:08 INFO Utils: Successfully started service 'org.apache.spark.network.netty.NettyBlockTransferService' on port 39909.
> spark-thrift_1 | 22/01/24 20:55:08 INFO NettyBlockTransferService: Server created on 9d3fb159061a:39909
> spark-thrift_1 | 22/01/24 20:55:08 INFO BlockManager: Using org.apache.spark.storage.RandomBlockReplicationPolicy for block replication policy
> spark-thrift_1 | 22/01/24 20:55:08 INFO BlockManagerMaster: Registering BlockManager BlockManagerId(driver, 9d3fb159061a, 39909, None)
> spark-thrift_1 | 22/01/24 20:55:08 INFO BlockManagerMasterEndpoint: Registering block manager 9d3fb159061a:39909 with 434.4 MiB RAM, BlockManagerId(driver, 9d3fb159061a, 39909, None)
> spark-thrift_1 | 22/01/24 20:55:08 INFO BlockManagerMaster: Registered BlockManager BlockManagerId(driver, 9d3fb159061a, 39909, None)
> spark-thrift_1 | 22/01/24 20:55:08 INFO BlockManager: Initialized BlockManager: BlockManagerId(driver, 9d3fb159061a, 39909, None)
> spark-thrift_1 | 22/01/24 20:55:08 INFO StandaloneAppClient$ClientEndpoint: Executor updated: app-20220124205508-0004/1 is now RUNNING
> spark-thrift_1 | 22/01/24 20:55:08 INFO StandaloneAppClient$ClientEndpoint: Executor updated: app-20220124205508-0004/0 is now RUNNING
> spark-thrift_1 | 22/01/24 20:55:08 INFO StandaloneSchedulerBackend: SchedulerBackend is ready for scheduling beginning after reached minRegisteredResourcesRatio: 0.0
> spark-thrift_1 | 22/01/24 20:55:08 INFO SharedState: Setting hive.metastore.warehouse.dir ('null') to the value of spark.sql.warehouse.dir ('file:/opt/spark/spark-warehouse').
> spark-thrift_1 | 22/01/24 20:55:08 INFO SharedState: Warehouse path is 'file:/opt/spark/spark-warehouse'.
> spark-thrift_1 | 22/01/24 20:55:08 INFO HiveUtils: Initializing HiveMetastoreConnection version 2.3.7 using Spark classes.
> spark-thrift_1 | 22/01/24 20:55:09 INFO SessionState: Created HDFS directory: /tmp/hive/root/9a5cefa7-7ee5-46cc-a7ff-2b8bb4eb7ac2
> spark-thrift_1 | 22/01/24 20:55:09 INFO SessionState: Created local directory: /tmp/root/9a5cefa7-7ee5-46cc-a7ff-2b8bb4eb7ac2
> spark-thrift_1 | 22/01/24 20:55:09 INFO SessionState: Created HDFS directory: /tmp/hive/root/9a5cefa7-7ee5-46cc-a7ff-2b8bb4eb7ac2/_tmp_space.db
> spark-thrift_1 | 22/01/24 20:55:09 INFO HiveClientImpl: Warehouse location for Hive client (version 2.3.7) is file:/opt/spark/spark-warehouse
> spark-thrift_1 | 22/01/24 20:55:10 WARN HiveConf: HiveConf of name hive.stats.jdbc.timeout does not exist
> spark-thrift_1 | 22/01/24 20:55:10 WARN HiveConf: HiveConf of name hive.stats.retries.wait does not exist
> spark-thrift_1 | 22/01/24 20:55:10 INFO HiveMetaStore: 0: Opening raw store with implementation class:org.apache.hadoop.hive.metastore.ObjectStore
> spark-thrift_1 | 22/01/24 20:55:10 INFO ObjectStore: ObjectStore, initialize called
> spark-thrift_1 | 22/01/24 20:55:10 INFO Persistence: Property hive.metastore.integral.jdo.pushdown unknown - will be ignored
> spark-thrift_1 | 22/01/24 20:55:10 INFO Persistence: Property datanucleus.cache.level2 unknown - will be ignored
> spark-thrift_1 | 22/01/24 20:55:10 INFO ResourceProfile: Default ResourceProfile created, executor resources: Map(cores -> name: cores, amount: 1, script: , vendor: , memory -> name: memory, amount: 1024, script: , vendor: ), task resources: Map(cpus -> name: cpus, amount: 1.0)
> spark-thrift_1 | 22/01/24 20:55:11 INFO CoarseGrainedSchedulerBackend$DriverEndpoint: Registered executor NettyRpcEndpointRef(spark-client://Executor) (172.28.0.4:46986) with ID 1
> spark-thrift_1 | 22/01/24 20:55:11 INFO CoarseGrainedSchedulerBackend$DriverEndpoint: Registered executor NettyRpcEndpointRef(spark-client://Executor) (172.28.0.3:48170) with ID 0
> spark-thrift_1 | 22/01/24 20:55:11 INFO BlockManagerMasterEndpoint: Registering block manager 172.28.0.4:34799 with 434.4 MiB RAM, BlockManagerId(1, 172.28.0.4, 34799, None)
> spark-thrift_1 | 22/01/24 20:55:11 INFO BlockManagerMasterEndpoint: Registering block manager 172.28.0.3:34961 with 434.4 MiB RAM, BlockManagerId(0, 172.28.0.3, 34961, None)
> spark-thrift_1 | 22/01/24 20:55:12 INFO ObjectStore: Setting MetaStore object pin classes with hive.metastore.cache.pinobjtypes="Table,StorageDescriptor,SerDeInfo,Partition,Database,Type,FieldSchema,Order"
> spark-thrift_1 | 22/01/24 20:55:14 INFO MetaStoreDirectSql: Using direct SQL, underlying DB is DERBY
> spark-thrift_1 | 22/01/24 20:55:14 INFO ObjectStore: Initialized ObjectStore
> spark-thrift_1 | 22/01/24 20:55:14 WARN ObjectStore: Version information not found in metastore. hive.metastore.schema.verification is not enabled so recording the schema version 2.3.0
> spark-thrift_1 | 22/01/24 20:55:14 WARN ObjectStore: setMetaStoreSchemaVersion called but recording version is disabled: version = 2.3.0, comment = Set by MetaStore UNKNOWN@172.28.0.5
> spark-thrift_1 | 22/01/24 20:55:14 INFO HiveMetaStore: Added admin role in metastore
> spark-thrift_1 | 22/01/24 20:55:14 INFO HiveMetaStore: Added public role in metastore
> spark-thrift_1 | 22/01/24 20:55:14 INFO HiveMetaStore: No user is added in admin role, since config is empty
> spark-thrift_1 | 22/01/24 20:55:14 INFO HiveMetaStore: 0: get_all_functions
> spark-thrift_1 | 22/01/24 20:55:14 INFO audit: ugi=root ip=unknown-ip-addr cmd=get_all_functions
> spark-thrift_1 | 22/01/24 20:55:14 INFO HiveMetaStore: 0: get_database: default
> spark-thrift_1 | 22/01/24 20:55:14 INFO audit: ugi=root ip=unknown-ip-addr cmd=get_database: default
> spark-thrift_1 | 22/01/24 20:55:14 INFO HiveUtils: Initializing execution hive, version 2.3.7
> spark-thrift_1 | 22/01/24 20:55:14 INFO HiveClientImpl: Warehouse location for Hive client (version 2.3.7) is file:/opt/spark/spark-warehouse
> spark-thrift_1 | 22/01/24 20:55:14 INFO SessionManager: Operation log root directory is created: /tmp/root/operation_logs
> spark-thrift_1 | 22/01/24 20:55:14 INFO SessionManager: HiveServer2: Background operation thread pool size: 100
> spark-thrift_1 | 22/01/24 20:55:14 INFO SessionManager: HiveServer2: Background operation thread wait queue size: 100
> spark-thrift_1 | 22/01/24 20:55:14 INFO SessionManager: HiveServer2: Background operation thread keepalive time: 10 seconds
> spark-thrift_1 | 22/01/24 20:55:14 INFO AbstractService: Service:OperationManager is inited.
> spark-thrift_1 | 22/01/24 20:55:14 INFO AbstractService: Service:SessionManager is inited.
> spark-thrift_1 | 22/01/24 20:55:14 INFO AbstractService: Service: CLIService is inited.
> spark-thrift_1 | 22/01/24 20:55:14 INFO AbstractService: Service:ThriftBinaryCLIService is inited.
> spark-thrift_1 | 22/01/24 20:55:14 INFO AbstractService: Service: HiveServer2 is inited.
> spark-thrift_1 | 22/01/24 20:55:14 INFO AbstractService: Service:OperationManager is started.
> spark-thrift_1 | 22/01/24 20:55:14 INFO AbstractService: Service:SessionManager is started.
> spark-thrift_1 | 22/01/24 20:55:14 INFO AbstractService: Service:CLIService is started.
> spark-thrift_1 | 22/01/24 20:55:14 INFO ObjectStore: ObjectStore, initialize called
> spark-thrift_1 | 22/01/24 20:55:14 INFO MetaStoreDirectSql: Using direct SQL, underlying DB is DERBY
> spark-thrift_1 | 22/01/24 20:55:14 INFO ObjectStore: Initialized ObjectStore
> spark-thrift_1 | 22/01/24 20:55:14 INFO HiveMetaStore: 0: get_databases: default
> spark-thrift_1 | 22/01/24 20:55:14 INFO audit: ugi=root ip=unknown-ip-addr cmd=get_databases: default
> spark-thrift_1 | 22/01/24 20:55:14 INFO HiveMetaStore: 0: Cleaning up thread local RawStore...
> spark-thrift_1 | 22/01/24 20:55:14 INFO audit: ugi=root ip=unknown-ip-addr cmd=Cleaning up thread local RawStore...
> spark-thrift_1 | 22/01/24 20:55:14 INFO HiveMetaStore: 0: Done cleaning up thread local RawStore
> spark-thrift_1 | 22/01/24 20:55:14 INFO audit: ugi=root ip=unknown-ip-addr cmd=Done cleaning up thread local RawStore
> spark-thrift_1 | 22/01/24 20:55:14 INFO AbstractService: Service:ThriftBinaryCLIService is started.
> spark-thrift_1 | 22/01/24 20:55:14 INFO AbstractService: Service:HiveServer2 is started.
> spark-thrift_1 | 22/01/24 20:55:14 INFO HiveThriftServer2: HiveThriftServer2 started
> spark-thrift_1 | 22/01/24 20:55:14 INFO ThriftCLIService: Starting ThriftBinaryCLIService on port 10001 with 5...500 worker threads
> spark-thrift_1 | 22/01/24 20:56:24 INFO ThriftCLIService: Client protocol version: HIVE_CLI_SERVICE_PROTOCOL_V8
> spark-thrift_1 | 22/01/24 20:56:24 INFO SessionState: Created HDFS directory: /tmp/hive/anonymous/77e79a4b-6753-4d2a-8d46-b1af590f3ab2
> spark-thrift_1 | 22/01/24 20:56:24 INFO SessionState: Created local directory: /tmp/root/77e79a4b-6753-4d2a-8d46-b1af590f3ab2
> spark-thrift_1 | 22/01/24 20:56:24 INFO SessionState: Created HDFS directory: /tmp/hive/anonymous/77e79a4b-6753-4d2a-8d46-b1af590f3ab2/_tmp_space.db
> spark-thrift_1 | 22/01/24 20:56:24 INFO HiveSessionImpl: Operation log session directory is created: /tmp/root/operation_logs/77e79a4b-6753-4d2a-8d46-b1af590f3ab2
> spark-thrift_1 | Exception in thread "HiveServer2-Handler-Pool: Thread-87" java.lang.ExceptionInInitializerError
> spark-thrift_1 | at org.apache.sedona.sql.UDF.UdfRegistrator$.registerAll(UdfRegistrator.scala:32)
> spark-thrift_1 | at org.apache.sedona.sql.utils.SedonaSQLRegistrator$.registerAll(SedonaSQLRegistrator.scala:34)
> spark-thrift_1 | at org.apache.sedona.sql.SedonaSqlExtensions.$anonfun$apply$1(SedonaSqlExtensions.scala:28)
> spark-thrift_1 | at org.apache.spark.sql.SparkSessionExtensions.$anonfun$buildCheckRules$1(SparkSessionExtensions.scala:174)
> spark-thrift_1 | at scala.collection.TraversableLike.$anonfun$map$1(TraversableLike.scala:238)
> spark-thrift_1 | at scala.collection.mutable.ResizableArray.foreach(ResizableArray.scala:62)
> spark-thrift_1 | at scala.collection.mutable.ResizableArray.foreach$(ResizableArray.scala:55)
> spark-thrift_1 | at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:49)
> spark-thrift_1 | at scala.collection.TraversableLike.map(TraversableLike.scala:238)
> spark-thrift_1 | at scala.collection.TraversableLike.map$(TraversableLike.scala:231)
> spark-thrift_1 | at scala.collection.AbstractTraversable.map(Traversable.scala:108)
> spark-thrift_1 | at org.apache.spark.sql.SparkSessionExtensions.buildCheckRules(SparkSessionExtensions.scala:174)
> spark-thrift_1 | at org.apache.spark.sql.internal.BaseSessionStateBuilder.customCheckRules(BaseSessionStateBuilder.scala:227)
> spark-thrift_1 | at org.apache.spark.sql.hive.HiveSessionStateBuilder$$anon$1.<init>(HiveSessionStateBuilder.scala:100)
> spark-thrift_1 | at org.apache.spark.sql.hive.HiveSessionStateBuilder.analyzer(HiveSessionStateBuilder.scala:74)
> spark-thrift_1 | at org.apache.spark.sql.internal.BaseSessionStateBuilder.$anonfun$build$2(BaseSessionStateBuilder.scala:336)
> spark-thrift_1 | at org.apache.spark.sql.internal.SessionState.analyzer$lzycompute(SessionState.scala:83)
> spark-thrift_1 | at org.apache.spark.sql.internal.SessionState.analyzer(SessionState.scala:83)
> spark-thrift_1 | at org.apache.spark.sql.execution.QueryExecution.$anonfun$analyzed$1(QueryExecution.scala:68)
> spark-thrift_1 | at org.apache.spark.sql.catalyst.QueryPlanningTracker.measurePhase(QueryPlanningTracker.scala:111)
> spark-thrift_1 | at org.apache.spark.sql.execution.QueryExecution.$anonfun$executePhase$1(QueryExecution.scala:138)
> spark-thrift_1 | at org.apache.spark.sql.SparkSession.withActive(SparkSession.scala:764)
> spark-thrift_1 | at org.apache.spark.sql.execution.QueryExecution.executePhase(QueryExecution.scala:138)
> spark-thrift_1 | at org.apache.spark.sql.execution.QueryExecution.analyzed$lzycompute(QueryExecution.scala:68)
> spark-thrift_1 | at org.apache.spark.sql.execution.QueryExecution.analyzed(QueryExecution.scala:66)
> spark-thrift_1 | at org.apache.spark.sql.execution.QueryExecution.assertAnalyzed(QueryExecution.scala:58)
> spark-thrift_1 | at org.apache.spark.sql.Dataset$.$anonfun$ofRows$2(Dataset.scala:99)
> spark-thrift_1 | at org.apache.spark.sql.SparkSession.withActive(SparkSession.scala:764)
> spark-thrift_1 | at org.apache.spark.sql.Dataset$.ofRows(Dataset.scala:97)
> spark-thrift_1 | at org.apache.spark.sql.SparkSession.$anonfun$sql$1(SparkSession.scala:607)
> spark-thrift_1 | at org.apache.spark.sql.SparkSession.withActive(SparkSession.scala:764)
> spark-thrift_1 | at org.apache.spark.sql.SparkSession.sql(SparkSession.scala:602)
> spark-thrift_1 | at org.apache.spark.sql.SQLContext.sql(SQLContext.scala:650)
> spark-thrift_1 | at org.apache.spark.sql.hive.thriftserver.SparkSQLSessionManager.openSession(SparkSQLSessionManager.scala:70)
> spark-thrift_1 | at org.apache.hive.service.cli.CLIService.openSessionWithImpersonation(CLIService.java:203)
> spark-thrift_1 | at org.apache.hive.service.cli.thrift.ThriftCLIService.getSessionHandle(ThriftCLIService.java:353)
> spark-thrift_1 | at org.apache.hive.service.cli.thrift.ThriftCLIService.OpenSession(ThriftCLIService.java:247)
> spark-thrift_1 | at org.apache.hive.service.rpc.thrift.TCLIService$Processor$OpenSession.getResult(TCLIService.java:1377)
> spark-thrift_1 | at org.apache.hive.service.rpc.thrift.TCLIService$Processor$OpenSession.getResult(TCLIService.java:1362)
> spark-thrift_1 | at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:38)
> spark-thrift_1 | at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
> spark-thrift_1 | at org.apache.hive.service.auth.TSetIpAddressProcessor.process(TSetIpAddressProcessor.java:53)
> spark-thrift_1 | at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:310)
> spark-thrift_1 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
> spark-thrift_1 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
> spark-thrift_1 | at java.base/java.lang.Thread.run(Unknown Source)
> spark-thrift_1 | Caused by: scala.ScalaReflectionException: class org.locationtech.jts.geom.Geometry in JavaMirror with org.apache.hadoop.hive.ql.exec.UDFClassLoader@5a71c867 of type class org.apache.hadoop.hive.ql.exec.UDFClassLoader with classpath [] and parent being jdk.internal.loader.ClassLoaders$AppClassLoader@3c153a1 of type class jdk.internal.loader.ClassLoaders$AppClassLoader with classpath [<unknown>] and parent being jdk.internal.loader.ClassLoaders$PlatformClassLoader@435dacd9 of type class jdk.internal.loader.ClassLoaders$PlatformClassLoader with classpath [<unknown>] and parent being primordial classloader with boot classpath [<unknown>] not found.
> spark-thrift_1 | at scala.reflect.internal.Mirrors$RootsBase.staticClass(Mirrors.scala:146)
> spark-thrift_1 | at scala.reflect.internal.Mirrors$RootsBase.staticClass(Mirrors.scala:29)
> spark-thrift_1 | at org.apache.spark.sql.sedona_sql.expressions.TraitSTAggregateExec$$typecreator1$1.apply(AggregateFunctions.scala:42)
> spark-thrift_1 | at scala.reflect.api.TypeTags$WeakTypeTagImpl.tpe$lzycompute(TypeTags.scala:237)
> spark-thrift_1 | at scala.reflect.api.TypeTags$WeakTypeTagImpl.tpe(TypeTags.scala:237)
> spark-thrift_1 | at org.apache.spark.sql.catalyst.encoders.ExpressionEncoder$.apply(ExpressionEncoder.scala:54)
> spark-thrift_1 | at org.apache.spark.sql.sedona_sql.expressions.TraitSTAggregateExec.$init$(AggregateFunctions.scala:42)
> spark-thrift_1 | at org.apache.spark.sql.sedona_sql.expressions.ST_Union_Aggr.<init>(AggregateFunctions.scala:56)
> spark-thrift_1 | at org.apache.sedona.sql.UDF.Catalog$.<init>(Catalog.scala:126)
> spark-thrift_1 | at org.apache.sedona.sql.UDF.Catalog$.<clinit>(Catalog.scala)
> spark-thrift_1 | ... 46 more
> {noformat}
--
This message was sent by Atlassian Jira
(v8.20.1#820001)