You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "soumilshah1995 (via GitHub)" <gi...@apache.org> on 2023/12/08 18:14:13 UTC
[I] [SUPPORT] Seeking Assistance with Hudi Integration Issue in Spark Thrift Server [hudi]
soumilshah1995 opened a new issue, #10287:
URL: https://github.com/apache/hudi/issues/10287
Hey community,
I hope you're doing well. I recently launched a Thrift server using Spark, incorporating the Hudi library. The server runs smoothly, and I can interact with it using Beeline to query data successfully.
```
spark-submit \
--master 'local[*]' \
--conf spark.executor.extraJavaOptions=-Duser.timezone=Etc/UTC \
--conf spark.eventLog.enabled=false \
--conf spark.sql.warehouse.dir=file:///Users/soumilshah/Desktop/soumil/sparkwarehouse \
--packages 'org.apache.hudi:hudi-spark3-bundle_2.12:0.14.0,org.apache.spark:spark-sql_2.12:3.4.0,org.apache.spark:spark-hive_2.12:3.4.0' \
--class org.apache.spark.sql.hive.thriftserver.HiveThriftServer2 \
--name "Thrift JDBC/ODBC Server" \
--executor-memory 512m \
--conf spark.serializer=org.apache.spark.serializer.KryoSerializer \
--conf spark.sql.extensions=org.apache.spark.sql.hudi.HoodieSparkSessionExtension \
--conf spark.kryo.registrator=org.apache.spark.HoodieSparkKryoRegistrar
```
BEELINE
```
beeline -u jdbc:hive2://localhost:10000/default
```
on BEELINE
```
CREATE TABLE hudi_table (
ts BIGINT,
uuid STRING,
rider STRING,
driver STRING,
fare DOUBLE,
city STRING
) USING HUDI
PARTITIONED BY (city);
```
Works fine
INSerted data
```
INSERT INTO hudi_table
VALUES
(1695159649087,'334e26e9-8355-45cc-97c6-c31daf0df330','rider-A','driver-K',19.10,'san_francisco'),
(1695091554788,'e96c4396-3fad-413a-a942-4cb36106d721','rider-C','driver-M',27.70 ,'san_francisco'),
(1695046462179,'9909a8b1-2d15-4d3d-8ec9-efc48c536a00','rider-D','driver-L',33.90 ,'san_francisco'),
(1695332066204,'1dced545-862b-4ceb-8b43-d2a568f6616b','rider-E','driver-O',93.50,'san_francisco'),
(1695516137016,'e3cf430c-889d-4015-bc98-59bdce1e530c','rider-F','driver-P',34.15,'sao_paulo' ),
(1695376420876,'7a84095f-737f-40bc-b62f-6b69664712d2','rider-G','driver-Q',43.40 ,'sao_paulo' ),
(1695173887231,'3eeb61f7-c2b0-4636-99bd-5d7a5a1d2c04','rider-I','driver-S',41.06 ,'chennai' ),
(1695115999911,'c8abbe79-8d89-47ea-b4ce-4d224bae5bfa','rider-J','driver-T',17.85,'chennai');
```
Now when i am trying to connect with DBT or DBeaver
to run SQL query against i see following error
```
QL Error: org.apache.hive.service.cli.HiveSQLException: Error running query: java.util.concurrent.ExecutionException: org.apache.spark.SparkClassNotFoundException: [DATA_SOURCE_NOT_FOUND] Failed to find the data source: hudi. Please find packages at `https://spark.apache.org/third-party-projects.html`.
```
I have successfully created a table and inserted data into Hudi tables using Beeline. The problem arises when I try to interact with Hudi tables using tools like DBT or DB Ever.
Any insights or guidance on resolving this issue would be greatly appreciated! If you have any experience with integrating Hudi into Spark Thrift Server and overcoming similar challenges, your expertise would be invaluable.
Thanks in advance for your help!
Regards
Soumil
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [SUPPORT] Seeking Assistance with Hudi Integration Issue in Spark Thrift Server and DBT [hudi]
Posted by "soumilshah1995 (via GitHub)" <gi...@apache.org>.
soumilshah1995 commented on issue #10287:
URL: https://github.com/apache/hudi/issues/10287#issuecomment-1848612805
got the Thrift server and Apache Debry working followed all steps mention
https://github.com/apache/hudi/tree/master/hudi-examples/hudi-examples-dbt
```
dbt debug works fine as well
Here is issue dbt run get stucks for very long time
18:47:28 adapter type: spark
18:47:28 adapter version: 1.7.1
18:47:28 Configuration:
18:47:28 profiles.yml file [OK found and valid]
18:47:28 dbt_project.yml file [OK found and valid]
18:47:28 Required dependencies:
18:47:28 - git [OK found]
18:47:28 Connection:
18:47:28 host: localhost
18:47:28 port: 10000
18:47:28 cluster: None
18:47:28 endpoint: None
18:47:28 schema: hudi_examples_dbt
18:47:28 organization: 0
18:47:28 Registered adapter: spark=1.7.1
18:47:29 Connection test: [OK connection ok]
18:47:29 All checks passed!
(dbt-env) soumilshah@Soumils-MBP dbtspark % clear
(dbt-env) soumilshah@Soumils-MBP dbtspark % dbt run
18:47:34 Running with dbt=1.7.3
18:47:34 Registered adapter: spark=1.7.1
18:47:34 Unable to do partial parsing because profile has changed
18:47:35 Found 2 models, 6 tests, 0 sources, 0 exposures, 0 metrics, 439 macros, 0 groups, 0 semantic models
18:47:35
18:47:36 Concurrency: 1 threads (target='dev')
18:47:36
18:47:36 1 of 2 START sql incremental model hudi_examples_dbt.hudi_insert_table ......... [RUN]
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [SUPPORT] Seeking Assistance with Hudi Integration Issue in Spark Thrift Server and DBT [hudi]
Posted by "soumilshah1995 (via GitHub)" <gi...@apache.org>.
soumilshah1995 commented on issue #10287:
URL: https://github.com/apache/hudi/issues/10287#issuecomment-1847838799
previous test i used spark parquet and had launched thrift server using similar technique worked fine
Thrift sever
spark-submit \
--master 'local[*]' \
--conf spark.executor.extraJavaOptions=-Duser.timezone=Etc/UTC \
--conf spark.eventLog.enabled=false \
--conf spark.sql.warehouse.dir=file:///Users/soumilshah/Desktop/soumil/sparkwarehouse \
--packages 'org.apache.spark:spark-sql_2.12:3.4.0,org.apache.spark:spark-hive_2.12:3.4.0' \
--class org.apache.spark.sql.hive.thriftserver.HiveThriftServer2 \
--name "Thrift JDBC/ODBC Server" \
--executor-memory 512m
this was just using spark and parquet
{{ config(
materialized='incremental',
file_format='parquet',
incremental_strategy='insert_overwrite',
) }}
SELECT
date(d) AS id,
d AS full_date,
EXTRACT (YEAR FROM d) AS YEAR,
EXTRACT (WEEK FROM d) AS year_week,
EXTRACT (DAY FROM d) AS year_day,
EXTRACT (YEAR FROM d) AS fiscal_year,
EXTRACT (QUARTER FROM d) AS fiscal_qtr,
EXTRACT (MONTH FROM d) AS MONTH,
date_format(d, 'MMMM') AS month_name,
EXTRACT (DOW FROM d) AS week_day,
date_format(d, 'EEEE') AS day_name,
(CASE WHEN date_format(d, 'EEEE') NOT IN ('Sunday', 'Saturday') THEN 0 ELSE 1 END) AS day_is_weekday
FROM (SELECT EXPLODE(months) AS d FROM (SELECT SEQUENCE (TO_DATE('2000-01-01'), TO_DATE('2023-01-01'), INTERVAL 1 DAY) AS months))
DBT project.yml
name: 'sparkdbt'
version: '1.0.0'
config-version: 2
profile: 'sparkdbt'
model-paths: ["models"]
analysis-paths: ["analyses"]
test-paths: ["tests"]
seed-paths: ["seeds"]
macro-paths: ["macros"]
snapshot-paths: ["snapshots"]
clean-targets:
- "target"
- "dbt_packages"
models:
sparkdbt:
core:
+enabled: true
+materialized: table
dim_date:
+materialized: table
works fine i was able to see table and query it using beeline and dbeaver as well i am seeing issue with Hudi tables not sure what CONF i am missing here
I also have read examples provided https://github.com/apache/hudi/blob/master/hudi-examples/hudi-examples-dbt/README.md
could not get it to work for some reason
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [SUPPORT] Seeking Assistance with Hudi Integration Issue in Spark Thrift Server and DBT [hudi]
Posted by "soumilshah1995 (via GitHub)" <gi...@apache.org>.
soumilshah1995 closed issue #10287: [SUPPORT] Seeking Assistance with Hudi Integration Issue in Spark Thrift Server and DBT
URL: https://github.com/apache/hudi/issues/10287
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [SUPPORT] Seeking Assistance with Hudi Integration Issue in Spark Thrift Server and DBT [hudi]
Posted by "ad1happy2go (via GitHub)" <gi...@apache.org>.
ad1happy2go commented on issue #10287:
URL: https://github.com/apache/hudi/issues/10287#issuecomment-1849367305
@soumilshah1995 I saw this error before. Can you put your hudi bundle jars in the jars folder and set spark configurations in spark-defaults.yaml.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [SUPPORT] Seeking Assistance with Hudi Integration Issue in Spark Thrift Server and DBT [hudi]
Posted by "soumilshah1995 (via GitHub)" <gi...@apache.org>.
soumilshah1995 commented on issue #10287:
URL: https://github.com/apache/hudi/issues/10287#issuecomment-1868324146

will be creating YouTube videos for this which will help everyone
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [SUPPORT] Seeking Assistance with Hudi Integration Issue in Spark Thrift Server and DBT [hudi]
Posted by "soumilshah1995 (via GitHub)" <gi...@apache.org>.
soumilshah1995 commented on issue #10287:
URL: https://github.com/apache/hudi/issues/10287#issuecomment-1848711118
launched
```
spark-submit \
--master 'local[*]' \
--conf spark.executor.extraJavaOptions=-Duser.timezone=Etc/UTC \
--conf spark.eventLog.enabled=false \
--conf spark.sql.warehouse.dir=file:///Users/soumilshah/Desktop/soumil/dbt \
--class org.apache.spark.sql.hive.thriftserver.HiveThriftServer2 \
--name "Thrift JDBC/ODBC Server" \
--executor-memory 3g \
--conf spark.serializer=org.apache.spark.serializer.KryoSerializer \
--conf spark.sql.extensions=org.apache.spark.sql.hudi.HoodieSparkSessionExtension \
--conf spark.sql.catalog.spark_catalog=org.apache.spark.sql.hudi.catalog.HoodieCatalog \
--conf hive.metastore.warehouse.dir=/Users/soumilshah/Desktop/soumil/dbt \
--conf hive.metastore.schema.verification=false \
--conf datanucleus.schema.autoCreateAll=true \
--conf javax.jdo.option.ConnectionDriverName=org.apache.derby.jdbc.ClientDriver \
--conf 'javax.jdo.option.ConnectionURL=jdbc:derby://localhost:1527/MyDatabase;create=true'
```
Launched
```
$DERBY_HOME/bin/startNetworkServer -h localhost
```
## DBT Debug
```
dbt debug
21:36:47 Running with dbt=1.7.3
21:36:47 dbt version: 1.7.3
21:36:47 python version: 3.9.6
21:36:47 python path: /Users/soumilshah/IdeaProjects/mdbt/Youtube/dbt-env/bin/python
21:36:47 os info: macOS-13.6.1-x86_64-i386-64bit
21:36:47 Using profiles dir at /Users/soumilshah/.dbt
21:36:47 Using profiles.yml file at /Users/soumilshah/.dbt/profiles.yml
21:36:47 Using dbt_project.yml file at /Users/soumilshah/IdeaProjects/mdbt/Youtube/dbtspark/dbt_project.yml
21:36:47 adapter type: spark
21:36:47 adapter version: 1.7.1
21:36:47 Configuration:
21:36:47 profiles.yml file [OK found and valid]
21:36:47 dbt_project.yml file [OK found and valid]
21:36:47 Required dependencies:
21:36:47 - git [OK found]
21:36:47 Connection:
21:36:47 host: localhost
21:36:47 port: 10000
21:36:47 cluster: None
21:36:47 endpoint: None
21:36:47 schema: hudi_examples_dbt
21:36:47 organization: 0
21:36:47 Registered adapter: spark=1.7.1
21:36:48 Connection test: [OK connection ok]
21:36:48 All checks passed!
(dbt-env) soumilshah@Soumils-MBP dbtspark %
```
after running dbt run terminal crashes

--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [SUPPORT] Seeking Assistance with Hudi Integration Issue in Spark Thrift Server and DBT [hudi]
Posted by "soumilshah1995 (via GitHub)" <gi...@apache.org>.
soumilshah1995 commented on issue #10287:
URL: https://github.com/apache/hudi/issues/10287#issuecomment-1850094087
I explored two pathways in addressing this challenge. The first route involved initiating the process using pure Hive and Spark SQL, coupled with Thrift Server. However, when attempting to run dbt, I encountered a specific issue.
The second route, a more intricate and time-consuming approach, required the installation of Apache Derby and Spark. Despite my efforts, an odd complication arose: when executing "dbt run," it led to the unexpected crash of both Thrift Server and Apache Derby.
## Step 1: Create DBT Environment
```bash
# Create virtual environment for DBT
python -m venv dbt-env
source dbt-env/bin/activate
# Install required packages
pip install dbt-core
pip install dbt-spark
pip install 'dbt-spark[PyHive]'
# Navigate to DBT directory
cd ~/.dbt/
# Set Java environment variable
export JAVA_HOME=/opt/homebrew/Cellar/openjdk@11/11.0.21/libexec/openjdk.jdk/Contents/Home
# Step 2: Download and Run Apache Derby
```
install Apache Derby
export DERBY_VERSION=10.14.2.0
curl -O https://archive.apache.org/dist/db/derby/db-derby-$DERBY_VERSION/db-derby-$DERBY_VERSION-bin.tar.gz -P /opt/
tar -xf db-derby-$DERBY_VERSION-bin.tar.gz
export DERBY_HOME=/Users/soumilshah/Desktop/soumil/dbt/db-derby-10.14.2.0-bin
echo $DERBY_HOME
rm -r db-derby-10.14.2.0-bin.tar.gz
$DERBY_HOME/bin/startNetworkServer -h localhost
```
Step 3: Install Apache Spark
```
# Specify Spark version
export SPARK_VERSION=3.2.3
# Download and extract Spark
curl -O https://archive.apache.org/dist/spark/spark-$SPARK_VERSION/spark-$SPARK_VERSION-bin-hadoop2.7.tgz
tar -xf spark-$SPARK_VERSION-bin-hadoop2.7.tgz
# Set Spark home
export SPARK_HOME=/Users/soumilshah/Desktop/soumil/dbt/spark-3.2.3-bin-hadoop2.7
echo $SPARK_HOME
# Clean up downloaded files
rm spark-3.2.3-bin-hadoop2.7.tgz
```
Step 4: Copy JAR files
```
# Copy JAR files to Spark JARS directory
cp /Users/soumilshah/Desktop/myjar/*.jar $SPARK_HOME/jars/
```
Step 5: Spark Submit Configuration
```
# Submit Spark job
spark-submit \
--master 'local[*]' \
--conf spark.executor.extraJavaOptions=-Duser.timezone=Etc/UTC \
--conf spark.eventLog.enabled=false \
--conf spark.sql.warehouse.dir=file:///Users/soumilshah/Desktop/soumil/dbt \
--class org.apache.spark.sql.hive.thriftserver.HiveThriftServer2 \
--packages 'org.apache.spark:spark-sql_2.12:3.2.3,org.apache.spark:spark-hive_2.12:3.2.3,org.apache.hudi:hudi-spark3.2-bundle_2.12:0.14.0' \
--name "Thrift JDBC/ODBC Server" \
--executor-memory 5g \
--conf spark.serializer=org.apache.spark.serializer.KryoSerializer \
--conf spark.sql.extensions=org.apache.spark.sql.hudi.HoodieSparkSessionExtension \
--conf spark.sql.catalog.spark_catalog=org.apache.spark.sql.hudi.catalog.HoodieCatalog \
--conf hive.metastore.warehouse.dir=/Users/soumilshah/Desktop/soumil/dbt \
--conf hive.metastore.schema.verification=false \
--conf datanucleus.schema.autoCreateAll=true \
--conf javax.jdo.option.ConnectionDriverName=org.apache.derby.jdbc.ClientDriver \
--conf 'javax.jdo.option.ConnectionURL=jdbc:derby://localhost:1527/MyDatabase;create=true'
```
Question why do I have to use apache derby I mean when I am simply using Spark SQL and hive server I am able to create Hudi Tables through beeline why does it fails on dbt run
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org