You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "soumilshah1995 (via GitHub)" <gi...@apache.org> on 2023/12/08 18:14:13 UTC

[I] [SUPPORT] Seeking Assistance with Hudi Integration Issue in Spark Thrift Server [hudi]

soumilshah1995 opened a new issue, #10287:
URL: https://github.com/apache/hudi/issues/10287

   Hey community,
   I hope you're doing well. I recently launched a Thrift server using Spark, incorporating the Hudi library. The server runs smoothly, and I can interact with it using Beeline to query data successfully.
   
   ```
   spark-submit \
     --master 'local[*]' \
     --conf spark.executor.extraJavaOptions=-Duser.timezone=Etc/UTC \
     --conf spark.eventLog.enabled=false \
     --conf spark.sql.warehouse.dir=file:///Users/soumilshah/Desktop/soumil/sparkwarehouse \
     --packages 'org.apache.hudi:hudi-spark3-bundle_2.12:0.14.0,org.apache.spark:spark-sql_2.12:3.4.0,org.apache.spark:spark-hive_2.12:3.4.0' \
     --class org.apache.spark.sql.hive.thriftserver.HiveThriftServer2 \
     --name "Thrift JDBC/ODBC Server" \
     --executor-memory 512m \
     --conf spark.serializer=org.apache.spark.serializer.KryoSerializer \
     --conf spark.sql.extensions=org.apache.spark.sql.hudi.HoodieSparkSessionExtension \
     --conf spark.kryo.registrator=org.apache.spark.HoodieSparkKryoRegistrar
   
   ```
   
   BEELINE
   ```
   beeline -u jdbc:hive2://localhost:10000/default
   ```
   
   on BEELINE
   
   ```
   CREATE TABLE hudi_table (
       ts BIGINT,
       uuid STRING,
       rider STRING,
       driver STRING,
       fare DOUBLE,
       city STRING
   ) USING HUDI
   PARTITIONED BY (city);
   ```
   
   Works fine
   
   INSerted data
   ```
   INSERT INTO hudi_table
   VALUES
   (1695159649087,'334e26e9-8355-45cc-97c6-c31daf0df330','rider-A','driver-K',19.10,'san_francisco'),
   (1695091554788,'e96c4396-3fad-413a-a942-4cb36106d721','rider-C','driver-M',27.70 ,'san_francisco'),
   (1695046462179,'9909a8b1-2d15-4d3d-8ec9-efc48c536a00','rider-D','driver-L',33.90 ,'san_francisco'),
   (1695332066204,'1dced545-862b-4ceb-8b43-d2a568f6616b','rider-E','driver-O',93.50,'san_francisco'),
   (1695516137016,'e3cf430c-889d-4015-bc98-59bdce1e530c','rider-F','driver-P',34.15,'sao_paulo'    ),
   (1695376420876,'7a84095f-737f-40bc-b62f-6b69664712d2','rider-G','driver-Q',43.40 ,'sao_paulo'    ),
   (1695173887231,'3eeb61f7-c2b0-4636-99bd-5d7a5a1d2c04','rider-I','driver-S',41.06 ,'chennai'      ),
   (1695115999911,'c8abbe79-8d89-47ea-b4ce-4d224bae5bfa','rider-J','driver-T',17.85,'chennai');
   ```
   
   
   Now when i am trying to connect with DBT or DBeaver
   to run SQL query against i see following error
   
   ```
   QL Error: org.apache.hive.service.cli.HiveSQLException: Error running query: java.util.concurrent.ExecutionException: org.apache.spark.SparkClassNotFoundException: [DATA_SOURCE_NOT_FOUND] Failed to find the data source: hudi. Please find packages at `https://spark.apache.org/third-party-projects.html`.
   ```
   
   
   I have successfully created a table and inserted data into Hudi tables using Beeline. The problem arises when I try to interact with Hudi tables using tools like DBT or DB Ever.
   
   Any insights or guidance on resolving this issue would be greatly appreciated! If you have any experience with integrating Hudi into Spark Thrift Server and overcoming similar challenges, your expertise would be invaluable.
   
   Thanks in advance for your help!
   
   Regards
   Soumil
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [SUPPORT] Seeking Assistance with Hudi Integration Issue in Spark Thrift Server and DBT [hudi]

Posted by "soumilshah1995 (via GitHub)" <gi...@apache.org>.
soumilshah1995 commented on issue #10287:
URL: https://github.com/apache/hudi/issues/10287#issuecomment-1848612805

    got the Thrift server and Apache Debry working followed all steps mention 
   
   https://github.com/apache/hudi/tree/master/hudi-examples/hudi-examples-dbt
   
   ```
   dbt debug works fine as well
   Here is issue dbt run get stucks for very long time
   18:47:28  adapter type: spark
   18:47:28  adapter version: 1.7.1
   18:47:28  Configuration:
   18:47:28    profiles.yml file [OK found and valid]
   18:47:28    dbt_project.yml file [OK found and valid]
   18:47:28  Required dependencies:
   18:47:28   - git [OK found]
   
   18:47:28  Connection:
   18:47:28    host: localhost
   18:47:28    port: 10000
   18:47:28    cluster: None
   18:47:28    endpoint: None
   18:47:28    schema: hudi_examples_dbt
   18:47:28    organization: 0
   18:47:28  Registered adapter: spark=1.7.1
   18:47:29    Connection test: [OK connection ok]
   
   18:47:29  All checks passed!
   (dbt-env) soumilshah@Soumils-MBP dbtspark % clear
   (dbt-env) soumilshah@Soumils-MBP dbtspark % dbt run
   18:47:34  Running with dbt=1.7.3
   18:47:34  Registered adapter: spark=1.7.1
   18:47:34  Unable to do partial parsing because profile has changed
   18:47:35  Found 2 models, 6 tests, 0 sources, 0 exposures, 0 metrics, 439 macros, 0 groups, 0 semantic models
   18:47:35  
   18:47:36  Concurrency: 1 threads (target='dev')
   18:47:36  
   18:47:36  1 of 2 START sql incremental model hudi_examples_dbt.hudi_insert_table ......... [RUN]
   
   ```


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [SUPPORT] Seeking Assistance with Hudi Integration Issue in Spark Thrift Server and DBT [hudi]

Posted by "soumilshah1995 (via GitHub)" <gi...@apache.org>.
soumilshah1995 commented on issue #10287:
URL: https://github.com/apache/hudi/issues/10287#issuecomment-1847838799

   previous test i used spark parquet and had launched thrift server using similar technique  worked fine
   
   Thrift sever 
   spark-submit \
     --master 'local[*]' \
     --conf spark.executor.extraJavaOptions=-Duser.timezone=Etc/UTC \
     --conf spark.eventLog.enabled=false \
     --conf spark.sql.warehouse.dir=file:///Users/soumilshah/Desktop/soumil/sparkwarehouse \
     --packages 'org.apache.spark:spark-sql_2.12:3.4.0,org.apache.spark:spark-hive_2.12:3.4.0' \
     --class org.apache.spark.sql.hive.thriftserver.HiveThriftServer2 \
     --name "Thrift JDBC/ODBC Server" \
     --executor-memory 512m
   
   this was just using spark and parquet
   {{ config(
       materialized='incremental',
       file_format='parquet',
       incremental_strategy='insert_overwrite',
   ) }}
   
   SELECT
       date(d) AS id,
       d AS full_date,
       EXTRACT (YEAR FROM d)  AS YEAR,
       EXTRACT (WEEK FROM d)  AS year_week,
       EXTRACT (DAY FROM d)   AS year_day,
       EXTRACT (YEAR FROM  d)  AS fiscal_year,
       EXTRACT (QUARTER FROM d) AS fiscal_qtr,
       EXTRACT (MONTH FROM d) AS MONTH,
       date_format(d, 'MMMM')  AS month_name,
       EXTRACT (DOW FROM d)  AS week_day,
       date_format(d, 'EEEE')  AS day_name,
       (CASE WHEN date_format(d, 'EEEE') NOT IN ('Sunday', 'Saturday') THEN 0 ELSE 1 END) AS day_is_weekday
   FROM (SELECT EXPLODE(months) AS d FROM (SELECT SEQUENCE (TO_DATE('2000-01-01'), TO_DATE('2023-01-01'), INTERVAL 1 DAY) AS months))
   DBT project.yml
   name: 'sparkdbt'
   version: '1.0.0'
   config-version: 2
   
   profile: 'sparkdbt'
   
   model-paths: ["models"]
   analysis-paths: ["analyses"]
   test-paths: ["tests"]
   seed-paths: ["seeds"]
   macro-paths: ["macros"]
   snapshot-paths: ["snapshots"]
   
   clean-targets:
     - "target"
     - "dbt_packages"
   
   models:
     sparkdbt:
       core:
         +enabled: true
         +materialized: table
       dim_date:
         +materialized: table
   
   works fine i was able to see table and query it using beeline and dbeaver as well i am seeing issue with Hudi tables not sure what CONF i am missing here
   
   I also have read examples provided https://github.com/apache/hudi/blob/master/hudi-examples/hudi-examples-dbt/README.md
   
   could not get it to work for some reason


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [SUPPORT] Seeking Assistance with Hudi Integration Issue in Spark Thrift Server and DBT [hudi]

Posted by "soumilshah1995 (via GitHub)" <gi...@apache.org>.
soumilshah1995 closed issue #10287: [SUPPORT] Seeking Assistance with Hudi Integration Issue in Spark Thrift Server and DBT
URL: https://github.com/apache/hudi/issues/10287


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [SUPPORT] Seeking Assistance with Hudi Integration Issue in Spark Thrift Server and DBT [hudi]

Posted by "ad1happy2go (via GitHub)" <gi...@apache.org>.
ad1happy2go commented on issue #10287:
URL: https://github.com/apache/hudi/issues/10287#issuecomment-1849367305

   @soumilshah1995 I saw this error before. Can you put your hudi bundle jars in the jars folder and set spark configurations in spark-defaults.yaml. 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [SUPPORT] Seeking Assistance with Hudi Integration Issue in Spark Thrift Server and DBT [hudi]

Posted by "soumilshah1995 (via GitHub)" <gi...@apache.org>.
soumilshah1995 commented on issue #10287:
URL: https://github.com/apache/hudi/issues/10287#issuecomment-1868324146

   ![Screenshot 2023-12-23 at 11 13 29 AM](https://github.com/apache/hudi/assets/39345855/1029e731-be52-4ff8-81b8-1753c342de44)
   
   
   will be creating YouTube videos for this which will help everyone 
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [SUPPORT] Seeking Assistance with Hudi Integration Issue in Spark Thrift Server and DBT [hudi]

Posted by "soumilshah1995 (via GitHub)" <gi...@apache.org>.
soumilshah1995 commented on issue #10287:
URL: https://github.com/apache/hudi/issues/10287#issuecomment-1848711118

   launched 
   ```
   spark-submit \
     --master 'local[*]' \
     --conf spark.executor.extraJavaOptions=-Duser.timezone=Etc/UTC \
     --conf spark.eventLog.enabled=false \
     --conf spark.sql.warehouse.dir=file:///Users/soumilshah/Desktop/soumil/dbt \
     --class org.apache.spark.sql.hive.thriftserver.HiveThriftServer2 \
     --name "Thrift JDBC/ODBC Server" \
     --executor-memory 3g \
     --conf spark.serializer=org.apache.spark.serializer.KryoSerializer \
     --conf spark.sql.extensions=org.apache.spark.sql.hudi.HoodieSparkSessionExtension \
     --conf spark.sql.catalog.spark_catalog=org.apache.spark.sql.hudi.catalog.HoodieCatalog \
     --conf hive.metastore.warehouse.dir=/Users/soumilshah/Desktop/soumil/dbt \
     --conf hive.metastore.schema.verification=false \
     --conf datanucleus.schema.autoCreateAll=true \
     --conf javax.jdo.option.ConnectionDriverName=org.apache.derby.jdbc.ClientDriver \
     --conf 'javax.jdo.option.ConnectionURL=jdbc:derby://localhost:1527/MyDatabase;create=true'
   ```
   
   Launched 
   ```
   $DERBY_HOME/bin/startNetworkServer -h localhost
   ```
   
   
   ## DBT Debug 
   ```
   
   dbt debug 
   21:36:47  Running with dbt=1.7.3
   21:36:47  dbt version: 1.7.3
   21:36:47  python version: 3.9.6
   21:36:47  python path: /Users/soumilshah/IdeaProjects/mdbt/Youtube/dbt-env/bin/python
   21:36:47  os info: macOS-13.6.1-x86_64-i386-64bit
   21:36:47  Using profiles dir at /Users/soumilshah/.dbt
   21:36:47  Using profiles.yml file at /Users/soumilshah/.dbt/profiles.yml
   21:36:47  Using dbt_project.yml file at /Users/soumilshah/IdeaProjects/mdbt/Youtube/dbtspark/dbt_project.yml
   21:36:47  adapter type: spark
   21:36:47  adapter version: 1.7.1
   21:36:47  Configuration:
   21:36:47    profiles.yml file [OK found and valid]
   21:36:47    dbt_project.yml file [OK found and valid]
   21:36:47  Required dependencies:
   21:36:47   - git [OK found]
   
   21:36:47  Connection:
   21:36:47    host: localhost
   21:36:47    port: 10000
   21:36:47    cluster: None
   21:36:47    endpoint: None
   21:36:47    schema: hudi_examples_dbt
   21:36:47    organization: 0
   21:36:47  Registered adapter: spark=1.7.1
   21:36:48    Connection test: [OK connection ok]
   
   21:36:48  All checks passed!
   (dbt-env) soumilshah@Soumils-MBP dbtspark %
   ```
   
   after running dbt run terminal crashes 
   ![Screenshot 2023-12-09 at 4 38 12 PM](https://github.com/apache/hudi/assets/39345855/7b1d1cb6-288b-411c-84de-05f25f1df964)
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [SUPPORT] Seeking Assistance with Hudi Integration Issue in Spark Thrift Server and DBT [hudi]

Posted by "soumilshah1995 (via GitHub)" <gi...@apache.org>.
soumilshah1995 commented on issue #10287:
URL: https://github.com/apache/hudi/issues/10287#issuecomment-1850094087

   I explored two pathways in addressing this challenge. The first route involved initiating the process using pure Hive and Spark SQL, coupled with Thrift Server. However, when attempting to run dbt, I encountered a specific issue.
   
   The second route, a more intricate and time-consuming approach, required the installation of Apache Derby and Spark. Despite my efforts, an odd complication arose: when executing "dbt run," it led to the unexpected crash of both Thrift Server and Apache Derby.
   
   ## Step 1: Create DBT Environment
   
   ```bash
   # Create virtual environment for DBT
   python -m venv dbt-env
   source dbt-env/bin/activate
   
   # Install required packages
   pip install dbt-core
   pip install dbt-spark
   pip install 'dbt-spark[PyHive]'
   
   # Navigate to DBT directory
   cd ~/.dbt/
   
   # Set Java environment variable
   export JAVA_HOME=/opt/homebrew/Cellar/openjdk@11/11.0.21/libexec/openjdk.jdk/Contents/Home
   
   # Step 2: Download and Run Apache Derby
   ```
    install Apache Derby
   
   export DERBY_VERSION=10.14.2.0
   curl -O https://archive.apache.org/dist/db/derby/db-derby-$DERBY_VERSION/db-derby-$DERBY_VERSION-bin.tar.gz -P /opt/
   tar -xf db-derby-$DERBY_VERSION-bin.tar.gz
   export DERBY_HOME=/Users/soumilshah/Desktop/soumil/dbt/db-derby-10.14.2.0-bin
   echo $DERBY_HOME
   rm -r db-derby-10.14.2.0-bin.tar.gz
   $DERBY_HOME/bin/startNetworkServer -h localhost
   ```
   
   
   Step 3: Install Apache Spark
   ```
   # Specify Spark version
   export SPARK_VERSION=3.2.3
   
   # Download and extract Spark
   curl -O https://archive.apache.org/dist/spark/spark-$SPARK_VERSION/spark-$SPARK_VERSION-bin-hadoop2.7.tgz
   tar -xf spark-$SPARK_VERSION-bin-hadoop2.7.tgz
   
   # Set Spark home
   export SPARK_HOME=/Users/soumilshah/Desktop/soumil/dbt/spark-3.2.3-bin-hadoop2.7
   echo $SPARK_HOME
   
   # Clean up downloaded files
   rm spark-3.2.3-bin-hadoop2.7.tgz
   
   ```
   Step 4: Copy JAR files
   
   ```
   # Copy JAR files to Spark JARS directory
   cp /Users/soumilshah/Desktop/myjar/*.jar $SPARK_HOME/jars/
   
   ```
   Step 5: Spark Submit Configuration
   
   ```
   # Submit Spark job
   spark-submit \
     --master 'local[*]' \
     --conf spark.executor.extraJavaOptions=-Duser.timezone=Etc/UTC \
     --conf spark.eventLog.enabled=false \
     --conf spark.sql.warehouse.dir=file:///Users/soumilshah/Desktop/soumil/dbt \
     --class org.apache.spark.sql.hive.thriftserver.HiveThriftServer2 \
     --packages 'org.apache.spark:spark-sql_2.12:3.2.3,org.apache.spark:spark-hive_2.12:3.2.3,org.apache.hudi:hudi-spark3.2-bundle_2.12:0.14.0' \
     --name "Thrift JDBC/ODBC Server" \
     --executor-memory 5g \
     --conf spark.serializer=org.apache.spark.serializer.KryoSerializer \
     --conf spark.sql.extensions=org.apache.spark.sql.hudi.HoodieSparkSessionExtension \
     --conf spark.sql.catalog.spark_catalog=org.apache.spark.sql.hudi.catalog.HoodieCatalog \
     --conf hive.metastore.warehouse.dir=/Users/soumilshah/Desktop/soumil/dbt \
     --conf hive.metastore.schema.verification=false \
     --conf datanucleus.schema.autoCreateAll=true \
     --conf javax.jdo.option.ConnectionDriverName=org.apache.derby.jdbc.ClientDriver \
     --conf 'javax.jdo.option.ConnectionURL=jdbc:derby://localhost:1527/MyDatabase;create=true'
   
   
   ```
   
   
   Question why do I have to use apache derby I mean when I am simply using Spark SQL and hive server I am able to create Hudi Tables through beeline why does it fails on dbt run
    
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org