[GitHub] [hudi] JB-data edited a comment on issue #3905: [SUPPORT] Transform from kafka complains about table not found when using transformer.sql

[GitBox <gi...@apache.org>]

JB-data edited a comment on issue #3905:
URL: https://github.com/apache/hudi/issues/3905#issuecomment-971693314


   Indeed, original issue clear.
   Thanks for feedback.
   Thanks for links: related to the SRC table-my confusion was: where does this SRC come from, what table is it?
   
   But after looking at the java code you suggested (https://github.com/a0x8o/hudi/blob/master/hudi-utilities/src/main/java/org/apache/hudi/utilities/transform/SqlQueryBasedTransformer.java), this is I believe what happens:
   1)WITH SQL TRANSFORMATION
   When you are using a transformation he will register  a temporary table in hive in order to perform the SQL transformation. So the data from the kafka topic is pulled into hive for a bit (in the default db), the SQL transformation is performed, and then the data is written according to the target schema.
   This means we need to make sure that the user that is running the job has all permissions to access the hive metastore's default db as this is where he will temporarily register this <SRC> table, which in reality wil be called "HOODIE_SRC_TMP_TABLE_<some long random number>".
   2)WITHOUT SQL TRANSFORMATION
   While if you dont do any sql transformation, the data is directly landed in the target table and target db specified in the props.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org