[I] Only when the target table to be inserted/merged is a hudi table should the meta fields be eliminated [hudi]

["TheR1sing3un (via GitHub)" <gi...@apache.org>]

TheR1sing3un opened a new issue, #14229:
URL: https://github.com/apache/hudi/issues/14229

   ### Task Description
   
   **What needs to be done:**
   Only when the target table to be inserted/merged is a hudi table should the meta fields be eliminated
   **Why this task is needed:**
   We should not make any pruning to the meta fields when inserting non-Hudi tables. 
   If we do not want users to read the meta fields, we should mask them when building datasource relations rather than making pruning when writing. When writing, we should only consider that when a Hudi table needs to be inserted, to prevent meta column conflicts, as meta columns should be removed from both the query and target.
   
   I encountered this problem in a real scenario. We needed to read out all the columns included metadata fields from the hudi table and write them into a hive table for backup, auditing, and other purposes. However, we found that the meta columns in the query were cut, but the meta columns in the target table were still there. This has led to the following exceptions:
   
   ```
   inserted have the same number of columns as the target table: target table has 73 column(s) but the inserted data has 68
   ```
   
   ### Task Type
   
   Code improvement/refactoring
   
   ### Related Issues
   
   **Parent feature issue:** (if applicable )
   **Related issues:**
   NOTE: Use `Relationships` button to add parent/blocking issues after issue is created.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] Only when the target table to be inserted/merged is a hudi table should the meta fields be eliminated [hudi]

["yihua (via GitHub)" <gi...@apache.org>]

yihua closed issue #14229: Only when the target table to be inserted/merged is a hudi table should the meta fields be eliminated
URL: https://github.com/apache/hudi/issues/14229


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org