You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by GitBox <gi...@apache.org> on 2022/08/25 08:45:57 UTC

[GitHub] [hudi] Armelabdelkbir opened a new issue, #6496: [SUPPORT] Hudi schema evolution, Null for oldest values

Armelabdelkbir opened a new issue, #6496:
URL: https://github.com/apache/hudi/issues/6496

   
   
   Hi everyone, i'm trying to test schema evolution for my cdc pipline (debizium + kafka)  with hudi 0.11.0  and spark structured streming , i follow this documentation, https://hudi.apache.org/docs/0.11.0/schema_evolution,
    does hudi manage well the schema evolution, it is necessary to restart the job, once it is done all the old values will be null, it takes into account only the values of the last commits, so the data not match my Postgres source  ? on my schema registry i can see V1 and V2  
   
     any ideas thanks
   
   A clear and concise description of the problem.
   
   **To Reproduce**
   
   Steps to reproduce the behavior:
   
   1. Stop hudi streams, and drop hive tables
   2. add some columns ALTER TABLE <table> ADD COLUMN <column_name> character varying(50) DEFAULT 'toto'  ;
   3.  restart hudi spark jobs 
   4. select * from hudi _ro / _rt table ( or read parquet hudi format using spark)
   
   **Expected behavior**
   
   when i select my data it expected to see default value on the added column and not null values
   
   data on postgres source:
   ```
   cdc_hudi=> select test, test2, test3 from hudipart
   ;
    test | test2 | test3 
   ------+-------+-------
    toto | f     | Toto
    test | t     | Toto
    test | t     | Toto
    test | t     | Toto
    toto | f     | Toto
    toto | f     | Toto
    toto | f     | Toto
    toto | f     | Toto
    toto | f     | Toto
    toto | f     | Toto
    toto | f     | Toto
    test | t     | Toto
    test | t     | Toto
    test | t     | test3
    test | t     | test3
    toto | f     | Toto
    toto | f     | Toto
    toto | f     | Toto
    test | t     | test3
    toto | f     | Toto
    toto | f     | Toto
    toto | f     | Toto
   ```
   data on hudi parquets / hive tables :
   ```
   spark.sql("select _hoodie_commit_time as commitTime, test, test2, test3 from evolution ").show()
   ---------------------------------------
   +-----------------+----+-----+-----+
   |       commitTime|test|test2|test3|
   +-----------------+----+-----+-----+
   |20220824102514494|null| null| null|
   |20220824102514494|null| null| null|
   |20220824102514494|null| null| null|
   |20220824102514494|null| null| null|
   |20220824102514494|null| null| null|
   |20220824102514494|null| null| null|
   |20220824102514494|null| null| null|
   |20220824102514494|null| null| null|
   |20220824102514494|null| null| null|
   |20220824102514494|null| null| null|
   |20220824102514494|null| null| null|
   |20220824102514494|null| null| null|
   |20220824132039517|null| null| null|
   |20220824132113066|null| null| null|
   |20220824132113066|null| null| null|
   |20220824132934016|test| true| null|
   |20220824135050368|test| true| null|
   |20220824135411903|test| true| null|
   |20220824135446080|test| true| null|
   |20220824135921176|test| true|test3|
   ```
   
   
   
   **Environment Description**
   
   * Hudi version  : 0.11.0
   
   * Spark version :3.1.4
   
   * Hive version :1.2.1000
   
   * Hadoop version : 2.7.3
   
   * Storage (HDFS)
   * schema registry
   * Kafka
   * debezium 
   
   
   
   
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [hudi] xiarixiaoyao commented on issue #6496: [SUPPORT] Hudi schema evolution, Null for oldest values

Posted by GitBox <gi...@apache.org>.
xiarixiaoyao commented on issue #6496:
URL: https://github.com/apache/hudi/issues/6496#issuecomment-1231091867

   @Armelabdelkbir   if you has this requirement for spark 3.1x  pls  raise a pr, and i will fixed it as soon as possible


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [hudi] xiarixiaoyao commented on issue #6496: [SUPPORT] Hudi schema evolution, Null for oldest values

Posted by GitBox <gi...@apache.org>.
xiarixiaoyao commented on issue #6496:
URL: https://github.com/apache/hudi/issues/6496#issuecomment-1228023484

   @Armelabdelkbir   spark now not support default value, maybe https://github.com/apache/spark/pull/36672/files can help you, thanks  


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [hudi] codope closed issue #6496: [SUPPORT] Hudi schema evolution, Null for oldest values

Posted by GitBox <gi...@apache.org>.
codope closed issue #6496: [SUPPORT] Hudi schema evolution, Null for oldest values
URL: https://github.com/apache/hudi/issues/6496


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [hudi] codope commented on issue #6496: [SUPPORT] Hudi schema evolution, Null for oldest values

Posted by GitBox <gi...@apache.org>.
codope commented on issue #6496:
URL: https://github.com/apache/hudi/issues/6496#issuecomment-1263428666

   Closing as the issue has been triaged and we have a patch as suggested in https://github.com/apache/hudi/issues/6496#issuecomment-1228023484


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org