You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by GitBox <gi...@apache.org> on 2021/07/22 10:41:26 UTC

[GitHub] [hudi] fengjian428 opened a new issue #3327: [SUPPORT] ingetst avro nested array field error occur

fengjian428 opened a new issue #3327:
URL: https://github.com/apache/hudi/issues/3327


   **_Tips before filing an issue_**
   
   - Have you gone through our [FAQs](https://cwiki.apache.org/confluence/display/HUDI/FAQ)? yes
   
   - Join the mailing list to engage in conversations and get faster support at dev-subscribe@hudi.apache.org.
   
   - If you have triaged this as a bug, then file an [issue](https://issues.apache.org/jira/projects/HUDI/issues) directly.
   
   **Describe the problem you faced**
   
   when ingest data into hudi table ,I found if the avro schema has a array field with nested array field and has no other fields, error happens
   
   ![](https://jira.shopee.io/secure/attachment/1006609/1006609_image-2021-06-16-20-43-50-611.png)
   
   
   ![](https://jira.shopee.io/secure/attachment/1006610/1006610_image-2021-06-16-20-44-15-478.png)
   
   but if I add a dummy field  like below(’source‘) , ingestion works fine
   
   ![](https://jira.shopee.io/secure/attachment/1006619/1006619_image-2021-06-16-20-48-47-000.png)
   
   anyone know how to fix this?  is this a parquet-avro ‘s bug? neither delta-streamer or spark data source can trigger this error 
   
   
   **Environment Description**
   
   * Hudi version : 0.8.0 and 0.7.0
   
   * Spark version :2.4.7
   
   * Hive version :
   
   * Hadoop version :2.10
   
   * Storage (HDFS/S3/GCS..) :HDFS
   
   * Running on Docker? (yes/no) :no
   * 
   
   
   **Additional context**
   
   Add any other context about the problem here.
   
   **Stacktrace**
   
   ![](https://jira.shopee.io/secure/attachment/1006610/1006610_image-2021-06-16-20-44-15-478.png)
   
   
   ```Add the stacktrace of the error.```
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [hudi] fengjian428 commented on issue #3327: [SUPPORT] ingetst avro nested array field error occur

Posted by GitBox <gi...@apache.org>.
fengjian428 commented on issue #3327:
URL: https://github.com/apache/hudi/issues/3327#issuecomment-889576271


   > @fengjian428 I will try to reproduce this. Please take a look at https://hudi.apache.org/docs/writing_data.html#schema-evolution
   > The exception says that 'array' field not found. Was the scehma changed in between and there are partitions with older schema as well?
   
   no,I can reproduce it with new table


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [hudi] codope commented on issue #3327: [SUPPORT] ingetst avro nested array field error occur

Posted by GitBox <gi...@apache.org>.
codope commented on issue #3327:
URL: https://github.com/apache/hudi/issues/3327#issuecomment-889864471


   @fengjian428 I could reproduce this and I believe this is a bug in parquet-avro. See following open issues:
   https://issues.apache.org/jira/browse/PARQUET-1254
   https://issues.apache.org/jira/browse/PARQUET-1681


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [hudi] nsivabalan commented on issue #3327: [SUPPORT] ingetst avro nested array field error occur

Posted by GitBox <gi...@apache.org>.
nsivabalan commented on issue #3327:
URL: https://github.com/apache/hudi/issues/3327#issuecomment-905192770


   Closing this out since this is not hudi issue and we don't have any action item from our end. thanks ! 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [hudi] codope commented on issue #3327: [SUPPORT] ingetst avro nested array field error occur

Posted by GitBox <gi...@apache.org>.
codope commented on issue #3327:
URL: https://github.com/apache/hudi/issues/3327#issuecomment-889304560


   @fengjian428  I will try to reproduce this. Please take a look at https://hudi.apache.org/docs/writing_data.html#schema-evolution
   The exception says that 'array' field not found. Was the scehma changed in between and there are partitions with older schema as well?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [hudi] nsivabalan closed issue #3327: [SUPPORT] ingetst avro nested array field error occur

Posted by GitBox <gi...@apache.org>.
nsivabalan closed issue #3327:
URL: https://github.com/apache/hudi/issues/3327


   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org