You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by GitBox <gi...@apache.org> on 2021/07/22 10:41:26 UTC
[GitHub] [hudi] fengjian428 opened a new issue #3327: [SUPPORT] ingetst avro nested array field error occur
fengjian428 opened a new issue #3327:
URL: https://github.com/apache/hudi/issues/3327
**_Tips before filing an issue_**
- Have you gone through our [FAQs](https://cwiki.apache.org/confluence/display/HUDI/FAQ)? yes
- Join the mailing list to engage in conversations and get faster support at dev-subscribe@hudi.apache.org.
- If you have triaged this as a bug, then file an [issue](https://issues.apache.org/jira/projects/HUDI/issues) directly.
**Describe the problem you faced**
when ingest data into hudi table ,I found if the avro schema has a array field with nested array field and has no other fields, error happens


but if I add a dummy field like below(’source‘) , ingestion works fine

anyone know how to fix this? is this a parquet-avro ‘s bug? neither delta-streamer or spark data source can trigger this error
**Environment Description**
* Hudi version : 0.8.0 and 0.7.0
* Spark version :2.4.7
* Hive version :
* Hadoop version :2.10
* Storage (HDFS/S3/GCS..) :HDFS
* Running on Docker? (yes/no) :no
*
**Additional context**
Add any other context about the problem here.
**Stacktrace**

```Add the stacktrace of the error.```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] fengjian428 commented on issue #3327: [SUPPORT] ingetst avro nested array field error occur
Posted by GitBox <gi...@apache.org>.
fengjian428 commented on issue #3327:
URL: https://github.com/apache/hudi/issues/3327#issuecomment-889576271
> @fengjian428 I will try to reproduce this. Please take a look at https://hudi.apache.org/docs/writing_data.html#schema-evolution
> The exception says that 'array' field not found. Was the scehma changed in between and there are partitions with older schema as well?
no,I can reproduce it with new table
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] codope commented on issue #3327: [SUPPORT] ingetst avro nested array field error occur
Posted by GitBox <gi...@apache.org>.
codope commented on issue #3327:
URL: https://github.com/apache/hudi/issues/3327#issuecomment-889864471
@fengjian428 I could reproduce this and I believe this is a bug in parquet-avro. See following open issues:
https://issues.apache.org/jira/browse/PARQUET-1254
https://issues.apache.org/jira/browse/PARQUET-1681
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan commented on issue #3327: [SUPPORT] ingetst avro nested array field error occur
Posted by GitBox <gi...@apache.org>.
nsivabalan commented on issue #3327:
URL: https://github.com/apache/hudi/issues/3327#issuecomment-905192770
Closing this out since this is not hudi issue and we don't have any action item from our end. thanks !
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] codope commented on issue #3327: [SUPPORT] ingetst avro nested array field error occur
Posted by GitBox <gi...@apache.org>.
codope commented on issue #3327:
URL: https://github.com/apache/hudi/issues/3327#issuecomment-889304560
@fengjian428 I will try to reproduce this. Please take a look at https://hudi.apache.org/docs/writing_data.html#schema-evolution
The exception says that 'array' field not found. Was the scehma changed in between and there are partitions with older schema as well?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan closed issue #3327: [SUPPORT] ingetst avro nested array field error occur
Posted by GitBox <gi...@apache.org>.
nsivabalan closed issue #3327:
URL: https://github.com/apache/hudi/issues/3327
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org