You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by GitBox <gi...@apache.org> on 2021/11/09 11:03:28 UTC
[GitHub] [hudi] Limess opened a new issue #3953: [SUPPORT]
Limess opened a new issue #3953:
URL: https://github.com/apache/hudi/issues/3953
**Describe the problem you faced**
Schema validation using:
```
hoodie.avro.schema.validate=true
```
always fails due to mismatched namespaces when writing using Deltasteamer with `RowBasedSchemaProvider` and Spark Datasource writer against the same Hudi table.
Deltastreamer always sets the Avro namespace and name to
```
public static final String HOODIE_RECORD_NAMESPACE = "hoodie.source";
public static final String HOODIE_RECORD_STRUCT_NAME = "hoodie_source";
```
https://github.com/apache/hudi/blob/e057a10499729301ebe96d7cd54902b113af1811/hudi-utilities/src/main/java/org/apache/hudi/utilities/schema/RowBasedSchemaProvider.java#L31-L32
Whereas dataframes written via spark always add the schema using a record and struct name based on the table name:
https://github.com/apache/hudi/blob/e057a10499729301ebe96d7cd54902b113af1811/hudi-client/hudi-spark-client/src/main/scala/org/apache/hudi/AvroConversionUtils.scala#L129-L132
This seems to fail this check in validation: https://github.com/apache/hudi/blob/178767948e906f673d6d4a357c65c11bc574f619/hudi-common/src/main/java/org/apache/hudi/common/table/TableSchemaResolver.java#L340-L342 as the namespaces are not equal, even if the schema itself is identical
**Expected behavior**
Validation succeeds if the schema is identical or evolvable, regardless of internal namespace usage.
**Environment Description**
EMR 6.4.0
* Hudi version: 0.9.0
* Spark version :
3.1.2
* Hive version :
Hive 3.1.2
* Hadoop version :
Amazon 3.2.1
* Storage (HDFS/S3/GCS..) :
S3
* Running on Docker? (yes/no) :
no
*Additional Context*
Full stack trace.
The Hudi table was written by deltastreamer, and we are trying to upsert from Spark datasource with validation enabled
```
21/11/09 10:24:26 ERROR ApplicationMaster: User class threw exception: org.apache.hudi.exception.HoodieUpsertException: Failed upsert schema compatibility check.
org.apache.hudi.exception.HoodieUpsertException: Failed upsert schema compatibility check.
at org.apache.hudi.table.HoodieTable.validateUpsertSchema(HoodieTable.java:639)
at org.apache.hudi.client.SparkRDDWriteClient.upsert(SparkRDDWriteClient.java:155)
at org.apache.hudi.utilities.deltastreamer.DeltaSync.writeToSink(DeltaSync.java:484)
at org.apache.hudi.utilities.deltastreamer.DeltaSync.syncOnce(DeltaSync.java:303)
at org.apache.hudi.utilities.deltastreamer.HoodieDeltaStreamer.lambda$sync$2(HoodieDeltaStreamer.java:182)
at org.apache.hudi.common.util.Option.ifPresent(Option.java:96)
at org.apache.hudi.utilities.deltastreamer.HoodieDeltaStreamer.sync(HoodieDeltaStreamer.java:180)
at org.apache.hudi.utilities.deltastreamer.HoodieDeltaStreamer.main(HoodieDeltaStreamer.java:509)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.spark.deploy.yarn.ApplicationMaster$$anon$2.run(ApplicationMaster.scala:735)
Caused by: org.apache.hudi.exception.HoodieException: Failed schema compatibility check for writerSchema :{"type":"record","name":"hoodie_source","namespace":"hoodie.source","fields":[{"name":"_hoodie_commit_time","type":["null","string"],"doc":"","default":null},{"name":"_hoodie_commit_seqno","type":["null","string"],"doc":"","default":null},{"name":"_hoodie_record_key","type":["null","string"],"doc":"","default":null},{"name":"_hoodie_partition_path","type":["null","string"],"doc":"","default":null},{"name":"_hoodie_file_name","type":["null","string"],"doc":"","default":null},{"name":"id","type":["null","string"],"default":null},{"name":"version","type":["null","long"],"default":null},{"name":"aggregation_id","type":["null","string"],"default":null},{"name":"area_on_page","type":["null","long"],"default":null},{"name":"article_cursor","type":["null","string"],"default":null},{"name":"article_deduplication_id","type":["null","string"],"default":null},{"name":"article_earliest_pub
lished_date","type":["null","string"],"default":null},{"name":"author","type":["null","string"],"default":null},{"name":"canonical_source_id","type":["null","string"],"default":null},{"name":"canonical_source_location","type":["null",{"type":"record","name":"canonical_source_location","namespace":"hoodie.source.hoodie_source","fields":[{"name":"country","type":["null","string"],"default":null},{"name":"region","type":["null","string"],"default":null},{"name":"subregion","type":["null","string"],"default":null}]}],"default":null},{"name":"canonical_source_name","type":["null","string"],"default":null},{"name":"canonical_source_reach","type":["null",{"type":"record","name":"canonical_source_reach","namespace":"hoodie.source.hoodie_source","fields":[{"name":"reach_origin","type":["null","string"],"default":null},{"name":"reach_provider","type":["null","string"],"default":null},{"name":"source_id","type":["null","string"],"default":null},{"name":"type","type":["null","string"],"default"
:null},{"name":"value","type":["null","long"],"default":null}]}],"default":null},{"name":"content","type":["null","string"],"default":null},{"name":"deleted_date","type":["null","string"],"default":null},{"name":"document_type","type":["null","string"],"default":null},{"name":"eclips_web_url","type":["null","string"],"default":null},{"name":"embargoed_until_date","type":["null","string"],"default":null},{"name":"entities","type":["null",{"type":"array","items":[{"type":"record","name":"entities","namespace":"hoodie.source.hoodie_source","fields":[{"name":"offset","type":["null","long"],"default":null},{"name":"overlapping","type":["null","boolean"],"default":null},{"name":"position","type":["null","string"],"default":null},{"name":"rule_based_entity","type":["null","boolean"],"default":null},{"name":"sentiment","type":["null",{"type":"record","name":"sentiment","namespace":"hoodie.source.hoodie_source.entities","fields":[{"name":"compound","type":["null","double"],"default":null},{"
name":"neg","type":["null","double"],"default":null},{"name":"neu","type":["null","double"],"default":null},{"name":"pos","type":["null","double"],"default":null}]}],"default":null},{"name":"signal_type","type":["null","string"],"default":null},{"name":"surface_form","type":["null","string"],"default":null},{"name":"wiki_title","type":["null","string"],"default":null}]},"null"]}],"default":null},{"name":"entity_salience","type":["null",{"type":"array","items":[{"type":"record","name":"entity_salience","namespace":"hoodie.source.hoodie_source","fields":[{"name":"salience","type":["null","double"],"default":null},{"name":"salience_rank","type":["null","long"],"default":null},{"name":"wiki_title","type":["null","string"],"default":null}]},"null"]}],"default":null},{"name":"feed","type":["null","string"],"default":null},{"name":"format_version","type":["null","long"],"default":null},{"name":"ingestion_id","type":["null","string"],"default":null},{"name":"ingestion_metadata","type":["nul
l",{"type":"record","name":"ingestion_metadata","namespace":"hoodie.source.hoodie_source","fields":[{"name":"replay_time","type":["null","string"],"default":null},{"name":"replay_type","type":["null","string"],"default":null}]}],"default":null},{"name":"journalist_id","type":["null","string"],"default":null},{"name":"journalist_name","type":["null","string"],"default":null},{"name":"journalistic_quality","type":["null","double"],"default":null},{"name":"language","type":["null","string"],"default":null},{"name":"licence_id","type":["null","string"],"default":null},{"name":"media_type","type":["null","string"],"default":null},{"name":"metadata_keys","type":["null",{"type":"array","items":["string","null"]}],"default":null},{"name":"native_content","type":["null",{"type":"record","name":"native_content","namespace":"hoodie.source.hoodie_source","fields":[{"name":"content","type":["null","string"],"default":null},{"name":"summary","type":["null","string"],"default":null},{"name":"title
","type":["null","string"],"default":null}]}],"default":null},{"name":"old_article_ids","type":["null","string"],"default":null},{"name":"original_id","type":["null","string"],"default":null},{"name":"original_source_id","type":["null","string"],"default":null},{"name":"original_source_name","type":["null","string"],"default":null},{"name":"original_url","type":["null","string"],"default":null},{"name":"page","type":["null","long"],"default":null},{"name":"page_section","type":["null","string"],"default":null},{"name":"pdf_url","type":["null","string"],"default":null},{"name":"processed_date","type":["null","string"],"default":null},{"name":"provider_data","type":["null",{"type":"record","name":"provider_data","namespace":"hoodie.source.hoodie_source","fields":[{"name":"podcast_link","type":["null","string"],"default":null},{"name":"copyright","type":["null","string"],"default":null},{"name":"formatted_content","type":["null","string"],"default":null},{"name":"nla_publisher","type":
["null","string"],"default":null},{"name":"provider_hosted_url","type":["null","string"],"default":null},{"name":"publication_time","type":["null","string"],"default":null},{"name":"tvplayer","type":["null",{"type":"record","name":"tvplayer","namespace":"hoodie.source.hoodie_source.provider_data","fields":[{"name":"end_date","type":["null","string"],"default":null},{"name":"start_date","type":["null","string"],"default":null},{"name":"station_id","type":["null","string"],"default":null},{"name":"asset_id","type":["null","string"],"default":null},{"name":"partner_id","type":["null","long"],"default":null}]}],"default":null}]}],"default":null},{"name":"published_date","type":["null","string"],"default":null},{"name":"quotes","type":["null",{"type":"array","items":[{"type":"record","name":"quotes","namespace":"hoodie.source.hoodie_source","fields":[{"name":"end","type":["null","long"],"default":null},{"name":"start","type":["null","long"],"default":null},{"name":"text","type":["null","
string"],"default":null}]},"null"]}],"default":null},{"name":"received","type":["null","string"],"default":null},{"name":"signal_importance","type":["null","long"],"default":null},{"name":"social_engagement","type":["null",{"type":"record","name":"social_engagement","namespace":"hoodie.source.hoodie_source","fields":[{"name":"twitter","type":["null",{"type":"record","name":"twitter","namespace":"hoodie.source.hoodie_source.social_engagement","fields":[{"name":"shares","type":["null","long"],"default":null}]}],"default":null}]}],"default":null},{"name":"source_groups","type":["null",{"type":"array","items":["string","null"]}],"default":null},{"name":"story_cursor","type":["null","string"],"default":null},{"name":"story_id","type":["null","string"],"default":null},{"name":"story_index","type":["null","string"],"default":null},{"name":"story_processed_date","type":["null","string"],"default":null},{"name":"story_published_date","type":["null","string"],"default":null},{"name":"summary"
,"type":["null","string"],"default":null},{"name":"summary_origin","type":["null","string"],"default":null},{"name":"taxonomy_categories","type":["null",{"type":"array","items":[{"type":"record","name":"taxonomy_categories","namespace":"hoodie.source.hoodie_source","fields":[{"name":"id","type":["null","string"],"default":null},{"name":"score","type":["null","double"],"default":null}]},"null"]}],"default":null},{"name":"title","type":["null","string"],"default":null},{"name":"topic_predictions","type":["null",{"type":"array","items":[{"type":"record","name":"topic_predictions","namespace":"hoodie.source.hoodie_source","fields":[{"name":"probability","type":["null","double"],"default":null},{"name":"topic_id","type":["null","string"],"default":null}]},"null"]}],"default":null},{"name":"topics","type":["null",{"type":"array","items":["string","null"]}],"default":null},{"name":"tracking_url","type":["null","string"],"default":null},{"name":"translated_from","type":["null","string"],"de
fault":null},{"name":"story_published_partition_date","type":["null","string"],"default":null}]}, table schema :{"type":"record","name":"articles_hudi_copy_on_write_record","namespace":"hoodie.articles_hudi_copy_on_write","fields":[{"name":"_hoodie_commit_time","type":["null","string"],"doc":"","default":null},{"name":"_hoodie_commit_seqno","type":["null","string"],"doc":"","default":null},{"name":"_hoodie_record_key","type":["null","string"],"doc":"","default":null},{"name":"_hoodie_partition_path","type":["null","string"],"doc":"","default":null},{"name":"_hoodie_file_name","type":["null","string"],"doc":"","default":null},{"name":"id","type":["null","string"],"default":null},{"name":"version","type":["null","long"],"default":null},{"name":"aggregation_id","type":["null","string"],"default":null},{"name":"area_on_page","type":["null","long"],"default":null},{"name":"article_cursor","type":["null","string"],"default":null},{"name":"article_deduplication_id","type":["null","string"]
,"default":null},{"name":"article_earliest_published_date","type":["null","string"],"default":null},{"name":"author","type":["null","string"],"default":null},{"name":"canonical_source_id","type":["null","string"],"default":null},{"name":"canonical_source_location","type":["null",{"type":"record","name":"canonical_source_location","namespace":"hoodie.articles_hudi_copy_on_write.articles_hudi_copy_on_write_record","fields":[{"name":"country","type":["null","string"],"default":null},{"name":"region","type":["null","string"],"default":null},{"name":"subregion","type":["null","string"],"default":null}]}],"default":null},{"name":"canonical_source_name","type":["null","string"],"default":null},{"name":"canonical_source_reach","type":["null",{"type":"record","name":"canonical_source_reach","namespace":"hoodie.articles_hudi_copy_on_write.articles_hudi_copy_on_write_record","fields":[{"name":"reach_origin","type":["null","string"],"default":null},{"name":"reach_provider","type":["null","strin
g"],"default":null},{"name":"source_id","type":["null","string"],"default":null},{"name":"type","type":["null","string"],"default":null},{"name":"value","type":["null","long"],"default":null}]}],"default":null},{"name":"content","type":["null","string"],"default":null},{"name":"deleted_date","type":["null","string"],"default":null},{"name":"document_type","type":["null","string"],"default":null},{"name":"eclips_web_url","type":["null","string"],"default":null},{"name":"embargoed_until_date","type":["null","string"],"default":null},{"name":"entities","type":["null",{"type":"array","items":[{"type":"record","name":"entities","namespace":"hoodie.articles_hudi_copy_on_write.articles_hudi_copy_on_write_record","fields":[{"name":"offset","type":["null","long"],"default":null},{"name":"overlapping","type":["null","boolean"],"default":null},{"name":"position","type":["null","string"],"default":null},{"name":"rule_based_entity","type":["null","boolean"],"default":null},{"name":"sentiment","t
ype":["null",{"type":"record","name":"sentiment","namespace":"hoodie.articles_hudi_copy_on_write.articles_hudi_copy_on_write_record.entities","fields":[{"name":"compound","type":["null","double"],"default":null},{"name":"neg","type":["null","double"],"default":null},{"name":"neu","type":["null","double"],"default":null},{"name":"pos","type":["null","double"],"default":null}]}],"default":null},{"name":"signal_type","type":["null","string"],"default":null},{"name":"surface_form","type":["null","string"],"default":null},{"name":"wiki_title","type":["null","string"],"default":null}]},"null"]}],"default":null},{"name":"entity_salience","type":["null",{"type":"array","items":[{"type":"record","name":"entity_salience","namespace":"hoodie.articles_hudi_copy_on_write.articles_hudi_copy_on_write_record","fields":[{"name":"salience","type":["null","double"],"default":null},{"name":"salience_rank","type":["null","long"],"default":null},{"name":"wiki_title","type":["null","string"],"default":nul
l}]},"null"]}],"default":null},{"name":"feed","type":["null","string"],"default":null},{"name":"format_version","type":["null","long"],"default":null},{"name":"ingestion_id","type":["null","string"],"default":null},{"name":"ingestion_metadata","type":["null",{"type":"record","name":"ingestion_metadata","namespace":"hoodie.articles_hudi_copy_on_write.articles_hudi_copy_on_write_record","fields":[{"name":"replay_time","type":["null","string"],"default":null},{"name":"replay_type","type":["null","string"],"default":null}]}],"default":null},{"name":"journalist_id","type":["null","string"],"default":null},{"name":"journalist_name","type":["null","string"],"default":null},{"name":"journalistic_quality","type":["null","double"],"default":null},{"name":"language","type":["null","string"],"default":null},{"name":"licence_id","type":["null","string"],"default":null},{"name":"media_type","type":["null","string"],"default":null},{"name":"metadata_keys","type":["null",{"type":"array","items":["s
tring","null"]}],"default":null},{"name":"native_content","type":["null",{"type":"record","name":"native_content","namespace":"hoodie.articles_hudi_copy_on_write.articles_hudi_copy_on_write_record","fields":[{"name":"content","type":["null","string"],"default":null},{"name":"summary","type":["null","string"],"default":null},{"name":"title","type":["null","string"],"default":null}]}],"default":null},{"name":"old_article_ids","type":["null","string"],"default":null},{"name":"original_id","type":["null","string"],"default":null},{"name":"original_source_id","type":["null","string"],"default":null},{"name":"original_source_name","type":["null","string"],"default":null},{"name":"original_url","type":["null","string"],"default":null},{"name":"page","type":["null","long"],"default":null},{"name":"page_section","type":["null","string"],"default":null},{"name":"pdf_url","type":["null","string"],"default":null},{"name":"processed_date","type":["null","string"],"default":null},{"name":"provide
r_data","type":["null",{"type":"record","name":"provider_data","namespace":"hoodie.articles_hudi_copy_on_write.articles_hudi_copy_on_write_record","fields":[{"name":"podcast_link","type":["null","string"],"default":null},{"name":"copyright","type":["null","string"],"default":null},{"name":"formatted_content","type":["null","string"],"default":null},{"name":"nla_publisher","type":["null","string"],"default":null},{"name":"provider_hosted_url","type":["null","string"],"default":null},{"name":"publication_time","type":["null","string"],"default":null},{"name":"tvplayer","type":["null",{"type":"record","name":"tvplayer","namespace":"hoodie.articles_hudi_copy_on_write.articles_hudi_copy_on_write_record.provider_data","fields":[{"name":"end_date","type":["null","string"],"default":null},{"name":"start_date","type":["null","string"],"default":null},{"name":"station_id","type":["null","string"],"default":null},{"name":"asset_id","type":["null","string"],"default":null},{"name":"partner_id",
"type":["null","long"],"default":null}]}],"default":null}]}],"default":null},{"name":"published_date","type":["null","string"],"default":null},{"name":"quotes","type":["null",{"type":"array","items":[{"type":"record","name":"quotes","namespace":"hoodie.articles_hudi_copy_on_write.articles_hudi_copy_on_write_record","fields":[{"name":"end","type":["null","long"],"default":null},{"name":"start","type":["null","long"],"default":null},{"name":"text","type":["null","string"],"default":null}]},"null"]}],"default":null},{"name":"received","type":["null","string"],"default":null},{"name":"signal_importance","type":["null","long"],"default":null},{"name":"social_engagement","type":["null",{"type":"record","name":"social_engagement","namespace":"hoodie.articles_hudi_copy_on_write.articles_hudi_copy_on_write_record","fields":[{"name":"twitter","type":["null",{"type":"record","name":"twitter","namespace":"hoodie.articles_hudi_copy_on_write.articles_hudi_copy_on_write_record.social_engagement","
fields":[{"name":"shares","type":["null","long"],"default":null}]}],"default":null}]}],"default":null},{"name":"source_groups","type":["null",{"type":"array","items":["string","null"]}],"default":null},{"name":"story_cursor","type":["null","string"],"default":null},{"name":"story_id","type":["null","string"],"default":null},{"name":"story_index","type":["null","string"],"default":null},{"name":"story_processed_date","type":["null","string"],"default":null},{"name":"story_published_date","type":["null","string"],"default":null},{"name":"summary","type":["null","string"],"default":null},{"name":"summary_origin","type":["null","string"],"default":null},{"name":"taxonomy_categories","type":["null",{"type":"array","items":[{"type":"record","name":"taxonomy_categories","namespace":"hoodie.articles_hudi_copy_on_write.articles_hudi_copy_on_write_record","fields":[{"name":"id","type":["null","string"],"default":null},{"name":"score","type":["null","double"],"default":null}]},"null"]}],"defau
lt":null},{"name":"title","type":["null","string"],"default":null},{"name":"topic_predictions","type":["null",{"type":"array","items":[{"type":"record","name":"topic_predictions","namespace":"hoodie.articles_hudi_copy_on_write.articles_hudi_copy_on_write_record","fields":[{"name":"probability","type":["null","double"],"default":null},{"name":"topic_id","type":["null","string"],"default":null}]},"null"]}],"default":null},{"name":"topics","type":["null",{"type":"array","items":["string","null"]}],"default":null},{"name":"tracking_url","type":["null","string"],"default":null},{"name":"translated_from","type":["null","string"],"default":null},{"name":"story_published_partition_date","type":["null","string"],"default":null}]}, base path :s3://prod-signal-articles-store/articles_hudi_copy_on_write
at org.apache.hudi.table.HoodieTable.validateSchema(HoodieTable.java:631)
at org.apache.hudi.table.HoodieTable.validateUpsertSchema(HoodieTable.java:637)
... 12 more
21/11/09 10:24:26 INFO ApplicationMaster: Final app status: FAILED, exitCode: 15, (reason: User class threw exception: org.apache.hudi.exception.HoodieUpsertException: Failed upsert schema compatibility check.
at org.apache.hudi.table.HoodieTable.validateUpsertSchema(HoodieTable.java:639)
at org.apache.hudi.client.SparkRDDWriteClient.upsert(SparkRDDWriteClient.java:155)
at org.apache.hudi.utilities.deltastreamer.DeltaSync.writeToSink(DeltaSync.java:484)
at org.apache.hudi.utilities.deltastreamer.DeltaSync.syncOnce(DeltaSync.java:303)
at org.apache.hudi.utilities.deltastreamer.HoodieDeltaStreamer.lambda$sync$2(HoodieDeltaStreamer.java:182)
at org.apache.hudi.common.util.Option.ifPresent(Option.java:96)
at org.apache.hudi.utilities.deltastreamer.HoodieDeltaStreamer.sync(HoodieDeltaStreamer.java:180)
at org.apache.hudi.utilities.deltastreamer.HoodieDeltaStreamer.main(HoodieDeltaStreamer.java:509)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.spark.deploy.yarn.ApplicationMaster$$anon$2.run(ApplicationMaster.scala:735)
Caused by: org.apache.hudi.exception.HoodieException: Failed schema compatibility check for writerSchema :{"type":"record","name":"hoodie_source","namespace":"hoodie.source","fields":[{"name":"_hoodie_commit_time","type":["null","string"],"doc":"","default":null},{"name":"_hoodie_commit_seqno","type":["null","string"],"doc":"","default":null},{"name":"_hoodie_record_key","type":["null","string"],"doc":"","default":null},{"name":"_hoodie_partition_path","type":["null","string"],"doc":"","default":null},{"name":"_hoodie_file_name","type":["null","string"],"doc":"","default":null},{"name":"id","type":["null","string"],"default":null},{"name":"version","type":["null","long"],"default":null},{"name":"aggregation_id","type":["null","string"],"default":null},{"name":"area_on_page","type":["null","long"],"default":null},{"name":"article_cursor","type":["null","string"],"default":null},{"name":"article_deduplication_id","type":["null","string"],"default":null},{"name":"article_earliest_pub
lished_date","type":["null","string"],"default":null},{"name":"author","type":["null","string"],"default":null},{"name":"canonical_source_id","type":["null","string"],"default":null},{"name":"canonical_source_location","type":["null",{"type":"record","name":"canonical_source_location","namespace":"hoodie.source.hoodie_source","fields":[{"name":"country","type":["null","string"],"default":null},{"name":"region","type":["null","string"],"default":null},{"name":"subregion","type":["null","string"],"default":null}]}],"default":null},{"name":"canonical_source_name","type":["null","string"],"default":null},{"name":"canonical_source_reach","type":["null",{"type":"record","name":"canonical_source_reach","namespace":"hoodie.source.hoodie_source","fields":[{"name":"reach_origin","type":["null","string"],"default":null},{"name":"reach_provider","type":["null","string"],"default":null},{"name":"source_id","type":["null","string"],"default":null},{"name":"type","type":["null","string"],"default"
:null},{"name":"value","type":["null","long"],"default":null}]}],"default":null},{"name":"content","type":["null","string"],"default":null},{"name":"deleted_date","type":["null","string"],"default":null},{"name":"document_type","type":["null","string"],"default":null},{"name":"eclips_web_url","type":["null","string"],"default":null},{"name":"embargoed_until_date","type":["null","string"],"default":null},{"name":"entities","type":["null",{"type":"array","items":[{"type":"record","name":"entities","namespace":"hoodie.source.hoodie_source","fields":[{"name":"offset","type":["null","long"],"default":null},{"name":"overlapping","type":["null","boolean"],"default":null},{"name":"position","type":["null","string"],"default":null},{"name":"rule_based_entity","type":["null","boolean"],"default":null},{"name":"sentiment","type":["null",{"type":"record","name":"sentiment","namespace":"hoodie.source.hoodie_source.entities","fields":[{"name":"compound","type":["null","double"],"default":null},{"
name":"neg","type":["null","double"],"default":null},{"name":"neu","type":["null","double"],"default":null},{"name":"pos","type":["null","double"],"default":null}]}],"default":null},{"name":"signal_type","type":["null","string"],"default":null},{"name":"surface_form","type":["null","string"],"default":null},{"name":"wiki_title","type":["null","string"],"default":null}]},"null"]}],"default":null},{"name":"entity_salience","type":["null",{"type":"array","items":[{"type":"record","name":"entity_salience","namespace":"hoodie.source.hoodie_source","fields":[{"name":"salience","type":["null","double"],"default":null},{"name":"salience_rank","type":["null","long"],"default":null},{"name":"wiki_title","type":["null","string"],"default":null}]},"null"]}],"default":null},{"name":"feed","type":["null","string"],"default":null},{"name":"format_version","type":["null","long"],"default":null},{"name":"ingestion_id","type":["null","string"],"default":null},{"name":"ingestion_metadata","type":["nul
l",{"type":"record","name":"ingestion_metadata","namespace":"hoodie.source.hoodie_source","fields":[{"name":"replay_time","type":["null","string"],"default":null},{"name":"replay_type","type":["null","string"],"default":null}]}],"default":null},{"name":"journalist_id","type":["null","string"],"default":null},{"name":"journalist_name","type":["null","string"],"default":null},{"name":"journalistic_quality","type":["null","double"],"default":null},{"name":"language","type":["null","string"],"default":null},{"name":"licence_id","type":["null","string"],"default":null},{"name":"media_type","type":["null","string"],"default":null},{"name":"metadata_keys","type":["null",{"type":"array","items":["string","null"]}],"default":null},{"name":"native_content","type":["null",{"type":"record","name":"native_content","namespace":"hoodie.source.hoodie_source","fields":[{"name":"content","type":["null","string"],"default":null},{"name":"summary","type":["null","string"],"default":null},{"name":"title
","type":["null","string"],"default":null}]}],"default":null},{"name":"old_article_ids","type":["null","string"],"default":null},{"name":"original_id","type":["null","string"],"default":null},{"name":"original_source_id","type":["null","string"],"default":null},{"name":"original_source_name","type":["null","string"],"default":null},{"name":"original_url","type":["null","string"],"default":null},{"name":"page","type":["null","long"],"default":null},{"name":"page_section","type":["null","string"],"default":null},{"name":"pdf_url","type":["null","string"],"default":null},{"name":"processed_date","type":["null","string"],"default":null},{"name":"provider_data","type":["null",{"type":"record","name":"provider_data","namespace":"hoodie.source.hoodie_source","fields":[{"name":"podcast_link","type":["null","string"],"default":null},{"name":"copyright","type":["null","string"],"default":null},{"name":"formatted_content","type":["null","string"],"default":null},{"name":"nla_publisher","type":
["null","string"],"default":null},{"name":"provider_hosted_url","type":["null","string"],"default":null},{"name":"publication_time","type":["null","string"],"default":null},{"name":"tvplayer","type":["null",{"type":"record","name":"tvplayer","namespace":"hoodie.source.hoodie_source.provider_data","fields":[{"name":"end_date","type":["null","string"],"default":null},{"name":"start_date","type":["null","string"],"default":null},{"name":"station_id","type":["null","string"],"default":null},{"name":"asset_id","type":["null","string"],"default":null},{"name":"partner_id","type":["null","long"],"default":null}]}],"default":null}]}],"default":null},{"name":"published_date","type":["null","string"],"default":null},{"name":"quotes","type":["null",{"type":"array","items":[{"type":"record","name":"quotes","namespace":"hoodie.source.hoodie_source","fields":[{"name":"end","type":["null","long"],"default":null},{"name":"start","type":["null","long"],"default":null},{"name":"text","type":["null","
string"],"default":null}]},"null"]}],"default":null},{"name":"received","type":["null","string"],"default":null},{"name":"signal_importance","type":["null","long"],"default":null},{"name":"social_engagement","type":["null",{"type":"record","name":"social_engagement","namespace":"hoodie.source.hoodie_source","fields":[{"name":"twitter","type":["null",{"type":"record","name":"twitter","namespace":"hoodie.source.hoodie_source.social_engagement","fields":[{"name":"shares","type":["null","long"],"default":null}]}],"default":null}]}],"default":null},{"name":"source_groups","type":["null",{"type":"array","items":["string","null"]}],"default":null},{"name":"story_cursor","type":["null","string"],"default":null},{"name":"story_id","type":["null","string"],"default":null},{"name":"story_index","type":["null","string"],"default":null},{"name":"story_processed_date","type":["null","string"],"default":null},{"name":"story_published_date","type":["null","string"],"default":null},{"name":"summary"
,"type":["null","string"],"default":null},{"name":"summary_origin","type":["null","string"],"default":null},{"name":"taxonomy_categories","type":["null",{"type":"array","items":[{"type":"record","name":"taxonomy_categories","namespace":"hoodie.source.hoodie_source","fields":[{"name":"id","type":["null","string"],"default":null},{"name":"score","type":["null","double"],"default":null}]},"null"]}],"default":null},{"name":"title","type":["null","string"],"default":null},{"name":"topic_predictions","type":["null",{"type":"array","items":[{"type":"record","name":"topic_predictions","namespace":"hoodie.source.hoodie_source","fields":[{"name":"probability","type":["null","double"],"default":null},{"name":"topic_id","type":["null","string"],"default":null}]},"null"]}],"default":null},{"name":"topics","type":["null",{"type":"array","items":["string","null"]}],"default":null},{"name":"tracking_url","type":["null","string"],"default":null},{"name":"translated_from","type":["null","string"],"de
fault":null},{"name":"story_published_partition_date","type":["null","string"],"default":null}]}, table schema :{"type":"record","name":"articles_hudi_copy_on_write_record","namespace":"hoodie.articles_hudi_copy_on_write","fields":[{"name":"_hoodie_commit_time","type":["null","string"],"doc":"","default":null},{"name":"_hoodie_commit_seqno","type":["null","string"],"doc":"","default":null},{"name":"_hoodie_record_key","type":["null","string"],"doc":"","default":null},{"name":"_hoodie_partition_path","type":["null","string"],"doc":"","default":null},{"name":"_hoodie_file_name","type":["null","string"],"doc":"","default":null},{"name":"id","type":["null","string"],"default":null},{"name":"version","type":["null","long"],"default":null},{"name":"aggregation_id","type":["null","string"],"default":null},{"name":"area_on_page","type":["null","long"],"default":null},{"name":"article_cursor","type":["null","string"],"default":null},{"name":"article_deduplication_id","type":["null","string"]
,"default":null},{"name":"article_earliest_published_date","type":["null","string"],"default":null},{"name":"author","type":["null","string"],"default":null},{"name":"canonical_source_id","type":["null","string"],"default":null},{"name":"canonical_source_location","type":["null",{"type":"record","name":"canonical_source_location","namespace":"hoodie.articles_hudi_copy_on_write.articles_hudi_copy_on_write_record","fields":[{"name":"country","type":["null","string"],"default":null},{"name":"region","type":["null","string"],"default":null},{"name":"subregion","type":["null","string"],"default":null}]}],"default":null},{"name":"canonical_source_name","type":["null","string"],"default":null},{"name":"canonical_source_reach","type":["null",{"type":"record","name":"canonical_source_reach","namespace":"hoodie.articles_hudi_copy_on_write.articles_hudi_copy_on_write_record","fields":[{"name":"reach_origin","type":["null","string"],"default":null},{"name":"reach_provider","type":["null","strin
g"],"default":null},{"name":"source_id","type":["null","string"],"default":null},{"name":"type","type":["null","string"],"default":null},{"name":"value","type":["null","long"],"default":null}]}],"default":null},{"name":"content","type":["null","string"],"default":null},{"name":"deleted_date","type":["null","string"],"default":null},{"name":"document_type","type":["null","string"],"default":null},{"name":"eclips_web_url","type":["null","string"],"default":null},{"name":"embargoed_until_date","type":["null","string"],"default":null},{"name":"entities","type":["null",{"type":"array","items":[{"type":"record","name":"entities","namespace":"hoodie.articles_hudi_copy_on_write.articles_hudi_copy_on_write_record","fields":[{"name":"offset","type":["null","long"],"default":null},{"name":"overlapping","type":["null","boolean"],"default":null},{"name":"position","type":["null","string"],"default":null},{"name":"rule_based_entity","type":["null","boolean"],"default":null},{"name":"sentiment","t
ype":["null",{"type":"record","name":"sentiment","namespace":"hoodie.articles_hudi_copy_on_write.articles_hudi_copy_on_write_record.entities","fields":[{"name":"compound","type":["null","double"],"default":null},{"name":"neg","type":["null","double"],"default":null},{"name":"neu","type":["null","double"],"default":null},{"name":"pos","type":["null","double"],"default":null}]}],"default":null},{"name":"signal_type","type":["null","string"],"default":null},{"name":"surface_form","type":["null","string"],"default":null},{"name":"wiki_title","type":["null","string"],"default":null}]},"null"]}],"default":null},{"name":"entity_salience","type":["null",{"type":"array","items":[{"type":"record","name":"entity_salience","namespace":"hoodie.articles_hudi_copy_on_write.articles_hudi_copy_on_write_record","fields":[{"name":"salience","type":["null","double"],"default":null},{"name":"salience_rank","type":["null","long"],"default":null},{"name":"wiki_title","type":["null","string"],"default":nul
l}]},"null"]}],"default":null},{"name":"feed","type":["null","string"],"default":null},{"name":"format_version","type":["null","long"],"default":null},{"name":"ingestion_id","type":["null","string"],"default":null},{"name":"ingestion_metadata","type":["null",{"type":"record","name":"ingestion_metadata","namespace":"hoodie.articles_hudi_copy_on_write.articles_hudi_copy_on_write_record","fields":[{"name":"replay_time","type":["null","string"],"default":null},{"name":"replay_type","type":["null","string"],"default":null}]}],"default":null},{"name":"journalist_id","type":["null","string"],"default":null},{"name":"journalist_name","type":["null","string"],"default":null},{"name":"journalistic_quality","type":["null","double"],"default":null},{"name":"language","type":["null","string"],"default":null},{"name":"licence_id","type":["null","string"],"default":null},{"name":"media_type","type":["null","string"],"default":null},{"name":"metadata_keys","type":["null",{"type":"array","items":["s
tring","null"]}],"default":null},{"name":"native_content","type":["null",{"type":"record","name":"native_content","namespace":"hoodie.articles_hudi_copy_on_write.articles_hudi_copy_on_write_record","fields":[{"name":"content","type":["null","string"],"default":null},{"name":"summary","type":["null","string"],"default":null},{"name":"title","type":["null","string"],"default":null}]}],"default":null},{"name":"old_article_ids","type":["null","string"],"default":null},{"name":"original_id","type":["null","string"],"default":null},{"name":"original_source_id","type":["null","string"],"default":null},{"name":"original_source_name","type":["null","string"],"default":null},{"name":"original_url","type":["null","string"],"default":null},{"name":"page","type":["null","long"],"default":null},{"name":"page_section","type":["null","string"],"default":null},{"name":"pdf_url","type":["null","string"],"default":null},{"name":"processed_date","type":["null","string"],"default":null},{"name":"provide
r_data","type":["null",{"type":"record","name":"provider_data","namespace":"hoodie.articles_hudi_copy_on_write.articles_hudi_copy_on_write_record","fields":[{"name":"podcast_link","type":["null","string"],"default":null},{"name":"copyright","type":["null","string"],"default":null},{"name":"formatted_content","type":["null","string"],"default":null},{"name":"nla_publisher","type":["null","string"],"default":null},{"name":"provider_hosted_url","type":["null","string"],"default":null},{"name":"publication_time","type":["null","string"],"default":null},{"name":"tvplayer","type":["null",{"type":"record","name":"tvplayer","namespace":"hoodie.articles_hudi_copy_on_write.articles_hudi_copy_on_write_record.provider_data","fields":[{"name":"end_date","type":["null","string"],"default":null},{"name":"start_date","type":["null","string"],"default":null},{"name":"station_id","type":["null","string"],"default":null},{"name":"asset_id","type":["null","string"],"default":null},{"name":"partner_id",
"type":["null","long"],"default":null}]}],"default":null}]}],"default":null},{"name":"published_date","type":["null","string"],"default":null},{"name":"quotes","type":["null",{"type":"array","items":[{"type":"record","name":"quotes","namespace":"hoodie.articles_hudi_copy_on_write.articles_hudi_copy_on_write_record","fields":[{"name":"end","type":["null","long"],"default":null},{"name":"start","type":["null","long"],"default":null},{"name":"text","type":["null","string"],"default":null}]},"null"]}],"default":null},{"name":"received","type":["null","string"],"default":null},{"name":"signal_importance","type":["null","long"],"default":null},{"name":"social_engagement","type":["null",{"type":"record","name":"social_engagement","namespace":"hoodie.articles_hudi_copy_on_write.articles_hudi_copy_on_write_record","fields":[{"name":"twitter","type":["null",{"type":"record","name":"twitter","namespace":"hoodie.articles_hudi_copy_on_write.articles_hudi_copy_on_write_record.social_engagement","
fields":[{"name":"shares","type":["null","long"],"default":null}]}],"default":null}]}],"default":null},{"name":"source_groups","type":["null",{"type":"array","items":["string","null"]}],"default":null},{"name":"story_cursor","type":["null","string"],"default":null},{"name":"story_id","type":["null","string"],"default":null},{"name":"story_index","type":["null","string"],"default":null},{"name":"story_processed_date","type":["null","string"],"default":null},{"name":"story_published_date","type":["null","string"],"default":null},{"name":"summary","type":["null","string"],"default":null},{"name":"summary_origin","type":["null","string"],"default":null},{"name":"taxonomy_categories","type":["null",{"type":"array","items":[{"type":"record","name":"taxonomy_categories","namespace":"hoodie.articles_hudi_copy_on_write.articles_hudi_copy_on_write_record","fields":[{"name":"id","type":["null","string"],"default":null},{"name":"score","type":["null","double"],"default":null}]},"null"]}],"defau
lt":null},{"name":"title","type":["null","string"],"default":null},{"name":"topic_predictions","type":["null",{"type":"array","items":[{"type":"record","name":"topic_predictions","namespace":"hoodie.articles_hudi_copy_on_write.articles_hudi_copy_on_write_record","fields":[{"name":"probability","type":["null","double"],"default":null},{"name":"topic_id","type":["null","string"],"default":null}]},"null"]}],"default":null},{"name":"topics","type":["null",{"type":"array","items":["string","null"]}],"default":null},{"name":"tracking_url","type":["null","string"],"default":null},{"name":"translated_from","type":["null","string"],"default":null},{"name":"story_published_partition_date","type":["null","string"],"default":null}]}, base path :s3://prod-signal-articles-store/articles_hudi_copy_on_write
at org.apache.hudi.table.HoodieTable.validateSchema(HoodieTable.java:631)
at org.apache.hudi.table.HoodieTable.validateUpsertSchema(HoodieTable.java:637)
... 12 more
)
21/11/09 10:24:26 INFO ApplicationMaster: Unregistering ApplicationMaster with FAILED (diag message: User class threw exception: org.apache.hudi.exception.HoodieUpsertException: Failed upsert schema compatibility check.
at org.apache.hudi.table.HoodieTable.validateUpsertSchema(HoodieTable.java:639)
at org.apache.hudi.client.SparkRDDWriteClient.upsert(SparkRDDWriteClient.java:155)
at org.apache.hudi.utilities.deltastreamer.DeltaSync.writeToSink(DeltaSync.java:484)
at org.apache.hudi.utilities.deltastreamer.DeltaSync.syncOnce(DeltaSync.java:303)
at org.apache.hudi.utilities.deltastreamer.HoodieDeltaStreamer.lambda$sync$2(HoodieDeltaStreamer.java:182)
at org.apache.hudi.common.util.Option.ifPresent(Option.java:96)
at org.apache.hudi.utilities.deltastreamer.HoodieDeltaStreamer.sync(HoodieDeltaStreamer.java:180)
at org.apache.hudi.utilities.deltastreamer.HoodieDeltaStreamer.main(HoodieDeltaStreamer.java:509)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.spark.deploy.yarn.ApplicationMaster$$anon$2.run(ApplicationMaster.scala:735)
Caused by: org.apache.hudi.exception.HoodieException: Failed schema compatibility check for writerSchema :{"type":"record","name":"hoodie_source","namespace":"hoodie.source","fields":[{"name":"_hoodie_commit_time","type":["null","string"],"doc":"","default":null},{"name":"_hoodie_commit_seqno","type":["null","string"],"doc":"","default":null},{"name":"_hoodie_record_key","type":["null","string"],"doc":"","default":null},{"name":"_hoodie_partition_path","type":["null","string"],"doc":"","default":null},{"name":"_hoodie_file_name","type":["null","string"],"doc":"","default":null},{"name":"id","type":["null","string"],"default":null},{"name":"version","type":["null","long"],"default":null},{"name":"aggregation_id","type":["null","string"],"default":null},{"name":"area_on_page","type":["null","long"],"default":null},{"name":"article_cursor","type":["null","string"],"default":null},{"name":"article_deduplication_id","type":["null","string"],"default":null},{"name":"article_earliest_pub
lished_date","type":["null","string"],"default":null},{"name":"author","type":["null","string"],"default":null},{"name":"canonical_source_id","type":["null","string"],"default":null},{"name":"canonical_source_location","type":["null",{"type":"record","name":"canonical_source_location","namespace":"hoodie.source.hoodie_source","fields":[{"name":"country","type":["null","string"],"default":null},{"name":"region","type":["null","string"],"default":null},{"name":"subregion","type":["null","string"],"default":null}]}],"default":null},{"name":"canonical_source_name","type":["null","string"],"default":null},{"name":"canonical_source_reach","type":["null",{"type":"record","name":"canonical_source_reach","namespace":"hoodie.source.hoodie_source","fields":[{"name":"reach_origin","type":["null","string"],"default":null},{"name":"reach_provider","type":["null","string"],"default":null},{"name":"source_id","type":["null","string"],"default":null},{"name":"type","type":["null","string"],"default"
:null},{"name":"value","type":["null","long"],"default":null}]}],"default":null},{"name":"content","type":["null","string"],"default":null},{"name":"deleted_date","type":["null","string"],"default":null},{"name":"document_type","type":["null","string"],"default":null},{"name":"eclips_web_url","type":["null","string"],"default":null},{"name":"embargoed_until_date","type":["null","string"],"default":null},{"name":"entities","type":["null",{"type":"array","items":[{"type":"record","name":"entities","namespace":"hoodie.source.hoodie_source","fields":[{"name":"offset","type":["null","long"],"default":null},{"name":"overlapping","type":["null","boolean"],"default":null},{"name":"position","type":["null","string"],"default":null},{"name":"rule_based_entity","type":["null","boolean"],"default":null},{"name":"sentiment","type":["null",{"type":"record","name":"sentiment","namespace":"hoodie.source.hoodie_source.entities","fields":[{"name":"compound","type":["null","double"],"default":null},{"
name":"neg","type":["null","double"],"default":null},{"name":"neu","type":["null","double"],"default":null},{"name":"pos","type":["null","double"],"default":null}]}],"default":null},{"name":"signal_type","type":["null","string"],"default":null},{"name":"surface_form","type":["null","string"],"default":null},{"name":"wiki_title","type":["null","string"],"default":null}]},"null"]}],"default":null},{"name":"entity_salience","type":["null",{"type":"array","items":[{"type":"record","name":"entity_salience","namespace":"hoodie.source.hoodie_source","fields":[{"name":"salience","type":["null","double"],"default":null},{"name":"salience_rank","type":["null","long"],"default":null},{"name":"wiki_title","type":["null","string"],"default":null}]},"null"]}],"default":null},{"name":"feed","type":["null","string"],"default":null},{"name":"format_version","type":["null","long"],"default":null},{"name":"ingestion_id","type":["null","string"],"default":null},{"name":"ingestion_metadata","type":["nul
l",{"type":"record","name":"ingestion_metadata","namespace":"hoodie.source.hoodie_source","fields":[{"name":"replay_time","type":["null","string"],"default":null},{"name":"replay_type","type":["null","string"],"default":null}]}],"default":null},{"name":"journalist_id","type":["null","string"],"default":null},{"name":"journalist_name","type":["null","string"],"default":null},{"name":"journalistic_quality","type":["null","double"],"default":null},{"name":"language","type":["null","string"],"default":null},{"name":"licence_id","type":["null","string"],"default":null},{"name":"media_type","type":["null","string"],"default":null},{"name":"metadata_keys","type":["null",{"type":"array","items":["string","null"]}],"default":null},{"name":"native_content","type":["null",{"type":"record","name":"native_content","namespace":"hoodie.source.hoodie_source","fields":[{"name":"content","type":["null","string"],"default":null},{"name":"summary","type":["null","string"],"default":null},{"name":"title
","type":["null","string"],"default":null}]}],"default":null},{"name":"old_article_ids","type":["null","string"],"default":null},{"name":"original_id","type":["null","string"],"default":null},{"name":"original_source_id","type":["null","string"],"default":null},{"name":"original_source_name","type":["null","string"],"default":null},{"name":"original_url","type":["null","string"],"default":null},{"name":"page","type":["null","long"],"default":null},{"name":"page_section","type":["null","string"],"default":null},{"name":"pdf_url","type":["null","string"],"default":null},{"name":"processed_date","type":["null","string"],"default":null},{"name":"provider_data","type":["null",{"type":"record","name":"provider_data","namespace":"hoodie.source.hoodie_source","fields":[{"name":"podcast_link","type":["null","string"],"default":null},{"name":"copyright","type":["null","string"],"default":null},{"name":"formatted_content","type":["null","string"],"default":null},{"name":"nla_publisher","type":
["null","string"],"default":null},{"name":"provider_hosted_url","type":["null","string"],"default":null},{"name":"publication_time","type":["null","string"],"default":null},{"name":"tvplayer","type":["null",{"type":"record","name":"tvplayer","namespace":"hoodie.source.hoodie_source.provider_data","fields":[{"name":"end_date","type":["null","string"],"default":null},{"name":"start_date","type":["null","string"],"default":null},{"name":"station_id","type":["null","string"],"default":null},{"name":"asset_id","type":["null","string"],"default":null},{"name":"partner_id","type":["null","long"],"default":null}]}],"default":null}]}],"default":null},{"name":"published_date","type":["null","string"],"default":null},{"name":"quotes","type":["null",{"type":"array","items":[{"type":"record","name":"quotes","namespace":"hoodie.source.hoodie_source","fields":[{"name":"end","type":["null","long"],"default":null},{"name":"start","type":["null","long"],"default":null},{"name":"text","type":["null","
string"],"default":null}]},"null"]}],"default":null},{"name":"received","type":["null","string"],"default":null},{"name":"signal_importance","type":["null","long"],"default":null},{"name":"social_engagement","type":["null",{"type":"record","name":"social_engagement","namespace":"hoodie.source.hoodie_source","fields":[{"name":"twitter","type":["null",{"type":"record","name":"twitter","namespace":"hoodie.source.hoodie_source.social_engagement","fields":[{"name":"shares","type":["null","long"],"default":null}]}],"default":null}]}],"default":null},{"name":"source_groups","type":["null",{"type":"array","items":["string","null"]}],"default":null},{"name":"story_cursor","type":["null","string"],"default":null},{"name":"story_id","type":["null","string"],"default":null},{"name":"story_index","type":["null","string"],"default":null},{"name":"story_processed_date","type":["null","string"],"default":null},{"name":"story_published_date","type":["null","string"],"default":null},{"name":"summary"
,"type":["null","string"],"default":null},{"name":"summary_origin","type":["null","string"],"default":null},{"name":"taxonomy_categories","type":["null",{"type":"array","items":[{"type":"record","name":"taxonomy_categories","namespace":"hoodie.source.hoodie_source","fields":[{"name":"id","type":["null","string"],"default":null},{"name":"score","type":["null","double"],"default":null}]},"null"]}],"default":null},{"name":"title","type":["null","string"],"default":null},{"name":"topic_predictions","type":["null",{"type":"array","items":[{"type":"record","name":"topic_predictions","namespace":"hoodie.source.hoodie_source","fields":[{"name":"probability","type":["null","double"],"default":null},{"name":"topic_id","type":["null","string"],"default":null}]},"null"]}],"default":null},{"name":"topics","type":["null",{"type":"array","items":["string","null"]}],"default":null},{"name":"tracking_url","type":["null","string"],"default":null},{"name":"translated_from","type":["null","string"],"de
fault":null},{"name":"story_published_partition_date","type":["null","string"],"default":null}]}, table schema :{"type":"record","name":"articles_hudi_copy_on_write_record","namespace":"hoodie.articles_hudi_copy_on_write","fields":[{"name":"_hoodie_commit_time","type":["null","string"],"doc":"","default":null},{"name":"_hoodie_commit_seqno","type":["null","string"],"doc":"","default":null},{"name":"_hoodie_record_key","type":["null","string"],"doc":"","default":null},{"name":"_hoodie_partition_path","type":["null","string"],"doc":"","default":null},{"name":"_hoodie_file_name","type":["null","string"],"doc":"","default":null},{"name":"id","type":["null","string"],"default":null},{"name":"version","type":["null","long"],"default":null},{"name":"aggregation_id","type":["null","string"],"default":null},{"name":"area_on_page","type":["null","long"],"default":null},{"name":"article_cursor","type":["null","string"],"default":null},{"name":"article_deduplication_id","type":["null","string"]
,"default":null},{"name":"article_earliest_published_date","type":["null","string"],"default":null},{"name":"author","type":["null","string"],"default":null},{"name":"canonical_source_id","type":["null","string"],"default":null},{"name":"canonical_source_location","type":["null",{"type":"record","name":"canonical_source_location","namespace":"hoodie.articles_hudi_copy_on_write.articles_hudi_copy_on_write_record","fields":[{"name":"country","type":["null","string"],"default":null},{"name":"region","type":["null","string"],"default":null},{"name":"subregion","type":["null","string"],"default":null}]}],"default":null},{"name":"canonical_source_name","type":["null","string"],"default":null},{"name":"canonical_source_reach","type":["null",{"type":"record","name":"canonical_source_reach","namespace":"hoodie.articles_hudi_copy_on_write.articles_hudi_copy_on_write_record","fields":[{"name":"reach_origin","type":["null","string"],"default":null},{"name":"reach_provider","type":["null","strin
g"],"default":null},{"name":"source_id","type":["null","string"],"default":null},{"name":"type","type":["null","string"],"default":null},{"name":"value","type":["null","long"],"default":null}]}],"default":null},{"name":"content","type":["null","string"],"default":null},{"name":"deleted_date","type":["null","string"],"default":null},{"name":"document_type","type":["null","string"],"default":null},{"name":"eclips_web_url","type":["null","string"],"default":null},{"name":"embargoed_until_date","type":["null","string"],"default":null},{"name":"entities","type":["null",{"type":"array","items":[{"type":"record","name":"entities","namespace":"hoodie.articles_hudi_copy_on_write.articles_hudi_copy_on_write_record","fields":[{"name":"offset","type":["null","long"],"default":null},{"name":"overlapping","type":["null","boolean"],"default":null},{"name":"position","type":["null","string"],"default":null},{"name":"rule_based_entity","type":["null","boolean"],"default":null},{"name":"sentiment","t
ype":["null",{"type":"record","name":"sentiment","namespace":"hoodie.articles_hudi_copy_on_write.articles_hudi_copy_on_write_record.entities","fields":[{"name":"compound","type":["null","double"],"default":null},{"name":"neg","type":["null","double"],"default":null},{"name":"neu","type":["null","double"],"default":null},{"name":"pos","type":["null","double"],"default":null}]}],"default":null},{"name":"signal_type","type":["null","string"],"default":null},{"name":"surface_form","type":["null","string"],"default":null},{"name":"wiki_title","type":["null","string"],"default":null}]},"null"]}],"default":null},{"name":"entity_salience","type":["null",{"type":"array","items":[{"type":"record","name":"entity_salience","namespace":"hoodie.articles_hudi_copy_on_write.articles_hudi_copy_on_write_record","fields":[{"name":"salience","type":["null","double"],"default":null},{"name":"salience_rank","type":["null","long"],"default":null},{"name":"wiki_title","type":["null","string"],"default":nul
l}]},"null"]}],"default":null},{"name":"feed","type":["null","string"],"default":null},{"name":"format_version","type":["null","long"],"default":null},{"name":"ingestion_id","type":["null","string"],"default":null},{"name":"ingestion_metadata","type":["null",{"type":"record","name":"ingestion_metadata","namespace":"hoodie.articles_hudi_copy_on_write.articles_hudi_copy_on_write_record","fields":[{"name":"replay_time","type":["null","string"],"default":null},{"name":"replay_type","type":["null","string"],"default":null}]}],"default":null},{"name":"journalist_id","type":["null","string"],"default":null},{"name":"journalist_name","type":["null","string"],"default":null},{"name":"journalistic_quality","type":["null","double"],"default":null},{"name":"language","type":["null","string"],"default":null},{"name":"licence_id","type":["null","string"],"default":null},{"name":"media_type","type":["null","string"],"default":null},{"name":"metadata_keys","type":["null",{"type":"array","items":["s
tring","null"]}],"default":null},{"name":"native_content","type":["null",{"type":"record","name":"native_content","namespace":"hoodie.articles_hudi_copy_on_write.articles_hudi_copy_on_write_record","fields":[{"name":"content","type":["null","string"],"default":null},{"name":"summary","type":["null","string"],"default":null},{"name":"title","type":["null","string"],"default":null}]}],"default":null},{"name":"old_article_ids","type":["null","string"],"default":null},{"name":"original_id","type":["null","string"],"default":null},{"name":"original_source_id","type":["null","string"],"default":null},{"name":"original_source_name","type":["null","string"],"default":null},{"name":"original_url","type":["null","string"],"default":null},{"name":"page","type":["null","long"],"default":null},{"name":"page_section","type":["null","string"],"default":null},{"name":"pdf_url","type":["null","string"],"default":null},{"name":"processed_date","type":["null","string"],"default":null},{"name":"provide
r_data","type":["null",{"type":"record","name":"provider_data","namespace":"hoodie.articles_hudi_copy_on_write.articles_hudi_copy_on_write_record","fields":[{"name":"podcast_link","type":["null","string"],"default":null},{"name":"copyright","type":["null","string"],"default":null},{"name":"formatted_content","type":["null","string"],"default":null},{"name":"nla_publisher","type":["null","string"],"default":null},{"name":"provider_hosted_url","type":["null","string"],"default":null},{"name":"publication_time","type":["null","string"],"default":null},{"name":"tvplayer","type":["null",{"type":"record","name":"tvplayer","namespace":"hoodie.articles_hudi_copy_on_write.articles_hudi_copy_on_write_record.provider_data","fields":[{"name":"end_date","type":["null","string"],"default":null},{"name":"start_date","type":["null","string"],"default":null},{"name":"station_id","type":["null","string"],"default":null},{"name":"asset_id","type":["null","string"],"default":null},{"name":"partner_id",
"type":["null","long"],"default":null}]}],"default":null}]}],"default":null},{"name":"published_date","type":["null","string"],"default":null},{"name":"quotes","type":["null",{"type":"array","items":[{"type":"record","name":"quotes","namespace":"hoodie.articles_hudi_copy_on_write.articles_hudi_copy_on_write_record","fields":[{"name":"end","type":["null","long"],"default":null},{"name":"start","type":["null","long"],"default":null},{"name":"text","type":["null","string"],"default":null}]},"null"]}],"default":null},{"name":"received","type":["null","string"],"default":null},{"name":"signal_importance","type":["null","long"],"default":null},{"name":"social_engagement","type":["null",{"type":"record","name":"social_engagement","namespace":"hoodie.articles_hudi_copy_on_write.articles_hudi_copy_on_write_record","fields":[{"name":"twitter","type":["null",{"type":"record","name":"twitter","namespace":"hoodie.articles_hudi_copy_on_write.articles_hudi_copy_on_write_record.social_engagement","
fields":[{"name":"shares","type":["null","long"],"default":null}]}],"default":null}]}],"default":null},{"name":"source_groups","type":["null",{"type":"array","items":["string","null"]}],"default":null},{"name":"story_cursor","type":["null","string"],"default":null},{"name":"story_id","type":["null","string"],"default":null},{"name":"story_index","type":["null","string"],"default":null},{"name":"story_processed_date","type":["null","string"],"default":null},{"name":"story_published_date","type":["null","string"],"default":null},{"name":"summary","type":["null","string"],"default":null},{"name":"summary_origin","type":["null","string"],"default":null},{"name":"taxonomy_categories","type":["null",{"type":"array","items":[{"type":"record","name":"taxonomy_categories","namespace":"hoodie.articles_hudi_copy_on_write.articles_hudi_copy_on_write_record","fields":[{"name":"id","type":["null","string"],"default":null},{"name":"score","type":["null","double"],"default":null}]},"null"]}],"defau
lt":null},{"name":"title","type":["null","string"],"default":null},{"name":"topic_predictions","type":["null",{"type":"array","items":[{"type":"record","name":"topic_predictions","namespace":"hoodie.articles_hudi_copy_on_write.articles_hudi_copy_on_write_record","fields":[{"name":"probability","type":["null","double"],"default":null},{"name":"topic_id","type":["null","string"],"default":null}]},"null"]}],"default":null},{"name":"topics","type":["null",{"type":"array","items":["string","null"]}],"default":null},{"name":"tracking_url","type":["null","string"],"default":null},{"name":"translated_from","type":["null","string"],"default":null},{"name":"story_published_partition_date","type":["null","string"],"default":null}]}, base path :s3://prod-signal-articles-store/articles_hudi_copy_on_write
at org.apache.hudi.table.HoodieTable.validateSchema(HoodieTable.java:631)
at org.apache.hudi.table.HoodieTable.validateUpsertSchema(HoodieTable.java:637)
... 12 more
)
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] xushiyan commented on issue #3953: [SUPPORT] Schema validation always fails if mixing writers (Deltastreamer/Spark)
Posted by GitBox <gi...@apache.org>.
xushiyan commented on issue #3953:
URL: https://github.com/apache/hudi/issues/3953#issuecomment-967735057
@Limess thanks for reporting this. yes it should be consistent. I took a further look at `org.apache.avro.SchemaCompatibility#schemaNameEquals()` it also checks for `if (reader.getAliases().contains(writerFullName))`. So we can add aliases on deltastreamer path and datasource writer path to make each other compatible.
https://issues.apache.org/jira/browse/HUDI-2755
Filed this ticket and closing the issue.
@Limess would you be keen taking that ticket, since you have a good understanding of the problem already?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] xushiyan closed issue #3953: [SUPPORT] Schema validation always fails if mixing writers (Deltastreamer/Spark)
Posted by GitBox <gi...@apache.org>.
xushiyan closed issue #3953:
URL: https://github.com/apache/hudi/issues/3953
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org