You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by GitBox <gi...@apache.org> on 2022/08/04 17:09:04 UTC
[GitHub] [hudi] SubashRanganathan opened a new issue, #6304: [SUPPORT]
SubashRanganathan opened a new issue, #6304:
URL: https://github.com/apache/hudi/issues/6304
Issue: Hudi MultiTable Deltastreamer not updating glue catalog when new column added on Source.
Hudi Version: 0.11
Description:
Hudi Multi Table deltaStreamer job does not update AWS Glue Catalog when new column is added on files ingested by DMS. The following property is used in the hudi configurations. "hoodie.meta.sync.client.tool.class=org.apache.hudi.aws.sync.AwsGlueCatalogSyncTool"
The hudi table.properties are as follows:
hoodie.datasource.write.recordkey.field=col_name
hoodie.datasource.write.precombine.field=ts
hoodie.datasource.write.partitionpath_field_opt_key=""
hoodie.deltastreamer.ingestion.targetBasePath=s3://hudi_s3_bucket_name/hudi/hudi_table_name/
hoodie.deltastreamer.source.dfs.root=s3://s3_dms_landing_bucket_name/source_folder
hoodie.avro.schema.validate=false
hoodie.datasource.write.keygenerator.class=org.apache.hudi.keygen.NonpartitionedKeyGenerator
hoodie.datasource.hive_sync.enable=true
hoodie.datasource.hive_sync.auto_create_database=true
hoodie.datasource.hive_sync.database=database_name
hoodie.datasource.hive_sync.table=table_name
hoodie.datasource.hive.partition_fields_opt_key=""
hoodie.schema.on.read.enable=true
hoodie.meta.sync.client.tool.class=org.apache.hudi.aws.sync.AwsGlueCatalogSyncTool
Config_properties are as follows:
hoodie.deltastreamer.ingestion.tablesToBeIngested= table_name
hoodie.deltastreamer.ingestion.default.res_bestand.configFile=file:///home/hadoop/hudi/config-table_name.properties
hoodie.datasource.write.keygenerator.class:org.apache.hudi.keygen.NonpartitionedKeyGenerator
hoodie.datasource.hive_sync.enable=true
hoodie.datasource.hive_sync.auto_create_database=true
hoodie.schema.on.read.enable=true
hoodie.meta.sync.client.tool.class=org.apache.hudi.aws.sync.AwsGlueCatalogSyncTool
The delta streamer spark-submit command:
spark-submit --jars "/usr/lib/spark/external/lib/spark-avro.jar,/usr/lib/hudi/hudi-spark-bundle_2.12-0.11.1.jar" \
--conf spark.sql.legacy.parquet.int96RebaseModeInRead=CORRECTED \
--conf spark.sql.legacy.parquet.int96RebaseModeInWrite=CORRECTED \
--conf spark.sql.legacy.parquet.datetimeRebaseModeInRead=corrected \
--conf spark.sql.legacy.parquet.datetimeRebaseModeInRead=legacy \
--conf spark.sql.legacy.parquet.datetimeRebaseModeInWrite=CORRECTED \
--conf spark.sql.parquet.mergeSchema=true \
--class org.apache.hudi.utilities.deltastreamer.HoodieMultiTableDeltaStreamer \
--master yarn --deploy-mode client /home/hadoop/hudi/hudi-utilities-bundle_2.12-0.11.1.jar \
--table-type COPY_ON_WRITE \
--props "file:///home/hadoop/hudi/config-source.properties" \
--config-folder "file:///home/hadoop/hudi/" \
--base-path-prefix "s3://hudi_s3_bucket_name/hudi/" \
--source-class org.apache.hudi.utilities.sources.ParquetDFSSource \
--source-ordering-field event_ts \
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan commented on issue #6304: Hudi MultiTable Deltastreamer not updating glue catalog when new column added on Source
Posted by GitBox <gi...@apache.org>.
nsivabalan commented on issue #6304:
URL: https://github.com/apache/hudi/issues/6304#issuecomment-1229262916
can you furnish driver logs when the sync happens. also, could you do spark.read.format("hudi") and verify that you could read latest hudi data w/o issues and could see the new column as well.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] codope closed issue #6304: Hudi MultiTable Deltastreamer not updating glue catalog when new column added on Source
Posted by "codope (via GitHub)" <gi...@apache.org>.
codope closed issue #6304: Hudi MultiTable Deltastreamer not updating glue catalog when new column added on Source
URL: https://github.com/apache/hudi/issues/6304
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] codope commented on issue #6304: Hudi MultiTable Deltastreamer not updating glue catalog when new column added on Source
Posted by GitBox <gi...@apache.org>.
codope commented on issue #6304:
URL: https://github.com/apache/hudi/issues/6304#issuecomment-1263443677
@brskiran1 @SubashRanganathan gentle ping^
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan commented on issue #6304: Hudi MultiTable Deltastreamer not updating glue catalog when new column added on Source
Posted by GitBox <gi...@apache.org>.
nsivabalan commented on issue #6304:
URL: https://github.com/apache/hudi/issues/6304#issuecomment-1301569129
@brskiran1 @SubashRanganathan : can you folks respond us w/ more info and logs please.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] yihua commented on issue #6304: Hudi MultiTable Deltastreamer not updating glue catalog when new column added on Source
Posted by GitBox <gi...@apache.org>.
yihua commented on issue #6304:
URL: https://github.com/apache/hudi/issues/6304#issuecomment-1207686416
@SubashRanganathan have you contacted AWS support regarding this?
cc @zhedoubushishi @rahil-c who may also help here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] brskiran1 commented on issue #6304: Hudi MultiTable Deltastreamer not updating glue catalog when new column added on Source
Posted by GitBox <gi...@apache.org>.
brskiran1 commented on issue #6304:
URL: https://github.com/apache/hudi/issues/6304#issuecomment-1227907870
@rmahindra123 please let me know if you have an update on this? I have tried with hoodie.schema.on.read.enable ==true but still no change
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] brskiran1 commented on issue #6304: Hudi MultiTable Deltastreamer not updating glue catalog when new column added on Source
Posted by GitBox <gi...@apache.org>.
brskiran1 commented on issue #6304:
URL: https://github.com/apache/hudi/issues/6304#issuecomment-1225173640
@rmahindra123 responding on behalf of @SubashRanganathan . we have tried this without the flag hoodie.schema.on.read.enable set to true. Still dont see glue catalog updated with new column.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] rmahindra123 commented on issue #6304: Hudi MultiTable Deltastreamer not updating glue catalog when new column added on Source
Posted by GitBox <gi...@apache.org>.
rmahindra123 commented on issue #6304:
URL: https://github.com/apache/hudi/issues/6304#issuecomment-1223751364
@SubashRanganathan Adding a column is backward compatible, so you do not read to enable schema evolution, that is still experimental. Could you try re-runnning without the flag: hoodie.schema.on.read.enable set to true. Thanks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] codope commented on issue #6304: Hudi MultiTable Deltastreamer not updating glue catalog when new column added on Source
Posted by "codope (via GitHub)" <gi...@apache.org>.
codope commented on issue #6304:
URL: https://github.com/apache/hudi/issues/6304#issuecomment-1488046159
Closing due to inactivity. Have also tested the adding column scenario with glue sync for upcoming 0.12.3 release.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org