You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by GitBox <gi...@apache.org> on 2023/01/13 08:32:03 UTC
[GitHub] [hudi] pushpavanthar opened a new issue, #7667: [SUPPORT] ambiguous table type. Seeing both MERGE_ON_READ and COPY_ON_WRITE
pushpavanthar opened a new issue, #7667:
URL: https://github.com/apache/hudi/issues/7667
**Describe the problem you faced**
In Hudi 0.11.1, hudi has made `hoodie.metadata.enable=true` by default. In logs I see that `HoodieTableMetaClient` loading table as type `COPY_ON_WRITE(version=1, baseFileFormat=PARQUET)` from base path and same class loading table as `MERGE_ON_READ(version=1, baseFileFormat=HFILE)` from the metadata path.
Steps to reproduce the behavior:
1. Deploy HoodieDeltaStreamer in continuous mode with `hoodie.metadata.enable=true` and table type `COPY_ON_WRITE` with below configs
```
acks: all
auto.offset.reset: earliest
bootstrap.servers: kafka:9092
client.dns.lookup: use_all_dns_ips
group.id: hudi-cow-continuous-credit-analysis-data
hive.metastore.disallow.incompatible.col.type.changes: false
hoodie.archive.async: true
hoodie.archive.automatic: true
hoodie.archive.delete.parallelism: 500
hoodie.archive.merge.enable: true
hoodie.archive.merge.files.batch.size: 20
hoodie.auto.adjust.lock.configs: true
hoodie.bloom.index.update.partition.path: false
hoodie.bloom.index.use.metadata: true
hoodie.clean.allow.multiple: false
hoodie.clean.async: true
hoodie.clean.automatic: true
hoodie.clean.max.commits: 10
hoodie.cleaner.hours.retained: 1
hoodie.cleaner.incremental.mode: true
hoodie.cleaner.parallelism: 500
hoodie.cleaner.policy: KEEP_LATEST_BY_HOURS
hoodie.clustering.async.enabled: false
hoodie.clustering.async.max.commits: 1
hoodie.clustering.execution.strategy.class: org.apache.hudi.client.clustering.run.strategy.SparkSortAndSizeExecutionStrategy
hoodie.clustering.inline: false
hoodie.clustering.plan.strategy.class: org.apache.hudi.client.clustering.plan.strategy.SparkSizeBasedClusteringPlanStrategy
hoodie.clustering.plan.strategy.small.file.limit: 629145600
hoodie.clustering.plan.strategy.target.file.max.bytes: 1073741824
hoodie.commits.archival.batch: 20
hoodie.datasource.hive_sync.database: test_clustering
hoodie.datasource.hive_sync.partition_extractor_class: org.apache.hudi.hive.NonPartitionedExtractor
hoodie.datasource.hive_sync.table: cow_credit_analysis_data
hoodie.datasource.write.keygenerator.class: org.apache.hudi.keygen.NonpartitionedKeyGenerator
hoodie.datasource.write.partitionpath.field: ''
hoodie.datasource.write.precombine.field: __lsn
hoodie.datasource.write.reconcile.schema: false
hoodie.datasource.write.recordkey.field: id
hoodie.deltastreamer.schemaprovider.registry.url: https://schema_registry/subjects/lending_customer_service.public.credit_analysis_data-value/versions/latest
hoodie.deltastreamer.schemaprovider.spark_avro_post_processor.enable: false
hoodie.deltastreamer.source.kafka.auto.reset.offsets: earliest
hoodie.deltastreamer.source.kafka.enable.commit.offset: true
hoodie.deltastreamer.source.kafka.topic: lending_customer_service.public.credit_analysis_data
hoodie.index.type: BLOOM
hoodie.keep.max.commits: 800
hoodie.keep.min.commits: 600
hoodie.metrics.on: true
hoodie.metrics.pushgateway.delete.on.shutdown: false
hoodie.metrics.pushgateway.host: pushgateway
hoodie.metrics.pushgateway.job.name: hudi_cow_continuous_credit_analysis_data
hoodie.metrics.pushgateway.port: 443
hoodie.metrics.pushgateway.random.job.name.suffix: false
hoodie.metrics.reporter.metricsname.prefix: hudi
hoodie.metrics.reporter.type: PROMETHEUS_PUSHGATEWAY
hoodie.parquet.compression.codec: snappy
partition.assignment.strategy: org.apache.kafka.clients.consumer.RangeAssignor
sasl.jaas.config: org.apache.kafka.common.security.plain.PlainLoginModule required username='***************' password='***************';
sasl.mechanism: PLAIN
schema.registry.url: https://schema_registry
security.protocol: SASL_SSL
```
2. Check logs for `HoodieTableMetaClient` of the pattern `HoodieTableMetaClient: Finished Loading Table of type `
**Expected behavior**
Is this expected behaviour for the table config read from metadata hoodie.properties? isn't this is misleading. Aren't we supposed to have consistent table type within metadata as well? I've been noticing lot of issues when metadata is enabled, not sure if this is the root cause.
**Environment Description**
* Hudi version : 0.11.1
* Spark version : 3.1.1
* Hive version : 3.1.2
* Hadoop version :
* Storage (HDFS/S3/GCS..) : S3
* Running on Docker? (yes/no) : no
**Additional context**
I'm attaching driver logs for better understanding of the problem. I can share entire driver logs if required.
```
23/01/13 07:19:55 INFO [pool-31-thread-1] HoodieTableMetaClient: Loading HoodieTableMetaClient from s3://datalake_bucket/test/hudi_poc/continuous_cow/cow_credit_analysis_data
23/01/13 07:19:55 INFO [pool-31-thread-1] HoodieTableConfig: Loading table properties from s3://datalake_bucket/test/hudi_poc/continuous_cow/cow_credit_analysis_data/.hoodie/hoodie.properties
23/01/13 07:19:55 INFO [pool-31-thread-1] S3NativeFileSystem: Opening 's3://datalake_bucket/test/hudi_poc/continuous_cow/cow_credit_analysis_data/.hoodie/hoodie.properties' for reading
23/01/13 07:19:55 INFO [pool-31-thread-1] HoodieTableMetaClient: Finished Loading Table of type COPY_ON_WRITE(version=1, baseFileFormat=PARQUET) from s3://datalake_bucket/test/hudi_poc/continuous_cow/cow_credit_analysis_data
23/01/13 07:19:55 INFO [pool-31-thread-1] HoodieTableMetaClient: Loading HoodieTableMetaClient from s3://datalake_bucket/test/hudi_poc/continuous_cow/cow_credit_analysis_data/.hoodie/metadata
23/01/13 07:19:55 INFO [pool-31-thread-1] HoodieTableConfig: Loading table properties from s3://datalake_bucket/test/hudi_poc/continuous_cow/cow_credit_analysis_data/.hoodie/metadata/.hoodie/hoodie.properties
23/01/13 07:19:55 INFO [pool-31-thread-1] S3NativeFileSystem: Opening 's3://datalake_bucket/test/hudi_poc/continuous_cow/cow_credit_analysis_data/.hoodie/metadata/.hoodie/hoodie.properties' for reading
23/01/13 07:19:55 INFO [pool-31-thread-1] HoodieTableMetaClient: Finished Loading Table of type MERGE_ON_READ(version=1, baseFileFormat=HFILE) from s3://datalake_bucket/test/hudi_poc/continuous_cow/cow_credit_analysis_data/.hoodie/metadata
23/01/13 07:19:55 INFO [pool-31-thread-1] HoodieTableMetaClient: Loading HoodieTableMetaClient from s3://datalake_bucket/test/hudi_poc/continuous_cow/cow_credit_analysis_data
23/01/13 07:19:55 INFO [pool-31-thread-1] HoodieTableConfig: Loading table properties from s3://datalake_bucket/test/hudi_poc/continuous_cow/cow_credit_analysis_data/.hoodie/hoodie.properties
23/01/13 07:19:55 INFO [pool-31-thread-1] S3NativeFileSystem: Opening 's3://datalake_bucket/test/hudi_poc/continuous_cow/cow_credit_analysis_data/.hoodie/hoodie.properties' for reading
23/01/13 07:19:55 INFO [pool-31-thread-1] HoodieTableMetaClient: Finished Loading Table of type COPY_ON_WRITE(version=1, baseFileFormat=PARQUET) from s3://datalake_bucket/test/hudi_poc/continuous_cow/cow_credit_analysis_data
23/01/13 07:19:55 INFO [pool-31-thread-1] HoodieTableMetaClient: Loading HoodieTableMetaClient from s3://datalake_bucket/test/hudi_poc/continuous_cow/cow_credit_analysis_data/.hoodie/metadata
23/01/13 07:19:55 INFO [pool-31-thread-1] HoodieTableConfig: Loading table properties from s3://datalake_bucket/test/hudi_poc/continuous_cow/cow_credit_analysis_data/.hoodie/metadata/.hoodie/hoodie.properties
23/01/13 07:19:55 INFO [pool-31-thread-1] S3NativeFileSystem: Opening 's3://datalake_bucket/test/hudi_poc/continuous_cow/cow_credit_analysis_data/.hoodie/metadata/.hoodie/hoodie.properties' for reading
23/01/13 07:19:55 INFO [pool-31-thread-1] HoodieTableMetaClient: Finished Loading Table of type MERGE_ON_READ(version=1, baseFileFormat=HFILE) from s3://datalake_bucket/test/hudi_poc/continuous_cow/cow_credit_analysis_data/.hoodie/metadata
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] danny0405 commented on issue #7667: [SUPPORT] ambiguous table type. Seeing both MERGE_ON_READ and COPY_ON_WRITE
Posted by GitBox <gi...@apache.org>.
danny0405 commented on issue #7667:
URL: https://github.com/apache/hudi/issues/7667#issuecomment-1381521905
That is as expected and by design, the metadata table is actually a builtin Hudi table and it's always MOR table type(mianly for write efficiency). For internals to explain, the avro log records the Hfile data block and finally these logs can be compacted into HFile.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] danny0405 closed issue #7667: [SUPPORT] ambiguous table type. Seeing both MERGE_ON_READ and COPY_ON_WRITE
Posted by GitBox <gi...@apache.org>.
danny0405 closed issue #7667: [SUPPORT] ambiguous table type. Seeing both MERGE_ON_READ and COPY_ON_WRITE
URL: https://github.com/apache/hudi/issues/7667
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org