You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by GitBox <gi...@apache.org> on 2021/05/25 23:59:43 UTC
[GitHub] [hudi] jtmzheng opened a new issue #2995: [SUPPORT] Upserts creating duplicates after enabling metadata table in Hudi 0.7 indexing pipeline
jtmzheng opened a new issue #2995:
URL: https://github.com/apache/hudi/issues/2995
**Describe the problem you faced**
**Background**: We run a Spark Streaming application that ingests messages from Kinesis and upserts/deletes objects from a date-partitioned Hudi 0.6 MOR dataset. This runs on EMR 5.31.
We’ve been running into serious performance issues and S3 503s due to file listing when ingesting into Hudi. To solve this issue we are prototyping upgrading to 0.7 on EMR 5.33 and enabling the [metadata table](https://hudi.apache.org/releases.html#metadata-table) . A small prototype for upgrading from 0.6 to 0.7 with the metadata table enabled worked without any issues, so to test out the upgrade on our production dataset we set up a shadow pipeline that ingests the same messages from Kinesis and upserts/deletes objects from a replica Hudi dataset.
The logical plan from Spark for upserting data is below. The only change we made was adding `hoodie.metadata.enable -> true`
```
== Parsed Logical Plan ==
SaveIntoDataSourceCommand org.apache.hudi.DefaultSource@68b6dfc3, Map(
hoodie.compaction.target.io -> 3000000,
path -> s3://plaid-test-emr/hudi/production/transactions/,
hoodie.datasource.write.precombine.field -> publishedAtUnixNano,
hoodie.metadata.enable -> true,
hoodie.datasource.write.operation -> upsert,
hoodie.datasource.write.recordkey.field -> id.value,
hoodie.table.name -> transactions,
hoodie.datasource.write.table.type -> MERGE_ON_READ,
hoodie.datasource.write.table.name -> transactions,
hoodie.compact.inline -> true,
hoodie.datasource.write.keygenerator.class -> org.apache.hudi.keygen.ComplexKeyGenerator,
hoodie.cleaner.commits.retained -> 1,
hoodie.datasource.write.partitionpath.field -> year,month,day,
hoodie.compact.inline.max.delta.commits -> 10
), Append
```
**Problem**: It seems like the shadow indexing pipeline created [duplicates](https://hudi.apache.org/docs/deployment.html#duplicates) that span multiple files in the same partition. For example querying a partition using a Spark snapshot query returned (NB: this is a dummy record key):
```
[
Row(
_hoodie_commit_time="20210524010032",
_hoodie_commit_seqno="20210524010032_590_32004",
_hoodie_record_key="id.value:aaaaa",
_hoodie_partition_path="2021/5/21",
_hoodie_file_name="08a5e3cf-2faa-4118-9d46-39f7f63b1ee4-0_590-44-133796_20210524010032.parquet",
publishedAtUnixNano=1621643431452628308,
day=21,
month=5,
year=2021,
),
Row(
_hoodie_commit_time="20210523210035",
_hoodie_commit_seqno="20210523210035_4583_15295543",
_hoodie_record_key="id.value:aaaaa",
_hoodie_partition_path="2021/5/21",
_hoodie_file_name="6b52bbb6-fdea-44ec-b10e-b991105d1e57-0_4583-1605-8354526_20210523210035.parquet",
publishedAtUnixNano=1621621876551104247,
day=21,
month=5,
year=2021,
),
]
```
It seems like when the shadow consumer started ingesting using Hudi 0.7 commits stopped updating existing files (“commits show” output below). 20210520220038 (the 780.6 mb written commit) is the commit time that the shadow consumer started and after this commit every commit had 0 files updated and no updates to records. For context the alternating commits are for upserts and hard deletes (ie. 20210520220310 is a delete).
**To Reproduce**
Unsure how to reproduce. My hunch is that the metadata table was corrupted on the initial write for commit 20210520220038, but I’m not sure how to check this. I tried to run “metadata list-partitions” and ran into this error (java.lang.ClassNotFoundException: org.apache.hudi.org.apache.hadoop.hbase.KeyValue$KeyComparator)
```
21/05/25 20:55:54 INFO fs.FSUtils: Hadoop Configuration: fs.defaultFS: [hdfs://ip-10-105-7-17.ec2.internal:8020], Config:[Configuration: core-default.xml, core-site.xml, mapred-default.xml, mapred-site.xml, yarn-default.xml, yarn-site.xml, hdfs-default.xml, hdfs-site.xml, emrfs-site.xml, resource-types.xml, __spark_hadoop_conf__.xml], FileSystem: [com.amazon.ws.emr.hadoop.fs.s3n.S3NativeFileSystem@5541e141]
21/05/25 20:55:54 INFO s3n.S3NativeFileSystem: Opening 's3://plaid-test-emr/hudi/production/transactions/.hoodie/metadata/files/17c82a98-3625-4a36-9852-9d7417f64378-0_0-38-68923_20201216222013.hfile' for reading
21/05/25 20:55:54 ERROR metadata.BaseTableMetadata: Failed to retrieve list of partition from metadata
org.apache.hudi.exception.HoodieIOException: Error merging records from metadata table for key :__all_partitions__
at org.apache.hudi.metadata.HoodieBackedTableMetadata.getRecordByKeyFromMetadata(HoodieBackedTableMetadata.java:153)
at org.apache.hudi.metadata.BaseTableMetadata.getMergedRecordByKey(BaseTableMetadata.java:268)
at org.apache.hudi.metadata.BaseTableMetadata.fetchAllPartitionPaths(BaseTableMetadata.java:149)
at org.apache.hudi.metadata.BaseTableMetadata.getAllPartitionPaths(BaseTableMetadata.java:102)
at org.apache.hudi.cli.commands.MetadataCommand.listPartitions(MetadataCommand.java:176)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.springframework.util.ReflectionUtils.invokeMethod(ReflectionUtils.java:216)
at org.springframework.shell.core.SimpleExecutionStrategy.invoke(SimpleExecutionStrategy.java:68)
at org.springframework.shell.core.SimpleExecutionStrategy.execute(SimpleExecutionStrategy.java:59)
at org.springframework.shell.core.AbstractShell.executeCommand(AbstractShell.java:134)
at org.springframework.shell.core.JLineShell.promptLoop(JLineShell.java:533)
at org.springframework.shell.core.JLineShell.run(JLineShell.java:179)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.apache.hadoop.hbase.io.hfile.CorruptHFileException: Problem reading HFile Trailer from file s3://plaid-test-emr/hudi/production/transactions/.hoodie/metadata/files/17c82a98-3625-4a36-9852-9d7417f64378-0_0-38-68923_20201216222013.hfile
at org.apache.hadoop.hbase.io.hfile.HFile.pickReaderVersion(HFile.java:494)
at org.apache.hadoop.hbase.io.hfile.HFile.createReader(HFile.java:537)
at org.apache.hudi.io.storage.HoodieHFileReader.<init>(HoodieHFileReader.java:69)
at org.apache.hudi.io.storage.HoodieFileReaderFactory.newHFileFileReader(HoodieFileReaderFactory.java:53)
at org.apache.hudi.io.storage.HoodieFileReaderFactory.getFileReader(HoodieFileReaderFactory.java:41)
at org.apache.hudi.metadata.HoodieBackedTableMetadata.openFileSliceIfNeeded(HoodieBackedTableMetadata.java:177)
at org.apache.hudi.metadata.HoodieBackedTableMetadata.getRecordByKeyFromMetadata(HoodieBackedTableMetadata.java:118)
... 15 more
Caused by: java.io.IOException: java.lang.ClassNotFoundException: org.apache.hudi.org.apache.hadoop.hbase.KeyValue$KeyComparator
at org.apache.hadoop.hbase.io.hfile.FixedFileTrailer.getComparatorClass(FixedFileTrailer.java:587)
at org.apache.hadoop.hbase.io.hfile.FixedFileTrailer.deserializeFromPB(FixedFileTrailer.java:299)
at org.apache.hadoop.hbase.io.hfile.FixedFileTrailer.deserialize(FixedFileTrailer.java:241)
at org.apache.hadoop.hbase.io.hfile.FixedFileTrailer.readFromStream(FixedFileTrailer.java:406)
at org.apache.hadoop.hbase.io.hfile.HFile.pickReaderVersion(HFile.java:479)
... 21 more
Caused by: java.lang.ClassNotFoundException: org.apache.hudi.org.apache.hadoop.hbase.KeyValue$KeyComparator
at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
at java.lang.ClassLoader.loadClass(ClassLoader.java:418)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:352)
at java.lang.ClassLoader.loadClass(ClassLoader.java:351)
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:264)
at org.apache.hadoop.hbase.io.hfile.FixedFileTrailer.getComparatorClass(FixedFileTrailer.java:584)
... 25 more
```
Properties (from “desc”):
```
╔═════════════════════════════════╤══════════════════════════════════════════════════════════════════════════════════╗
║ Property │ Value ║
╠═════════════════════════════════╪══════════════════════════════════════════════════════════════════════════════════╣
║ basePath │ s3://plaid-test-emr/hudi/production/transactions/ ║
╟─────────────────────────────────┼──────────────────────────────────────────────────────────────────────────────────╢
║ metaPath │ s3://plaid-test-emr/hudi/production/transactions/.hoodie ║
╟─────────────────────────────────┼──────────────────────────────────────────────────────────────────────────────────╢
║ fileSystem │ s3 ║
╟─────────────────────────────────┼──────────────────────────────────────────────────────────────────────────────────╢
║ hoodie.table.name │ transactions ║
╟─────────────────────────────────┼──────────────────────────────────────────────────────────────────────────────────╢
║ hoodie.compaction.payload.class │ org.apache.hudi.common.model.OverwriteWithLatestAvroPayload ║
╟─────────────────────────────────┼──────────────────────────────────────────────────────────────────────────────────╢
║ hoodie.table.type │ MERGE_ON_READ ║
╟─────────────────────────────────┼──────────────────────────────────────────────────────────────────────────────────╢
║ hoodie.archivelog.folder │ archived ║
╟─────────────────────────────────┼──────────────────────────────────────────────────────────────────────────────────╢
║ hoodie.timeline.layout.version │ 1 ║
╟─────────────────────────────────┼──────────────────────────────────────────────────────────────────────────────────╢
║ hoodie.table.version │ 1 ║
╚═════════════════════════════════╧══════════════════════════════════════════════════════════════════════════════════╝
```
**Expected behavior**
Same behavior as Hudi 0.6 but now using the metadata table to track files/partitions. Happy to provide whatever info I can.
Questions:
1. What is causing these duplicates to occur? Since no errors happened as far as I can tell, what info can I look at to debug/RCA? I’ve verified there are no duplicates (ie. checked some partitions) on 0.6 dataset.
2. How can the metadata table be inspected? I can’t tell from https://cwiki.apache.org/confluence/display/HUDI/RFC+-+15%3A+HUDI+File+Listing+Improvements
3. Should `hoodie.metadata.validate` be enabled? My understanding is this is a “dry run” config where S3 file listing will still happen as before while also updating the metadata table
4. How do we recover when duplicates occur? I see “records deduplicate” is suggested in https://hudi.apache.org/docs/deployment.html#duplicates (NB: seems like this should be “repair deduplicate”?), do we need to turn off ingestion first and then run over every affected partition?
5. How do we recover if the metadata table is corrupted? Should we delete the existing metadata table from the CLI and recreate? Is this safe to do?
6. What upgrade path is suggested from 0.6 to 0.7 with metadata table enabled? Should the metadata table be created from the CLI pre-ingestion and then starting up the consumer after?
**Environment Description**
* Hudi version : 0.7
* Spark version : 2.4.7 (EMR 5.33)
* Hive version : 2.3.7
* Hadoop version : Amazon 2.10.1
* Storage (HDFS/S3/GCS..) : S3
* Running on Docker? (yes/no) : No
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] prashantwason commented on issue #2995: [SUPPORT] Upserts creating duplicates after enabling metadata table in Hudi 0.7 indexing pipeline
Posted by GitBox <gi...@apache.org>.
prashantwason commented on issue #2995:
URL: https://github.com/apache/hudi/issues/2995#issuecomment-875313926
>> What I'm not understanding is how the table is only updated until 20201216222013.deltacommit since that commit was long before [1].
Metadata Table bootstrap had some bug which caused it to only bootrap till 20201216222013. Future changes (commits / cleans etc) were not committed to metadata table which caused it to always report data from dec.
This was not caused by your actions.
I have been filing the various patches to fix this. 0.9 will have all the fixes for metadata table. But if you want to try now, you need these patches over current master:
https://github.com/apache/hudi/pull/3084
https://github.com/apache/hudi/pull/3210
Once you have these patches, you can create your shadow pipeline again and it should work.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] jtmzheng commented on issue #2995: [SUPPORT] Upserts creating duplicates after enabling metadata table in Hudi 0.7 indexing pipeline
Posted by GitBox <gi...@apache.org>.
jtmzheng commented on issue #2995:
URL: https://github.com/apache/hudi/issues/2995#issuecomment-857903531
Interesting, @prashantwason so the timeline of how I ran the shadow pipeline was:
1. Stopped ingestion pipeline (0.6 without metadata table)
2. Ran s3-dist-cp job to replicate dataset to a separate path
3. Ran shadow pipeline (0.7 with metadata table enabled) against replica
What I'm not understanding is how the table is only updated until 20201216222013.deltacommit since that commit was long before [1]. How is the metadata table initialized? Happy to dig through docs if you can point me to the right place :)
Would you recommend waiting until 0.9?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] n3nash edited a comment on issue #2995: [SUPPORT] Upserts creating duplicates after enabling metadata table in Hudi 0.7 indexing pipeline
Posted by GitBox <gi...@apache.org>.
n3nash edited a comment on issue #2995:
URL: https://github.com/apache/hudi/issues/2995#issuecomment-851715268
@jtmzheng Thanks for the very detailed information that helps to understand the problem. Let me answer some of your questions inline before attempting to debug the underlying root cause of duplicates.
> **Expected behavior**
>
> Same behavior as Hudi 0.6 but now using the metadata table to track files/partitions. Happy to provide whatever info I can.
>
> Questions:
>
> 1. What is causing these duplicates to occur? Since no errors happened as far as I can tell, what info can I look at to debug/RCA? I’ve verified there are no duplicates (ie. checked some partitions) on 0.6 dataset.
Can you check if the files from the dataset is the same as in the metadata table ? This would require you to
1. Perform a listing on the entire dataset and get the unique files
2. Read the metadata table and get the list of files from there
3. Diff the 2 results to see if there are files in your dataset which are not present in the metadata table.
Additionally, if you have the logs from the first application where you turned on the `hoodie.metadata.enable` flag to true. Can you grep for the following log lines:
`Creating a new metadata table in`
`Initializing metadata table by using file listings in`
`files to metadata`
Please share the output of the grep of the above lines.
> 2. How can the metadata table be inspected? I can’t tell from https://cwiki.apache.org/confluence/display/HUDI/RFC+-+15%3A+HUDI+File+Listing+Improvements
You can use the `MetadataCommand` to inspect this using the Hudi CLI. You can read about how to initialize the CLI here -> https://hudi.apache.org/docs/deployment.html#cli
> 3. Should `hoodie.metadata.validate` be enabled? My understanding is this is a “dry run” config where S3 file listing will still happen as before while also updating the metadata table
That is correct. In this scenario, enabling that before the duplicates happened would've helped. But it doesn't help to enable this after the corruption has happened. At a high level, you don't need to enable it unless this issue is fully reproducible.
> 4. How do we recover when duplicates occur? I see “records deduplicate” is suggested in https://hudi.apache.org/docs/deployment.html#duplicates (NB: seems like this should be “repair deduplicate”?), do we need to turn off ingestion first and then run over every affected partition?
Yes. You will need to turn off ingestion first before deduping. Unfortunately, this command is pretty old and has not been maintained since we don't see duplicate issues. You might need to re-bootstrap you dataset to recover this or start the shadow pipeline fresh.
> 5. How do we recover if the metadata table is corrupted? Should we delete the existing metadata table from the CLI and recreate? Is this safe to do?
Yes. You can delete the existing metadata table using the CLI or manually but you need to stop ingestion during this time. It is safe to do this. Then, you should disable `hoodie.metadata.enable` if there is a corruption since the code may have had bug. After this, you can resume your pipeline.
> 6. What upgrade path is suggested from 0.6 to 0.7 with metadata table enabled? Should the metadata table be created from the CLI pre-ingestion and then starting up the consumer after?
As long as you don't have concurrently running jobs, simply turning on the metadata table before the next ingestion run suffices. No need to stop the job.
Please share your logs to help debug this problem further.
@prashantwason This seems like a serious issue. Can you please check if this is legitimately possible ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan closed issue #2995: [SUPPORT] Upserts creating duplicates after enabling metadata table in Hudi 0.7 indexing pipeline
Posted by GitBox <gi...@apache.org>.
nsivabalan closed issue #2995:
URL: https://github.com/apache/hudi/issues/2995
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] jtmzheng commented on issue #2995: [SUPPORT] Upserts creating duplicates after enabling metadata table in Hudi 0.7 indexing pipeline
Posted by GitBox <gi...@apache.org>.
jtmzheng commented on issue #2995:
URL: https://github.com/apache/hudi/issues/2995#issuecomment-853465482
Update, I was able to run the metadata commands from the CLI and check `list-partitions` and `list-files`. It seems like every partition is there, but spot checking some partitions its clear not all the files are there:
```
21/06/03 00:10:32 INFO log.HoodieMergedLogRecordScanner: Number of log files scanned => 0
21/06/03 00:10:32 INFO log.HoodieMergedLogRecordScanner: MaxMemoryInBytes allowed for compaction => 0
21/06/03 00:10:32 INFO log.HoodieMergedLogRecordScanner: Number of entries in MemoryBasedMap in ExternalSpillableMap => 0
21/06/03 00:10:32 INFO log.HoodieMergedLogRecordScanner: Total size in bytes of MemoryBasedMap in ExternalSpillableMap => 0
21/06/03 00:10:32 INFO log.HoodieMergedLogRecordScanner: Number of entries in DiskBasedMap in ExternalSpillableMap => 0
21/06/03 00:10:32 INFO log.HoodieMergedLogRecordScanner: Size of file spilled to disk => 0
21/06/03 00:10:32 INFO metadata.HoodieBackedTableMetadata: Opened metadata log files from [] at instant 20210524060712(dataset instant=20210524060712, metadata instant=20201216222013)
21/06/03 00:10:32 INFO compress.CodecPool: Got brand-new decompressor [.gz]
21/06/03 00:10:32 INFO metadata.HoodieBackedTableMetadata: Metadata read for key 2020/9/4 took [open, baseFileRead, logMerge] [94, 71, 0] ms
21/06/03 00:10:32 INFO metadata.BaseTableMetadata: Listed file in partition from metadata: partition=2020/9/4, #files=121
.f8a8f054-6d0e-43e8-9412-3dfec79d7d53-0_20210509151344.log.1_10765-6377-69863781
.f75e9845-a3a2-4b41-b59a-8effc2ee049a-0_20210429061318.log.1_12603-1679-22268308
.f745a448-31fa-4a95-a3c4-5f88f6c95bb2-0_20210515000505.log.2_11136-1920-26157888
.f745a448-31fa-4a95-a3c4-5f88f6c95bb2-0_20210515000505.log.1_12428-158-1954195
...
```
Listing shows 1,539 files under that partition.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] prashantwason commented on issue #2995: [SUPPORT] Upserts creating duplicates after enabling metadata table in Hudi 0.7 indexing pipeline
Posted by GitBox <gi...@apache.org>.
prashantwason commented on issue #2995:
URL: https://github.com/apache/hudi/issues/2995#issuecomment-852423034
@jtmzheng if you have logs from the spark applications, please grep to check if any HoodieMetadataException was ever raised. There may have been an exception which got swallowed so the pipeline did not fail but there was an error anyways.
Also, can you check if you have any non-completed instants on your hoodie timeline (i.e. instants which have .requested and .inflight but not the third file). Alternatively, you can provide the listing of the following two dirs and I can check:
s3://plaid-test-emr/hudi/production/transactions/.hoodie
s3://plaid-test-emr/hudi/production/transactions/.hoodie/metadata/.hoodie
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] jtmzheng commented on issue #2995: [SUPPORT] Upserts creating duplicates after enabling metadata table in Hudi 0.7 indexing pipeline
Posted by GitBox <gi...@apache.org>.
jtmzheng commented on issue #2995:
URL: https://github.com/apache/hudi/issues/2995#issuecomment-852468587
Got it, thanks for the help! I will comb through the Spark logs again and inspect the metadata table later today but the default log level for Hudi on EMR is unfortunately WARN so some logs might not be available.
@prashantwason the listings:
s3://plaid-test-emr/hudi/production/transactions/.hoodie:
[main_path.txt](https://github.com/apache/hudi/files/6580694/main_path.txt)
s3://plaid-test-emr/hudi/production/transactions/.hoodie/metadata/.hoodie
[metadata_path.txt](https://github.com/apache/hudi/files/6580699/metadata_path.txt)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] jtmzheng commented on issue #2995: [SUPPORT] Upserts creating duplicates after enabling metadata table in Hudi 0.7 indexing pipeline
Posted by GitBox <gi...@apache.org>.
jtmzheng commented on issue #2995:
URL: https://github.com/apache/hudi/issues/2995#issuecomment-852761651
I grepped for HoodieMetadataException and "Creating a new metadata table in"/"Initializing metadata table by using file listings in"/"files to metadata" in the EMR logs but did not find anything. I can try replicating this from scratch later this week with more verbose Hudi logging enabled.
@n3nash re:
> You can use the MetadataCommand to inspect this using the Hudi CLI. You can read about how to initialize the CLI here -> https://hudi.apache.org/docs/deployment.html#cli
Not sure if this is what you're referring to, but I tried to run "metadata list-partitions" which errored out with `(java.lang.ClassNotFoundException: org.apache.hudi.org.apache.hadoop.hbase.KeyValue$KeyComparator)` (the stacktrace is in the main description). I'm running the CLI on EMR 5.33 (https://docs.aws.amazon.com/emr/latest/ReleaseGuide/emr-hudi-cli.html), seems like this could be a separate packaging issue on EMR. Any idea?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] prashantwason commented on issue #2995: [SUPPORT] Upserts creating duplicates after enabling metadata table in Hudi 0.7 indexing pipeline
Posted by GitBox <gi...@apache.org>.
prashantwason commented on issue #2995:
URL: https://github.com/apache/hudi/issues/2995#issuecomment-856050504
I have found the issue - The metadata table timeline is updated only till 20201216222013.deltacommit.
2021-05-20 15:00:47 0 .temp_$folder$
2021-05-20 15:01:09 2001 20201216222013.deltacommit
2021-05-20 15:01:03 985 20201216222013.deltacommit.inflight
2021-05-20 15:01:02 0 20201216222013.deltacommit.requested
So the table is returning the old listing (which was accurate in Dec) causing the duplicate issue. I think the reason the table the stuck is one of the following issues: https://issues.apache.org/jira/browse/HUDI-1649
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] n3nash commented on issue #2995: [SUPPORT] Upserts creating duplicates after enabling metadata table in Hudi 0.7 indexing pipeline
Posted by GitBox <gi...@apache.org>.
n3nash commented on issue #2995:
URL: https://github.com/apache/hudi/issues/2995#issuecomment-851715268
@jtmzheng Thanks for the very detailed information that helps to understand the problem. Let me answer some of your questions inline before attempting to debug the underlying root cause of duplicates.
> **Expected behavior**
>
> Same behavior as Hudi 0.6 but now using the metadata table to track files/partitions. Happy to provide whatever info I can.
>
> Questions:
>
> 1. What is causing these duplicates to occur? Since no errors happened as far as I can tell, what info can I look at to debug/RCA? I’ve verified there are no duplicates (ie. checked some partitions) on 0.6 dataset.
Can you check if the files from the dataset is the same as in the metadata table ? This would require you to
1. Perform a listing on the entire dataset and get the unique files
2. Read the metadata table and get the list of files from there
3. Diff the 2 results to see if there are files in your dataset which are not present in the metadata table.
Additionally, if you have the logs from the first application where you turned on the `hoodie.metadata.enable` flag to true. Can you grep for the following log lines:
`Creating a new metadata table in`
`Initializing metadata table by using file listings in`
`files to metadata`
Please share the output of the grep of the above lines.
> 2. How can the metadata table be inspected? I can’t tell from https://cwiki.apache.org/confluence/display/HUDI/RFC+-+15%3A+HUDI+File+Listing+Improvements
You can use the `MetadataCommand` to inspect this using the Hudi CLI. You can read about how to initialize the CLI here -> https://hudi.apache.org/docs/deployment.html#cli
> 3. Should `hoodie.metadata.validate` be enabled? My understanding is this is a “dry run” config where S3 file listing will still happen as before while also updating the metadata table
That is correct. In this scenario, enabling that before the duplicates happened would've helped. But it doesn't help to enable this after the corruption has happened. At a high level, you don't need to enable it unless this issue is fully reproducible.
> 4. How do we recover when duplicates occur? I see “records deduplicate” is suggested in https://hudi.apache.org/docs/deployment.html#duplicates (NB: seems like this should be “repair deduplicate”?), do we need to turn off ingestion first and then run over every affected partition?
Yes. You will need to turn off ingestion first before deduping. Unfortunately, this command is pretty old and has not been maintained since we don't see duplicate issues. You might need to re-bootstrap you dataset to recover this or start the shadow pipeline fresh.
> 5. How do we recover if the metadata table is corrupted? Should we delete the existing metadata table from the CLI and recreate? Is this safe to do?
Yes. You can delete the existing metadata table using the CLI or manually but you need to stop ingestion during this time. It is safe to do this. Then, you should disable `hoodie.metadata.enable` if there is a corruption since the code may have had bug. After this, you can resume your pipeline.
> 6. What upgrade path is suggested from 0.6 to 0.7 with metadata table enabled? Should the metadata table be created from the CLI pre-ingestion and then starting up the consumer after?
As long as you don't have concurrently running jobs, simply turning on the metadata table before the next ingestion run suffices. No need to stop the job.
Please share your logs to help debug this problem further.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan commented on issue #2995: [SUPPORT] Upserts creating duplicates after enabling metadata table in Hudi 0.7 indexing pipeline
Posted by GitBox <gi...@apache.org>.
nsivabalan commented on issue #2995:
URL: https://github.com/apache/hudi/issues/2995#issuecomment-905174311
We are closing the issue. Please reopen if you run into issues. thanks for reporting.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] codope commented on issue #2995: [SUPPORT] Upserts creating duplicates after enabling metadata table in Hudi 0.7 indexing pipeline
Posted by GitBox <gi...@apache.org>.
codope commented on issue #2995:
URL: https://github.com/apache/hudi/issues/2995#issuecomment-904591711
@jtmzheng Did you get a chance to test with the patched shared above?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] jtmzheng commented on issue #2995: [SUPPORT] Upserts creating duplicates after enabling metadata table in Hudi 0.7 indexing pipeline
Posted by GitBox <gi...@apache.org>.
jtmzheng commented on issue #2995:
URL: https://github.com/apache/hudi/issues/2995#issuecomment-853465482
Update, I was able to run the metadata commands from the CLI and check `list-partitions` and `list-files`. It seems like every partition is there, but spot checking some partitions its clear not all the files are there:
```
21/06/03 00:10:32 INFO log.HoodieMergedLogRecordScanner: Number of log files scanned => 0
21/06/03 00:10:32 INFO log.HoodieMergedLogRecordScanner: MaxMemoryInBytes allowed for compaction => 0
21/06/03 00:10:32 INFO log.HoodieMergedLogRecordScanner: Number of entries in MemoryBasedMap in ExternalSpillableMap => 0
21/06/03 00:10:32 INFO log.HoodieMergedLogRecordScanner: Total size in bytes of MemoryBasedMap in ExternalSpillableMap => 0
21/06/03 00:10:32 INFO log.HoodieMergedLogRecordScanner: Number of entries in DiskBasedMap in ExternalSpillableMap => 0
21/06/03 00:10:32 INFO log.HoodieMergedLogRecordScanner: Size of file spilled to disk => 0
21/06/03 00:10:32 INFO metadata.HoodieBackedTableMetadata: Opened metadata log files from [] at instant 20210524060712(dataset instant=20210524060712, metadata instant=20201216222013)
21/06/03 00:10:32 INFO compress.CodecPool: Got brand-new decompressor [.gz]
21/06/03 00:10:32 INFO metadata.HoodieBackedTableMetadata: Metadata read for key 2020/9/4 took [open, baseFileRead, logMerge] [94, 71, 0] ms
21/06/03 00:10:32 INFO metadata.BaseTableMetadata: Listed file in partition from metadata: partition=2020/9/4, #files=121
.f8a8f054-6d0e-43e8-9412-3dfec79d7d53-0_20210509151344.log.1_10765-6377-69863781
.f75e9845-a3a2-4b41-b59a-8effc2ee049a-0_20210429061318.log.1_12603-1679-22268308
.f745a448-31fa-4a95-a3c4-5f88f6c95bb2-0_20210515000505.log.2_11136-1920-26157888
.f745a448-31fa-4a95-a3c4-5f88f6c95bb2-0_20210515000505.log.1_12428-158-1954195
...
```
Listing shows 1,539 files under that partition.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org