You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by GitBox <gi...@apache.org> on 2021/04/01 01:57:11 UTC

[GitHub] [hudi] li36909 opened a new pull request #2751: [HUDI-1748] Read operation will possiblity fail on mor table rt view when a write operations is concurrency running

li36909 opened a new pull request #2751:
URL: https://github.com/apache/hudi/pull/2751


   ## *Tips*
   - *Thank you very much for contributing to Apache Hudi.*
   - *Please review https://hudi.apache.org/contributing.html before opening a pull request.*
   
   ## What is the purpose of the pull request
   
   Solve read write concurrency bug on mor table rt view
   
   ## Brief change log
   
   *(for example:)*
     - *Modify AnnotationLocation checkstyle rule in checkstyle.xml*
   
   ## Verify this pull request
   
   Testing concurrency probabilistic problems in UT is difficult. I add sleep stability to the getRealtimeSplits method to reproduce the problem and verify the fix.
   
   
   ## Committer checklist
   
    - [ ] Has a corresponding JIRA in PR title & commit
    
    - [ ] Commit message is descriptive of the change
    
    - [ ] CI is green
   
    - [ ] Necessary doc changes done or have another open PR
          
    - [ ] For large changes, please consider breaking it into sub-tasks under an umbrella JIRA.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] nsivabalan edited a comment on pull request #2751: [HUDI-1748] Read operation will possiblity fail on mor table rt view when a write operations is concurrency running

Posted by GitBox <gi...@apache.org>.
nsivabalan edited a comment on pull request #2751:
URL: https://github.com/apache/hudi/pull/2751#issuecomment-812037473


   @li36909 : can you clarify what do you mean by concurrent operations. Ignoring any table services, you mean to say that just a batch of write and rt snapshot view happening concurrently is running into issues? which hudi version are you running.   
   @n3nash : you may want to check it out. 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] li36909 commented on pull request #2751: [HUDI-1748] Read operation will possiblity fail on mor table rt view when a write operations is concurrency running

Posted by GitBox <gi...@apache.org>.
li36909 commented on pull request #2751:
URL: https://github.com/apache/hudi/pull/2751#issuecomment-829861322


   cc @n3nash could you help to take a look again, thank you


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] hudi-bot edited a comment on pull request #2751: [HUDI-1748] Read operation will possiblity fail on mor table rt view when a write operations is concurrency running

Posted by GitBox <gi...@apache.org>.
hudi-bot edited a comment on pull request #2751:
URL: https://github.com/apache/hudi/pull/2751#issuecomment-914646698


   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "2de7140cbdf98568f63110b88e2ff65cab14c0b1",
       "status" : "FAILURE",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2082",
       "triggerID" : "2de7140cbdf98568f63110b88e2ff65cab14c0b1",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 2de7140cbdf98568f63110b88e2ff65cab14c0b1 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2082) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run travis` re-run the last Travis build
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] hudi-bot commented on pull request #2751: [HUDI-1748] Read operation will possiblity fail on mor table rt view when a write operations is concurrency running

Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #2751:
URL: https://github.com/apache/hudi/pull/2751#issuecomment-961587651


   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "2de7140cbdf98568f63110b88e2ff65cab14c0b1",
       "status" : "FAILURE",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2082",
       "triggerID" : "2de7140cbdf98568f63110b88e2ff65cab14c0b1",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 2de7140cbdf98568f63110b88e2ff65cab14c0b1 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2082) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] n3nash commented on pull request #2751: [HUDI-1748] Read operation will possiblity fail on mor table rt view when a write operations is concurrency running

Posted by GitBox <gi...@apache.org>.
n3nash commented on pull request #2751:
URL: https://github.com/apache/hudi/pull/2751#issuecomment-812357498


   Okay, thanks for the confirmation, I will try to reproduce this issue on my end and get back. 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] hudi-bot commented on pull request #2751: [HUDI-1748] Read operation will possiblity fail on mor table rt view when a write operations is concurrency running

Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #2751:
URL: https://github.com/apache/hudi/pull/2751#issuecomment-961587651


   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "2de7140cbdf98568f63110b88e2ff65cab14c0b1",
       "status" : "FAILURE",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2082",
       "triggerID" : "2de7140cbdf98568f63110b88e2ff65cab14c0b1",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 2de7140cbdf98568f63110b88e2ff65cab14c0b1 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2082) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] li36909 commented on pull request #2751: [HUDI-1748] Read operation will possiblity fail on mor table rt view when a write operations is concurrency running

Posted by GitBox <gi...@apache.org>.
li36909 commented on pull request #2751:
URL: https://github.com/apache/hudi/pull/2751#issuecomment-812341340


   @n3nash yes


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] hudi-bot removed a comment on pull request #2751: [HUDI-1748] Read operation will possiblity fail on mor table rt view when a write operations is concurrency running

Posted by GitBox <gi...@apache.org>.
hudi-bot removed a comment on pull request #2751:
URL: https://github.com/apache/hudi/pull/2751#issuecomment-914646698


   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "2de7140cbdf98568f63110b88e2ff65cab14c0b1",
       "status" : "FAILURE",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2082",
       "triggerID" : "2de7140cbdf98568f63110b88e2ff65cab14c0b1",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 2de7140cbdf98568f63110b88e2ff65cab14c0b1 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2082) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run travis` re-run the last Travis build
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] hudi-bot commented on pull request #2751: [HUDI-1748] Read operation will possiblity fail on mor table rt view when a write operations is concurrency running

Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #2751:
URL: https://github.com/apache/hudi/pull/2751#issuecomment-961587651


   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "2de7140cbdf98568f63110b88e2ff65cab14c0b1",
       "status" : "FAILURE",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2082",
       "triggerID" : "2de7140cbdf98568f63110b88e2ff65cab14c0b1",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 2de7140cbdf98568f63110b88e2ff65cab14c0b1 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2082) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] hudi-bot commented on pull request #2751: [HUDI-1748] Read operation will possiblity fail on mor table rt view when a write operations is concurrency running

Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #2751:
URL: https://github.com/apache/hudi/pull/2751#issuecomment-914646698


   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "2de7140cbdf98568f63110b88e2ff65cab14c0b1",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "2de7140cbdf98568f63110b88e2ff65cab14c0b1",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 2de7140cbdf98568f63110b88e2ff65cab14c0b1 UNKNOWN
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run travis` re-run the last Travis build
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] li36909 commented on pull request #2751: [HUDI-1748] Read operation will possiblity fail on mor table rt view when a write operations is concurrency running

Posted by GitBox <gi...@apache.org>.
li36909 commented on pull request #2751:
URL: https://github.com/apache/hudi/pull/2751#issuecomment-812271790


   @nsivabalan I run the test at hudi 0.7. yes, you are right, I start a spark-shell for upserting, and query the same table by spark datasouce api, then the problem arises. The cause of the problem is clear, during the query, hudi get partitions at MergeOnReadSnapshotRelation, and build a new fsview at HoodieRealtimeInputFormatUtils.groupLogsByBaseFile, when a write operation is happening,  HoodieRealtimeInputFormatUtils.groupLogsByBaseFile will find some new base files.
   we can reproduce this issue by add a Thread.sleep(60000) at HoodieRealtimeInputFormatUtils.groupLogsByBaseFile, then run this test:
   step 1: write first batch into a hudi table:
   spark-shell
   
   import org.apache.hudi.QuickstartUtils.
   import scala.collection.JavaConversions.
   import org.apache.spark.sql.SaveMode.
   import org.apache.hudi.DataSourceReadOptions.
   import org.apache.hudi.DataSourceWriteOptions.
   import org.apache.hudi.config.HoodieWriteConfig.
   
   
   val tableName = "hudi_mor_table"
   val basePath = "hdfs://hacluster/tmp/hudi_mor_table"
   val dataGen = new DataGenerator
   val inserts = convertToStringList(dataGen.generateInserts(10))
   val df = spark.read.json(spark.sparkContext.parallelize(inserts, 2))
   df.write.format("org.apache.hudi").
   option("hoodie.datasource.write.table.type", "MERGE_ON_READ").
   option("hoodie.datasource.write.operation", "bulk_insert").
   options(getQuickstartWriteConfigs).
   option(PRECOMBINE_FIELD_OPT_KEY, "ts").
   option(RECORDKEY_FIELD_OPT_KEY, "uuid").
   option(PARTITIONPATH_FIELD_OPT_KEY, "partitionpath").
   option("hoodie.datasource.write.hive_style_partitioning", "false").
   option(TABLE_NAME, tableName).
   option("hoodie.datasource.hive_sync.enable", "true").
   option("hoodie.datasource.hive_sync.use_jdbc", "false").
   option("hoodie.datasource.hive_sync.table", "hudi_mor_test").
   option("hoodie.datasource.hive_sync.partition_extractor_class", "org.apache.hudi.hive.MultiPartKeysValueExtractor").
   option("hoodie.datasource.hive_sync.partition_fields", "continent,country,city").
   option("hoodie.datasource.hive_sync.assume_date_partitioning", "false").
   option("hoodie.insert.shuffle.parallelism", "2").
   option("hoodie.upsert.shuffle.parallelism","2").
   option("hoodie.bulkinsert.shuffle.parallelism", "2").
   option("hoodie.delete.shuffle.parallelism","2").
   mode(Append).
   save(basePath);
   
   step 2: run a query at new spark-shell (when the query hang at Thread.sleep, start to write a new batch at step3)
   spark.read.format("hudi").load("hdfs://hacluster/tmp/hudi_mor_table/*/*/*/*").count
   
   setp 3:  go to the spark-shell at step1, write a new batch:
   df.write.format("org.apache.hudi").
   option("hoodie.datasource.write.table.type", "MERGE_ON_READ").
   option("hoodie.datasource.write.operation", "bulk_insert").
   options(getQuickstartWriteConfigs).
   option(PRECOMBINE_FIELD_OPT_KEY, "ts").
   option(RECORDKEY_FIELD_OPT_KEY, "uuid").
   option(PARTITIONPATH_FIELD_OPT_KEY, "partitionpath").
   option("hoodie.datasource.write.hive_style_partitioning", "false").
   option(TABLE_NAME, tableName).
   option("hoodie.datasource.hive_sync.enable", "true").
   option("hoodie.datasource.hive_sync.use_jdbc", "false").
   option("hoodie.datasource.hive_sync.table", "hudi_mor_test").
   option("hoodie.datasource.hive_sync.partition_extractor_class", "org.apache.hudi.hive.MultiPartKeysValueExtractor").
   option("hoodie.datasource.hive_sync.partition_fields", "continent,country,city").
   option("hoodie.datasource.hive_sync.assume_date_partitioning", "false").
   option("hoodie.insert.shuffle.parallelism", "2").
   option("hoodie.upsert.shuffle.parallelism","2").
   option("hoodie.bulkinsert.shuffle.parallelism", "2").
   option("hoodie.delete.shuffle.parallelism","2").
   mode(Append).
   save(basePath);
   
   we can see the step2 will throw a exception


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] codecov-io commented on pull request #2751: [HUDI-1748] Read operation will possiblity fail on mor table rt view when a write operations is concurrency running

Posted by GitBox <gi...@apache.org>.
codecov-io commented on pull request #2751:
URL: https://github.com/apache/hudi/pull/2751#issuecomment-811590982


   # [Codecov](https://codecov.io/gh/apache/hudi/pull/2751?src=pr&el=h1) Report
   > Merging [#2751](https://codecov.io/gh/apache/hudi/pull/2751?src=pr&el=desc) (2de7140) into [master](https://codecov.io/gh/apache/hudi/commit/fe16d0de7c76105775c887b700751241bc82624c?el=desc) (fe16d0d) will **increase** coverage by `17.69%`.
   > The diff coverage is `n/a`.
   
   [![Impacted file tree graph](https://codecov.io/gh/apache/hudi/pull/2751/graphs/tree.svg?width=650&height=150&src=pr&token=VTTXabwbs2)](https://codecov.io/gh/apache/hudi/pull/2751?src=pr&el=tree)
   
   ```diff
   @@              Coverage Diff              @@
   ##             master    #2751       +/-   ##
   =============================================
   + Coverage     52.04%   69.73%   +17.69%     
   + Complexity     3625      371     -3254     
   =============================================
     Files           479       54      -425     
     Lines         22804     1989    -20815     
     Branches       2415      236     -2179     
   =============================================
   - Hits          11868     1387    -10481     
   + Misses         9911      471     -9440     
   + Partials       1025      131      -894     
   ```
   
   | Flag | Coverage Δ | Complexity Δ | |
   |---|---|---|---|
   | hudicli | `?` | `?` | |
   | hudiclient | `?` | `?` | |
   | hudicommon | `?` | `?` | |
   | hudiflink | `?` | `?` | |
   | hudihadoopmr | `?` | `?` | |
   | hudisparkdatasource | `?` | `?` | |
   | hudisync | `?` | `?` | |
   | huditimelineservice | `?` | `?` | |
   | hudiutilities | `69.73% <ø> (-0.06%)` | `0.00 <ø> (ø)` | |
   
   Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags#carryforward-flags-in-the-pull-request-comment) to find out more.
   
   | [Impacted Files](https://codecov.io/gh/apache/hudi/pull/2751?src=pr&el=tree) | Coverage Δ | Complexity Δ | |
   |---|---|---|---|
   | [...apache/hudi/utilities/deltastreamer/DeltaSync.java](https://codecov.io/gh/apache/hudi/pull/2751/diff?src=pr&el=tree#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL2RlbHRhc3RyZWFtZXIvRGVsdGFTeW5jLmphdmE=) | `71.37% <0.00%> (-0.35%)` | `55.00% <0.00%> (-1.00%)` | |
   | [...sioning/clean/CleanMetadataV2MigrationHandler.java](https://codecov.io/gh/apache/hudi/pull/2751/diff?src=pr&el=tree#diff-aHVkaS1jb21tb24vc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvY29tbW9uL3RhYmxlL3RpbWVsaW5lL3ZlcnNpb25pbmcvY2xlYW4vQ2xlYW5NZXRhZGF0YVYyTWlncmF0aW9uSGFuZGxlci5qYXZh) | | | |
   | [...org/apache/hudi/common/table/log/AppendResult.java](https://codecov.io/gh/apache/hudi/pull/2751/diff?src=pr&el=tree#diff-aHVkaS1jb21tb24vc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvY29tbW9uL3RhYmxlL2xvZy9BcHBlbmRSZXN1bHQuamF2YQ==) | | | |
   | [...oning/compaction/CompactionV2MigrationHandler.java](https://codecov.io/gh/apache/hudi/pull/2751/diff?src=pr&el=tree#diff-aHVkaS1jb21tb24vc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvY29tbW9uL3RhYmxlL3RpbWVsaW5lL3ZlcnNpb25pbmcvY29tcGFjdGlvbi9Db21wYWN0aW9uVjJNaWdyYXRpb25IYW5kbGVyLmphdmE=) | | | |
   | [...on/table/timeline/versioning/MetadataMigrator.java](https://codecov.io/gh/apache/hudi/pull/2751/diff?src=pr&el=tree#diff-aHVkaS1jb21tb24vc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvY29tbW9uL3RhYmxlL3RpbWVsaW5lL3ZlcnNpb25pbmcvTWV0YWRhdGFNaWdyYXRvci5qYXZh) | | | |
   | [...e/hudi/common/table/timeline/dto/FileGroupDTO.java](https://codecov.io/gh/apache/hudi/pull/2751/diff?src=pr&el=tree#diff-aHVkaS1jb21tb24vc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvY29tbW9uL3RhYmxlL3RpbWVsaW5lL2R0by9GaWxlR3JvdXBEVE8uamF2YQ==) | | | |
   | [...di/sink/partitioner/delta/DeltaBucketAssigner.java](https://codecov.io/gh/apache/hudi/pull/2751/diff?src=pr&el=tree#diff-aHVkaS1mbGluay9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvaHVkaS9zaW5rL3BhcnRpdGlvbmVyL2RlbHRhL0RlbHRhQnVja2V0QXNzaWduZXIuamF2YQ==) | | | |
   | [...che/hudi/metadata/TimelineMergedTableMetadata.java](https://codecov.io/gh/apache/hudi/pull/2751/diff?src=pr&el=tree#diff-aHVkaS1jb21tb24vc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvbWV0YWRhdGEvVGltZWxpbmVNZXJnZWRUYWJsZU1ldGFkYXRhLmphdmE=) | | | |
   | [...pache/hudi/io/storage/HoodieFileReaderFactory.java](https://codecov.io/gh/apache/hudi/pull/2751/diff?src=pr&el=tree#diff-aHVkaS1jb21tb24vc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvaW8vc3RvcmFnZS9Ib29kaWVGaWxlUmVhZGVyRmFjdG9yeS5qYXZh) | | | |
   | [...oning/compaction/CompactionV1MigrationHandler.java](https://codecov.io/gh/apache/hudi/pull/2751/diff?src=pr&el=tree#diff-aHVkaS1jb21tb24vc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvY29tbW9uL3RhYmxlL3RpbWVsaW5lL3ZlcnNpb25pbmcvY29tcGFjdGlvbi9Db21wYWN0aW9uVjFNaWdyYXRpb25IYW5kbGVyLmphdmE=) | | | |
   | ... and [415 more](https://codecov.io/gh/apache/hudi/pull/2751/diff?src=pr&el=tree-more) | |
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] hudi-bot removed a comment on pull request #2751: [HUDI-1748] Read operation will possiblity fail on mor table rt view when a write operations is concurrency running

Posted by GitBox <gi...@apache.org>.
hudi-bot removed a comment on pull request #2751:
URL: https://github.com/apache/hudi/pull/2751#issuecomment-914646698


   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "2de7140cbdf98568f63110b88e2ff65cab14c0b1",
       "status" : "FAILURE",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2082",
       "triggerID" : "2de7140cbdf98568f63110b88e2ff65cab14c0b1",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 2de7140cbdf98568f63110b88e2ff65cab14c0b1 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2082) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run travis` re-run the last Travis build
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] hudi-bot removed a comment on pull request #2751: [HUDI-1748] Read operation will possiblity fail on mor table rt view when a write operations is concurrency running

Posted by GitBox <gi...@apache.org>.
hudi-bot removed a comment on pull request #2751:
URL: https://github.com/apache/hudi/pull/2751#issuecomment-914646698


   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "2de7140cbdf98568f63110b88e2ff65cab14c0b1",
       "status" : "FAILURE",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2082",
       "triggerID" : "2de7140cbdf98568f63110b88e2ff65cab14c0b1",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 2de7140cbdf98568f63110b88e2ff65cab14c0b1 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2082) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run travis` re-run the last Travis build
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] n3nash commented on pull request #2751: [HUDI-1748] Read operation will possiblity fail on mor table rt view when a write operations is concurrency running

Posted by GitBox <gi...@apache.org>.
n3nash commented on pull request #2751:
URL: https://github.com/apache/hudi/pull/2751#issuecomment-812329664


   @li36909 If I understand this correctly, you are saying reading a MOR table `_rt` view can fail if a concurrent write is happening ?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] nsivabalan commented on pull request #2751: [HUDI-1748] Read operation will possiblity fail on mor table rt view when a write operations is concurrency running

Posted by GitBox <gi...@apache.org>.
nsivabalan commented on pull request #2751:
URL: https://github.com/apache/hudi/pull/2751#issuecomment-846058878


   @n3nash : once you verify the issue, please move it to "ready to review" col in the PR board. 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] nsivabalan commented on pull request #2751: [HUDI-1748] Read operation will possiblity fail on mor table rt view when a write operations is concurrency running

Posted by GitBox <gi...@apache.org>.
nsivabalan commented on pull request #2751:
URL: https://github.com/apache/hudi/pull/2751#issuecomment-812037473


   @li36909 : can you clarify what do you mean by concurrent operations. Is it cleaning/archiving along with a batch of write or do you say two batches of write going concurrently ? 
   @n3nash : you may want to check it out. 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] li36909 commented on pull request #2751: [HUDI-1748] Read operation will possiblity fail on mor table rt view when a write operations is concurrency running

Posted by GitBox <gi...@apache.org>.
li36909 commented on pull request #2751:
URL: https://github.com/apache/hudi/pull/2751#issuecomment-811582115


   cc @nsivabalan could you help to take a look, thank you


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] nsivabalan commented on pull request #2751: [HUDI-1748] Read operation will possiblity fail on mor table rt view when a write operations is concurrency running

Posted by GitBox <gi...@apache.org>.
nsivabalan commented on pull request #2751:
URL: https://github.com/apache/hudi/pull/2751#issuecomment-1032116706


   @alexeykudinkin : this has been a long pending PR. Can you follow up on this please. Guess there are steps to reproduce. If its not a valid issue, we can close the patch and the jira. 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org