You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "imhy (via GitHub)" <gi...@apache.org> on 2024/09/05 12:31:40 UTC

[PR] [MINOR] fix: LongWritable cannot be cast to TimestampWritable for MOR table with timestamp column and schema evolution enabled. [hudi]

imhy opened a new pull request, #11890:
URL: https://github.com/apache/hudi/pull/11890

   For hudi MOR table with timestamp column In some cases we get exception:
   org.apache.hadoop.io.LongWritable cannot be cast to org.apache.hadoop.hive.serde2.io.TimestampWritable*
   
   This happens because timestamp long value in log file converted to LongWritable but must be converted to TimestampWritable.
   
   Reason: field AbstractRealtimeRecordReader.supportTimestamp not initialized if Schema evolution context contain internal schema.
   
   This example reproduce problem (tested on spark3.3, hive 3.1):
   spark-sql>
   set hoodie.schema.on.read.enable=true;
   create table hudi_test1 (col0 int, col1 float, col2 string, col3 timestamp)
   using hudi
   tblproperties (
       type='mor',
       primaryKey='col0',
       preCombineField='col1',
       'hoodie.compaction.payload.class'='org.apache.hudi.common.model.OverwriteWithLatestAvroPayload');
   
   insert into hudi_test1 values(1, 1.1, 'text', timestamp('2021-12-25 12:01:01'));
   update hudi_test1 set col2 = 'text2' where col0 = 1;
   alter table hudi_test1 rename column col2 to col2_new;
   
   hive>
   set hoodie.schema.on.read.enable=true;
   select * from hudi_test1;
   Failed with exception java.io.IOException:java.lang.ClassCastException: org.apache.hadoop.io.LongWritable cannot be cast to org.apache.hadoop.hive.serde2.io.TimestampWritableV2
   
   ### Change Logs
   
   Logic of initialization of  supportTimestamp property moved to right place.
   
   ### Impact
   
   none
   
   ### Risk level (write none, low medium or high below)
   
   none
   
   ### Documentation Update
   
   none
   
   ### Contributor's checklist
   
   - [ ] Read through [contributor's guide](https://hudi.apache.org/contribute/how-to-contribute)
   - [ ] Change Logs and Impact were stated clearly
   - [ ] Adequate tests were added if applicable
   - [ ] CI passed
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] [HUDI-8175] Fix LongWritable cannot be cast to TimestampWritable for MOR table with timestamp column and schema evolution enabled [hudi]

Posted by "imhy (via GitHub)" <gi...@apache.org>.
imhy commented on PR #11890:
URL: https://github.com/apache/hudi/pull/11890#issuecomment-2333229190

   > @imhy could you raise another PR on [`branch-0.x`](https://github.com/apache/hudi/tree/branch-0.x) for 0.15.1 release?
   
   done
   https://github.com/apache/hudi/pull/11898


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] [MINOR] fix: LongWritable cannot be cast to TimestampWritable for MOR table with timestamp column and schema evolution enabled. [hudi]

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #11890:
URL: https://github.com/apache/hudi/pull/11890#issuecomment-2331744136

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "5bec9e77943a23fca0732198890caeb252e68a2f",
       "status" : "SUCCESS",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=274",
       "triggerID" : "5bec9e77943a23fca0732198890caeb252e68a2f",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 5bec9e77943a23fca0732198890caeb252e68a2f Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=274) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] [MINOR] fix: LongWritable cannot be cast to TimestampWritable for MOR table with timestamp column and schema evolution enabled. [hudi]

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #11890:
URL: https://github.com/apache/hudi/pull/11890#issuecomment-2331451647

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "5bec9e77943a23fca0732198890caeb252e68a2f",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=274",
       "triggerID" : "5bec9e77943a23fca0732198890caeb252e68a2f",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 5bec9e77943a23fca0732198890caeb252e68a2f Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=274) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] [MINOR] fix: LongWritable cannot be cast to TimestampWritable for MOR table with timestamp column and schema evolution enabled. [hudi]

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #11890:
URL: https://github.com/apache/hudi/pull/11890#issuecomment-2331442464

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "5bec9e77943a23fca0732198890caeb252e68a2f",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "5bec9e77943a23fca0732198890caeb252e68a2f",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 5bec9e77943a23fca0732198890caeb252e68a2f UNKNOWN
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] [HUDI-8175] Fix LongWritable cannot be cast to TimestampWritable for MOR table with timestamp column and schema evolution enabled [hudi]

Posted by "yihua (via GitHub)" <gi...@apache.org>.
yihua merged PR #11890:
URL: https://github.com/apache/hudi/pull/11890


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org