[GitHub] [incubator-hudi] lamber-ken edited a comment on issue #1466: [HUDI-742] Fix Java Math Exception

[GitBox <gi...@apache.org>]

lamber-ken edited a comment on issue #1466: [HUDI-742] Fix Java Math Exception
URL: https://github.com/apache/incubator-hudi/pull/1466#issuecomment-606271560
 
 
   hi, @vinothchandar let me summarize the analysis process
   
   #### 1. The exception stackstrace
   ```
   Caused by: java.lang.NoSuchMethodError: java.lang.Math.floorMod(JI)I
   at org.apache.hudi.index.bloom.BucketizedBloomCheckPartitioner.getPartition(BucketizedBloomCheckPartitioner.java:148)
   at org.apache.spark.shuffle.sort.BypassMergeSortShuffleWriter.write(BypassMergeSortShuffleWriter.java:151)
   ```
   
   #### 2. Check the codebase (BucketizedBloomCheckPartitioner#getPartition)
   ```
   @Override
   public int getPartition(Object key) {
     final Pair<String, String> parts = (Pair<String, String>) key;
     final long hashOfKey = Hashing.md5().hashString(parts.getRight(), StandardCharsets.UTF_8).asLong();
     final List<Integer> candidatePartitions = fileGroupToPartitions.get(parts.getLeft());
     final int idx = (int) Math.floorMod(hashOfKey, candidatePartitions.size());
     assert idx >= 0;
     return candidatePartitions.get(idx);
   }
   ```
   
   #### 3. Analyze 
   1, The code snippet above seems very normal, we cann't find any problem just only from it.
   2, What caused `java.lang.NoSuchMethodError: java.lang.Math.floorMod(JI)I`?
   
   #### 4. Some thing about JVM specification
   from the JVM specification we know that:
   `I: Integer`, `J:Long`
   
   ![image](https://user-images.githubusercontent.com/20113411/77780070-6592ae00-708e-11ea-9980-2e79a0f8055c.png)
   
   https://docs.oracle.com/javase/specs/jvms/se8/html/jvms-4.html#jvms-4.3.2-200
   
   #### 5. Check released version `0.5.0` / `0.5.1` hudi version
   1) parse bytecode 
   ```
   jar xf hudi-client-0.5.2-incubating.jar
   javap -v org/apache/hudi/index/bloom/BucketizedBloomCheckPartitioner.class | grep Math.floor
   ```
   2) output
   ```
      #14 = Methodref          #184.#187     // java/lang/Math.floor:(D)D
      #64 = Methodref          #184.#242     // java/lang/Math.floorMod:(JJ)J
           88: invokestatic  #14                 // Method java/lang/Math.floor:(D)D
           54: invokestatic  #64                 // Method java/lang/Math.floorMod:(JJ)J
   ```
   3) from the output above, one thing we knew for certain was that official hudi-jar is ok.
   
   #### 6. Where is `java.lang.Math.floorMod(JI)I` from?
   At last we found the answer, @EdwinGuo use `jdk11` build the hudi project and then run on `jdk8`, it may caused by `jdk11` jvm compile mechanism or `floorMod(JI)I` exists in jdk11.
   
   #### 7. End
   So, the changes LGTM, thanks @EdwinGuo 
   
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services