You are viewing a plain text version of this content. The canonical link for it is here.
Posted to jira@kafka.apache.org by "msillence (via GitHub)" <gi...@apache.org> on 2024/10/15 14:02:37 UTC

[PR] Kafka 17792 header parsing times out processing and using large quantities of memory if the string looks like a number [kafka]

msillence opened a new pull request, #17510:
URL: https://github.com/apache/kafka/pull/17510

   Jira [KAFKA-17792](https://issues.apache.org/jira/browse/KAFKA-17792)
   
   We have trace headers such as:
   
   "X-B3-SpanId": "74320e6e26adc8f8"
   
   if however the value happens to be: "407127e212797209"
   
   This is then treated as a numeric value and it tries to convert this as a numeric representation and an exact value using BigDecimal
   Specifically the calls to setScale with floor and ceiling seem to cause the problem
   
   testing this the behaviour of biginteger and setScale calls shows there are already limits on the number range possible sadly my particular number falls in the range of about 25 minutes and 4 gig of memory rather than quick failure
   
   1e+1 time 0.0 totalMemory 532676608
   1e+10 time 0.0 totalMemory 532676608
   1e+100 time 0.0 totalMemory 532676608
   1e+1000 time 0.0 totalMemory 532676608
   1e+10000 time 0.005 totalMemory 532676608
   1e+100000 time 0.035 totalMemory 532676608
   1e+1000000 time 0.228 totalMemory 532676608
   1e+10000000 time 4.308 totalMemory 926941184
   1e+100000000 time 117.119 totalMemory 3221225472
   1e+1000000000 time 0.0 totalMemory 3221225472 BigInteger would overflow supported range
   1e+10000000000 time 0.001 totalMemory 3221225472 Too many nonzero exponent digits.
   1e+100000000000 time 0.0 totalMemory 3221225472 Too many nonzero exponent digits.
   
   ### Committer Checklist (excluded from commit message)
   - [ ] Verify design and implementation 
   - [ ] Verify test coverage and CI build status
   - [ ] Verify documentation (including upgrade notes)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Kafka 17792 header parsing times out processing and using large quantities of memory if the string looks like a number [kafka]

Posted by "gharris1727 (via GitHub)" <gi...@apache.org>.
gharris1727 commented on code in PR #17510:
URL: https://github.com/apache/kafka/pull/17510#discussion_r1817251244


##########
connect/api/src/main/java/org/apache/kafka/connect/data/Values.java:
##########
@@ -1041,6 +1043,10 @@ private static SchemaAndValue parseAsNumber(String token) {
         }
 
         private static SchemaAndValue parseAsExactDecimal(BigDecimal decimal) {
+            BigDecimal abs = decimal.abs();
+            if (abs.compareTo(TOO_BIG) > 0 || (abs.compareTo(TOO_SMALL) < 0 && BigDecimal.ZERO.compareTo(abs) != 0)) {
+                throw new NumberFormatException("outside efficient parsing range");

Review Comment:
   At this stage in parsing, we've successfully parsed a BigDecimal. This method tries to losslessly cast the BigDecimal to a primitive integer type. If the cast would be lossy, it returns null and lets the caller figure out whether it's a float32, float64, or has to be an arbitrary precision decimal.
   
   Rather than throwing NumberFormatException and making this a STRING, we should return null, and let the other fallback logic happen, so this shows up as an arbitrary precision Decimal.
   
   I also suspect that the comparison could be a little tighter in that case: TOO_BIG could be LONG_MAX, and TOO_SMALL could be ONE, eliminating calling setScale for `1e20` which cannot possibly fit in a LONG.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Kafka 17792 header parsing times out processing and using large quantities of memory if the string looks like a number [kafka]

Posted by "sillencem (via GitHub)" <gi...@apache.org>.
sillencem commented on PR #17510:
URL: https://github.com/apache/kafka/pull/17510#issuecomment-2612947561

   I rebased and the build is now clean


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Kafka 17792 header parsing times out processing and using large quantities of memory if the string looks like a number [kafka]

Posted by "mimaison (via GitHub)" <gi...@apache.org>.
mimaison commented on PR #17510:
URL: https://github.com/apache/kafka/pull/17510#issuecomment-2414556510

   Thanks for the PR. Please make sure it only contains changes related to the issue this is addressing. Currently I see a lot of other changes including some non-Apache Kafka files.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Kafka 17792 header parsing times out processing and using large quantities of memory if the string looks like a number [kafka]

Posted by "msillence (via GitHub)" <gi...@apache.org>.
msillence commented on PR #17510:
URL: https://github.com/apache/kafka/pull/17510#issuecomment-2420378656

   I feel like the correct fix is the stackoverflow test:
   `bd.signum() == 0 || bd.scale() <= 0 || bd.stripTrailingZeros().scale() <= 0`
   sadly there are quite a few tests that expect the perhaps incorrect behavior where Float.MAX_VALUE which is 3.4028235E38 looks like a integer to me but the ceil/floor calls failed due to the number of zeros so it fell though to another part of the parsing and returned a float.
   To further confound the issue the very small numbers such as 1e-100000000 will with the new integer test fall though to the float conversion which only checks the upper range and will simply convert to zero as float doesn't have the precision. Fixing that breaks yet more tests.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Kafka 17792 header parsing times out processing and using large quantities of memory if the string looks like a number [kafka]

Posted by "gharris1727 (via GitHub)" <gi...@apache.org>.
gharris1727 commented on PR #17510:
URL: https://github.com/apache/kafka/pull/17510#issuecomment-2617005007

   Thanks so much @msillence for reporting and fixing this!


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Kafka 17792 header parsing times out processing and using large quantities of memory if the string looks like a number [kafka]

Posted by "msillence (via GitHub)" <gi...@apache.org>.
msillence commented on PR #17510:
URL: https://github.com/apache/kafka/pull/17510#issuecomment-2416330859

   Oh now I see the problem I've mixed up repos this change is against github.com:confluentinc/kafka
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Kafka 17792 header parsing times out processing and using large quantities of memory if the string looks like a number [kafka]

Posted by "sillencem (via GitHub)" <gi...@apache.org>.
sillencem commented on code in PR #17510:
URL: https://github.com/apache/kafka/pull/17510#discussion_r1928459255


##########
connect/api/src/main/java/org/apache/kafka/connect/data/Values.java:
##########
@@ -1039,30 +1041,34 @@ private static SchemaAndValue parseAsNumber(String token) {
                 return new SchemaAndValue(schema, decimal);
             }
         }
+        
+    	public static boolean isBigInteger(BigDecimal bd) {
+    		return bd.signum() == 0 || bd.scale() <= 0 || bd.stripTrailingZeros().scale() <= 0;
+    	}        
 
         private static SchemaAndValue parseAsExactDecimal(BigDecimal decimal) {
+            BigDecimal abs = decimal.abs();

Review Comment:
   I think I've addressed this now



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Kafka 17792 header parsing times out processing and using large quantities of memory if the string looks like a number [kafka]

Posted by "gharris1727 (via GitHub)" <gi...@apache.org>.
gharris1727 commented on code in PR #17510:
URL: https://github.com/apache/kafka/pull/17510#discussion_r1924063426


##########
connect/api/src/main/java/org/apache/kafka/connect/data/Values.java:
##########
@@ -71,6 +71,8 @@ public class Values {
     static final String ISO_8601_DATE_FORMAT_PATTERN = "yyyy-MM-dd";
     static final String ISO_8601_TIME_FORMAT_PATTERN = "HH:mm:ss.SSS'Z'";
     static final String ISO_8601_TIMESTAMP_FORMAT_PATTERN = ISO_8601_DATE_FORMAT_PATTERN + "'T'" + ISO_8601_TIME_FORMAT_PATTERN;
+    private static BigDecimal TOO_BIG = new BigDecimal("1e1000000");
+    private static BigDecimal TOO_SMALL = new BigDecimal("1e-1000000");

Review Comment:
   These constants are unused.



##########
connect/api/src/test/java/org/apache/kafka/connect/data/ValuesTest.java:
##########
@@ -1154,7 +1152,17 @@ public void shouldParseFractionalPartsAsIntegerWhenNoFractionalPart() {
         assertEquals(new SchemaAndValue(Schema.INT32_SCHEMA, 66000), Values.parseString("66000.0"));
         assertEquals(new SchemaAndValue(Schema.FLOAT32_SCHEMA, 66000.0008f), Values.parseString("66000.0008"));
     }
+    @Test
+    public void avoidCpuAndMemoryIssuesConvertingExtremeBigDecimals() {
+    	long s = System.currentTimeMillis();
+        String veryBig = "1e+100000000"; // new BigDecimal().setScale(0, RoundingMode.FLOOR) takes around two minutes and uses 3GB;
+        System.out.println("time taken" + System.currentTimeMillis() - s)
+        assertEquals(new SchemaAndValue(Schema.BYTES_SCHEMA, veryBig), Values.parseString(veryBig));
 
+        String verySmall = "1e-100000000";
+        assertEquals(new SchemaAndValue(Schema.STRING_SCHEMA, verySmall), Values.parseString(verySmall));
+        assertFalse(true);

Review Comment:
   This will always fail.



##########
connect/api/src/test/java/org/apache/kafka/connect/data/ValuesTest.java:
##########
@@ -1154,7 +1152,17 @@ public void shouldParseFractionalPartsAsIntegerWhenNoFractionalPart() {
         assertEquals(new SchemaAndValue(Schema.INT32_SCHEMA, 66000), Values.parseString("66000.0"));
         assertEquals(new SchemaAndValue(Schema.FLOAT32_SCHEMA, 66000.0008f), Values.parseString("66000.0008"));
     }
+    @Test
+    public void avoidCpuAndMemoryIssuesConvertingExtremeBigDecimals() {
+    	long s = System.currentTimeMillis();
+        String veryBig = "1e+100000000"; // new BigDecimal().setScale(0, RoundingMode.FLOOR) takes around two minutes and uses 3GB;
+        System.out.println("time taken" + System.currentTimeMillis() - s)

Review Comment:
   I think we should remove this debug print, and have a Timeout annotation on this method to cause fails on regressions.



##########
connect/api/src/main/java/org/apache/kafka/connect/data/Values.java:
##########
@@ -1039,30 +1041,34 @@ private static SchemaAndValue parseAsNumber(String token) {
                 return new SchemaAndValue(schema, decimal);
             }
         }
+        
+    	public static boolean isBigInteger(BigDecimal bd) {

Review Comment:
   This uses tabs, which is disallowed by our linter.



##########
connect/api/src/test/java/org/apache/kafka/connect/data/ValuesTest.java:
##########
@@ -45,6 +41,8 @@
 import static org.junit.jupiter.api.Assertions.assertThrows;
 import static org.junit.jupiter.api.Assertions.assertTrue;
 import static org.junit.jupiter.api.Assertions.fail;
+import org.junit.jupiter.api.Test;

Review Comment:
   If you run into problems with the imports, run `./gradlew spotlessApply`



##########
connect/api/src/test/java/org/apache/kafka/connect/data/ValuesTest.java:
##########
@@ -1154,7 +1152,17 @@ public void shouldParseFractionalPartsAsIntegerWhenNoFractionalPart() {
         assertEquals(new SchemaAndValue(Schema.INT32_SCHEMA, 66000), Values.parseString("66000.0"));
         assertEquals(new SchemaAndValue(Schema.FLOAT32_SCHEMA, 66000.0008f), Values.parseString("66000.0008"));
     }
+    @Test
+    public void avoidCpuAndMemoryIssuesConvertingExtremeBigDecimals() {
+    	long s = System.currentTimeMillis();
+        String veryBig = "1e+100000000"; // new BigDecimal().setScale(0, RoundingMode.FLOOR) takes around two minutes and uses 3GB;
+        System.out.println("time taken" + System.currentTimeMillis() - s)
+        assertEquals(new SchemaAndValue(Schema.BYTES_SCHEMA, veryBig), Values.parseString(veryBig));
 
+        String verySmall = "1e-100000000";
+        assertEquals(new SchemaAndValue(Schema.STRING_SCHEMA, verySmall), Values.parseString(verySmall));

Review Comment:
   This actually gets parsed as a FLOAT32. Arguably that's a bug and we're losing precision, but it's the legacy behavior and we shouldn't change it, or at least have a separate change for this. For now, we should just update this test to match the behavior.



##########
connect/api/src/main/java/org/apache/kafka/connect/data/Values.java:
##########
@@ -1039,30 +1041,34 @@ private static SchemaAndValue parseAsNumber(String token) {
                 return new SchemaAndValue(schema, decimal);
             }
         }
+        
+    	public static boolean isBigInteger(BigDecimal bd) {
+    		return bd.signum() == 0 || bd.scale() <= 0 || bd.stripTrailingZeros().scale() <= 0;
+    	}        
 
         private static SchemaAndValue parseAsExactDecimal(BigDecimal decimal) {
+            BigDecimal abs = decimal.abs();

Review Comment:
   This value is unused.



##########
connect/api/src/test/java/org/apache/kafka/connect/data/ValuesTest.java:
##########
@@ -1154,7 +1152,17 @@ public void shouldParseFractionalPartsAsIntegerWhenNoFractionalPart() {
         assertEquals(new SchemaAndValue(Schema.INT32_SCHEMA, 66000), Values.parseString("66000.0"));
         assertEquals(new SchemaAndValue(Schema.FLOAT32_SCHEMA, 66000.0008f), Values.parseString("66000.0008"));
     }
+    @Test
+    public void avoidCpuAndMemoryIssuesConvertingExtremeBigDecimals() {
+    	long s = System.currentTimeMillis();
+        String veryBig = "1e+100000000"; // new BigDecimal().setScale(0, RoundingMode.FLOOR) takes around two minutes and uses 3GB;

Review Comment:
   I think this example actually still has a regression. i.e. the bug is still there because ceil takes forever.



##########
connect/api/src/main/java/org/apache/kafka/connect/data/Values.java:
##########
@@ -1039,30 +1041,34 @@ private static SchemaAndValue parseAsNumber(String token) {
                 return new SchemaAndValue(schema, decimal);
             }
         }
+        
+    	public static boolean isBigInteger(BigDecimal bd) {
+    		return bd.signum() == 0 || bd.scale() <= 0 || bd.stripTrailingZeros().scale() <= 0;

Review Comment:
   This stripTrailingZeros covers the cases that the ceil/floor comparison was supposed to cover, great find!



##########
connect/api/src/test/java/org/apache/kafka/connect/data/ValuesTest.java:
##########
@@ -1154,7 +1152,17 @@ public void shouldParseFractionalPartsAsIntegerWhenNoFractionalPart() {
         assertEquals(new SchemaAndValue(Schema.INT32_SCHEMA, 66000), Values.parseString("66000.0"));
         assertEquals(new SchemaAndValue(Schema.FLOAT32_SCHEMA, 66000.0008f), Values.parseString("66000.0008"));
     }
+    @Test
+    public void avoidCpuAndMemoryIssuesConvertingExtremeBigDecimals() {
+    	long s = System.currentTimeMillis();
+        String veryBig = "1e+100000000"; // new BigDecimal().setScale(0, RoundingMode.FLOOR) takes around two minutes and uses 3GB;
+        System.out.println("time taken" + System.currentTimeMillis() - s)
+        assertEquals(new SchemaAndValue(Schema.BYTES_SCHEMA, veryBig), Values.parseString(veryBig));

Review Comment:
   This shouldn't be a plain BYTES schema, it should be a connect `Decimal` type. And veryBig needs to be parsed into a BigDecimal.



##########
connect/api/src/test/java/org/apache/kafka/connect/data/ValuesTest.java:
##########
@@ -1154,7 +1152,17 @@ public void shouldParseFractionalPartsAsIntegerWhenNoFractionalPart() {
         assertEquals(new SchemaAndValue(Schema.INT32_SCHEMA, 66000), Values.parseString("66000.0"));
         assertEquals(new SchemaAndValue(Schema.FLOAT32_SCHEMA, 66000.0008f), Values.parseString("66000.0008"));
     }
+    @Test

Review Comment:
   nit: missing a newline above this test.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Kafka 17792 header parsing times out processing and using large quantities of memory if the string looks like a number [kafka]

Posted by "gharris1727 (via GitHub)" <gi...@apache.org>.
gharris1727 merged PR #17510:
URL: https://github.com/apache/kafka/pull/17510


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Kafka 17792 header parsing times out processing and using large quantities of memory if the string looks like a number [kafka]

Posted by "msillence (via GitHub)" <gi...@apache.org>.
msillence commented on code in PR #17510:
URL: https://github.com/apache/kafka/pull/17510#discussion_r1819229664


##########
connect/api/src/main/java/org/apache/kafka/connect/data/Values.java:
##########
@@ -1041,6 +1043,10 @@ private static SchemaAndValue parseAsNumber(String token) {
         }
 
         private static SchemaAndValue parseAsExactDecimal(BigDecimal decimal) {
+            BigDecimal abs = decimal.abs();
+            if (abs.compareTo(TOO_BIG) > 0 || (abs.compareTo(TOO_SMALL) < 0 && BigDecimal.ZERO.compareTo(abs) != 0)) {
+                throw new NumberFormatException("outside efficient parsing range");

Review Comment:
   I thought we were converting to a BigInteger at this point?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Kafka 17792 header parsing times out processing and using large quantities of memory if the string looks like a number [kafka]

Posted by "msillence (via GitHub)" <gi...@apache.org>.
msillence commented on PR #17510:
URL: https://github.com/apache/kafka/pull/17510#issuecomment-2416390523

   actually there is a simpler solution https://stackoverflow.com/a/12748321


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Kafka 17792 header parsing times out processing and using large quantities of memory if the string looks like a number [kafka]

Posted by "msillence (via GitHub)" <gi...@apache.org>.
msillence commented on code in PR #17510:
URL: https://github.com/apache/kafka/pull/17510#discussion_r1819229664


##########
connect/api/src/main/java/org/apache/kafka/connect/data/Values.java:
##########
@@ -1041,6 +1043,10 @@ private static SchemaAndValue parseAsNumber(String token) {
         }
 
         private static SchemaAndValue parseAsExactDecimal(BigDecimal decimal) {
+            BigDecimal abs = decimal.abs();
+            if (abs.compareTo(TOO_BIG) > 0 || (abs.compareTo(TOO_SMALL) < 0 && BigDecimal.ZERO.compareTo(abs) != 0)) {
+                throw new NumberFormatException("outside efficient parsing range");

Review Comment:
   I thought we were converting to a BigInteger at this point?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Kafka 17792 header parsing times out processing and using large quantities of memory if the string looks like a number [kafka]

Posted by "msillence (via GitHub)" <gi...@apache.org>.
msillence commented on PR #17510:
URL: https://github.com/apache/kafka/pull/17510#issuecomment-2416327765

   > Thanks for the PR. Please make sure it only contains changes related to the issue this is addressing. Currently I see a lot of other changes including some non-Apache Kafka files.
   
   I'm not quite sure what I've done wrong, I cloned the repo and applied my change again and forced pushed it to my fork and it still looks wrong
   
   The change is here https://github.com/msillence/kafka/commit/372095166a2d495af333554ca3476fd5076f64fa


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Kafka 17792 header parsing times out processing and using large quantities of memory if the string looks like a number [kafka]

Posted by "showuon (via GitHub)" <gi...@apache.org>.
showuon commented on PR #17510:
URL: https://github.com/apache/kafka/pull/17510#issuecomment-2604223295

   @gharris1727 , do you have any further comments for this?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org