You are viewing a plain text version of this content. The canonical link for it is here.
Posted to jira@kafka.apache.org by "bachmanity1 (via GitHub)" <gi...@apache.org> on 2023/02/14 09:28:20 UTC
[GitHub] [kafka] bachmanity1 opened a new pull request, #13246: MINOR: Remove useless array size check
bachmanity1 opened a new pull request, #13246:
URL: https://github.com/apache/kafka/pull/13246
```
if (size > buffer.remaining())
throw new SchemaException("Error reading array of size " + size + ", only " + buffer.remaining() + " bytes available");
```
Array type has the above code snippet which checks if there are enough bytes in the buffer to read all the array elements. However, there is a unit mismatch because `size` is the number of elements in the array, not the number of bytes. Thus, even if the above check passes there still might be not enough bytes remaining in the buffer. It seems to be impossible to compute byte size of the array before reading the whole array because each element of the array might have a different size (for example, array of strings). In my opinion this code can be removed because it doesn't solve the problem it was written to solve.
### Committer Checklist (excluded from commit message)
- [ ] Verify design and implementation
- [ ] Verify test coverage and CI build status
- [ ] Verify documentation (including upgrade notes)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] bachmanity1 commented on pull request #13246: MINOR: Remove useless array size check
Posted by "bachmanity1 (via GitHub)" <gi...@apache.org>.
bachmanity1 commented on PR #13246:
URL: https://github.com/apache/kafka/pull/13246#issuecomment-1429404900
@cmccabe @dongjinleekr could you please have a look?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] bachmanity1 commented on a diff in pull request #13246: KAFKA-14707: Improve array size check
Posted by "bachmanity1 (via GitHub)" <gi...@apache.org>.
bachmanity1 commented on code in PR #13246:
URL: https://github.com/apache/kafka/pull/13246#discussion_r1106542835
##########
clients/src/main/java/org/apache/kafka/common/protocol/types/ArrayOf.java:
##########
@@ -72,10 +74,10 @@ public Object read(ByteBuffer buffer) {
else if (size < 0)
throw new SchemaException("Array size " + size + " cannot be negative");
- Object[] objs = new Object[size];
+ List<Object> objs = new ArrayList<>();
for (int i = 0; i < size; i++)
- objs[i] = type.read(buffer);
- return objs;
+ objs.add(type.read(buffer));
+ return objs.toArray();
Review Comment:
> Checking for OOM is a security issue, so we cannot do that as the main approach. We need to do pre-validation.
@ijuma
Then how about using a list instead of an array? I know this might have a negative impact on performance but it seems to resolve OOM issue.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] ijuma commented on pull request #13246: KAFKA-14707: Improve array size check
Posted by "ijuma (via GitHub)" <gi...@apache.org>.
ijuma commented on PR #13246:
URL: https://github.com/apache/kafka/pull/13246#issuecomment-1430570552
Checking for OOM is a security issue, so we cannot do that.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] bachmanity1 commented on pull request #13246: KAFKA-14707: Improve array size check
Posted by "bachmanity1 (via GitHub)" <gi...@apache.org>.
bachmanity1 commented on PR #13246:
URL: https://github.com/apache/kafka/pull/13246#issuecomment-1433802673
@showuon
> Basically, the size > buffer.remaining() check is originally not trying to check if the size can be allocated in the JVM, it's just a sanity check to see if this size is abnormal, right? What we're trying to protect is just the array allocation can be fit into JVM
yeah, then this check definitely makes sense. It's just not immediately clear from the exception message that this check is done to prevent OOM error.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] bachmanity1 closed pull request #13246: KAFKA-14707: Improve array size check
Posted by "bachmanity1 (via GitHub)" <gi...@apache.org>.
bachmanity1 closed pull request #13246: KAFKA-14707: Improve array size check
URL: https://github.com/apache/kafka/pull/13246
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org