You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@avro.apache.org by GitBox <gi...@apache.org> on 2020/11/20 18:36:23 UTC
[GitHub] [avro] bryjeep opened a new pull request #1013: AVRO-2983: Set Span length of ArrayPool Rent buffer
bryjeep opened a new pull request #1013:
URL: https://github.com/apache/avro/pull/1013
ArrayPool<T>.Rent(Int32) returns a buffer that is at least
minimumLength in length, but can be more. The span that is used is
read into until it is empty. When this buffer returned from ArrayPool
is not exactly the same as the requested minimumLength this causes an
Avro.AvroException : End of stream reached
Make sure you have checked _all_ steps below.
### Jira
- [x] My PR addresses the following [Avro Jira](https://issues.apache.org/jira/browse/AVRO/) issues and references them in the PR title. For example, "AVRO-1234: My Avro PR"
- https://issues.apache.org/jira/browse/AVRO-2983
### Tests
- [x] My PR adds the following unit tests __OR__ does not need testing for this extremely good reason:
- Avro.Test.BinaryCodecTests.TestLargeString which generates a 16KB buffer in array pool and then tries to read 16000 string, and this shows how the system currently fails and the patch fixes. The existing unit tests for string reading didn't catch this because the ArrayPool was in a clean state for each of those reads, this simulates a real world case where the ArrayPool contains buffers of different sizes.
### Commits
- [x] My commits all reference Jira issues in their subject lines. In addition, my commits follow the guidelines from "[How to write a good git commit message](https://chris.beams.io/posts/git-commit/)":
1. Subject is separated from body by a blank line
1. Subject is limited to 50 characters (not including Jira issue reference)
1. Subject does not end with a period
1. Subject uses the imperative mood ("add", not "adding")
1. Body wraps at 72 characters
1. Body explains "what" and "why", not "how"
### Documentation
- [x] In case of new functionality, my PR adds documentation that describes how to use it.
- All the public functions and the classes in the PR contain Javadoc that explain what it does
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [avro] eerhardt commented on a change in pull request #1013: AVRO-2983: Set Span length of ArrayPool Rent buffer
Posted by GitBox <gi...@apache.org>.
eerhardt commented on a change in pull request #1013:
URL: https://github.com/apache/avro/pull/1013#discussion_r547356202
##########
File path: lang/csharp/src/apache/test/IO/BinaryCodecTests.cs
##########
@@ -214,6 +213,23 @@ public void TestString(string n, int overhead)
TestSkip(n, (Decoder d) => d.SkipString(), (Encoder e, string t) => e.WriteString(t), overhead + n.Length);
}
+#if NETCOREAPP3_1
Review comment:
How do you feel about also adding the test from https://issues.apache.org/jira/secure/attachment/13017389/AVRO-3005.patch ?
That way `net461` gets test coverage for a longer string as well.
BTW - AVRO-3005 appears to be a duplicate of AVRO-2983.
```diff
[TestCase("", 1)]
[TestCase("hello", 1)]
[TestCase("1234567890123456789012345678901234567890123456789012345678901234", 2)]
+ [TestCase("01234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456", 2)]
public void TestString(string n, int overhead)
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [avro] JarrodJ83 commented on pull request #1013: AVRO-2983: Set Span length of ArrayPool Rent buffer
Posted by GitBox <gi...@apache.org>.
JarrodJ83 commented on pull request #1013:
URL: https://github.com/apache/avro/pull/1013#issuecomment-745323299
Any idea if/when this will get merged? I'm currently running into issues in regard to this and am trying to figure out what my options are.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [avro] yxmm-wxe commented on pull request #1013: AVRO-2983: Set Span length of ArrayPool Rent buffer
Posted by GitBox <gi...@apache.org>.
yxmm-wxe commented on pull request #1013:
URL: https://github.com/apache/avro/pull/1013#issuecomment-767389072
When will the 1.10.2 be released?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [avro] RyanSkraba commented on pull request #1013: AVRO-2983: Set Span length of ArrayPool Rent buffer
Posted by GitBox <gi...@apache.org>.
RyanSkraba commented on pull request #1013:
URL: https://github.com/apache/avro/pull/1013#issuecomment-760160459
Cherry picked to [branch-1.10](https://github.com/apache/avro/commit/266ba6e7afa8f67a11eed747a9688c8058e67b34), so you will see it in the 1.10.2 release.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [avro] RyanSkraba commented on pull request #1013: AVRO-2983: Set Span length of ArrayPool Rent buffer
Posted by GitBox <gi...@apache.org>.
RyanSkraba commented on pull request #1013:
URL: https://github.com/apache/avro/pull/1013#issuecomment-760156219
Hello! I've run the provided unit test -- it works with the fix, and it fails without the fix in BinaryDecoder. Since three developers have come up with (nearly) identical solutions and this has been reviewed by @eerhardt I'm going to merge this and mark it for the next minor release of Avro.
Thanks for your patience! Ryan
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [avro] eerhardt commented on pull request #1013: AVRO-2983: Set Span length of ArrayPool Rent buffer
Posted by GitBox <gi...@apache.org>.
eerhardt commented on pull request #1013:
URL: https://github.com/apache/avro/pull/1013#issuecomment-760288352
Thank you for merging the fix, @RyanSkraba!
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [avro] eerhardt commented on pull request #1013: AVRO-2983: Set Span length of ArrayPool Rent buffer
Posted by GitBox <gi...@apache.org>.
eerhardt commented on pull request #1013:
URL: https://github.com/apache/avro/pull/1013#issuecomment-749623539
@blachniet @Fokko @dkulp - This issue is pretty prevalent, it's been logged 3 times in JIRA so far:
* AVRO-2929
* AVRO-2983
* AVRO-3005
Any chance this could get merged sometime soon?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [avro] RyanSkraba merged pull request #1013: AVRO-2983: Set Span length of ArrayPool Rent buffer
Posted by GitBox <gi...@apache.org>.
RyanSkraba merged pull request #1013:
URL: https://github.com/apache/avro/pull/1013
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [avro] bryjeep commented on pull request #1013: AVRO-2983: Set Span length of ArrayPool Rent buffer
Posted by GitBox <gi...@apache.org>.
bryjeep commented on pull request #1013:
URL: https://github.com/apache/avro/pull/1013#issuecomment-731344045
After making this PR I noticed another PR someone already independently made #957 for AVRO-2929 which is the same issue. Either is fine, this one has unit tests.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org