You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@avro.apache.org by GitBox <gi...@apache.org> on 2021/10/09 11:57:07 UTC
[GitHub] [avro] blachniet commented on a change in pull request #1357: AVRO-3225 & AVRO-3226: Fix possible StackOverflowException and OutOfMemoryException on invalid input
blachniet commented on a change in pull request #1357:
URL: https://github.com/apache/avro/pull/1357#discussion_r725478365
##########
File path: lang/csharp/src/apache/test/IO/BinaryCodecTests.cs
##########
@@ -216,21 +217,98 @@ public void TestString(string n, int overhead)
#if NETCOREAPP3_1
[Test]
- public void TestLargeString()
+ public void TestStringReadIntoArrayPool()
{
+ const int maxFastReadLength = 4096;
+
// Create a 16KB buffer in the Array Pool
var largeBufferToSeedPool = ArrayPool<byte>.Shared.Rent(2 << 14);
ArrayPool<byte>.Shared.Return(largeBufferToSeedPool);
- // Create a slightly less than 16KB buffer, which will use the 16KB buffer in the pool
- var n = string.Concat(Enumerable.Repeat("1234567890", 1600));
- var overhead = 3;
+ var n = string.Concat(Enumerable.Repeat("A", maxFastReadLength));
+ var overhead = 2;
TestRead(n, (Decoder d) => d.ReadString(), (Encoder e, string t) => e.WriteString(t), overhead + n.Length);
- TestSkip(n, (Decoder d) => d.SkipString(), (Encoder e, string t) => e.WriteString(t), overhead + n.Length);
+ }
+
+ [Test]
+ public void TestStringReadByBinaryReader()
+ {
+ const int overhead = 2;
+ const int maxFastReadLength = 4096;
+ const int expectedStringLength = maxFastReadLength + 1;
+ var n = string.Concat(Enumerable.Repeat("A", expectedStringLength));
+
+ TestRead(n, (Decoder d) => d.ReadString(), (Encoder e, string t) => e.WriteString(t), expectedStringLength + overhead);
}
#endif
+ [Test]
+ public void TestInvalidInputWithNegativeStringLength()
+ {
+ using (MemoryStream iostr = new MemoryStream())
+ {
+ Encoder e = new BinaryEncoder(iostr);
+
+ e.WriteLong(-1);
+
+ iostr.Flush();
+ iostr.Position = 0;
+ Decoder d = new BinaryDecoder(iostr);
+
+ var exception = Assert.Throws<AvroException>(() => d.ReadString());
+
+ Assert.NotNull(exception);
+ Assert.AreEqual("Can not deserialize a string with negative length!", exception.Message);
+ iostr.Close();
+ }
+ }
+
+ [Test]
+ public void TestInvalidInputWithMaxIntAsStringLength()
+ {
+ using (MemoryStream iostr = new MemoryStream())
+ {
+ Encoder e = new BinaryEncoder(iostr);
+
+ e.WriteLong(int.MaxValue);
+ e.WriteBytes(Encoding.UTF8.GetBytes("SomeSmallString"));
+
+ iostr.Flush();
+ iostr.Position = 0;
+ Decoder d = new BinaryDecoder(iostr);
+
+ var exception = Assert.Throws<AvroException>(() => d.ReadString());
+
+ Assert.NotNull(exception);
+ Assert.AreEqual("String length is not supported!", exception.Message);
+ iostr.Close();
+ }
+ }
+
+ [Test]
+ public void TestInvalidInputWithMaxArrayLengthAsStringLength()
Review comment:
This test is failing when running on .NET Framework 4.6.1. It looks like we aren't running the Windows builds/tests in GitHub actions anymore, which would explain why that didn't catch it.
```
+ dotnet test --configuration Release --no-build --filter 'TestCategory!=Interop' Avro.sln
Test run for C:\Users\Brian.Lachniet\code\github.com\apache\avro\lang\csharp\src\apache\test\bin\Release\net461\Avro.test.dll (.NETFramework,Version=v4.6.1)
Microsoft (R) Test Execution Command Line Tool Version 16.11.0
Copyright (c) Microsoft Corporation. All rights reserved.
Starting test execution, please wait...
A total of 1 test files matched the specified pattern.
Failed TestInvalidInputWithMaxArrayLengthAsStringLength [75 ms]
Error Message:
Expected: <Avro.AvroException>
But was: <System.OutOfMemoryException: Exception of type 'System.OutOfMemoryException' was thrown.
at System.IO.BinaryReader.ReadBytes(Int32 count)
at Avro.IO.BinaryDecoder.ReadString() in C:\Users\Brian.Lachniet\code\github.com\apache\avro\lang\csharp\src\apache\main\IO\BinaryDecoder.netstandard2.0.cs:line 90
at Avro.Test.BinaryCodecTests.<>c__DisplayClass11_0.<TestInvalidInputWithMaxArrayLengthAsStringLength>b__0() in C:\Users\Brian.Lachniet\code\github.com\apache\avro\lang\csharp\src\apache\test\IO\BinaryCodecTests.cs:line 304
at NUnit.Framework.Assert.Throws(IResolveConstraint expression, TestDelegate code, String message, Object[] args)>
Stack Trace:
at Avro.Test.BinaryCodecTests.TestInvalidInputWithMaxArrayLengthAsStringLength() in C:\Users\Brian.Lachniet\code\github.com\apache\avro\lang\csharp\src\apache\test\IO\BinaryCodecTests.cs:line 304
Failed! - Failed: 1, Passed: 626, Skipped: 0, Total: 627, Duration: 3 s - Avro.test.dll (net461)
Test run for C:\Users\Brian.Lachniet\code\github.com\apache\avro\lang\csharp\src\apache\test\bin\Release\netcoreapp2.1\Avro.test.dll (.NETCoreApp,Version=v2.1)
```
The [Remarks on the Array Class](https://docs.microsoft.com/en-us/dotnet/api/system.array?view=net-5.0#remarks) indicate that the .NET Framework may have a smaller size limit. However, even after setting `MaxDotNetArrayLength = 0x77359400;`, I'm getting the same test failure (only on the net461 tests).
@RyanSkraba , how does the Java implementation handle large strings? Does it specify some arbitrary limit?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@avro.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org