You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@avro.apache.org by "mkeskells (via GitHub)" <gi...@apache.org> on 2026/04/29 22:23:24 UTC
[PR] AVRO-4249: [java] provide a cache of schema to avoid building [avro]
mkeskells opened a new pull request, #3746:
URL: https://github.com/apache/avro/pull/3746
## What is the purpose of the change
To improve the performance of parsing files.
In an environment where we parse 10k to 100K (generally very small) files that are small, and use the same schema, or a handful of schemas we see many Tb of garbage generation with duplicate schemas being parsed
this PR is a simple fix to enable a cache to be inserted in the reader, so that the cache lookup can replace the parse where there is an exact match already
The shape of the API changes - I leave this to the reviewers to comment, and I am happy to work to their steer, and will generate tests when I have agreement of the approach to this
## Verifying this change
*(Please pick one of the following options)*
I will add tests once the chnages to the API have been agreed. There are many ways that this chnage could be implemented so I dont want to spend th time until the shape can be agreed
## Documentation
- Does this pull request introduce a new feature? yes - i presume there would be some pluggable system setting, and an API chnage
- If yes, how is the feature documented? not documented as yet
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@avro.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] AVRO-4249: [java] provide a cache of schema to avoid building [avro]
Posted by "RyanSkraba (via GitHub)" <gi...@apache.org>.
RyanSkraba commented on PR #3746:
URL: https://github.com/apache/avro/pull/3746#issuecomment-4390786168
Hey, pardon me! I appreciate the work, and I'll take a closer look soon -- we're going to do a 1.13.0 release just after the next one 1.12.2, and this should be in it!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@avro.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] AVRO-4249: [java] provide a cache of schema to avoid building [avro]
Posted by "mkeskells (via GitHub)" <gi...@apache.org>.
mkeskells commented on PR #3746:
URL: https://github.com/apache/avro/pull/3746#issuecomment-4364813598
Could a maintainer please add the performance label to this PR?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@avro.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] AVRO-4249: [java] provide a cache of schema to avoid building [avro]
Posted by "mkeskells (via GitHub)" <gi...@apache.org>.
mkeskells commented on PR #3746:
URL: https://github.com/apache/avro/pull/3746#issuecomment-4416667776
I don think that there is an oppertunity to use the same logic on a broader level, a soft cache of a String + parse options -> schema, but don't know the innards of the parsing expectations to know if a new Schema is guaranteed
handling duplicate schemas seems to have secondary costs, like duplicate data in `FastReaderBuilder`, which can be avoided
Anyway - its outside of the scope of this PR, and outside my knowledge, but maybe a manttainer can have an opinion
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@avro.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] AVRO-4249: [java] provide a cache of schema to avoid building [avro]
Posted by "mkeskells (via GitHub)" <gi...@apache.org>.
mkeskells commented on code in PR #3746:
URL: https://github.com/apache/avro/pull/3746#discussion_r3230406434
##########
lang/java/avro/src/main/java/org/apache/avro/file/DataFileStream.java:
##########
@@ -127,7 +151,7 @@ void initialize(InputStream in, byte[] magic) throws IOException {
if (l > 0) {
do {
for (long i = 0; i < l; i++) {
- String key = vin.readString(null).toString();
+ String key = vin.readString();
Review Comment:
not related to the schema cache, but removes the allocation of the ByteBuffer, which is disguarded immediately.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@avro.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] AVRO-4249: [java] provide a cache of schema to avoid building [avro]
Posted by "mkeskells (via GitHub)" <gi...@apache.org>.
mkeskells commented on PR #3746:
URL: https://github.com/apache/avro/pull/3746#issuecomment-4364800674
Benchmark results for PR change. Nothing unexpected
Significant recuction in allocation and CPU time with this change
```
Benchmark (cacheType) (numFields) (numRecords) Mode Cnt Score Error Units
SchemaCacheEffectTest.benchmarkDataReading none 5 1 thrpt 10 160708.485 ± 66123.198 ops/s
SchemaCacheEffectTest.benchmarkDataReading:gc.alloc.rate none 5 1 thrpt 10 3148.068 ± 1289.209 MB/sec
SchemaCacheEffectTest.benchmarkDataReading:gc.alloc.rate.norm none 5 1 thrpt 10 20558.844 ± 269.523 B/op
SchemaCacheEffectTest.benchmarkDataReading:gc.count none 5 1 thrpt 10 50.000 counts
SchemaCacheEffectTest.benchmarkDataReading:gc.time none 5 1 thrpt 10 6083.000 ms
SchemaCacheEffectTest.benchmarkDataReading none 5 10 thrpt 10 130146.190 ± 22400.139 ops/s
SchemaCacheEffectTest.benchmarkDataReading:gc.alloc.rate none 5 10 thrpt 10 3006.175 ± 514.267 MB/sec
SchemaCacheEffectTest.benchmarkDataReading:gc.alloc.rate.norm none 5 10 thrpt 10 24253.534 ± 51.819 B/op
SchemaCacheEffectTest.benchmarkDataReading:gc.count none 5 10 thrpt 10 21.000 counts
SchemaCacheEffectTest.benchmarkDataReading:gc.time none 5 10 thrpt 10 3193.000 ms
SchemaCacheEffectTest.benchmarkDataReading none 5 100 thrpt 10 68919.081 ± 7302.996 ops/s
SchemaCacheEffectTest.benchmarkDataReading:gc.alloc.rate none 5 100 thrpt 10 4151.588 ± 439.714 MB/sec
SchemaCacheEffectTest.benchmarkDataReading:gc.alloc.rate.norm none 5 100 thrpt 10 63250.291 ± 59.216 B/op
SchemaCacheEffectTest.benchmarkDataReading:gc.count none 5 100 thrpt 10 16.000 counts
SchemaCacheEffectTest.benchmarkDataReading:gc.time none 5 100 thrpt 10 2092.000 ms
SchemaCacheEffectTest.benchmarkDataReading none 50 1 thrpt 10 16111.848 ± 20899.702 ops/s
SchemaCacheEffectTest.benchmarkDataReading:gc.alloc.rate none 50 1 thrpt 10 1271.894 ± 1649.321 MB/sec
SchemaCacheEffectTest.benchmarkDataReading:gc.alloc.rate.norm none 50 1 thrpt 10 82818.178 ± 85.805 B/op
SchemaCacheEffectTest.benchmarkDataReading:gc.count none 50 1 thrpt 10 53.000 counts
SchemaCacheEffectTest.benchmarkDataReading:gc.time none 50 1 thrpt 10 12251.000 ms
SchemaCacheEffectTest.benchmarkDataReading none 50 10 thrpt 10 17872.197 ± 16429.837 ops/s
SchemaCacheEffectTest.benchmarkDataReading:gc.alloc.rate none 50 10 thrpt 10 2001.422 ± 1839.438 MB/sec
SchemaCacheEffectTest.benchmarkDataReading:gc.alloc.rate.norm none 50 10 thrpt 10 117502.133 ± 82.087 B/op
SchemaCacheEffectTest.benchmarkDataReading:gc.count none 50 10 thrpt 10 50.000 counts
SchemaCacheEffectTest.benchmarkDataReading:gc.time none 50 10 thrpt 10 9733.000 ms
SchemaCacheEffectTest.benchmarkDataReading none 50 100 thrpt 10 11193.744 ± 1047.678 ops/s
SchemaCacheEffectTest.benchmarkDataReading:gc.alloc.rate none 50 100 thrpt 10 5074.400 ± 471.679 MB/sec
SchemaCacheEffectTest.benchmarkDataReading:gc.alloc.rate.norm none 50 100 thrpt 10 475709.595 ± 94.557 B/op
SchemaCacheEffectTest.benchmarkDataReading:gc.count none 50 100 thrpt 10 21.000 counts
SchemaCacheEffectTest.benchmarkDataReading:gc.time none 50 100 thrpt 10 2190.000 ms
SchemaCacheEffectTest.benchmarkDataReading none 500 1 thrpt 10 1239.421 ± 2417.645 ops/s
SchemaCacheEffectTest.benchmarkDataReading:gc.alloc.rate none 500 1 thrpt 10 827.063 ± 1606.939 MB/sec
SchemaCacheEffectTest.benchmarkDataReading:gc.alloc.rate.norm none 500 1 thrpt 10 698100.419 ± 89.206 B/op
SchemaCacheEffectTest.benchmarkDataReading:gc.count none 500 1 thrpt 10 44.000 counts
SchemaCacheEffectTest.benchmarkDataReading:gc.time none 500 1 thrpt 10 14395.000 ms
SchemaCacheEffectTest.benchmarkDataReading none 500 10 thrpt 10 1809.982 ± 2127.079 ops/s
SchemaCacheEffectTest.benchmarkDataReading:gc.alloc.rate none 500 10 thrpt 10 1817.693 ± 2130.299 MB/sec
SchemaCacheEffectTest.benchmarkDataReading:gc.alloc.rate.norm none 500 10 thrpt 10 1052253.600 ± 84.368 B/op
SchemaCacheEffectTest.benchmarkDataReading:gc.count none 500 10 thrpt 10 54.000 counts
SchemaCacheEffectTest.benchmarkDataReading:gc.time none 500 10 thrpt 10 10312.000 ms
SchemaCacheEffectTest.benchmarkDataReading none 500 100 thrpt 10 1132.882 ± 120.509 ops/s
SchemaCacheEffectTest.benchmarkDataReading:gc.alloc.rate none 500 100 thrpt 10 4457.240 ± 462.910 MB/sec
SchemaCacheEffectTest.benchmarkDataReading:gc.alloc.rate.norm none 500 100 thrpt 10 4127075.031 ± 185.786 B/op
SchemaCacheEffectTest.benchmarkDataReading:gc.count none 500 100 thrpt 10 20.000 counts
SchemaCacheEffectTest.benchmarkDataReading:gc.time none 500 100 thrpt 10 2194.000 ms
SchemaCacheEffectTest.benchmarkDataReading weak 5 1 thrpt 10 857058.952 ± 41590.581 ops/s
SchemaCacheEffectTest.benchmarkDataReading:gc.alloc.rate weak 5 1 thrpt 10 9055.300 ± 440.460 MB/sec
SchemaCacheEffectTest.benchmarkDataReading:gc.alloc.rate.norm weak 5 1 thrpt 10 11088.012 ± 0.001 B/op
SchemaCacheEffectTest.benchmarkDataReading:gc.count weak 5 1 thrpt 10 106.000 counts
SchemaCacheEffectTest.benchmarkDataReading:gc.time weak 5 1 thrpt 10 164.000 ms
SchemaCacheEffectTest.benchmarkDataReading weak 5 10 thrpt 10 558125.787 ± 48281.646 ops/s
SchemaCacheEffectTest.benchmarkDataReading:gc.alloc.rate weak 5 10 thrpt 10 7804.731 ± 679.401 MB/sec
SchemaCacheEffectTest.benchmarkDataReading:gc.alloc.rate.norm weak 5 10 thrpt 10 14676.020 ± 44.620 B/op
SchemaCacheEffectTest.benchmarkDataReading:gc.count weak 5 10 thrpt 10 99.000 counts
SchemaCacheEffectTest.benchmarkDataReading:gc.time weak 5 10 thrpt 10 146.000 ms
SchemaCacheEffectTest.benchmarkDataReading weak 5 100 thrpt 10 122035.807 ± 19252.663 ops/s
SchemaCacheEffectTest.benchmarkDataReading:gc.alloc.rate weak 5 100 thrpt 10 6238.761 ± 984.634 MB/sec
SchemaCacheEffectTest.benchmarkDataReading:gc.alloc.rate.norm weak 5 100 thrpt 10 53656.085 ± 0.014 B/op
SchemaCacheEffectTest.benchmarkDataReading:gc.count weak 5 100 thrpt 10 89.000 counts
SchemaCacheEffectTest.benchmarkDataReading:gc.time weak 5 100 thrpt 10 132.000 ms
SchemaCacheEffectTest.benchmarkDataReading weak 50 1 thrpt 10 381117.522 ± 30016.980 ops/s
SchemaCacheEffectTest.benchmarkDataReading:gc.alloc.rate weak 50 1 thrpt 10 7444.513 ± 598.525 MB/sec
SchemaCacheEffectTest.benchmarkDataReading:gc.alloc.rate.norm weak 50 1 thrpt 10 20500.029 ± 44.617 B/op
SchemaCacheEffectTest.benchmarkDataReading:gc.count weak 50 1 thrpt 10 91.000 counts
SchemaCacheEffectTest.benchmarkDataReading:gc.time weak 50 1 thrpt 10 131.000 ms
SchemaCacheEffectTest.benchmarkDataReading weak 50 10 thrpt 10 120204.520 ± 12184.146 ops/s
SchemaCacheEffectTest.benchmarkDataReading:gc.alloc.rate weak 50 10 thrpt 10 6324.247 ± 640.856 MB/sec
SchemaCacheEffectTest.benchmarkDataReading:gc.alloc.rate.norm weak 50 10 thrpt 10 55232.086 ± 0.009 B/op
SchemaCacheEffectTest.benchmarkDataReading:gc.count weak 50 10 thrpt 10 82.000 counts
SchemaCacheEffectTest.benchmarkDataReading:gc.time weak 50 10 thrpt 10 122.000 ms
SchemaCacheEffectTest.benchmarkDataReading weak 50 100 thrpt 10 17391.522 ± 2058.825 ops/s
SchemaCacheEffectTest.benchmarkDataReading:gc.alloc.rate weak 50 100 thrpt 10 6848.725 ± 808.000 MB/sec
SchemaCacheEffectTest.benchmarkDataReading:gc.alloc.rate.norm weak 50 100 thrpt 10 413376.598 ± 0.067 B/op
SchemaCacheEffectTest.benchmarkDataReading:gc.count weak 50 100 thrpt 10 94.000 counts
SchemaCacheEffectTest.benchmarkDataReading:gc.time weak 50 100 thrpt 10 168.000 ms
SchemaCacheEffectTest.benchmarkDataReading weak 500 1 thrpt 10 69282.529 ± 4269.320 ops/s
SchemaCacheEffectTest.benchmarkDataReading:gc.alloc.rate weak 500 1 thrpt 10 7729.057 ± 476.657 MB/sec
SchemaCacheEffectTest.benchmarkDataReading:gc.alloc.rate.norm weak 500 1 thrpt 10 117080.148 ± 0.011 B/op
SchemaCacheEffectTest.benchmarkDataReading:gc.count weak 500 1 thrpt 10 91.000 counts
SchemaCacheEffectTest.benchmarkDataReading:gc.time weak 500 1 thrpt 10 152.000 ms
SchemaCacheEffectTest.benchmarkDataReading weak 500 10 thrpt 10 15909.052 ± 1503.023 ops/s
SchemaCacheEffectTest.benchmarkDataReading:gc.alloc.rate weak 500 10 thrpt 10 7142.377 ± 677.104 MB/sec
SchemaCacheEffectTest.benchmarkDataReading:gc.alloc.rate.norm weak 500 10 thrpt 10 471271.854 ± 40.894 B/op
SchemaCacheEffectTest.benchmarkDataReading:gc.count weak 500 10 thrpt 10 88.000 counts
SchemaCacheEffectTest.benchmarkDataReading:gc.time weak 500 10 thrpt 10 155.000 ms
SchemaCacheEffectTest.benchmarkDataReading weak 500 100 thrpt 10 1843.039 ± 127.651 ops/s
SchemaCacheEffectTest.benchmarkDataReading:gc.alloc.rate weak 500 100 thrpt 10 6227.871 ± 427.151 MB/sec
SchemaCacheEffectTest.benchmarkDataReading:gc.alloc.rate.norm weak 500 100 thrpt 10 3548077.588 ± 0.376 B/op
SchemaCacheEffectTest.benchmarkDataReading:gc.count weak 500 100 thrpt 10 77.000 counts
SchemaCacheEffectTest.benchmarkDataReading:gc.time weak 500 100 thrpt 10 124.000 ms
Benchmark (cacheType) (numFields) Mode Cnt Score Error Units
SchemaCacheTest.benchmarkSchemaParsing NO_CACHE 5 thrpt 10 404585.474 ± 9206.027 ops/s
SchemaCacheTest.benchmarkSchemaParsing:gc.alloc.rate NO_CACHE 5 thrpt 10 3486.962 ± 79.322 MB/sec
SchemaCacheTest.benchmarkSchemaParsing:gc.alloc.rate.norm NO_CACHE 5 thrpt 10 9040.017 ± 0.001 B/op
SchemaCacheTest.benchmarkSchemaParsing:gc.count NO_CACHE 5 thrpt 10 85.000 counts
SchemaCacheTest.benchmarkSchemaParsing:gc.time NO_CACHE 5 thrpt 10 85.000 ms
SchemaCacheTest.benchmarkSchemaParsing NO_CACHE 50 thrpt 10 53144.889 ± 1996.919 ops/s
SchemaCacheTest.benchmarkSchemaParsing:gc.alloc.rate NO_CACHE 50 thrpt 10 3062.896 ± 114.963 MB/sec
SchemaCacheTest.benchmarkSchemaParsing:gc.alloc.rate.norm NO_CACHE 50 thrpt 10 60448.130 ± 0.005 B/op
SchemaCacheTest.benchmarkSchemaParsing:gc.count NO_CACHE 50 thrpt 10 76.000 counts
SchemaCacheTest.benchmarkSchemaParsing:gc.time NO_CACHE 50 thrpt 10 79.000 ms
SchemaCacheTest.benchmarkSchemaParsing NO_CACHE 500 thrpt 10 5854.415 ± 404.753 ops/s
SchemaCacheTest.benchmarkSchemaParsing:gc.alloc.rate NO_CACHE 500 thrpt 10 3150.857 ± 217.770 MB/sec
SchemaCacheTest.benchmarkSchemaParsing:gc.alloc.rate.norm NO_CACHE 500 thrpt 10 564505.190 ± 25.482 B/op
SchemaCacheTest.benchmarkSchemaParsing:gc.count NO_CACHE 500 thrpt 10 78.000 counts
SchemaCacheTest.benchmarkSchemaParsing:gc.time NO_CACHE 500 thrpt 10 76.000 ms
SchemaCacheTest.benchmarkSchemaParsing CONCURRENT 5 thrpt 10 213069364.030 ± 2674481.309 ops/s
SchemaCacheTest.benchmarkSchemaParsing:gc.alloc.rate CONCURRENT 5 thrpt 10 3250.295 ± 40.752 MB/sec
SchemaCacheTest.benchmarkSchemaParsing:gc.alloc.rate.norm CONCURRENT 5 thrpt 10 16.000 ± 0.001 B/op
SchemaCacheTest.benchmarkSchemaParsing:gc.count CONCURRENT 5 thrpt 10 81.000 counts
SchemaCacheTest.benchmarkSchemaParsing:gc.time CONCURRENT 5 thrpt 10 75.000 ms
SchemaCacheTest.benchmarkSchemaParsing CONCURRENT 50 thrpt 10 212597784.922 ± 3598523.893 ops/s
SchemaCacheTest.benchmarkSchemaParsing:gc.alloc.rate CONCURRENT 50 thrpt 10 3243.227 ± 54.879 MB/sec
SchemaCacheTest.benchmarkSchemaParsing:gc.alloc.rate.norm CONCURRENT 50 thrpt 10 16.000 ± 0.001 B/op
SchemaCacheTest.benchmarkSchemaParsing:gc.count CONCURRENT 50 thrpt 10 80.000 counts
SchemaCacheTest.benchmarkSchemaParsing:gc.time CONCURRENT 50 thrpt 10 72.000 ms
SchemaCacheTest.benchmarkSchemaParsing CONCURRENT 500 thrpt 10 210281467.875 ± 4954734.313 ops/s
SchemaCacheTest.benchmarkSchemaParsing:gc.alloc.rate CONCURRENT 500 thrpt 10 3207.787 ± 75.475 MB/sec
SchemaCacheTest.benchmarkSchemaParsing:gc.alloc.rate.norm CONCURRENT 500 thrpt 10 16.000 ± 0.001 B/op
SchemaCacheTest.benchmarkSchemaParsing:gc.count CONCURRENT 500 thrpt 10 86.000 counts
SchemaCacheTest.benchmarkSchemaParsing:gc.time CONCURRENT 500 thrpt 10 76.000 ms
SchemaCacheTest.benchmarkSchemaParsing WEAK 5 thrpt 10 38493704.277 ± 1283442.559 ops/s
SchemaCacheTest.benchmarkSchemaParsing:gc.alloc.rate WEAK 5 thrpt 10 880.873 ± 29.379 MB/sec
SchemaCacheTest.benchmarkSchemaParsing:gc.alloc.rate.norm WEAK 5 thrpt 10 24.001 ± 0.001 B/op
SchemaCacheTest.benchmarkSchemaParsing:gc.count WEAK 5 thrpt 10 39.000 counts
SchemaCacheTest.benchmarkSchemaParsing:gc.time WEAK 5 thrpt 10 25.000 ms
SchemaCacheTest.benchmarkSchemaParsing WEAK 50 thrpt 10 39070048.693 ± 426068.207 ops/s
SchemaCacheTest.benchmarkSchemaParsing:gc.alloc.rate WEAK 50 thrpt 10 894.238 ± 9.681 MB/sec
SchemaCacheTest.benchmarkSchemaParsing:gc.alloc.rate.norm WEAK 50 thrpt 10 24.007 ± 0.001 B/op
SchemaCacheTest.benchmarkSchemaParsing:gc.count WEAK 50 thrpt 10 40.000 counts
SchemaCacheTest.benchmarkSchemaParsing:gc.time WEAK 50 thrpt 10 28.000 ms
SchemaCacheTest.benchmarkSchemaParsing WEAK 500 thrpt 10 37408903.367 ± 3244572.982 ops/s
SchemaCacheTest.benchmarkSchemaParsing:gc.alloc.rate WEAK 500 thrpt 10 858.073 ± 74.413 MB/sec
SchemaCacheTest.benchmarkSchemaParsing:gc.alloc.rate.norm WEAK 500 thrpt 10 24.058 ± 0.009 B/op
SchemaCacheTest.benchmarkSchemaParsing:gc.count WEAK 500 thrpt 10 38.000 counts
SchemaCacheTest.benchmarkSchemaParsing:gc.time WEAK 500 thrpt 10 30.000 ms
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@avro.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] AVRO-4249: [java] provide a cache of schema to avoid building [avro]
Posted by "mkeskells (via GitHub)" <gi...@apache.org>.
mkeskells commented on PR #3746:
URL: https://github.com/apache/avro/pull/3746#issuecomment-4366433567
fixed the licence issues reported in the build
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@avro.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] AVRO-4249: [java] provide a cache of schema to avoid building [avro]
Posted by "mkeskells (via GitHub)" <gi...@apache.org>.
mkeskells commented on PR #3746:
URL: https://github.com/apache/avro/pull/3746#issuecomment-4395202004
> Hey, pardon me! I appreciate the work, and I'll take a closer look soon -- we're going to do a 1.13.0 release just after the next one 1.12.2, and this should be in it!
Hi @RyanSkraba I have fixed the checkstyle error I hope (unfortunately checkstyle doesnt run on this project fo me so I couldn't check before)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@avro.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] AVRO-4249: [java] provide a cache of schema to avoid building [avro]
Posted by "mkeskells (via GitHub)" <gi...@apache.org>.
mkeskells commented on code in PR #3746:
URL: https://github.com/apache/avro/pull/3746#discussion_r3230406434
##########
lang/java/avro/src/main/java/org/apache/avro/file/DataFileStream.java:
##########
@@ -127,7 +151,7 @@ void initialize(InputStream in, byte[] magic) throws IOException {
if (l > 0) {
do {
for (long i = 0; i < l; i++) {
- String key = vin.readString(null).toString();
+ String key = vin.readString();
Review Comment:
not related to the schema cache, but removes the allocation of the ByteBuffer, which is disguarded immediately. This was done after the benchmark results were published and will reduce allocations by the size of the schema for each file parsed, for all scenarios
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@avro.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] AVRO-4249: [java] provide a cache of schema to avoid building [avro]
Posted by "mkeskells (via GitHub)" <gi...@apache.org>.
mkeskells commented on code in PR #3746:
URL: https://github.com/apache/avro/pull/3746#discussion_r3230409728
##########
lang/java/avro/src/main/java/org/apache/avro/file/DataFileStream.java:
##########
@@ -127,7 +151,7 @@ void initialize(InputStream in, byte[] magic) throws IOException {
if (l > 0) {
do {
for (long i = 0; i < l; i++) {
- String key = vin.readString(null).toString();
+ String key = vin.readString();
ByteBuffer value = vin.readBytes(null);
Review Comment:
We could reuse the byte buffer, but an impovement in https://github.com/apache/avro/pull/3776 removes it anyway
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@avro.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] AVRO-4249: [java] provide a cache of schema to avoid building [avro]
Posted by "mkeskells (via GitHub)" <gi...@apache.org>.
mkeskells commented on PR #3746:
URL: https://github.com/apache/avro/pull/3746#issuecomment-4367351138
I finally got o the bottom of the memmory leak I was chasing when I observed the problem that this fixes- its https://issues.apache.org/jira/browse/AVRO-4253, a memory leak, which in my environment was holding only 200Gb of Schemas due to the leak. Mostly fixed by this PR
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@avro.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] AVRO-4249: [java] provide a cache of schema to avoid building [avro]
Posted by "mkeskells (via GitHub)" <gi...@apache.org>.
mkeskells commented on PR #3746:
URL: https://github.com/apache/avro/pull/3746#issuecomment-6171744865
@RyanSkraba could you review/advise on next steps
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@avro.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] AVRO-4249: [java] provide a cache of schema to avoid building [avro]
Posted by "mkeskells (via GitHub)" <gi...@apache.org>.
mkeskells commented on PR #3746:
URL: https://github.com/apache/avro/pull/3746#issuecomment-4410114645
@RyanSkraba I have added a soft cache as well.
QQ - what should the default cache behaviour be?
I can't see much downside in having a weak cache as the default, but it is a change in behaviour
Would it be a good idea to set it to weak now, or wait to 1.13
I don't know the avro policy on changes like this so looking for steer
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@avro.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] AVRO-4249: [java] provide a cache of schema to avoid building [avro]
Posted by "mkeskells (via GitHub)" <gi...@apache.org>.
mkeskells commented on PR #3746:
URL: https://github.com/apache/avro/pull/3746#issuecomment-4435636187
I did notice that the bytes for the schema gets allocated 3 times
1. in a `ByteBuffer`
2. in a `byte[]`, copies to the header
3. in a `String` to parse (or cache)
https://github.com/apache/avro/pull/3776 removes one of these, but it seems a waste to have the schema stored twice in the header, both as a schema and a byte[]. Are there any use cases where the schema is read post construction of the stream?, and if so could we just store the schema and regenerate the byte[] if needed. This would reduce the allocation and the retained size in an application
WDYT?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@avro.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org