You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@avro.apache.org by "Hernan Otero (JIRA)" <ji...@apache.org> on 2012/09/18 20:54:07 UTC
[jira] [Created] (AVRO-1169) Reduce memory footprint of
ResolvingGrammarGenerator
Hernan Otero created AVRO-1169:
----------------------------------
Summary: Reduce memory footprint of ResolvingGrammarGenerator
Key: AVRO-1169
URL: https://issues.apache.org/jira/browse/AVRO-1169
Project: Avro
Issue Type: Improvement
Components: java
Affects Versions: 1.7.0
Reporter: Hernan Otero
We noticed an OutOfMemory error in one of our client applications (GUI) when trying to decode messages that require resolving.
It turns out the use of Schema.toString(true) in method generate(Schema, Schema, Map<LitS,Symbol>) (and to a less extent resolveRecords(Schema, Schema, Map<LitS,Symbol>) too) in our very-large-Schema use-case is the main reason for this.
We found that patching this class to use Schema.getFullName() instead of Schema.toString(true) resolves our issue, and we thought this might benefit other users (and shouldn't affect anybody as this message is not actually being logged/reported).
The change involves changing this (in method generate()):
{code}
return Symbol.error("Found " + writer.toString(true)
+ ", expecting " + reader.toString(true));
{code}
into this:
{code}
return Symbol.error("Found " + writer.getFullName()
+ ", expecting " + reader.getFullName());
{code}
And this (in method resolveRecords()):
{code}
result = Symbol.error("Found " + writer.toString(true)
+ ", expecting " + reader.toString(true));
{code}
into this:
{code}
result = Symbol.error("Found " + writer.getFullName()
+ ", expecting " + reader.getFullName());
{code}
Thanks in advance,
Hernan
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (AVRO-1169) Reduce memory footprint of
ResolvingGrammarGenerator
Posted by "Doug Cutting (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/AVRO-1169?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Doug Cutting updated AVRO-1169:
-------------------------------
Resolution: Fixed
Fix Version/s: 1.7.3
Assignee: Hernan Otero
Status: Resolved (was: Patch Available)
I just committed this. Thanks, Hernan!
> Reduce memory footprint of ResolvingGrammarGenerator
> ----------------------------------------------------
>
> Key: AVRO-1169
> URL: https://issues.apache.org/jira/browse/AVRO-1169
> Project: Avro
> Issue Type: Improvement
> Components: java
> Affects Versions: 1.7.0
> Reporter: Hernan Otero
> Assignee: Hernan Otero
> Fix For: 1.7.3
>
> Attachments: AVRO-1169.patch
>
>
> We noticed an OutOfMemory error in one of our client applications (GUI) when trying to decode messages that require resolving.
> It turns out the use of Schema.toString(true) in method generate(Schema, Schema, Map<LitS,Symbol>) (and to a less extent resolveRecords(Schema, Schema, Map<LitS,Symbol>) too) in our very-large-Schema use-case is the main reason for this.
> We found that patching this class to use Schema.getFullName() instead of Schema.toString(true) resolves our issue, and we thought this might benefit other users (and shouldn't affect anybody as this message is not actually being logged/reported).
> The change involves changing this (in method generate()):
> {code}
> return Symbol.error("Found " + writer.toString(true)
> + ", expecting " + reader.toString(true));
> {code}
> into this:
> {code}
> return Symbol.error("Found " + writer.getFullName()
> + ", expecting " + reader.getFullName());
> {code}
> And this (in method resolveRecords()):
> {code}
> result = Symbol.error("Found " + writer.toString(true)
> + ", expecting " + reader.toString(true));
> {code}
> into this:
> {code}
> result = Symbol.error("Found " + writer.getFullName()
> + ", expecting " + reader.getFullName());
> {code}
> Thanks in advance,
> Hernan
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (AVRO-1169) Reduce memory footprint of
ResolvingGrammarGenerator
Posted by "Doug Cutting (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/AVRO-1169?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Doug Cutting updated AVRO-1169:
-------------------------------
Attachment: AVRO-1169.patch
Here is the proposed change as a patch file.
> Reduce memory footprint of ResolvingGrammarGenerator
> ----------------------------------------------------
>
> Key: AVRO-1169
> URL: https://issues.apache.org/jira/browse/AVRO-1169
> Project: Avro
> Issue Type: Improvement
> Components: java
> Affects Versions: 1.7.0
> Reporter: Hernan Otero
> Attachments: AVRO-1169.patch
>
>
> We noticed an OutOfMemory error in one of our client applications (GUI) when trying to decode messages that require resolving.
> It turns out the use of Schema.toString(true) in method generate(Schema, Schema, Map<LitS,Symbol>) (and to a less extent resolveRecords(Schema, Schema, Map<LitS,Symbol>) too) in our very-large-Schema use-case is the main reason for this.
> We found that patching this class to use Schema.getFullName() instead of Schema.toString(true) resolves our issue, and we thought this might benefit other users (and shouldn't affect anybody as this message is not actually being logged/reported).
> The change involves changing this (in method generate()):
> {code}
> return Symbol.error("Found " + writer.toString(true)
> + ", expecting " + reader.toString(true));
> {code}
> into this:
> {code}
> return Symbol.error("Found " + writer.getFullName()
> + ", expecting " + reader.getFullName());
> {code}
> And this (in method resolveRecords()):
> {code}
> result = Symbol.error("Found " + writer.toString(true)
> + ", expecting " + reader.toString(true));
> {code}
> into this:
> {code}
> result = Symbol.error("Found " + writer.getFullName()
> + ", expecting " + reader.getFullName());
> {code}
> Thanks in advance,
> Hernan
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (AVRO-1169) Reduce memory footprint of
ResolvingGrammarGenerator
Posted by "Doug Cutting (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/AVRO-1169?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Doug Cutting updated AVRO-1169:
-------------------------------
Status: Patch Available (was: Open)
This seems like a reasonable change to me.
If there are no objections I'll commit it.
> Reduce memory footprint of ResolvingGrammarGenerator
> ----------------------------------------------------
>
> Key: AVRO-1169
> URL: https://issues.apache.org/jira/browse/AVRO-1169
> Project: Avro
> Issue Type: Improvement
> Components: java
> Affects Versions: 1.7.0
> Reporter: Hernan Otero
> Attachments: AVRO-1169.patch
>
>
> We noticed an OutOfMemory error in one of our client applications (GUI) when trying to decode messages that require resolving.
> It turns out the use of Schema.toString(true) in method generate(Schema, Schema, Map<LitS,Symbol>) (and to a less extent resolveRecords(Schema, Schema, Map<LitS,Symbol>) too) in our very-large-Schema use-case is the main reason for this.
> We found that patching this class to use Schema.getFullName() instead of Schema.toString(true) resolves our issue, and we thought this might benefit other users (and shouldn't affect anybody as this message is not actually being logged/reported).
> The change involves changing this (in method generate()):
> {code}
> return Symbol.error("Found " + writer.toString(true)
> + ", expecting " + reader.toString(true));
> {code}
> into this:
> {code}
> return Symbol.error("Found " + writer.getFullName()
> + ", expecting " + reader.getFullName());
> {code}
> And this (in method resolveRecords()):
> {code}
> result = Symbol.error("Found " + writer.toString(true)
> + ", expecting " + reader.toString(true));
> {code}
> into this:
> {code}
> result = Symbol.error("Found " + writer.getFullName()
> + ", expecting " + reader.getFullName());
> {code}
> Thanks in advance,
> Hernan
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira