[jira] [Comment Edited] (AVRO-3451) fix poor Avro write performance

["Kevin (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/AVRO-3451?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17507937#comment-17507937 ] 

Kevin edited comment on AVRO-3451 at 3/17/22, 1:42 AM:
-------------------------------------------------------

Ok, so building against that PR our application still works. However, performance is even worse than without PR 1602.

I will look and see if I can figure out why encode performance still so poor on that branch. It looks like some of what we discovered has been done – do not clone Schema when caching for namespace lookup – but for some reason still very bad write performance.


was (Author: JIRAUSER285274):
Ok, so building against that PR our application still works. However, performance is even worse than without PR 1602.

I will look and see if I can figure out why encode performance still so poor on that branch. It looks like some of what we discovered has been done – do not clone Schema when caching for namespace lookup – but for some reason does not help.

> fix poor Avro write performance
> -------------------------------
>
>                 Key: AVRO-3451
>                 URL: https://issues.apache.org/jira/browse/AVRO-3451
>             Project: Apache Avro
>          Issue Type: Improvement
>          Components: rust
>    Affects Versions: 1.11.0
>         Environment: Mac OS X Big Sur
> {code:java}
> installed toolchains
> --------------------
> stable-x86_64-apple-darwin (default)
> nightly-x86_64-apple-darwin
> active toolchain
> ----------------
> stable-x86_64-apple-darwin (default)
> rustc 1.56.1 (59eed8a2a 2021-11-01) {code}
>            Reporter: Kevin
>            Priority: Major
>         Attachments: Screen Shot 2022-03-14 at 7.30.24 PM.png
>
>   Original Estimate: 1h
>  Remaining Estimate: 1h
>
> Rust implementation of Apache Avro library – apache-avro (née avro-rs) – demonstrates poor write performance when serializing Rust structures to Avro. Profiling indicates that this implementation spends an inordinate amount of time in the function {{encode::encode_ref}} performing {{clone()}} and {{drop}} operations related to a HashMap<String, Schema> type.
> We modified the function {{encode_ref0}} as follows:
> {code:java}
> -pub fn encode_ref(value: &Value, schema: &Schema, buffer: &mut Vec<u8>) {
> -    fn encode_ref0(
> +pub fn encode_ref<'a>(value: &Value, schema: &'a Schema, buffer: &mut Vec<u8>) {
> +    fn encode_ref0<'a>(
>          value: &Value,
> -        schema: &Schema,
> +        schema: &'a Schema,
>          buffer: &mut Vec<u8>,
> -        schemas_by_name: &mut HashMap<String, Schema>,
> +        schemas_by_name: &mut HashMap<&'a str, &'a Schema>,
>      ) {
>          match &schema {
>              Schema::Ref { ref name } => {
> -                let resolved = schemas_by_name.get(name.name.as_str()).unwrap();
> +                let resolved = schemas_by_name.get(&name.name as &str).unwrap();
>                  return encode_ref0(value, resolved, buffer, &mut schemas_by_name.clone());
>              }
>              Schema::Record { ref name, .. }
>              | Schema::Enum { ref name, .. }
>              | Schema::Fixed { ref name, .. } => {
> -                schemas_by_name.insert(name.name.clone(), schema.clone());
> +                schemas_by_name.insert(&name.name, &schema);
>              }
>              _ => (),
>          }{code}
> to remove any need for Clone in the {{schemas_by_name}} cache and see a notable improvement (factor of 4 to 5) in our application with this change.
> After this change, all Cargo Tests still pass and Benchmarks display a very significant improvement in Write performance across the board. Attached below is one example benchmark for {{big schema, write 10k records}} with Before on the Left and After on the Right.



--
This message was sent by Atlassian Jira
(v8.20.1#820001)