[GitHub] [arrow-rs] rockyzhengwu opened a new issue, #3150: Optimize memory usage of json reader

[GitBox <gi...@apache.org>]

rockyzhengwu opened a new issue, #3150:
URL: https://github.com/apache/arrow-rs/issues/3150

   **Is your feature request related to a problem or challenge? Please describe what you are trying to do.**
   
   The implementation of decode json to arrow array need convert batch_size of  json str to serde_json Value . 
   this  equires a lot of memory for serde_json Value. if with a big batch_size will OOM , usually a large batch_size will have a good compression rate.
   
   
    https://github.com/apache/arrow-rs/blob/e1b5657eb1206ce67eb079f6e72615982a70480a/arrow-json/src/reader.rs#L685 
   
   **Describe the solution you'd like**
   current implementation in pseudocode: 
   ``` rust
   for batch in value_iter{
      let mut rows: Vec<Value> = Vec::with_capacity(batch_size);
      let arrays = convert_function(rows)
   }
   ```
   If convert ony one json str to serde_json Value will save 3x-5x memory or more, i didn't record carefully .  
   I had implement a version in our online product in this way , because we use a large batch_size . the pseudocde is 
   ``` rust
   let  field_builder:  Vec<Box<dyn ArrayBuilder>> = create_array_builder(batch_size);
   for (i, row) in value_iter.enumerate(){
       let value = serde_json::from_str(row);
        for (index, field) in shema.field.fields{
            let col_name = field.name();
            field_builder[i].append(value.get(col_name))
        }
      if i == batch_size{
         let array_refs = builder.iter_mut().map(|builder| builder.finish()).collect();
        .....
      }
   }
   
   ```
   this implementation didn't effect the performance. 
    But it didn't support deep nested list and  map. 
    I'm not sure this is a elegant way for this. or it's possiable to support deep nested list and map. 
   if this is a good idea ,  I can try to make PR for this . 
   
   **Describe alternatives you've considered**
   <!--
   A clear and concise description of any alternative solutions or features you've considered.
   -->
   
   **Additional context**
   <!--
   Add any other context or screenshots about the feature request here.
   -->
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [arrow-rs] rockyzhengwu closed issue #3150: Optimize memory usage of json reader

[GitBox <gi...@apache.org>]

rockyzhengwu closed issue #3150: Optimize memory usage of json reader
URL: https://github.com/apache/arrow-rs/issues/3150


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org