[I] What's the best way to measure the size of arrow.Record in Golang? [arrow]

["Yifeng-Sigma (via GitHub)" <gi...@apache.org>]

Yifeng-Sigma opened a new issue, #38836:
URL: https://github.com/apache/arrow/issues/38836

   ### Describe the usage question you have. Please include as many useful details as  possible.
   
   
   I want to do some record splitting/merging based on the size, but didn't find a reliable way to estimate the size. 
   There are two ways:
   
   ```
   	for _, col := range rec.Columns() {
   		for _, buffer := range col.Data().Buffers() {
   			if buffer != nil {
   				bytes += uint64(buffer.Cap())
   			}
   		}
   	
   ```}
   
   or 
   
   ```
   func computeColumnSize(col arrow.Array) uint64 {
   	switch colType := col.DataType().(type) {
   	case arrow.BinaryDataType:
   		switch arr := col.(type) {
   		case *array.String:
   			return uint64(arr.ValueOffset(arr.Len()) - arr.ValueOffset(0))
                         // ...
   		}
   	}
   }
   ```
   
   I'm wondering what's the recommended way to compute the size of `arrow.Record`.
   
   ### Component(s)
   
   Go


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@arrow.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] What's the best way to measure the size of arrow.Record in Golang? [arrow]

["zeroshade (via GitHub)" <gi...@apache.org>]

zeroshade commented on issue #38836:
URL: https://github.com/apache/arrow/issues/38836#issuecomment-1821730163

   My recommendation would be a variation on the first one:
   
   ```go
   func calcSize(arr arrow.ArrayData) (sz uint64) {
       if arr == nil {
           return
       }
   
       for _, b := range arr.Buffers() {
           sz += uint64(b.Len())
       }
       for _, c := range arr.Children() {
           sz += calcSize(c)
       }
       sz += calcSize(arr.Dictionary())
       return
   }
   ```
   
   That would be my recommendation, it might be reasonable to add this as a utility into the arrow library directly via a PR


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] What's the best way to measure the size of arrow.Record in Golang? [arrow]

["Yifeng-Sigma (via GitHub)" <gi...@apache.org>]

Yifeng-Sigma commented on issue #38836:
URL: https://github.com/apache/arrow/issues/38836#issuecomment-1822081896

   > My recommendation would be a variation on the first one:
   > 
   > ```go
   > func calcSize(arr arrow.ArrayData) (sz uint64) {
   >     if arr == nil {
   >         return
   >     }
   > 
   >     for _, b := range arr.Buffers() {
   >         sz += uint64(b.Len())
   >     }
   >     for _, c := range arr.Children() {
   >         sz += calcSize(c)
   >     }
   >     sz += calcSize(arr.Dictionary())
   >     return
   > }
   > ```
   > 
   > That would be my recommendation, it might be reasonable to add this as a utility into the arrow library directly via a PR
   
   Thanks, opened https://github.com/apache/arrow/pull/38839


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] What's the best way to measure the size of arrow.Record in Golang? [arrow]

["zeroshade (via GitHub)" <gi...@apache.org>]

zeroshade closed issue #38836: What's the best way to measure the size of arrow.Record in Golang?
URL: https://github.com/apache/arrow/issues/38836


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org