[jira] [Comment Edited] (ARROW-855) Arrow Memory Leak

["xufusheng (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/ARROW-855?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15975929#comment-15975929 ] 

xufusheng edited comment on ARROW-855 at 4/20/17 2:28 AM:
----------------------------------------------------------

Our programming language is scala.
1.Through HashMap to store pillar
private val entries = new util.LinkedHashMap[PillarId, Pillar](32, 0.75f, true)
2.Pillar encapsulates the data structure of arrow
entries.put(pillarId, newPillar(getFieldDesc(pillarId.tableId, pillarId.colIdx)))
 def newPillar(fieldDesc: FieldDescription): Pillar = {
    ArrowPillar.getPillar(fieldDesc, root)
  }
object ArrowPillar {
  @NeedTest
  def getPillar(field: FieldDescription, allocator: BufferAllocator): Pillar = {
    field.desc.getType match {
      case MemoType.Boolean => new ArrowNullableBoolPillar(field, allocator)
      case MemoType.Short => new ArrowNullableShortPillar(field, allocator)
      ......
         }
  }
}
trait Pillar{
   def append(value:Option[Any]):Unit
   ......
}
3.The following is our free memory function
def remove(pillarId: PillarId): Boolean = {
    entries.synchronized {
      var pillar = entries.remove(pillarId)
      if (pillar != null) {
        pillar.close()
        pillar = null
        true
      } else {
        false
      }
    }
  }


was (Author: fushengxu):
Our programming language is scala.
1.Through HashMap to store pillar
private val entries = new util.LinkedHashMap[PillarId, Pillar](32, 0.75f, true)
2.Pillar encapsulates the data structure of arrow
entries.put(pillarId, newPillar(getFieldDesc(pillarId.tableId, pillarId.colIdx)))
 def newPillar(fieldDesc: FieldDescription): Pillar = {
    ArrowPillar.getPillar(fieldDesc, root)
  }
object ArrowPillar {
  @NeedTest
  def getPillar(field: FieldDescription, allocator: BufferAllocator): Pillar = {
    field.desc.getType match {
      case MemoType.Boolean => new ArrowNullableBoolPillar(field, allocator)
      case MemoType.Short => new ArrowNullableShortPillar(field, allocator)
      ......
         }
  }
trait Pillar{
   def append(value:Option[Any]):Unit
   ......
}
3.The following is our release function
def remove(pillarId: PillarId): Boolean = {
    entries.synchronized {
      var pillar = entries.remove(pillarId)
      if (pillar != null) {
        pillar.close()
        pillar = null
        true
      } else {
        false
      }
    }
  }

> Arrow Memory Leak
> -----------------
>
>                 Key: ARROW-855
>                 URL: https://issues.apache.org/jira/browse/ARROW-855
>             Project: Apache Arrow
>          Issue Type: Bug
>          Components: Java - Memory, Java - Vectors
>    Affects Versions: 0.1.0
>         Environment: CentOS release 6.7+Indellij IDEA
>            Reporter: xufusheng
>            Priority: Critical
>              Labels: test
>
> we create a memory table by arrow and the source data come from HBase.
> Create a memory table and then drop the table,there will be a memory leak.
> Hundreds of times,There will be OutOfMemoryError.
> anyone encounter similar problems?



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)