You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@gluten.apache.org by "FelixYBW (via GitHub)" <gi...@apache.org> on 2024/07/19 23:12:18 UTC
[I] [VL] BlockAllocationListener isn't thread safe [incubator-gluten]
FelixYBW opened a new issue, #6525:
URL: https://github.com/apache/incubator-gluten/issues/6525
### Backend
VL (Velox)
### Bug description
arrowPool_ and blockListener_ is created by VeloxMemoryManager which is part of VeloxRunTime.
But both may be called by multiple threads, like below example:
It's called by two threads:
```
139843515967232
libvelox.so(_ZN6gluten23BlockAllocationListener17allocationChangedEl+0x15a) [0x7f304afcecaa]
libgluten.so(_ZN6gluten25ListenableMemoryAllocator11updateUsageEl+0x1a) [0x7f3054aec16a]
libgluten.so(_ZN6gluten25ListenableMemoryAllocator4freeEPvl+0x29) [0x7f3054aec1b9]
libgluten.so(_ZN5arrow10PoolBufferD0Ev+0x4b) [0x7f3055057eeb]
libvelox.so(+0x14c49d8) [0x7f304af989d8]
libvelox.so(+0x155bf00) [0x7f304b02ff00]
libvelox.so(_ZNSt23_Sp_counted_ptr_inplaceIN8facebook5velox10FlatVectorINS1_10StringViewEEESaIS4_ELN9__gnu_cxx12_Lock_policyE2EE10_M_disposeEv+0x1d3) [0x7f304b044513]
libvelox.so(_ZNSt23_Sp_counted_ptr_inplaceIN8facebook5velox16DictionaryVectorINS1_10StringViewEEESaIS4_ELN9__gnu_cxx12_Lock_policyE2EE10_M_disposeEv+0x1ca) [0x7f304d57298a]
libvelox.so(_ZNSt23_Sp_counted_ptr_inplaceIN8facebook5velox16DictionaryVectorINS1_10StringViewEEESaIS4_ELN9__gnu_cxx12_Lock_policyE2EE10_M_disposeEv+0x1ca) [0x7f304d57298a]
libvelox.so(+0x14c49d8) [0x7f304af989d8]
libvelox.so(_ZN8facebook5velox10BaseVector14ensureWritableERKNS0_17SelectivityVectorERKSt10shared_ptrIKNS0_4TypeEEPNS0_6memory10MemoryPoolERS5_IS1_EPNS0_10VectorPoolE+0x103) [0x7f304e0b49c3]
libvelox.so(_ZN6gluten18VeloxColumnarBatch15ensureFlattenedEv+0xe2) [0x7f304afc5d42]
libvelox.so(_ZN6gluten18VeloxColumnarBatch21getFlattenedRowVectorEv+0x19) [0x7f304afc5e29]
libvelox.so(_ZN6gluten22VeloxParquetDatasource5writeERKSt10shared_ptrINS_13ColumnarBatchEE+0x86) [0x7f304b02d196]
libgluten.so(Java_org_apache_gluten_datasource_DatasourceJniWrapper_write+0x1b8) [0x7f3054ae2d98][0x7f30a87bf427]
```
```
139845779318528
libvelox.so(_ZN6gluten23BlockAllocationListener17allocationChangedEl+0x15a) [0x7f304afcecaa]
libgluten.so(_ZN6gluten25ListenableMemoryAllocator11updateUsageEl+0x1a) [0x7f3054aec16a]
libgluten.so(_ZN6gluten25ListenableMemoryAllocator15allocateAlignedEmlPPv+0x22) [0x7f3054aec542]
libgluten.so(_ZN6gluten15ArrowMemoryPool8AllocateEllPPh+0x3f) [0x7f3054aecf9f]
libgluten.so(_ZN5arrow14AllocateBufferEllPNS_10MemoryPoolE+0x32e) [0x7f30550572be]
libgluten.so(_ZN5arrow14AllocateBufferElPNS_10MemoryPoolE+0x2a) [0x7f305505758a]
libgluten.so(+0x988bea) [0x7f3054b0bbea]
libgluten.so(_ZN6gluten12BlockPayload11deserializeEPN5arrow2io11InputStreamERKSt10shared_ptrINS1_6SchemaEERKS5_INS1_4util5CodecEEPNS1_10MemoryPoolERjRl+0x2f5) [0x7f3054b0d125]
libvelox.so(_ZN6gluten30VeloxColumnarBatchDeserializer4nextEv+0xde) [0x7f304b0358de]
libgluten.so(Java_org_apache_gluten_vectorized_ColumnarBatchOutIterator_nativeNext+0x259) [0x7f3054ae2889][0x7f30a9656768]
```
### Spark version
Spark-3.2.x
### Spark configurations
_No response_
### System information
Velox System Info v0.0.2
Commit: 34dbec25d204fcb302893429350d37081feb5edf
CMake Version: 3.29.4
System: Linux-5.4.0-1063-aws
Arch: x86_64
CPU Name: Model name: Intel(R) Xeon(R) Platinum 8488C
C++ Compiler: /usr/bin/c++
C++ Compiler Version: 9.4.0
C Compiler: /usr/bin/cc
C Compiler Version: 9.4.0
CMake Prefix Path: /usr/local;/usr;/;/usr;/usr/local;/usr/X11R6;/usr/pkg;/opt
### Relevant logs
_No response_
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [I] [VL] BlockAllocationListener isn't thread safe [incubator-gluten]
Posted by "FelixYBW (via GitHub)" <gi...@apache.org>.
FelixYBW closed issue #6525: [VL] BlockAllocationListener isn't thread safe
URL: https://github.com/apache/incubator-gluten/issues/6525
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [I] [VL] BlockAllocationListener isn't thread safe [incubator-gluten]
Posted by "FelixYBW (via GitHub)" <gi...@apache.org>.
FelixYBW commented on issue #6525:
URL: https://github.com/apache/incubator-gluten/issues/6525#issuecomment-2240902101
fixe by 6526
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org