[jira] [Issue Comment Deleted] (IMPALA-4539) Parquet scanner memory bug: I/O buffer is attached to output batch while scratch batch rows still reference it

["Fu Lili (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/IMPALA-4539?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Fu Lili updated IMPALA-4539:
----------------------------
    Comment: was deleted

(was: @Tim Armstrong 
Thank you very much，and sorry for my poor english。)

> Parquet scanner memory bug: I/O buffer is attached to output batch while scratch batch rows still reference it
> --------------------------------------------------------------------------------------------------------------
>
>                 Key: IMPALA-4539
>                 URL: https://issues.apache.org/jira/browse/IMPALA-4539
>             Project: IMPALA
>          Issue Type: Bug
>          Components: Backend
>    Affects Versions: Impala 2.6.0, Impala 2.7.0, Impala 2.8.0
>            Reporter: Fu Lili
>            Assignee: Tim Armstrong
>            Priority: Blocker
>              Labels: crash, resource-management
>             Fix For: Impala 2.8.0
>
>
> in the HdfsScanner，RowBatch own some io_buffers when Scanner has complete some io read：
> {code:java}
>   // We need to pass the row batch to the scan node if there is too much memory attached,
>   // which can happen if the query is very selective. We need to release memory even
>   // if no rows passed predicates.
>   if (batch_->AtCapacity() || context_->num_completed_io_buffers() > 0) {
>     context_->ReleaseCompletedResources(batch_, /* done */ false);
>   }
> {code}
> when the row batch is reset, the io_buffers will be free or return to the mem_pool。
> {code:java}
>   if (!FLAGS_disable_mem_pools && free_buffers_[idx].size() < FLAGS_max_free_io_buffers) {
>     free_buffers_[idx].push_back(buffer);
>     if (ImpaladMetrics::IO_MGR_NUM_UNUSED_BUFFERS != NULL) {
>       ImpaladMetrics::IO_MGR_NUM_UNUSED_BUFFERS->Increment(1L);
>     }
>   } else {
>     process_mem_tracker_->Release(buffer_size);
>     num_allocated_buffers_.Add(-1);
>     delete[] buffer;
>     if (ImpaladMetrics::IO_MGR_NUM_BUFFERS != NULL) {
>       ImpaladMetrics::IO_MGR_NUM_BUFFERS->Increment(-1L);
>     }
>     if (ImpaladMetrics::IO_MGR_TOTAL_BYTES != NULL) {
>       ImpaladMetrics::IO_MGR_TOTAL_BYTES->Increment(-buffer_size);
>     }
>   }
> {code}
> here is the bug：the io_buffers owned by the row batch A may by  used by the row batch B in next ScanNode::GetNext at the same time，but when the row batch B is need to be read，the io_buffers may has been released because row batch A has been reset。for example:
> {code:java}
> 1. in scanner, get row batch A and owned the io_buffer O1。
> 2. row batch A has been consumed，and the io_buffer O1 is released。
> 3. in scanner, get row batch B，but some tuple in row batch B is pointed to io_buffer O1，for example，some string tuples。Especially when row batch A is AtCapacity()，the io_buffer is very likely not only used by row batch A。
> 4. when row batch B need to be consumed，some tuples will produce error data，because io_buffer O1 has been released。
> {code}
> this bug is easy to reproduce when use starts option: "disable_mem_pools=true"，because in this situation, the io_buffers will be free really instead of being returned to the mem_pool。



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-all-unsubscribe@impala.apache.org
For additional commands, e-mail: issues-all-help@impala.apache.org