[jira] [Commented] (ARROW-7051) [C++] Improve MakeArrayOfNull to support creation of multiple arrays

["Jeroen van Straten (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/ARROW-7051?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17470504#comment-17470504 ] 

Jeroen van Straten commented on ARROW-7051:
-------------------------------------------

I'll look at this for one of my first issues.

Idea: can we use mmap for this? On my machine anyway, mmap'ing a read-only, anonymous, and private region will give fully zeroed virtual memory where (I guess) all pages are mapped to the same physical page, or maybe some x86 MMU tricks are done to avoid even that. In any case, the following code results in no physical memory being allocated at all (not even a single 4kiB page, apparently):
{code:java|title=zero.c|borderStyle=solid}
#include <sys/mman.h>
#include <stdio.h>

int main() {
    char *addr;
    const size_t length = 1024ull*1024*1024*64; // 64 GiB, more memory than my machine has
    addr = (char*)mmap(0, length, PROT_READ, MAP_PRIVATE | MAP_ANONYMOUS | MAP_NORESERVE, -1, 0);
    if (!addr) {
        perror("mmap");
        return 1;
    }

    // Just to force the kernel to populate the page table by having the CPU read from every page in the mapping:
    size_t pos = 0;
    char total = 0;
    while (pos < length) {
        total += addr[pos];
        pos += 4096; // page size
    }
    printf("total: %d (should be 0)\n", (int)total);

    // Wait for input so resource usage can be inspected.
    getchar();

    munmap(addr, length);
}
{code}

Also:

 - CPU cache layers using physical addressing will not be filled with useless zeros.
 - If anything tries to write to the zeroed region, the application will immediately crash, rather than continuing with broken invariants.

The only problem I see is that I don't know how portable this is, if at all. But of course a fallback to some form of malloc + memset can always be done at compile time.

> [C++] Improve MakeArrayOfNull to support creation of multiple arrays
> --------------------------------------------------------------------
>
>                 Key: ARROW-7051
>                 URL: https://issues.apache.org/jira/browse/ARROW-7051
>             Project: Apache Arrow
>          Issue Type: Improvement
>          Components: C++
>    Affects Versions: 0.14.0
>            Reporter: Ben Kietzman
>            Assignee: Jeroen van Straten
>            Priority: Minor
>              Labels: beginner, good-first-issue
>          Time Spent: 1h 40m
>  Remaining Estimate: 0h
>
> MakeArrayOfNull reuses a single buffer of {{0}} for all buffers in the array it creates. It could be extended to reuse that same buffer for all buffers in multiple arrays. This optimization will make RecordBatchProjector and ConcatenateTablesWithPromotion more memory efficient



--
This message was sent by Atlassian Jira
(v8.20.1#820001)