[jira] [Updated] (ARROW-15281) [C++] Implement ability to retrieve fragment filename

["ASF GitHub Bot (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/ARROW-15281?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

ASF GitHub Bot updated ARROW-15281:
-----------------------------------
    Labels: dataset pull-request-available query-engine  (was: dataset query-engine)

> [C++] Implement ability to retrieve fragment filename
> -----------------------------------------------------
>
>                 Key: ARROW-15281
>                 URL: https://issues.apache.org/jira/browse/ARROW-15281
>             Project: Apache Arrow
>          Issue Type: Improvement
>          Components: C++
>            Reporter: Nicola Crane
>            Assignee: Sanjiban Sengupta
>            Priority: Major
>              Labels: dataset, pull-request-available, query-engine
>          Time Spent: 10m
>  Remaining Estimate: 0h
>
> A user has requested the ability to include the filename of the CSV in the dataset output - see discussion on ARROW-15260 for more context.
> Relevant info from that ticket:
>  
> "From a C++ perspective we've got many of the pieces needed already. One challenge is that the datasets API is written to work with "fragments" and not "files". For example, a dataset might be an in-memory table in which case we are working with InMemoryFragment and not FileFragment so there is no concept of "filename".
> That being said, the low level ScanBatchesAsync method actually returns a generator of TaggedRecordBatch for this very purpose. A TaggedRecordBatch is a struct with the record batch as well as the source fragment for that record batch.
> So if you were to execute scan, you could inspect the fragment and, if it is a FileFragment, you could extract the filename.
> Another challenge is that R is moving towards more and more access through an exec plan and not directly using a scanner. In order for that to work we would need to augment the scan results with the filename in C++ before sending into the exec plan. Luckily, we already do this a bit as well. We currently augment the scan results with fragment index, batch index, and whether the batch is the last batch in the fragment.
> Since ExecBatch can work with constants efficiently I don't think there will be much performance cost in always including the filename. So the work remaining is simply to add a new augmented field _{_}fragment_source_name which is always attached if the underlying fragment is a filename. Then users can get this field if they want by including "{_}_fragment_source_name" in the list of columns they query for."



--
This message was sent by Atlassian Jira
(v8.20.1#820001)