[jira] [Created] (ARROW-16761) [C++][Python] Track bytes_written on FileWriter / WrittenFile

["Will Jones (Jira)" <ji...@apache.org>]

Will Jones created ARROW-16761:
----------------------------------

             Summary: [C++][Python] Track bytes_written on FileWriter / WrittenFile
                 Key: ARROW-16761
                 URL: https://issues.apache.org/jira/browse/ARROW-16761
             Project: Apache Arrow
          Issue Type: New Feature
          Components: C++, Python
    Affects Versions: 8.0.0
            Reporter: Will Jones
             Fix For: 9.0.0


For Apache Iceberg and Delta Lake tables, we need to be able to get the size of the files written in bytes. In Iceberg, this is the required field {{file_size_in_bytes}} ([docs|https://iceberg.apache.org/spec/#manifests]). In Delta, this is the required field {{size}} as part of the Add action.

I think this could be exposed on [FileWriter|https://github.com/apache/arrow/blob/8c63788ff7d52812599a546989b7df10887cb01e/cpp/src/arrow/dataset/file_base.h#L305] and then through that [WrittenFile|https://github.com/apache/arrow/blob/8c63788ff7d52812599a546989b7df10887cb01e/python/pyarrow/_dataset.pyx#L766-L769]. But lower-level than that I'm not yet sure. {{FileWriter}} owns its {{OutputStream}}; would {{OutputStream::Tell()}} give the correct count?



--
This message was sent by Atlassian Jira
(v8.20.7#820007)