[jira] [Created] (ARROW-8950) [C++] Make head optional in s3fs

["Remi Dettai (Jira)" <ji...@apache.org>]

Remi Dettai created ARROW-8950:
----------------------------------

             Summary: [C++] Make head optional in s3fs
                 Key: ARROW-8950
                 URL: https://issues.apache.org/jira/browse/ARROW-8950
             Project: Apache Arrow
          Issue Type: Improvement
          Components: C++
            Reporter: Remi Dettai


When you open an input file with the f3fs, it issues a head request to S3 to check if the file is present/authorized and get the size (https://github.com/apache/arrow/blob/f16f76ab7693ae085e82f4269a0a0bc23770bef9/cpp/src/arrow/filesystem/s3fs.cc#L407).

This call comes with a non-neglictable cost:
 * adds latency
 * priced the same as a GET request by AWS

I fail to see usecases where this call is really crucial:
 * if the file is not present/authorized, failing at first read seems to have mostly the same effect as failing on opening. I agree that it is kind of "usual" for an _open_ call to fail eagerly, so to avoid surprises we could add a flag indicating if we don't need to fail when running _OpenInputFile_ on an inaccessible file.
 * getting the size can be done on the first read, and could be mostly avoided on caller side if the filesystem api provided read-from-end capabilities (compatible with fs reads using _ios::end_ and on http filesystems with _bytes=-xxx_). Worst case scenario the call to _head_ could be done lazily when calling _getSize()._

I agree that it makes things a bit more complex, and I understand that you would not want to complexify the generic fs api because of blob storage behavior. But obviously there are workloads where this has a significant impact.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)