IDEA: passing a known length into HadoopInputFile fromPath

[Aaron Niskode-Dossett via dev <de...@parquet.apache.org>]

A few months ago in a PR comment [1] Steve Loughran mentioned that tracking
the known length of a file could be helpful.  I was intrigued by that idea
and explored it this week.  I riffed on the idea with Codex and ended up
with a possible solution.  I'm bringing it up here because I'm not yet
confident enough in the change for a PR, but I can certainly open one.

Here's my understanding that I'm trying to validate:

The basic idea is that starting with Hadoop 3.3.5 (for reference we
currently depend on 3.3.0) a hint called `fs.option.openfile.length` can be
supplied when building a SeekableInputStream instead of supplying a full
FileStat object.  Catalogs like Iceberg (and I assume others) currently
construct the stat object, which entails a cloud store metadata lookup, but
they already track the length and could use that instead and save the
lookup cost.

So what if we add this method to HadoopInputFile?

HadoopInputFile fromPath(Path path, long length, Configuration conf)

and then the newStream() method uses the stat or the length depending on
which `from` method was used.

Stylistically, with this change instead of:

HadoopInputFile.fromStatus(stat, conf)`

Iceberg could

  HadoopInputFile.fromPath(
      input.getPath(),
      input.getLength(),
      input.getConf())

My understanding:  If the hadoop version used is at least 3.3.0 but less
than 3.3.5 then the option is ignored and a metadata lookup made anyway.
Below 3.3.0 this could be an error, but we're advertising a minimum
requirement of 3.3.0 so that seems acceptable and not a breaking change.
My PR would also include bumping the dependency to 3.3.5

Does this seem like a good idea?  Am I missing anything?  Is this idea
solid enough to open a PR and proceed?

Thanks!  -- Aaron

[1] https://github.com/apache/parquet-java/pull/3395#issuecomment-4268040711

-- 
Aaron Niskode-Dossett, Data Engineering -- Etsy

Re: IDEA: passing a known length into HadoopInputFile fromPath

[Aaron Niskode-Dossett via dev <de...@parquet.apache.org>]

On second thought, I decided to open a PR.
https://github.com/apache/parquet-java/pull/3678

On Sat, Jul 18, 2026 at 10:02 AM Aaron Niskode-Dossett <
aniskodedossett@etsy.com> wrote:

> A few months ago in a PR comment [1] Steve Loughran mentioned that
> tracking the known length of a file could be helpful.  I was intrigued by
> that idea and explored it this week.  I riffed on the idea with Codex and
> ended up with a possible solution.  I'm bringing it up here because I'm not
> yet confident enough in the change for a PR, but I can certainly open one.
>
> Here's my understanding that I'm trying to validate:
>
> The basic idea is that starting with Hadoop 3.3.5 (for reference we
> currently depend on 3.3.0) a hint called `fs.option.openfile.length` can be
> supplied when building a SeekableInputStream instead of supplying a full
> FileStat object.  Catalogs like Iceberg (and I assume others) currently
> construct the stat object, which entails a cloud store metadata lookup, but
> they already track the length and could use that instead and save the
> lookup cost.
>
> So what if we add this method to HadoopInputFile?
>
> HadoopInputFile fromPath(Path path, long length, Configuration conf)
>
> and then the newStream() method uses the stat or the length depending on
> which `from` method was used.
>
> Stylistically, with this change instead of:
>
> HadoopInputFile.fromStatus(stat, conf)`
>
> Iceberg could
>
>   HadoopInputFile.fromPath(
>       input.getPath(),
>       input.getLength(),
>       input.getConf())
>
> My understanding:  If the hadoop version used is at least 3.3.0 but less
> than 3.3.5 then the option is ignored and a metadata lookup made anyway.
> Below 3.3.0 this could be an error, but we're advertising a minimum
> requirement of 3.3.0 so that seems acceptable and not a breaking change.
> My PR would also include bumping the dependency to 3.3.5
>
> Does this seem like a good idea?  Am I missing anything?  Is this idea
> solid enough to open a PR and proceed?
>
> Thanks!  -- Aaron
>
> [1]
> https://github.com/apache/parquet-java/pull/3395#issuecomment-4268040711
>
> --
> Aaron Niskode-Dossett, Data Engineering -- Etsy
>


-- 
Aaron Niskode-Dossett, Data Engineering -- Etsy