You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-dev@hadoop.apache.org by Garret Wilson <ga...@globalmentor.com> on 2022/11/10 14:53:34 UTC

way forward for Winutils excision from `FileSystem`

Steve Loughran and I have been discussing on Stack Overflow 
<https://stackoverflow.com/q/73503205> a way forward for removing the 
Winutils requirement from the local `FileSystem` implementations.

Hadoop's FileSystem API has a lot of *nix assumptions which originally 
made it not possible to implement in pure Java for local file system 
access. The current implementation essentially creates shell processes 
that invoke *nix commands in order to e.g. access permissions. To get 
this working on Windows, Steve created Winutils 
<https://github.com/steveloughran/winutils>, a sort of Windows back-door 
subsystem of binary executables which must be installed separately 
(think a tiny .NET) and which Hadoop can invoke as a substitute for *nix 
calls. At the time it was no doubt a nifty quick workaround, but as a 
long-term solution it is horrible (for a long list of reasons which 
everyone here already knows so I won't go into them; see HADOOP-13223 
<https://issues.apache.org/jira/browse/HADOOP-13223> and HADOOP-17839 
<https://issues.apache.org/jira/browse/HADOOP-17839>.) There should be 
no need to install a separate set of executables maintained by a third 
party just to get Spark to write output to a local file on a Windows 
laptop, for example.

I have created the GlobalMentor Hadoop Bare Naked Local FileSystem 
<https://github.com/globalmentor/hadoop-bare-naked-local-fs>, an 
implementation of `FileSystem` for the local file system that extends 
`LocalFileSystem`/`RawLocalFileSystem` and "undoes" the Winutils code by 
accessing pure Java API calls instead. It is available on Maven, and 
using it with Spark is as simple as including it as a dependency at 
specifying the implementation in the configuration, e.g. programmatically:

```java
SparkSession spark = SparkSession.builder().appName("Foo 
Bar").master("local").getOrCreate();
spark.sparkContext().hadoopConfiguration().setClass("fs.file.impl", 
BareLocalFileSystem.class, FileSystem.class);
```

But Bare Naked Local File System is not the end of the story.

  * Bare Naked Local File System v0.1.0 doesn't (yet) support symlinks
    or the sticky bit.
  * But the bigger issue is how to excise Winutils completely in the
    existing Hadoop code. Winutils assumptions are hard-coded at a low
    level across various classes—even code that has nothing to do with
    the file system. The startup configuration for example calls
    `StringUtils.equalsIgnoreCase("true", valueString)` which loads the
    `StringUtils` class, which has a static reference to `Shell`, which
    has a static block that checks for `WINUTILS_EXE`.
  * For the most part there should no longer even be a need for anything
    but direct Java API access for the local file system. But muddling
    things further, the existing `RawLocalFileSystem` implementation has
    /four/ ways to access the local file system: Winutils, JNI calls,
    shell access, and a "new" approach using "stat". The "stat" approach
    has been switched off with a hard-coded `useDeprecatedFileStatus =
    true` because of HADOOP-9652
    <https://issues.apache.org/jira/browse/HADOOP-9652>.
  * Local file access is not contained within `RawLocalFileSystem` but
    is scattered across other classes; `FileUtil.readLink()` for example
    (which `RawLocalFileSystem` calls because of the deprecation issue
    above) uses the shell approach without any option to change it.
    (This implementation-specific decision should have been contained
    within the `FileSystem` implementation itself.)

In short, it's a mess that has accumulated over years and getting worse, 
charging high interest on what at first was a small, self-contained 
technical debt.

I would welcome the opportunity to clean up this mess. I'm probably as 
qualified as anyone to make the changes. This is one of my areas of 
expertise: I was designing a full abstract file system interface (with 
pure-Java from-scratch implementations for the local file system, 
Subversion, and WebDAV—even the WebDAV HTTP implementation was from 
scratch) around the time Apache Nutch was getting off the ground. Most 
recently I've worked on the Hadoop `FileSystem` API contracting for 
LinkedIn, discovering (what I consider to be) a huge bug in 
ViewFilesystem, HADOOP-18525 
<https://issues.apache.org/jira/browse/HADOOP-18525>.

The cleanup should be done in several stages (e.g. consolidating 
WinUtils access; replacing code with pure Java API calls; undeprecating 
the new Stat code and relegating it to a different class, etc.). 
Unfortunately it's not financially feasible for me to sit here for 
several months and revamp the Hadoop `FileSystem` subsystem for fun 
(even though I wish I could). Perhaps there is job opening at a company 
related to Hadoop that would be interested in hiring me and devoting a 
certain percentage of my time to fixing local `FileSystem` access. If 
so, let me know where I should send my resume 
<https://www.garretwilson.com/about/resume>.

Otherwise let me know if any ideas for a way forward. If there proves to 
be interest in GlobalMentor Hadoop Bare Naked Local FileSystem 
<https://github.com/globalmentor/hadoop-bare-naked-local-fs> on GitHub 
I'll try to maintain and improve it, but really what needs to be 
revamped is the Hadoop codebase itself. I'll be happy when Hadoop is 
fixed so that both Steve's code and my code are no longer needed.

Garret

Re: way forward for Winutils excision from `FileSystem`

Posted by Gautham Banasandra <ga...@apache.org>.
Hi Steve,

There are quite a few unit test failures on Windows. Some of them are major
ones.
I feel we need to fix those before we do a release. I'm trying to focus on
setting up the CI
so that we can incrementally fix the unit tests and avoid any possible
regressions.

Thanks,
--Gautham

On Tue, 15 Nov 2022 at 00:14, Steve Loughran <st...@cloudera.com> wrote:

> you up for doing a winutils build on the 3.3.5 release? i'm going to do
> the arm64 binaries
>
> On Mon, 14 Nov 2022 at 18:08, Gautham Banasandra <ga...@apache.org>
> wrote:
>
>> Also, I plan to do a Windows release once I setup the CI for Windows and
>> after I get
>> the major unit tests to pass. It would still contain winutils though.
>> However, we can do
>> another release after deprecating winutils.
>>
>> Thanks,
>> --Gautham
>>
>> On Mon, 14 Nov 2022 at 23:34, Gautham Banasandra <ga...@apache.org>
>> wrote:
>>
>> > Hi Iñigo,
>> >
>> > I would like to aim for winutils deprecation by the end of the first
>> > quarter of 2023.
>> > It really depends on how fast I can wrap up with setting up CI for
>> > Windows. Given
>> > that this involves getting Yetus to work properly on Windows, I feel
>> it's
>> > a bit
>> > ambitious. But if things fall into place, I think end of the first
>> quarter
>> > of 2023 would
>> > be a reachable timeline.
>> >
>> > Thanks,
>> > --Gautham
>> >
>> > On Sat, 12 Nov 2022 at 00:20, Iñigo Goiri <el...@gmail.com> wrote:
>> >
>> >> Gautham, thank you very much for the summary.
>> >> Do you have a time-line for when we can get rid of winutils?
>> >> My idea was to get this and the YARN federation hardening work into a
>> 3.4
>> >> release.
>> >>
>> >>
>> >>
>> >> On Fri, Nov 11, 2022, 10:15 Gautham Banasandra <ga...@apache.org>
>> >> wrote:
>> >>
>> >>> Hi folks,
>> >>>
>> >>>
>> >>> What have we done so far?
>> >>> ------------------------------------
>> >>> Inigo and I have been working for quite some time now on this topic,
>> >>> but our efforts have mostly been oriented towards making Hadoop
>> >>> cross-platform compatible. Our focus has been on streamlining the
>> >>> process of building Hadoop on Windows so that one can easily
>> >>> build and run Hadoop, just like on Linux. We reached this milestone
>> >>> quite recently and I've documented the steps for doing so here -
>> >>>
>> >>>
>> https://github.com/apache/hadoop/blob/5bb11cecea136acccac2563b37021b554e517012/BUILDING.txt#L493-L622
>> >>>
>> >>>
>> >>>
>> >>> Is winutils still required?
>> >>> -------------------------------
>> >>> As Steve mentioned, we would still require winutils for running
>> >>> Hadoop on Windows. The major change here is that winutils
>> >>> need not come from a third-party repository anymore, rather it
>> >>> gets built along with the Hadoop codebase itself henceforth.
>> >>> However, I agree that we need to deprecate winutils and
>> >>> replace it with something better so that Hadoop users can have
>> >>> a smoother experience.
>> >>>
>> >>>
>> >>> What's the best way to deprecate winutils?
>> >>> --------------------------------------------------------
>> >>> Over all the time that I've spent making Hadoop cross-platform
>> >>> compatible, I've realized that the best way would be to have a
>> >>> JNI interface that wraps around a native layer. This native layer
>> >>> could be implemented majorly in C++. C++17 provides the
>> >>> std::filesystem namespace that can satisfy most of the native
>> >>> filesystem API requirements. Since std::filesystem is part of "The
>> >>> Standard Libray", these APIs will be present on most/all the C++
>> >>> compilers of the various OS platforms. For those parts that can't
>> >>> be satisfied by std::filesystem, we'll have to delve into this part
>> >>> by writing C code that makes system calls. Please note that
>> >>> these C files will need to be implemented specifically for each
>> >>> platform. I took this approach when I wrote x-platform library to
>> >>> make HDFS native client cross-platform compatible -
>> >>>
>> >>>
>> https://github.com/apache/hadoop/tree/trunk/hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/lib/x-platform
>> >>>
>> >>>
>> >>> What am I focussing on currently?
>> >>> ------------------------------------------------
>> >>> So far, I've focussed on getting the build to work seamlessly
>> >>> on Windows. I'm now trying to protect this from breaking by
>> >>> setting up CI on Jenkins that builds Hadoop on Windows
>> >>> for the precommit validation -
>> >>> https://issues.apache.org/jira/browse/INFRA-23809
>> >>> Yes, it does involve getting
>> >>> Yetus to run on Windows. I can work on deprecating winutils
>> >>> after this.
>> >>>
>> >>> Thanks,
>> >>> --Gautham
>> >>>
>> >>> On Fri, 11 Nov 2022 at 19:51, Steve Loughran
>> <st...@cloudera.com.invalid>
>> >>> wrote:
>> >>>
>> >>>> It's time to reach for the axe.
>> >>>>
>> >>>> We haven't shipped eight version of Apache hadoop which builds and
>> runs
>> >>>> on
>> >>>> windows for a long long time. I the only people trying to use the
>> >>>> library
>> >>>> is on windows Will have been people trying to use spark on their
>> laptops
>> >>>> with "small" dataset of only a are few tens of gigabytes at a time,
>> the
>> >>>> kind of work where 32GB of ram and 16 cores is enough. Put
>> differently:
>> >>>> in
>> >>>> storage and performance of Single laptop means that it is perfectly
>> >>>> suitable for doing reasonable amounts of work and the main barrier to
>> >>>> doing
>> >>>> so is getting a copy of the winutils lib.
>> >>>>
>> >>>> I know Gautham and Inigo I trying to get windows to work as a
>> location
>> >>>> for
>> >>>> yarn again; not sure about hdfs. And there, yes, we have to say "they
>> >>>> likely to need an extra binary"
>> >>>>
>> >>>> But for someone wanting to count the number of rows in an avro file?
>> do
>> >>>> a
>> >>>> simple bit of filtering on some parquet data? Is these are the kind
>> of
>> >>>> things that anyone with a linux/mac laptop can do with ease and it is
>> >>>> not
>> >>>> fair to put suffering on to others. And well we could just say "why
>> do
>> >>>> you
>> >>>> just install Lynnox on that laptop then?", I have someone who has
>> had a
>> >>>> Linux laptop for many years I know the written strong arguments
>> against
>> >>>> it
>> >>>> even beyond the "my employer demand windows with their IT software"
>> as
>> >>>> "a
>> >>>> latop which comes out of sleep reliably" is kind of important too.
>> >>>>
>> >>>> I how can we let the people who have to live in this world – And we
>> have
>> >>>> someone who is clearly willing to help –Live a better life. Funnily
>> >>>> enough,
>> >>>> the fact that we have not shipped a working version of when you tails
>> >>>> for a
>> >>>> long time actually gives us an advantage: we can pick incompatible
>> >>>> changes
>> >>>> and be confident that most people aren't going to notice.
>> >>>>
>> >>>> I think a good first step would be for Shell to work well if winutils
>> >>>> isn't
>> >>>> around -get rid of that static, WINUTILS string and path/file
>> >>>> equivalents,
>> >>>> the ones deprecated in 2015. We can rip them out knowing no external
>> >>>> code
>> >>>> is using them.
>> >>>>
>> >>>> Then we should look very closely at FileUtil to see how much of that
>> is
>> >>>> needed and how can we isolate it better. If you look at the change
>> log
>> >>>> of
>> >>>> that file, we do have to consider that every time it execs a shell
>> >>>> command
>> >>>> I there's a security risk and more than once we've had to fix it. Not
>> >>>> executing any external shell commands is good everywhere.
>> >>>>
>> >>>>
>> >>>>
>> >>>>
>> >>>> On Thu, 10 Nov 2022 at 19:00, Chris Nauroth <cn...@apache.org>
>> >>>> wrote:
>> >>>>
>> >>>> > Symlink support on the local file system is still used. One
>> example I
>> >>>> can
>> >>>> > think of is YARN container launch [1].
>> >>>> >
>> >>>> > I would welcome removal of winutils, as already described in
>> various
>> >>>> JIRA
>> >>>> > issues. I think the biggest challenge we'll have is testing of a
>> >>>> transition
>> >>>> > from winutils to the newer Java APIs. The contract tests help, but
>> >>>> > historically there was also a tendency to break things in
>> downstream
>> >>>> > dependent projects.
>> >>>> >
>> >>>> > I'd suggest taking this on piecemeal, transitioning small pieces of
>> >>>> > FileSystem off of winutils one at a time.
>> >>>> >
>> >>>> > [1]
>> >>>> >
>> >>>> >
>> >>>>
>> https://github.com/apache/hadoop/blob/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/launcher/ContainerLaunch.java#L1508-L1509
>> >>>> >
>> >>>> > Chris Nauroth
>> >>>> >
>> >>>> >
>> >>>> > On Thu, Nov 10, 2022 at 10:33 AM Wei-Chiu Chuang <
>> weichiu@apache.org>
>> >>>> > wrote:
>> >>>> >
>> >>>> > > >
>> >>>> > > >
>> >>>> > > >
>> >>>> > > >   * Bare Naked Local File System v0.1.0 doesn't (yet) support
>> >>>> symlinks
>> >>>> > > >     or the sticky bit.
>> >>>> > > >
>> >>>> > > ok to not support symlinks. The symlinks of HDFS are not being
>> >>>> maintained
>> >>>> > > and I am not aware of anything relying on it.
>> >>>> > > So I assume people don't need it.
>> >>>> > >
>> >>>> > > Sticky bit would be useful, I guess.
>> >>>> > >
>> >>>> > > I suppose folks working at Microsoft would be more interested in
>> >>>> this
>> >>>> > work?
>> >>>> > > Last time I heard, Gautham and Inigo were revamping Hadoop's
>> Windows
>> >>>> > > support.
>> >>>> > >
>> >>>> > >
>> >>>> > > >   * But the bigger issue is how to excise Winutils completely
>> in
>> >>>> the
>> >>>> > > >     existing Hadoop code. Winutils assumptions are hard-coded
>> at
>> >>>> a low
>> >>>> > > >     level across various classes—even code that has nothing to
>> do
>> >>>> with
>> >>>> > > >     the file system. The startup configuration for example
>> calls
>> >>>> > > >     `StringUtils.equalsIgnoreCase("true", valueString)` which
>> >>>> loads the
>> >>>> > > >     `StringUtils` class, which has a static reference to
>> `Shell`,
>> >>>> which
>> >>>> > > >     has a static block that checks for `WINUTILS_EXE`.
>> >>>> > > >   * For the most part there should no longer even be a need for
>> >>>> > anything
>> >>>> > > >     but direct Java API access for the local file system. But
>> >>>> muddling
>> >>>> > > >     things further, the existing `RawLocalFileSystem`
>> >>>> implementation
>> >>>> > has
>> >>>> > > >     /four/ ways to access the local file system: Winutils, JNI
>> >>>> calls,
>> >>>> > > >     shell access, and a "new" approach using "stat". The "stat"
>> >>>> > approach
>> >>>> > > >     has been switched off with a hard-coded
>> >>>> `useDeprecatedFileStatus =
>> >>>> > > >     true` because of HADOOP-9652
>> >>>> > > >     <https://issues.apache.org/jira/browse/HADOOP-9652>.
>> >>>> > > >   * Local file access is not contained within
>> >>>> `RawLocalFileSystem` but
>> >>>> > > >     is scattered across other classes; `FileUtil.readLink()`
>> for
>> >>>> > example
>> >>>> > > >     (which `RawLocalFileSystem` calls because of the
>> deprecation
>> >>>> issue
>> >>>> > > >     above) uses the shell approach without any option to change
>> >>>> it.
>> >>>> > > >     (This implementation-specific decision should have been
>> >>>> contained
>> >>>> > > >     within the `FileSystem` implementation itself.)
>> >>>> > > >
>> >>>> > > > In short, it's a mess that has accumulated over years and
>> getting
>> >>>> > worse,
>> >>>> > > > charging high interest on what at first was a small,
>> >>>> self-contained
>> >>>> > > > technical debt.
>> >>>> > > >
>> >>>> > > > I would welcome the opportunity to clean up this mess. I'm
>> >>>> probably as
>> >>>> > > > qualified as anyone to make the changes. This is one of my
>> areas
>> >>>> of
>> >>>> > > > expertise: I was designing a full abstract file system
>> interface
>> >>>> (with
>> >>>> > > > pure-Java from-scratch implementations for the local file
>> system,
>> >>>> > > > Subversion, and WebDAV—even the WebDAV HTTP implementation was
>> >>>> from
>> >>>> > > > scratch) around the time Apache Nutch was getting off the
>> ground.
>> >>>> Most
>> >>>> > > > recently I've worked on the Hadoop `FileSystem` API contracting
>> >>>> for
>> >>>> > > > LinkedIn, discovering (what I consider to be) a huge bug in
>> >>>> > > > ViewFilesystem, HADOOP-18525
>> >>>> > > > <https://issues.apache.org/jira/browse/HADOOP-18525>.
>> >>>> > > >
>> >>>> > > > The cleanup should be done in several stages (e.g.
>> consolidating
>> >>>> > > > WinUtils access; replacing code with pure Java API calls;
>> >>>> undeprecating
>> >>>> > > > the new Stat code and relegating it to a different class,
>> etc.).
>> >>>> > > > Unfortunately it's not financially feasible for me to sit here
>> for
>> >>>> > > > several months and revamp the Hadoop `FileSystem` subsystem for
>> >>>> fun
>> >>>> > > > (even though I wish I could). Perhaps there is job opening at a
>> >>>> company
>> >>>> > > > related to Hadoop that would be interested in hiring me and
>> >>>> devoting a
>> >>>> > > > certain percentage of my time to fixing local `FileSystem`
>> >>>> access. If
>> >>>> > > > so, let me know where I should send my resume
>> >>>> > > > <https://www.garretwilson.com/about/resume>.
>> >>>> > > >
>> >>>> > > > Otherwise let me know if any ideas for a way forward. If there
>> >>>> proves
>> >>>> > to
>> >>>> > > > be interest in GlobalMentor Hadoop Bare Naked Local FileSystem
>> >>>> > > > <https://github.com/globalmentor/hadoop-bare-naked-local-fs>
>> on
>> >>>> GitHub
>> >>>> > > > I'll try to maintain and improve it, but really what needs to
>> be
>> >>>> > > > revamped is the Hadoop codebase itself. I'll be happy when
>> Hadoop
>> >>>> is
>> >>>> > > > fixed so that both Steve's code and my code are no longer
>> needed.
>> >>>> > > >
>> >>>> > > > Garret
>> >>>> > > >
>> >>>> > >
>> >>>> >
>> >>>>
>> >>>
>>
>

Re: way forward for Winutils excision from `FileSystem`

Posted by Steve Loughran <st...@cloudera.com.INVALID>.
you up for doing a winutils build on the 3.3.5 release? i'm going to do the
arm64 binaries

On Mon, 14 Nov 2022 at 18:08, Gautham Banasandra <ga...@apache.org> wrote:

> Also, I plan to do a Windows release once I setup the CI for Windows and
> after I get
> the major unit tests to pass. It would still contain winutils though.
> However, we can do
> another release after deprecating winutils.
>
> Thanks,
> --Gautham
>
> On Mon, 14 Nov 2022 at 23:34, Gautham Banasandra <ga...@apache.org>
> wrote:
>
> > Hi Iñigo,
> >
> > I would like to aim for winutils deprecation by the end of the first
> > quarter of 2023.
> > It really depends on how fast I can wrap up with setting up CI for
> > Windows. Given
> > that this involves getting Yetus to work properly on Windows, I feel it's
> > a bit
> > ambitious. But if things fall into place, I think end of the first
> quarter
> > of 2023 would
> > be a reachable timeline.
> >
> > Thanks,
> > --Gautham
> >
> > On Sat, 12 Nov 2022 at 00:20, Iñigo Goiri <el...@gmail.com> wrote:
> >
> >> Gautham, thank you very much for the summary.
> >> Do you have a time-line for when we can get rid of winutils?
> >> My idea was to get this and the YARN federation hardening work into a
> 3.4
> >> release.
> >>
> >>
> >>
> >> On Fri, Nov 11, 2022, 10:15 Gautham Banasandra <ga...@apache.org>
> >> wrote:
> >>
> >>> Hi folks,
> >>>
> >>>
> >>> What have we done so far?
> >>> ------------------------------------
> >>> Inigo and I have been working for quite some time now on this topic,
> >>> but our efforts have mostly been oriented towards making Hadoop
> >>> cross-platform compatible. Our focus has been on streamlining the
> >>> process of building Hadoop on Windows so that one can easily
> >>> build and run Hadoop, just like on Linux. We reached this milestone
> >>> quite recently and I've documented the steps for doing so here -
> >>>
> >>>
> https://github.com/apache/hadoop/blob/5bb11cecea136acccac2563b37021b554e517012/BUILDING.txt#L493-L622
> >>>
> >>>
> >>>
> >>> Is winutils still required?
> >>> -------------------------------
> >>> As Steve mentioned, we would still require winutils for running
> >>> Hadoop on Windows. The major change here is that winutils
> >>> need not come from a third-party repository anymore, rather it
> >>> gets built along with the Hadoop codebase itself henceforth.
> >>> However, I agree that we need to deprecate winutils and
> >>> replace it with something better so that Hadoop users can have
> >>> a smoother experience.
> >>>
> >>>
> >>> What's the best way to deprecate winutils?
> >>> --------------------------------------------------------
> >>> Over all the time that I've spent making Hadoop cross-platform
> >>> compatible, I've realized that the best way would be to have a
> >>> JNI interface that wraps around a native layer. This native layer
> >>> could be implemented majorly in C++. C++17 provides the
> >>> std::filesystem namespace that can satisfy most of the native
> >>> filesystem API requirements. Since std::filesystem is part of "The
> >>> Standard Libray", these APIs will be present on most/all the C++
> >>> compilers of the various OS platforms. For those parts that can't
> >>> be satisfied by std::filesystem, we'll have to delve into this part
> >>> by writing C code that makes system calls. Please note that
> >>> these C files will need to be implemented specifically for each
> >>> platform. I took this approach when I wrote x-platform library to
> >>> make HDFS native client cross-platform compatible -
> >>>
> >>>
> https://github.com/apache/hadoop/tree/trunk/hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/lib/x-platform
> >>>
> >>>
> >>> What am I focussing on currently?
> >>> ------------------------------------------------
> >>> So far, I've focussed on getting the build to work seamlessly
> >>> on Windows. I'm now trying to protect this from breaking by
> >>> setting up CI on Jenkins that builds Hadoop on Windows
> >>> for the precommit validation -
> >>> https://issues.apache.org/jira/browse/INFRA-23809
> >>> Yes, it does involve getting
> >>> Yetus to run on Windows. I can work on deprecating winutils
> >>> after this.
> >>>
> >>> Thanks,
> >>> --Gautham
> >>>
> >>> On Fri, 11 Nov 2022 at 19:51, Steve Loughran
> <st...@cloudera.com.invalid>
> >>> wrote:
> >>>
> >>>> It's time to reach for the axe.
> >>>>
> >>>> We haven't shipped eight version of Apache hadoop which builds and
> runs
> >>>> on
> >>>> windows for a long long time. I the only people trying to use the
> >>>> library
> >>>> is on windows Will have been people trying to use spark on their
> laptops
> >>>> with "small" dataset of only a are few tens of gigabytes at a time,
> the
> >>>> kind of work where 32GB of ram and 16 cores is enough. Put
> differently:
> >>>> in
> >>>> storage and performance of Single laptop means that it is perfectly
> >>>> suitable for doing reasonable amounts of work and the main barrier to
> >>>> doing
> >>>> so is getting a copy of the winutils lib.
> >>>>
> >>>> I know Gautham and Inigo I trying to get windows to work as a location
> >>>> for
> >>>> yarn again; not sure about hdfs. And there, yes, we have to say "they
> >>>> likely to need an extra binary"
> >>>>
> >>>> But for someone wanting to count the number of rows in an avro file?
> do
> >>>> a
> >>>> simple bit of filtering on some parquet data? Is these are the kind of
> >>>> things that anyone with a linux/mac laptop can do with ease and it is
> >>>> not
> >>>> fair to put suffering on to others. And well we could just say "why do
> >>>> you
> >>>> just install Lynnox on that laptop then?", I have someone who has had
> a
> >>>> Linux laptop for many years I know the written strong arguments
> against
> >>>> it
> >>>> even beyond the "my employer demand windows with their IT software" as
> >>>> "a
> >>>> latop which comes out of sleep reliably" is kind of important too.
> >>>>
> >>>> I how can we let the people who have to live in this world – And we
> have
> >>>> someone who is clearly willing to help –Live a better life. Funnily
> >>>> enough,
> >>>> the fact that we have not shipped a working version of when you tails
> >>>> for a
> >>>> long time actually gives us an advantage: we can pick incompatible
> >>>> changes
> >>>> and be confident that most people aren't going to notice.
> >>>>
> >>>> I think a good first step would be for Shell to work well if winutils
> >>>> isn't
> >>>> around -get rid of that static, WINUTILS string and path/file
> >>>> equivalents,
> >>>> the ones deprecated in 2015. We can rip them out knowing no external
> >>>> code
> >>>> is using them.
> >>>>
> >>>> Then we should look very closely at FileUtil to see how much of that
> is
> >>>> needed and how can we isolate it better. If you look at the change log
> >>>> of
> >>>> that file, we do have to consider that every time it execs a shell
> >>>> command
> >>>> I there's a security risk and more than once we've had to fix it. Not
> >>>> executing any external shell commands is good everywhere.
> >>>>
> >>>>
> >>>>
> >>>>
> >>>> On Thu, 10 Nov 2022 at 19:00, Chris Nauroth <cn...@apache.org>
> >>>> wrote:
> >>>>
> >>>> > Symlink support on the local file system is still used. One example
> I
> >>>> can
> >>>> > think of is YARN container launch [1].
> >>>> >
> >>>> > I would welcome removal of winutils, as already described in various
> >>>> JIRA
> >>>> > issues. I think the biggest challenge we'll have is testing of a
> >>>> transition
> >>>> > from winutils to the newer Java APIs. The contract tests help, but
> >>>> > historically there was also a tendency to break things in downstream
> >>>> > dependent projects.
> >>>> >
> >>>> > I'd suggest taking this on piecemeal, transitioning small pieces of
> >>>> > FileSystem off of winutils one at a time.
> >>>> >
> >>>> > [1]
> >>>> >
> >>>> >
> >>>>
> https://github.com/apache/hadoop/blob/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/launcher/ContainerLaunch.java#L1508-L1509
> >>>> >
> >>>> > Chris Nauroth
> >>>> >
> >>>> >
> >>>> > On Thu, Nov 10, 2022 at 10:33 AM Wei-Chiu Chuang <
> weichiu@apache.org>
> >>>> > wrote:
> >>>> >
> >>>> > > >
> >>>> > > >
> >>>> > > >
> >>>> > > >   * Bare Naked Local File System v0.1.0 doesn't (yet) support
> >>>> symlinks
> >>>> > > >     or the sticky bit.
> >>>> > > >
> >>>> > > ok to not support symlinks. The symlinks of HDFS are not being
> >>>> maintained
> >>>> > > and I am not aware of anything relying on it.
> >>>> > > So I assume people don't need it.
> >>>> > >
> >>>> > > Sticky bit would be useful, I guess.
> >>>> > >
> >>>> > > I suppose folks working at Microsoft would be more interested in
> >>>> this
> >>>> > work?
> >>>> > > Last time I heard, Gautham and Inigo were revamping Hadoop's
> Windows
> >>>> > > support.
> >>>> > >
> >>>> > >
> >>>> > > >   * But the bigger issue is how to excise Winutils completely in
> >>>> the
> >>>> > > >     existing Hadoop code. Winutils assumptions are hard-coded at
> >>>> a low
> >>>> > > >     level across various classes—even code that has nothing to
> do
> >>>> with
> >>>> > > >     the file system. The startup configuration for example calls
> >>>> > > >     `StringUtils.equalsIgnoreCase("true", valueString)` which
> >>>> loads the
> >>>> > > >     `StringUtils` class, which has a static reference to
> `Shell`,
> >>>> which
> >>>> > > >     has a static block that checks for `WINUTILS_EXE`.
> >>>> > > >   * For the most part there should no longer even be a need for
> >>>> > anything
> >>>> > > >     but direct Java API access for the local file system. But
> >>>> muddling
> >>>> > > >     things further, the existing `RawLocalFileSystem`
> >>>> implementation
> >>>> > has
> >>>> > > >     /four/ ways to access the local file system: Winutils, JNI
> >>>> calls,
> >>>> > > >     shell access, and a "new" approach using "stat". The "stat"
> >>>> > approach
> >>>> > > >     has been switched off with a hard-coded
> >>>> `useDeprecatedFileStatus =
> >>>> > > >     true` because of HADOOP-9652
> >>>> > > >     <https://issues.apache.org/jira/browse/HADOOP-9652>.
> >>>> > > >   * Local file access is not contained within
> >>>> `RawLocalFileSystem` but
> >>>> > > >     is scattered across other classes; `FileUtil.readLink()` for
> >>>> > example
> >>>> > > >     (which `RawLocalFileSystem` calls because of the deprecation
> >>>> issue
> >>>> > > >     above) uses the shell approach without any option to change
> >>>> it.
> >>>> > > >     (This implementation-specific decision should have been
> >>>> contained
> >>>> > > >     within the `FileSystem` implementation itself.)
> >>>> > > >
> >>>> > > > In short, it's a mess that has accumulated over years and
> getting
> >>>> > worse,
> >>>> > > > charging high interest on what at first was a small,
> >>>> self-contained
> >>>> > > > technical debt.
> >>>> > > >
> >>>> > > > I would welcome the opportunity to clean up this mess. I'm
> >>>> probably as
> >>>> > > > qualified as anyone to make the changes. This is one of my areas
> >>>> of
> >>>> > > > expertise: I was designing a full abstract file system interface
> >>>> (with
> >>>> > > > pure-Java from-scratch implementations for the local file
> system,
> >>>> > > > Subversion, and WebDAV—even the WebDAV HTTP implementation was
> >>>> from
> >>>> > > > scratch) around the time Apache Nutch was getting off the
> ground.
> >>>> Most
> >>>> > > > recently I've worked on the Hadoop `FileSystem` API contracting
> >>>> for
> >>>> > > > LinkedIn, discovering (what I consider to be) a huge bug in
> >>>> > > > ViewFilesystem, HADOOP-18525
> >>>> > > > <https://issues.apache.org/jira/browse/HADOOP-18525>.
> >>>> > > >
> >>>> > > > The cleanup should be done in several stages (e.g. consolidating
> >>>> > > > WinUtils access; replacing code with pure Java API calls;
> >>>> undeprecating
> >>>> > > > the new Stat code and relegating it to a different class, etc.).
> >>>> > > > Unfortunately it's not financially feasible for me to sit here
> for
> >>>> > > > several months and revamp the Hadoop `FileSystem` subsystem for
> >>>> fun
> >>>> > > > (even though I wish I could). Perhaps there is job opening at a
> >>>> company
> >>>> > > > related to Hadoop that would be interested in hiring me and
> >>>> devoting a
> >>>> > > > certain percentage of my time to fixing local `FileSystem`
> >>>> access. If
> >>>> > > > so, let me know where I should send my resume
> >>>> > > > <https://www.garretwilson.com/about/resume>.
> >>>> > > >
> >>>> > > > Otherwise let me know if any ideas for a way forward. If there
> >>>> proves
> >>>> > to
> >>>> > > > be interest in GlobalMentor Hadoop Bare Naked Local FileSystem
> >>>> > > > <https://github.com/globalmentor/hadoop-bare-naked-local-fs> on
> >>>> GitHub
> >>>> > > > I'll try to maintain and improve it, but really what needs to be
> >>>> > > > revamped is the Hadoop codebase itself. I'll be happy when
> Hadoop
> >>>> is
> >>>> > > > fixed so that both Steve's code and my code are no longer
> needed.
> >>>> > > >
> >>>> > > > Garret
> >>>> > > >
> >>>> > >
> >>>> >
> >>>>
> >>>
>

Re: way forward for Winutils excision from `FileSystem`

Posted by Gautham Banasandra <ga...@apache.org>.
Also, I plan to do a Windows release once I setup the CI for Windows and
after I get
the major unit tests to pass. It would still contain winutils though.
However, we can do
another release after deprecating winutils.

Thanks,
--Gautham

On Mon, 14 Nov 2022 at 23:34, Gautham Banasandra <ga...@apache.org> wrote:

> Hi Iñigo,
>
> I would like to aim for winutils deprecation by the end of the first
> quarter of 2023.
> It really depends on how fast I can wrap up with setting up CI for
> Windows. Given
> that this involves getting Yetus to work properly on Windows, I feel it's
> a bit
> ambitious. But if things fall into place, I think end of the first quarter
> of 2023 would
> be a reachable timeline.
>
> Thanks,
> --Gautham
>
> On Sat, 12 Nov 2022 at 00:20, Iñigo Goiri <el...@gmail.com> wrote:
>
>> Gautham, thank you very much for the summary.
>> Do you have a time-line for when we can get rid of winutils?
>> My idea was to get this and the YARN federation hardening work into a 3.4
>> release.
>>
>>
>>
>> On Fri, Nov 11, 2022, 10:15 Gautham Banasandra <ga...@apache.org>
>> wrote:
>>
>>> Hi folks,
>>>
>>>
>>> What have we done so far?
>>> ------------------------------------
>>> Inigo and I have been working for quite some time now on this topic,
>>> but our efforts have mostly been oriented towards making Hadoop
>>> cross-platform compatible. Our focus has been on streamlining the
>>> process of building Hadoop on Windows so that one can easily
>>> build and run Hadoop, just like on Linux. We reached this milestone
>>> quite recently and I've documented the steps for doing so here -
>>>
>>> https://github.com/apache/hadoop/blob/5bb11cecea136acccac2563b37021b554e517012/BUILDING.txt#L493-L622
>>>
>>>
>>>
>>> Is winutils still required?
>>> -------------------------------
>>> As Steve mentioned, we would still require winutils for running
>>> Hadoop on Windows. The major change here is that winutils
>>> need not come from a third-party repository anymore, rather it
>>> gets built along with the Hadoop codebase itself henceforth.
>>> However, I agree that we need to deprecate winutils and
>>> replace it with something better so that Hadoop users can have
>>> a smoother experience.
>>>
>>>
>>> What's the best way to deprecate winutils?
>>> --------------------------------------------------------
>>> Over all the time that I've spent making Hadoop cross-platform
>>> compatible, I've realized that the best way would be to have a
>>> JNI interface that wraps around a native layer. This native layer
>>> could be implemented majorly in C++. C++17 provides the
>>> std::filesystem namespace that can satisfy most of the native
>>> filesystem API requirements. Since std::filesystem is part of "The
>>> Standard Libray", these APIs will be present on most/all the C++
>>> compilers of the various OS platforms. For those parts that can't
>>> be satisfied by std::filesystem, we'll have to delve into this part
>>> by writing C code that makes system calls. Please note that
>>> these C files will need to be implemented specifically for each
>>> platform. I took this approach when I wrote x-platform library to
>>> make HDFS native client cross-platform compatible -
>>>
>>> https://github.com/apache/hadoop/tree/trunk/hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/lib/x-platform
>>>
>>>
>>> What am I focussing on currently?
>>> ------------------------------------------------
>>> So far, I've focussed on getting the build to work seamlessly
>>> on Windows. I'm now trying to protect this from breaking by
>>> setting up CI on Jenkins that builds Hadoop on Windows
>>> for the precommit validation -
>>> https://issues.apache.org/jira/browse/INFRA-23809
>>> Yes, it does involve getting
>>> Yetus to run on Windows. I can work on deprecating winutils
>>> after this.
>>>
>>> Thanks,
>>> --Gautham
>>>
>>> On Fri, 11 Nov 2022 at 19:51, Steve Loughran <st...@cloudera.com.invalid>
>>> wrote:
>>>
>>>> It's time to reach for the axe.
>>>>
>>>> We haven't shipped eight version of Apache hadoop which builds and runs
>>>> on
>>>> windows for a long long time. I the only people trying to use the
>>>> library
>>>> is on windows Will have been people trying to use spark on their laptops
>>>> with "small" dataset of only a are few tens of gigabytes at a time, the
>>>> kind of work where 32GB of ram and 16 cores is enough. Put differently:
>>>> in
>>>> storage and performance of Single laptop means that it is perfectly
>>>> suitable for doing reasonable amounts of work and the main barrier to
>>>> doing
>>>> so is getting a copy of the winutils lib.
>>>>
>>>> I know Gautham and Inigo I trying to get windows to work as a location
>>>> for
>>>> yarn again; not sure about hdfs. And there, yes, we have to say "they
>>>> likely to need an extra binary"
>>>>
>>>> But for someone wanting to count the number of rows in an avro file? do
>>>> a
>>>> simple bit of filtering on some parquet data? Is these are the kind of
>>>> things that anyone with a linux/mac laptop can do with ease and it is
>>>> not
>>>> fair to put suffering on to others. And well we could just say "why do
>>>> you
>>>> just install Lynnox on that laptop then?", I have someone who has had a
>>>> Linux laptop for many years I know the written strong arguments against
>>>> it
>>>> even beyond the "my employer demand windows with their IT software" as
>>>> "a
>>>> latop which comes out of sleep reliably" is kind of important too.
>>>>
>>>> I how can we let the people who have to live in this world – And we have
>>>> someone who is clearly willing to help –Live a better life. Funnily
>>>> enough,
>>>> the fact that we have not shipped a working version of when you tails
>>>> for a
>>>> long time actually gives us an advantage: we can pick incompatible
>>>> changes
>>>> and be confident that most people aren't going to notice.
>>>>
>>>> I think a good first step would be for Shell to work well if winutils
>>>> isn't
>>>> around -get rid of that static, WINUTILS string and path/file
>>>> equivalents,
>>>> the ones deprecated in 2015. We can rip them out knowing no external
>>>> code
>>>> is using them.
>>>>
>>>> Then we should look very closely at FileUtil to see how much of that is
>>>> needed and how can we isolate it better. If you look at the change log
>>>> of
>>>> that file, we do have to consider that every time it execs a shell
>>>> command
>>>> I there's a security risk and more than once we've had to fix it. Not
>>>> executing any external shell commands is good everywhere.
>>>>
>>>>
>>>>
>>>>
>>>> On Thu, 10 Nov 2022 at 19:00, Chris Nauroth <cn...@apache.org>
>>>> wrote:
>>>>
>>>> > Symlink support on the local file system is still used. One example I
>>>> can
>>>> > think of is YARN container launch [1].
>>>> >
>>>> > I would welcome removal of winutils, as already described in various
>>>> JIRA
>>>> > issues. I think the biggest challenge we'll have is testing of a
>>>> transition
>>>> > from winutils to the newer Java APIs. The contract tests help, but
>>>> > historically there was also a tendency to break things in downstream
>>>> > dependent projects.
>>>> >
>>>> > I'd suggest taking this on piecemeal, transitioning small pieces of
>>>> > FileSystem off of winutils one at a time.
>>>> >
>>>> > [1]
>>>> >
>>>> >
>>>> https://github.com/apache/hadoop/blob/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/launcher/ContainerLaunch.java#L1508-L1509
>>>> >
>>>> > Chris Nauroth
>>>> >
>>>> >
>>>> > On Thu, Nov 10, 2022 at 10:33 AM Wei-Chiu Chuang <we...@apache.org>
>>>> > wrote:
>>>> >
>>>> > > >
>>>> > > >
>>>> > > >
>>>> > > >   * Bare Naked Local File System v0.1.0 doesn't (yet) support
>>>> symlinks
>>>> > > >     or the sticky bit.
>>>> > > >
>>>> > > ok to not support symlinks. The symlinks of HDFS are not being
>>>> maintained
>>>> > > and I am not aware of anything relying on it.
>>>> > > So I assume people don't need it.
>>>> > >
>>>> > > Sticky bit would be useful, I guess.
>>>> > >
>>>> > > I suppose folks working at Microsoft would be more interested in
>>>> this
>>>> > work?
>>>> > > Last time I heard, Gautham and Inigo were revamping Hadoop's Windows
>>>> > > support.
>>>> > >
>>>> > >
>>>> > > >   * But the bigger issue is how to excise Winutils completely in
>>>> the
>>>> > > >     existing Hadoop code. Winutils assumptions are hard-coded at
>>>> a low
>>>> > > >     level across various classes—even code that has nothing to do
>>>> with
>>>> > > >     the file system. The startup configuration for example calls
>>>> > > >     `StringUtils.equalsIgnoreCase("true", valueString)` which
>>>> loads the
>>>> > > >     `StringUtils` class, which has a static reference to `Shell`,
>>>> which
>>>> > > >     has a static block that checks for `WINUTILS_EXE`.
>>>> > > >   * For the most part there should no longer even be a need for
>>>> > anything
>>>> > > >     but direct Java API access for the local file system. But
>>>> muddling
>>>> > > >     things further, the existing `RawLocalFileSystem`
>>>> implementation
>>>> > has
>>>> > > >     /four/ ways to access the local file system: Winutils, JNI
>>>> calls,
>>>> > > >     shell access, and a "new" approach using "stat". The "stat"
>>>> > approach
>>>> > > >     has been switched off with a hard-coded
>>>> `useDeprecatedFileStatus =
>>>> > > >     true` because of HADOOP-9652
>>>> > > >     <https://issues.apache.org/jira/browse/HADOOP-9652>.
>>>> > > >   * Local file access is not contained within
>>>> `RawLocalFileSystem` but
>>>> > > >     is scattered across other classes; `FileUtil.readLink()` for
>>>> > example
>>>> > > >     (which `RawLocalFileSystem` calls because of the deprecation
>>>> issue
>>>> > > >     above) uses the shell approach without any option to change
>>>> it.
>>>> > > >     (This implementation-specific decision should have been
>>>> contained
>>>> > > >     within the `FileSystem` implementation itself.)
>>>> > > >
>>>> > > > In short, it's a mess that has accumulated over years and getting
>>>> > worse,
>>>> > > > charging high interest on what at first was a small,
>>>> self-contained
>>>> > > > technical debt.
>>>> > > >
>>>> > > > I would welcome the opportunity to clean up this mess. I'm
>>>> probably as
>>>> > > > qualified as anyone to make the changes. This is one of my areas
>>>> of
>>>> > > > expertise: I was designing a full abstract file system interface
>>>> (with
>>>> > > > pure-Java from-scratch implementations for the local file system,
>>>> > > > Subversion, and WebDAV—even the WebDAV HTTP implementation was
>>>> from
>>>> > > > scratch) around the time Apache Nutch was getting off the ground.
>>>> Most
>>>> > > > recently I've worked on the Hadoop `FileSystem` API contracting
>>>> for
>>>> > > > LinkedIn, discovering (what I consider to be) a huge bug in
>>>> > > > ViewFilesystem, HADOOP-18525
>>>> > > > <https://issues.apache.org/jira/browse/HADOOP-18525>.
>>>> > > >
>>>> > > > The cleanup should be done in several stages (e.g. consolidating
>>>> > > > WinUtils access; replacing code with pure Java API calls;
>>>> undeprecating
>>>> > > > the new Stat code and relegating it to a different class, etc.).
>>>> > > > Unfortunately it's not financially feasible for me to sit here for
>>>> > > > several months and revamp the Hadoop `FileSystem` subsystem for
>>>> fun
>>>> > > > (even though I wish I could). Perhaps there is job opening at a
>>>> company
>>>> > > > related to Hadoop that would be interested in hiring me and
>>>> devoting a
>>>> > > > certain percentage of my time to fixing local `FileSystem`
>>>> access. If
>>>> > > > so, let me know where I should send my resume
>>>> > > > <https://www.garretwilson.com/about/resume>.
>>>> > > >
>>>> > > > Otherwise let me know if any ideas for a way forward. If there
>>>> proves
>>>> > to
>>>> > > > be interest in GlobalMentor Hadoop Bare Naked Local FileSystem
>>>> > > > <https://github.com/globalmentor/hadoop-bare-naked-local-fs> on
>>>> GitHub
>>>> > > > I'll try to maintain and improve it, but really what needs to be
>>>> > > > revamped is the Hadoop codebase itself. I'll be happy when Hadoop
>>>> is
>>>> > > > fixed so that both Steve's code and my code are no longer needed.
>>>> > > >
>>>> > > > Garret
>>>> > > >
>>>> > >
>>>> >
>>>>
>>>

Re: way forward for Winutils excision from `FileSystem`

Posted by Steve Loughran <st...@cloudera.com.INVALID>.
ok, so let's start: what the easy bits we do to make life easier for people
trying to use the hadoop file:// client on a windows box, which essentially
means "people trying to use spark standalone?"

torg.apache.hadoop.util.Shell is the obvious first target. i had a go at
getting it fail more meaningfully a while back, but it still trouble.


   1. there shouldn't be any need for static initialisation
   2. anything we tagged as deprecated can be cut now

maybe the whole static winutils path can be replaced with some
WinutilsBinding which you ask shell to create or fail with; we'd then
modify our internal uses for it

We can be 100% confident that no external code is using this stuff because
if they were, they'd be complaining about the lack of windows support.

On Thu, 1 Dec 2022 at 05:06, larry mccay <lm...@apache.org> wrote:

> As Chris mentioned earlier, it would be wise to do this in pieces that can
> be reviewed properly.
> Bringing large refactorings in all at once, as Garret mentioned, is not
> likely to just get a +1.
>
> We do have a feature branch process and criteria and we could determine
> specific criteria for such a critical piece of code going through such
> changes.
>
> The question of funding strikes me as odd on this list - though I
> understand the motivation and relevance to you, for sure.
> Feel free to look me up on LI and if there are contacts or anything that I
> can help with let me know.
>
> In the meantime, I would think this thread and list should discuss
> solutions to problems and our ability to get there with reasonable effort
> and practices that we are familiar with.
>
> On Wed, Nov 30, 2022 at 12:46 PM Garret Wilson <ga...@globalmentor.com>
> wrote:
>
> > On 11/29/2022 8:16 AM, Gautham Banasandra wrote:
> > > …
> > > However, I don't see anyone stopping you from working on removing
> > > winutils. I encourage you to put across a PR and I would be glad to
> > review
> > > the same.
> >
> > That's not how it works. This is an intense undertaking. If I spend six
> > months with no income, just rewriting all the native `FileSystem`
> > implementations, and you simply gave thumbs up to the PR, then yay,
> > Apache would integrate my changes into the codebase? I hardly think so.
> > There has to be official buy-in across the group and authorization to
> > make such extensive changes. It's naive to say, "oh, just go rewrite it,
> > and I'll review it and then it will be done".
> >
> > However I am interested in who funds your work. Do you work on Apache
> > for free in your extra time? Or does some corporation pay you? If the
> > latter, I'll be happy to submit my resume to them, so that they can fund
> > me as well and I can start the work immediately. But as I've mentioned a
> > couple of times already, financially I cannot justify sitting here
> > rewriting Hadoop file systems without any income. If you find a creative
> > way for it to be financially viable for me, I would love to do it.
> > > One question I've is - how will you validate that your changes work
> fine
> > > and don't regress the existing functionality, given that we don't yet
> > have
> > > a CI for Hadoop on Windows?
> >
> > It's tempting to start to give you a detailed answer here, because it's
> > a legitimate question. The more general answer is that we would discuss
> > and form a plan with the group; you'll likely find that 1) the existing
> > code doesn't even have sufficient tests, and 2) the existing API isn't
> > even sufficiently documented. But  your question was formulated in a way
> > completely different than I conceptualize the issue. What I would be
> > writing would be a completely native Java implementation of
> > `FileSystem`. The tests accordingly should be written agnostic to the
> > platform. If the tests run on Linux, they will run on Windows; if not,
> > we need to file a bug against the JDK. I'm not even thinking in terms of
> > a "CI for Hadoop for Windows". I just want to build the Java project,
> > whether I'm running on Mac or Linux or Windows or whatever. (That was
> > the point of my wanting to get rid of Winutils to begin with.)
> >
> > I also know that pragmatically whatever I do with the `FileSystem`
> > implementation, something will initially break—not because of anything I
> > did incorrectly, but because the Hadoop API is inadequate and people
> > have therefore made a thousand brittle assumptions in their use of the
> > API. Things will break already with or without my `FileSystem`
> > implementation; that's why Hadoop is still using
> > `DeprecatedRawLocalFileStatus`: someone made a new version but had to
> > switch it off because something broke (HADOOP-9652
> > <https://issues.apache.org/jira/browse/HADOOP-9652> according to the
> > comments).
> >
> > In summary, yes, if I ever get buy-in and funding to rewrite
> > `FileSystem` for native Java, we need to have a discussion with the
> > wider group to form a plan for improving the documentation and for
> > testing. But whatever discussion or plan we do, things will eventually
> > break because Hadoop doesn't have a well-documented API and doesn't
> > cleanly separate the interface from the implementation. If I were to
> > work on it, I would improve that situation so that things would be
> > better documented and less brittle.
> >
> > In the meantime my Bare Naked Local FileSystem
> > <https://github.com/globalmentor/hadoop-bare-naked-local-fs> is meeting
> > my needs pragmatically, and I'm leaving this mailing list—not to be
> > antisocial, but because the unrelated (mostly automated) chatter is
> > distracting to my other work.
> >
> > Have a wonderful holiday season, and feel free to reach out directly.
> >
> > Best,
> >
> > Garret
> >
>

Re: way forward for Winutils excision from `FileSystem`

Posted by larry mccay <lm...@apache.org>.
As Chris mentioned earlier, it would be wise to do this in pieces that can
be reviewed properly.
Bringing large refactorings in all at once, as Garret mentioned, is not
likely to just get a +1.

We do have a feature branch process and criteria and we could determine
specific criteria for such a critical piece of code going through such
changes.

The question of funding strikes me as odd on this list - though I
understand the motivation and relevance to you, for sure.
Feel free to look me up on LI and if there are contacts or anything that I
can help with let me know.

In the meantime, I would think this thread and list should discuss
solutions to problems and our ability to get there with reasonable effort
and practices that we are familiar with.

On Wed, Nov 30, 2022 at 12:46 PM Garret Wilson <ga...@globalmentor.com>
wrote:

> On 11/29/2022 8:16 AM, Gautham Banasandra wrote:
> > …
> > However, I don't see anyone stopping you from working on removing
> > winutils. I encourage you to put across a PR and I would be glad to
> review
> > the same.
>
> That's not how it works. This is an intense undertaking. If I spend six
> months with no income, just rewriting all the native `FileSystem`
> implementations, and you simply gave thumbs up to the PR, then yay,
> Apache would integrate my changes into the codebase? I hardly think so.
> There has to be official buy-in across the group and authorization to
> make such extensive changes. It's naive to say, "oh, just go rewrite it,
> and I'll review it and then it will be done".
>
> However I am interested in who funds your work. Do you work on Apache
> for free in your extra time? Or does some corporation pay you? If the
> latter, I'll be happy to submit my resume to them, so that they can fund
> me as well and I can start the work immediately. But as I've mentioned a
> couple of times already, financially I cannot justify sitting here
> rewriting Hadoop file systems without any income. If you find a creative
> way for it to be financially viable for me, I would love to do it.
> > One question I've is - how will you validate that your changes work fine
> > and don't regress the existing functionality, given that we don't yet
> have
> > a CI for Hadoop on Windows?
>
> It's tempting to start to give you a detailed answer here, because it's
> a legitimate question. The more general answer is that we would discuss
> and form a plan with the group; you'll likely find that 1) the existing
> code doesn't even have sufficient tests, and 2) the existing API isn't
> even sufficiently documented. But  your question was formulated in a way
> completely different than I conceptualize the issue. What I would be
> writing would be a completely native Java implementation of
> `FileSystem`. The tests accordingly should be written agnostic to the
> platform. If the tests run on Linux, they will run on Windows; if not,
> we need to file a bug against the JDK. I'm not even thinking in terms of
> a "CI for Hadoop for Windows". I just want to build the Java project,
> whether I'm running on Mac or Linux or Windows or whatever. (That was
> the point of my wanting to get rid of Winutils to begin with.)
>
> I also know that pragmatically whatever I do with the `FileSystem`
> implementation, something will initially break—not because of anything I
> did incorrectly, but because the Hadoop API is inadequate and people
> have therefore made a thousand brittle assumptions in their use of the
> API. Things will break already with or without my `FileSystem`
> implementation; that's why Hadoop is still using
> `DeprecatedRawLocalFileStatus`: someone made a new version but had to
> switch it off because something broke (HADOOP-9652
> <https://issues.apache.org/jira/browse/HADOOP-9652> according to the
> comments).
>
> In summary, yes, if I ever get buy-in and funding to rewrite
> `FileSystem` for native Java, we need to have a discussion with the
> wider group to form a plan for improving the documentation and for
> testing. But whatever discussion or plan we do, things will eventually
> break because Hadoop doesn't have a well-documented API and doesn't
> cleanly separate the interface from the implementation. If I were to
> work on it, I would improve that situation so that things would be
> better documented and less brittle.
>
> In the meantime my Bare Naked Local FileSystem
> <https://github.com/globalmentor/hadoop-bare-naked-local-fs> is meeting
> my needs pragmatically, and I'm leaving this mailing list—not to be
> antisocial, but because the unrelated (mostly automated) chatter is
> distracting to my other work.
>
> Have a wonderful holiday season, and feel free to reach out directly.
>
> Best,
>
> Garret
>

Re: way forward for Winutils excision from `FileSystem`

Posted by Garret Wilson <ga...@globalmentor.com>.
On 11/29/2022 8:16 AM, Gautham Banasandra wrote:
> …
> However, I don't see anyone stopping you from working on removing
> winutils. I encourage you to put across a PR and I would be glad to review
> the same.

That's not how it works. This is an intense undertaking. If I spend six 
months with no income, just rewriting all the native `FileSystem` 
implementations, and you simply gave thumbs up to the PR, then yay, 
Apache would integrate my changes into the codebase? I hardly think so. 
There has to be official buy-in across the group and authorization to 
make such extensive changes. It's naive to say, "oh, just go rewrite it, 
and I'll review it and then it will be done".

However I am interested in who funds your work. Do you work on Apache 
for free in your extra time? Or does some corporation pay you? If the 
latter, I'll be happy to submit my resume to them, so that they can fund 
me as well and I can start the work immediately. But as I've mentioned a 
couple of times already, financially I cannot justify sitting here 
rewriting Hadoop file systems without any income. If you find a creative 
way for it to be financially viable for me, I would love to do it.
> One question I've is - how will you validate that your changes work fine
> and don't regress the existing functionality, given that we don't yet have
> a CI for Hadoop on Windows?

It's tempting to start to give you a detailed answer here, because it's 
a legitimate question. The more general answer is that we would discuss 
and form a plan with the group; you'll likely find that 1) the existing 
code doesn't even have sufficient tests, and 2) the existing API isn't 
even sufficiently documented. But  your question was formulated in a way 
completely different than I conceptualize the issue. What I would be 
writing would be a completely native Java implementation of 
`FileSystem`. The tests accordingly should be written agnostic to the 
platform. If the tests run on Linux, they will run on Windows; if not, 
we need to file a bug against the JDK. I'm not even thinking in terms of 
a "CI for Hadoop for Windows". I just want to build the Java project, 
whether I'm running on Mac or Linux or Windows or whatever. (That was 
the point of my wanting to get rid of Winutils to begin with.)

I also know that pragmatically whatever I do with the `FileSystem` 
implementation, something will initially break—not because of anything I 
did incorrectly, but because the Hadoop API is inadequate and people 
have therefore made a thousand brittle assumptions in their use of the 
API. Things will break already with or without my `FileSystem` 
implementation; that's why Hadoop is still using 
`DeprecatedRawLocalFileStatus`: someone made a new version but had to 
switch it off because something broke (HADOOP-9652 
<https://issues.apache.org/jira/browse/HADOOP-9652> according to the 
comments).

In summary, yes, if I ever get buy-in and funding to rewrite 
`FileSystem` for native Java, we need to have a discussion with the 
wider group to form a plan for improving the documentation and for 
testing. But whatever discussion or plan we do, things will eventually 
break because Hadoop doesn't have a well-documented API and doesn't 
cleanly separate the interface from the implementation. If I were to 
work on it, I would improve that situation so that things would be 
better documented and less brittle.

In the meantime my Bare Naked Local FileSystem 
<https://github.com/globalmentor/hadoop-bare-naked-local-fs> is meeting 
my needs pragmatically, and I'm leaving this mailing list—not to be 
antisocial, but because the unrelated (mostly automated) chatter is 
distracting to my other work.

Have a wonderful holiday season, and feel free to reach out directly.

Best,

Garret

Re: way forward for Winutils excision from `FileSystem`

Posted by Gautham Banasandra <ga...@apache.org>.
Hi Garret,

My responses hitherto indicate that I'm currently focusing on setting up
a CI for Windows for building Hadoop and getting the unit tests to run on
it.
I believe, without these, no matter the improvements, the build is bound
to be broken someday. I would thus like to prioritize setting up of the CI
over
deprecation of winutils.

However, I don't see anyone stopping you from working on removing
winutils. I encourage you to put across a PR and I would be glad to review
the same.

One question I've is - how will you validate that your changes work fine
and don't regress the existing functionality, given that we don't yet have
a CI for Hadoop on Windows?

Thanks,
--Gautham

On Mon, 28 Nov 2022 at 23:42, Garret Wilson <ga...@globalmentor.com> wrote:

> On 11/14/2022 12:04 PM, Gautham Banasandra wrote:
>
> …
> I would like to aim for winutils deprecation by the end of the first
> quarter of 2023.
>
> The key word is "deprecation". Note that the sentence doesn't say the
> references in the code to Winutils will be removed by this time. I offered
> to remove it immediately. (What's the point of "deprecating" it when it's
> an an extra step you're forcing people to take to begin with? Just rip out
> the references.)
>
> … I feel it's a bit
> ambitious. But if things fall into place …
>
> Hardly anything ever "falls into place" the way it is expected.
>
> Peeps, I'm going to leave this list for the moment because it's bringing a
> lot of clutter to my inbox. Let me just summarize:
>
>    - I pointed out that Winutils, however much it was a necessary evil at
>    the time, needs to go, and go now. I implemented a completely working first
>    version wrapper. I offered to finish the work by revamping the actual
>    Apache code.
>    - Many people said, "Oh, we're already working on that; hold our beer
>    [that's an expression, for those who may not know] and soon it will be
>    done!"
>    - As the months and years roll on and everything remains as it is
>    (I'll be happy if you prove me wrong), don't forget about me and feel free
>    to reach out. I'll probably still be eager to fix the Apache code itself it
>    if someone can figure out a way that I won't have to do it for free and
>    live half a year off savings to do it.
>
> Best,
>
> Garret
>

Re: way forward for Winutils excision from `FileSystem`

Posted by Gautham Banasandra <ga...@apache.org>.
Hi Steve,

I feel we won't have the confidence with major changes until we have
a CI that runs the full suite of tests.

Thanks,
--Gautham

On Tue, 29 Nov 2022 at 16:12, Steve Loughran <st...@cloudera.com> wrote:

>  he's got a point.
>
> what is to stop us cutting it from trunk?
>
> On Mon, 28 Nov 2022 at 18:12, Garret Wilson <ga...@globalmentor.com>
> wrote:
>
>> On 11/14/2022 12:04 PM, Gautham Banasandra wrote:
>>
>> …
>> I would like to aim for winutils deprecation by the end of the first
>> quarter of 2023.
>>
>> The key word is "deprecation". Note that the sentence doesn't say the
>> references in the code to Winutils will be removed by this time. I offered
>> to remove it immediately. (What's the point of "deprecating" it when it's
>> an an extra step you're forcing people to take to begin with? Just rip out
>> the references.)
>>
>> … I feel it's a bit
>> ambitious. But if things fall into place …
>>
>> Hardly anything ever "falls into place" the way it is expected.
>>
>> Peeps, I'm going to leave this list for the moment because it's bringing
>> a lot of clutter to my inbox. Let me just summarize:
>>
>>    - I pointed out that Winutils, however much it was a necessary evil
>>    at the time, needs to go, and go now. I implemented a completely working
>>    first version wrapper. I offered to finish the work by revamping the actual
>>    Apache code.
>>    - Many people said, "Oh, we're already working on that; hold our beer
>>    [that's an expression, for those who may not know] and soon it will be
>>    done!"
>>    - As the months and years roll on and everything remains as it is
>>    (I'll be happy if you prove me wrong), don't forget about me and feel free
>>    to reach out. I'll probably still be eager to fix the Apache code itself it
>>    if someone can figure out a way that I won't have to do it for free and
>>    live half a year off savings to do it.
>>
>> Best,
>>
>> Garret
>>
>

Re: way forward for Winutils excision from `FileSystem`

Posted by Steve Loughran <st...@cloudera.com.INVALID>.
 he's got a point.

what is to stop us cutting it from trunk?

On Mon, 28 Nov 2022 at 18:12, Garret Wilson <ga...@globalmentor.com> wrote:

> On 11/14/2022 12:04 PM, Gautham Banasandra wrote:
>
> …
> I would like to aim for winutils deprecation by the end of the first
> quarter of 2023.
>
> The key word is "deprecation". Note that the sentence doesn't say the
> references in the code to Winutils will be removed by this time. I offered
> to remove it immediately. (What's the point of "deprecating" it when it's
> an an extra step you're forcing people to take to begin with? Just rip out
> the references.)
>
> … I feel it's a bit
> ambitious. But if things fall into place …
>
> Hardly anything ever "falls into place" the way it is expected.
>
> Peeps, I'm going to leave this list for the moment because it's bringing a
> lot of clutter to my inbox. Let me just summarize:
>
>    - I pointed out that Winutils, however much it was a necessary evil at
>    the time, needs to go, and go now. I implemented a completely working first
>    version wrapper. I offered to finish the work by revamping the actual
>    Apache code.
>    - Many people said, "Oh, we're already working on that; hold our beer
>    [that's an expression, for those who may not know] and soon it will be
>    done!"
>    - As the months and years roll on and everything remains as it is
>    (I'll be happy if you prove me wrong), don't forget about me and feel free
>    to reach out. I'll probably still be eager to fix the Apache code itself it
>    if someone can figure out a way that I won't have to do it for free and
>    live half a year off savings to do it.
>
> Best,
>
> Garret
>

Re: way forward for Winutils excision from `FileSystem`

Posted by Garret Wilson <ga...@globalmentor.com>.
On 11/14/2022 12:04 PM, Gautham Banasandra wrote:
> …
> I would like to aim for winutils deprecation by the end of the first 
> quarter of 2023.

The key word is "deprecation". Note that the sentence doesn't say the 
references in the code to Winutils will be removed by this time. I 
offered to remove it immediately. (What's the point of "deprecating" it 
when it's an an extra step you're forcing people to take to begin with? 
Just rip out the references.)

> … I feel it's a bit
> ambitious. But if things fall into place …

Hardly anything ever "falls into place" the way it is expected.

Peeps, I'm going to leave this list for the moment because it's bringing 
a lot of clutter to my inbox. Let me just summarize:

  * I pointed out that Winutils, however much it was a necessary evil at
    the time, needs to go, and go now. I implemented a completely
    working first version wrapper. I offered to finish the work by
    revamping the actual Apache code.
  * Many people said, "Oh, we're already working on that; hold our beer
    [that's an expression, for those who may not know] and soon it will
    be done!"
  * As the months and years roll on and everything remains as it is
    (I'll be happy if you prove me wrong), don't forget about me and
    feel free to reach out. I'll probably still be eager to fix the
    Apache code itself it if someone can figure out a way that I won't
    have to do it for free and live half a year off savings to do it.

Best,

Garret

Re: way forward for Winutils excision from `FileSystem`

Posted by Gautham Banasandra <ga...@apache.org>.
Hi Iñigo,

I would like to aim for winutils deprecation by the end of the first
quarter of 2023.
It really depends on how fast I can wrap up with setting up CI for Windows.
Given
that this involves getting Yetus to work properly on Windows, I feel it's a
bit
ambitious. But if things fall into place, I think end of the first quarter
of 2023 would
be a reachable timeline.

Thanks,
--Gautham

On Sat, 12 Nov 2022 at 00:20, Iñigo Goiri <el...@gmail.com> wrote:

> Gautham, thank you very much for the summary.
> Do you have a time-line for when we can get rid of winutils?
> My idea was to get this and the YARN federation hardening work into a 3.4
> release.
>
>
>
> On Fri, Nov 11, 2022, 10:15 Gautham Banasandra <ga...@apache.org> wrote:
>
>> Hi folks,
>>
>>
>> What have we done so far?
>> ------------------------------------
>> Inigo and I have been working for quite some time now on this topic,
>> but our efforts have mostly been oriented towards making Hadoop
>> cross-platform compatible. Our focus has been on streamlining the
>> process of building Hadoop on Windows so that one can easily
>> build and run Hadoop, just like on Linux. We reached this milestone
>> quite recently and I've documented the steps for doing so here -
>>
>> https://github.com/apache/hadoop/blob/5bb11cecea136acccac2563b37021b554e517012/BUILDING.txt#L493-L622
>>
>>
>>
>> Is winutils still required?
>> -------------------------------
>> As Steve mentioned, we would still require winutils for running
>> Hadoop on Windows. The major change here is that winutils
>> need not come from a third-party repository anymore, rather it
>> gets built along with the Hadoop codebase itself henceforth.
>> However, I agree that we need to deprecate winutils and
>> replace it with something better so that Hadoop users can have
>> a smoother experience.
>>
>>
>> What's the best way to deprecate winutils?
>> --------------------------------------------------------
>> Over all the time that I've spent making Hadoop cross-platform
>> compatible, I've realized that the best way would be to have a
>> JNI interface that wraps around a native layer. This native layer
>> could be implemented majorly in C++. C++17 provides the
>> std::filesystem namespace that can satisfy most of the native
>> filesystem API requirements. Since std::filesystem is part of "The
>> Standard Libray", these APIs will be present on most/all the C++
>> compilers of the various OS platforms. For those parts that can't
>> be satisfied by std::filesystem, we'll have to delve into this part
>> by writing C code that makes system calls. Please note that
>> these C files will need to be implemented specifically for each
>> platform. I took this approach when I wrote x-platform library to
>> make HDFS native client cross-platform compatible -
>>
>> https://github.com/apache/hadoop/tree/trunk/hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/lib/x-platform
>>
>>
>> What am I focussing on currently?
>> ------------------------------------------------
>> So far, I've focussed on getting the build to work seamlessly
>> on Windows. I'm now trying to protect this from breaking by
>> setting up CI on Jenkins that builds Hadoop on Windows
>> for the precommit validation -
>> https://issues.apache.org/jira/browse/INFRA-23809
>> Yes, it does involve getting
>> Yetus to run on Windows. I can work on deprecating winutils
>> after this.
>>
>> Thanks,
>> --Gautham
>>
>> On Fri, 11 Nov 2022 at 19:51, Steve Loughran <st...@cloudera.com.invalid>
>> wrote:
>>
>>> It's time to reach for the axe.
>>>
>>> We haven't shipped eight version of Apache hadoop which builds and runs
>>> on
>>> windows for a long long time. I the only people trying to use the library
>>> is on windows Will have been people trying to use spark on their laptops
>>> with "small" dataset of only a are few tens of gigabytes at a time, the
>>> kind of work where 32GB of ram and 16 cores is enough. Put differently:
>>> in
>>> storage and performance of Single laptop means that it is perfectly
>>> suitable for doing reasonable amounts of work and the main barrier to
>>> doing
>>> so is getting a copy of the winutils lib.
>>>
>>> I know Gautham and Inigo I trying to get windows to work as a location
>>> for
>>> yarn again; not sure about hdfs. And there, yes, we have to say "they
>>> likely to need an extra binary"
>>>
>>> But for someone wanting to count the number of rows in an avro file? do a
>>> simple bit of filtering on some parquet data? Is these are the kind of
>>> things that anyone with a linux/mac laptop can do with ease and it is not
>>> fair to put suffering on to others. And well we could just say "why do
>>> you
>>> just install Lynnox on that laptop then?", I have someone who has had a
>>> Linux laptop for many years I know the written strong arguments against
>>> it
>>> even beyond the "my employer demand windows with their IT software" as "a
>>> latop which comes out of sleep reliably" is kind of important too.
>>>
>>> I how can we let the people who have to live in this world – And we have
>>> someone who is clearly willing to help –Live a better life. Funnily
>>> enough,
>>> the fact that we have not shipped a working version of when you tails
>>> for a
>>> long time actually gives us an advantage: we can pick incompatible
>>> changes
>>> and be confident that most people aren't going to notice.
>>>
>>> I think a good first step would be for Shell to work well if winutils
>>> isn't
>>> around -get rid of that static, WINUTILS string and path/file
>>> equivalents,
>>> the ones deprecated in 2015. We can rip them out knowing no external code
>>> is using them.
>>>
>>> Then we should look very closely at FileUtil to see how much of that is
>>> needed and how can we isolate it better. If you look at the change log of
>>> that file, we do have to consider that every time it execs a shell
>>> command
>>> I there's a security risk and more than once we've had to fix it. Not
>>> executing any external shell commands is good everywhere.
>>>
>>>
>>>
>>>
>>> On Thu, 10 Nov 2022 at 19:00, Chris Nauroth <cn...@apache.org> wrote:
>>>
>>> > Symlink support on the local file system is still used. One example I
>>> can
>>> > think of is YARN container launch [1].
>>> >
>>> > I would welcome removal of winutils, as already described in various
>>> JIRA
>>> > issues. I think the biggest challenge we'll have is testing of a
>>> transition
>>> > from winutils to the newer Java APIs. The contract tests help, but
>>> > historically there was also a tendency to break things in downstream
>>> > dependent projects.
>>> >
>>> > I'd suggest taking this on piecemeal, transitioning small pieces of
>>> > FileSystem off of winutils one at a time.
>>> >
>>> > [1]
>>> >
>>> >
>>> https://github.com/apache/hadoop/blob/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/launcher/ContainerLaunch.java#L1508-L1509
>>> >
>>> > Chris Nauroth
>>> >
>>> >
>>> > On Thu, Nov 10, 2022 at 10:33 AM Wei-Chiu Chuang <we...@apache.org>
>>> > wrote:
>>> >
>>> > > >
>>> > > >
>>> > > >
>>> > > >   * Bare Naked Local File System v0.1.0 doesn't (yet) support
>>> symlinks
>>> > > >     or the sticky bit.
>>> > > >
>>> > > ok to not support symlinks. The symlinks of HDFS are not being
>>> maintained
>>> > > and I am not aware of anything relying on it.
>>> > > So I assume people don't need it.
>>> > >
>>> > > Sticky bit would be useful, I guess.
>>> > >
>>> > > I suppose folks working at Microsoft would be more interested in this
>>> > work?
>>> > > Last time I heard, Gautham and Inigo were revamping Hadoop's Windows
>>> > > support.
>>> > >
>>> > >
>>> > > >   * But the bigger issue is how to excise Winutils completely in
>>> the
>>> > > >     existing Hadoop code. Winutils assumptions are hard-coded at a
>>> low
>>> > > >     level across various classes—even code that has nothing to do
>>> with
>>> > > >     the file system. The startup configuration for example calls
>>> > > >     `StringUtils.equalsIgnoreCase("true", valueString)` which
>>> loads the
>>> > > >     `StringUtils` class, which has a static reference to `Shell`,
>>> which
>>> > > >     has a static block that checks for `WINUTILS_EXE`.
>>> > > >   * For the most part there should no longer even be a need for
>>> > anything
>>> > > >     but direct Java API access for the local file system. But
>>> muddling
>>> > > >     things further, the existing `RawLocalFileSystem`
>>> implementation
>>> > has
>>> > > >     /four/ ways to access the local file system: Winutils, JNI
>>> calls,
>>> > > >     shell access, and a "new" approach using "stat". The "stat"
>>> > approach
>>> > > >     has been switched off with a hard-coded
>>> `useDeprecatedFileStatus =
>>> > > >     true` because of HADOOP-9652
>>> > > >     <https://issues.apache.org/jira/browse/HADOOP-9652>.
>>> > > >   * Local file access is not contained within `RawLocalFileSystem`
>>> but
>>> > > >     is scattered across other classes; `FileUtil.readLink()` for
>>> > example
>>> > > >     (which `RawLocalFileSystem` calls because of the deprecation
>>> issue
>>> > > >     above) uses the shell approach without any option to change it.
>>> > > >     (This implementation-specific decision should have been
>>> contained
>>> > > >     within the `FileSystem` implementation itself.)
>>> > > >
>>> > > > In short, it's a mess that has accumulated over years and getting
>>> > worse,
>>> > > > charging high interest on what at first was a small, self-contained
>>> > > > technical debt.
>>> > > >
>>> > > > I would welcome the opportunity to clean up this mess. I'm
>>> probably as
>>> > > > qualified as anyone to make the changes. This is one of my areas of
>>> > > > expertise: I was designing a full abstract file system interface
>>> (with
>>> > > > pure-Java from-scratch implementations for the local file system,
>>> > > > Subversion, and WebDAV—even the WebDAV HTTP implementation was from
>>> > > > scratch) around the time Apache Nutch was getting off the ground.
>>> Most
>>> > > > recently I've worked on the Hadoop `FileSystem` API contracting for
>>> > > > LinkedIn, discovering (what I consider to be) a huge bug in
>>> > > > ViewFilesystem, HADOOP-18525
>>> > > > <https://issues.apache.org/jira/browse/HADOOP-18525>.
>>> > > >
>>> > > > The cleanup should be done in several stages (e.g. consolidating
>>> > > > WinUtils access; replacing code with pure Java API calls;
>>> undeprecating
>>> > > > the new Stat code and relegating it to a different class, etc.).
>>> > > > Unfortunately it's not financially feasible for me to sit here for
>>> > > > several months and revamp the Hadoop `FileSystem` subsystem for fun
>>> > > > (even though I wish I could). Perhaps there is job opening at a
>>> company
>>> > > > related to Hadoop that would be interested in hiring me and
>>> devoting a
>>> > > > certain percentage of my time to fixing local `FileSystem` access.
>>> If
>>> > > > so, let me know where I should send my resume
>>> > > > <https://www.garretwilson.com/about/resume>.
>>> > > >
>>> > > > Otherwise let me know if any ideas for a way forward. If there
>>> proves
>>> > to
>>> > > > be interest in GlobalMentor Hadoop Bare Naked Local FileSystem
>>> > > > <https://github.com/globalmentor/hadoop-bare-naked-local-fs> on
>>> GitHub
>>> > > > I'll try to maintain and improve it, but really what needs to be
>>> > > > revamped is the Hadoop codebase itself. I'll be happy when Hadoop
>>> is
>>> > > > fixed so that both Steve's code and my code are no longer needed.
>>> > > >
>>> > > > Garret
>>> > > >
>>> > >
>>> >
>>>
>>

Re: way forward for Winutils excision from `FileSystem`

Posted by Chris Nauroth <cn...@apache.org>.
Somewhat related, I'd love to see the overall JNI footprint go down where
possible. For example, HADOOP-17895 proposes switching to the newer JDK
APIs for chmod.

Chris Nauroth


On Fri, Nov 11, 2022 at 10:50 AM Iñigo Goiri <el...@gmail.com> wrote:

> Gautham, thank you very much for the summary.
> Do you have a time-line for when we can get rid of winutils?
> My idea was to get this and the YARN federation hardening work into a 3.4
> release.
>
>
>
> On Fri, Nov 11, 2022, 10:15 Gautham Banasandra <ga...@apache.org> wrote:
>
>> Hi folks,
>>
>>
>> What have we done so far?
>> ------------------------------------
>> Inigo and I have been working for quite some time now on this topic,
>> but our efforts have mostly been oriented towards making Hadoop
>> cross-platform compatible. Our focus has been on streamlining the
>> process of building Hadoop on Windows so that one can easily
>> build and run Hadoop, just like on Linux. We reached this milestone
>> quite recently and I've documented the steps for doing so here -
>>
>> https://github.com/apache/hadoop/blob/5bb11cecea136acccac2563b37021b554e517012/BUILDING.txt#L493-L622
>>
>>
>>
>> Is winutils still required?
>> -------------------------------
>> As Steve mentioned, we would still require winutils for running
>> Hadoop on Windows. The major change here is that winutils
>> need not come from a third-party repository anymore, rather it
>> gets built along with the Hadoop codebase itself henceforth.
>> However, I agree that we need to deprecate winutils and
>> replace it with something better so that Hadoop users can have
>> a smoother experience.
>>
>>
>> What's the best way to deprecate winutils?
>> --------------------------------------------------------
>> Over all the time that I've spent making Hadoop cross-platform
>> compatible, I've realized that the best way would be to have a
>> JNI interface that wraps around a native layer. This native layer
>> could be implemented majorly in C++. C++17 provides the
>> std::filesystem namespace that can satisfy most of the native
>> filesystem API requirements. Since std::filesystem is part of "The
>> Standard Libray", these APIs will be present on most/all the C++
>> compilers of the various OS platforms. For those parts that can't
>> be satisfied by std::filesystem, we'll have to delve into this part
>> by writing C code that makes system calls. Please note that
>> these C files will need to be implemented specifically for each
>> platform. I took this approach when I wrote x-platform library to
>> make HDFS native client cross-platform compatible -
>>
>> https://github.com/apache/hadoop/tree/trunk/hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/lib/x-platform
>>
>>
>> What am I focussing on currently?
>> ------------------------------------------------
>> So far, I've focussed on getting the build to work seamlessly
>> on Windows. I'm now trying to protect this from breaking by
>> setting up CI on Jenkins that builds Hadoop on Windows
>> for the precommit validation -
>> https://issues.apache.org/jira/browse/INFRA-23809
>> Yes, it does involve getting
>> Yetus to run on Windows. I can work on deprecating winutils
>> after this.
>>
>> Thanks,
>> --Gautham
>>
>> On Fri, 11 Nov 2022 at 19:51, Steve Loughran <st...@cloudera.com.invalid>
>> wrote:
>>
>>> It's time to reach for the axe.
>>>
>>> We haven't shipped eight version of Apache hadoop which builds and runs
>>> on
>>> windows for a long long time. I the only people trying to use the library
>>> is on windows Will have been people trying to use spark on their laptops
>>> with "small" dataset of only a are few tens of gigabytes at a time, the
>>> kind of work where 32GB of ram and 16 cores is enough. Put differently:
>>> in
>>> storage and performance of Single laptop means that it is perfectly
>>> suitable for doing reasonable amounts of work and the main barrier to
>>> doing
>>> so is getting a copy of the winutils lib.
>>>
>>> I know Gautham and Inigo I trying to get windows to work as a location
>>> for
>>> yarn again; not sure about hdfs. And there, yes, we have to say "they
>>> likely to need an extra binary"
>>>
>>> But for someone wanting to count the number of rows in an avro file? do a
>>> simple bit of filtering on some parquet data? Is these are the kind of
>>> things that anyone with a linux/mac laptop can do with ease and it is not
>>> fair to put suffering on to others. And well we could just say "why do
>>> you
>>> just install Lynnox on that laptop then?", I have someone who has had a
>>> Linux laptop for many years I know the written strong arguments against
>>> it
>>> even beyond the "my employer demand windows with their IT software" as "a
>>> latop which comes out of sleep reliably" is kind of important too.
>>>
>>> I how can we let the people who have to live in this world – And we have
>>> someone who is clearly willing to help –Live a better life. Funnily
>>> enough,
>>> the fact that we have not shipped a working version of when you tails
>>> for a
>>> long time actually gives us an advantage: we can pick incompatible
>>> changes
>>> and be confident that most people aren't going to notice.
>>>
>>> I think a good first step would be for Shell to work well if winutils
>>> isn't
>>> around -get rid of that static, WINUTILS string and path/file
>>> equivalents,
>>> the ones deprecated in 2015. We can rip them out knowing no external code
>>> is using them.
>>>
>>> Then we should look very closely at FileUtil to see how much of that is
>>> needed and how can we isolate it better. If you look at the change log of
>>> that file, we do have to consider that every time it execs a shell
>>> command
>>> I there's a security risk and more than once we've had to fix it. Not
>>> executing any external shell commands is good everywhere.
>>>
>>>
>>>
>>>
>>> On Thu, 10 Nov 2022 at 19:00, Chris Nauroth <cn...@apache.org> wrote:
>>>
>>> > Symlink support on the local file system is still used. One example I
>>> can
>>> > think of is YARN container launch [1].
>>> >
>>> > I would welcome removal of winutils, as already described in various
>>> JIRA
>>> > issues. I think the biggest challenge we'll have is testing of a
>>> transition
>>> > from winutils to the newer Java APIs. The contract tests help, but
>>> > historically there was also a tendency to break things in downstream
>>> > dependent projects.
>>> >
>>> > I'd suggest taking this on piecemeal, transitioning small pieces of
>>> > FileSystem off of winutils one at a time.
>>> >
>>> > [1]
>>> >
>>> >
>>> https://github.com/apache/hadoop/blob/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/launcher/ContainerLaunch.java#L1508-L1509
>>> >
>>> > Chris Nauroth
>>> >
>>> >
>>> > On Thu, Nov 10, 2022 at 10:33 AM Wei-Chiu Chuang <we...@apache.org>
>>> > wrote:
>>> >
>>> > > >
>>> > > >
>>> > > >
>>> > > >   * Bare Naked Local File System v0.1.0 doesn't (yet) support
>>> symlinks
>>> > > >     or the sticky bit.
>>> > > >
>>> > > ok to not support symlinks. The symlinks of HDFS are not being
>>> maintained
>>> > > and I am not aware of anything relying on it.
>>> > > So I assume people don't need it.
>>> > >
>>> > > Sticky bit would be useful, I guess.
>>> > >
>>> > > I suppose folks working at Microsoft would be more interested in this
>>> > work?
>>> > > Last time I heard, Gautham and Inigo were revamping Hadoop's Windows
>>> > > support.
>>> > >
>>> > >
>>> > > >   * But the bigger issue is how to excise Winutils completely in
>>> the
>>> > > >     existing Hadoop code. Winutils assumptions are hard-coded at a
>>> low
>>> > > >     level across various classes—even code that has nothing to do
>>> with
>>> > > >     the file system. The startup configuration for example calls
>>> > > >     `StringUtils.equalsIgnoreCase("true", valueString)` which
>>> loads the
>>> > > >     `StringUtils` class, which has a static reference to `Shell`,
>>> which
>>> > > >     has a static block that checks for `WINUTILS_EXE`.
>>> > > >   * For the most part there should no longer even be a need for
>>> > anything
>>> > > >     but direct Java API access for the local file system. But
>>> muddling
>>> > > >     things further, the existing `RawLocalFileSystem`
>>> implementation
>>> > has
>>> > > >     /four/ ways to access the local file system: Winutils, JNI
>>> calls,
>>> > > >     shell access, and a "new" approach using "stat". The "stat"
>>> > approach
>>> > > >     has been switched off with a hard-coded
>>> `useDeprecatedFileStatus =
>>> > > >     true` because of HADOOP-9652
>>> > > >     <https://issues.apache.org/jira/browse/HADOOP-9652>.
>>> > > >   * Local file access is not contained within `RawLocalFileSystem`
>>> but
>>> > > >     is scattered across other classes; `FileUtil.readLink()` for
>>> > example
>>> > > >     (which `RawLocalFileSystem` calls because of the deprecation
>>> issue
>>> > > >     above) uses the shell approach without any option to change it.
>>> > > >     (This implementation-specific decision should have been
>>> contained
>>> > > >     within the `FileSystem` implementation itself.)
>>> > > >
>>> > > > In short, it's a mess that has accumulated over years and getting
>>> > worse,
>>> > > > charging high interest on what at first was a small, self-contained
>>> > > > technical debt.
>>> > > >
>>> > > > I would welcome the opportunity to clean up this mess. I'm
>>> probably as
>>> > > > qualified as anyone to make the changes. This is one of my areas of
>>> > > > expertise: I was designing a full abstract file system interface
>>> (with
>>> > > > pure-Java from-scratch implementations for the local file system,
>>> > > > Subversion, and WebDAV—even the WebDAV HTTP implementation was from
>>> > > > scratch) around the time Apache Nutch was getting off the ground.
>>> Most
>>> > > > recently I've worked on the Hadoop `FileSystem` API contracting for
>>> > > > LinkedIn, discovering (what I consider to be) a huge bug in
>>> > > > ViewFilesystem, HADOOP-18525
>>> > > > <https://issues.apache.org/jira/browse/HADOOP-18525>.
>>> > > >
>>> > > > The cleanup should be done in several stages (e.g. consolidating
>>> > > > WinUtils access; replacing code with pure Java API calls;
>>> undeprecating
>>> > > > the new Stat code and relegating it to a different class, etc.).
>>> > > > Unfortunately it's not financially feasible for me to sit here for
>>> > > > several months and revamp the Hadoop `FileSystem` subsystem for fun
>>> > > > (even though I wish I could). Perhaps there is job opening at a
>>> company
>>> > > > related to Hadoop that would be interested in hiring me and
>>> devoting a
>>> > > > certain percentage of my time to fixing local `FileSystem` access.
>>> If
>>> > > > so, let me know where I should send my resume
>>> > > > <https://www.garretwilson.com/about/resume>.
>>> > > >
>>> > > > Otherwise let me know if any ideas for a way forward. If there
>>> proves
>>> > to
>>> > > > be interest in GlobalMentor Hadoop Bare Naked Local FileSystem
>>> > > > <https://github.com/globalmentor/hadoop-bare-naked-local-fs> on
>>> GitHub
>>> > > > I'll try to maintain and improve it, but really what needs to be
>>> > > > revamped is the Hadoop codebase itself. I'll be happy when Hadoop
>>> is
>>> > > > fixed so that both Steve's code and my code are no longer needed.
>>> > > >
>>> > > > Garret
>>> > > >
>>> > >
>>> >
>>>
>>

Re: way forward for Winutils excision from `FileSystem`

Posted by Iñigo Goiri <el...@gmail.com>.
Gautham, thank you very much for the summary.
Do you have a time-line for when we can get rid of winutils?
My idea was to get this and the YARN federation hardening work into a 3.4
release.



On Fri, Nov 11, 2022, 10:15 Gautham Banasandra <ga...@apache.org> wrote:

> Hi folks,
>
>
> What have we done so far?
> ------------------------------------
> Inigo and I have been working for quite some time now on this topic,
> but our efforts have mostly been oriented towards making Hadoop
> cross-platform compatible. Our focus has been on streamlining the
> process of building Hadoop on Windows so that one can easily
> build and run Hadoop, just like on Linux. We reached this milestone
> quite recently and I've documented the steps for doing so here -
>
> https://github.com/apache/hadoop/blob/5bb11cecea136acccac2563b37021b554e517012/BUILDING.txt#L493-L622
>
>
>
> Is winutils still required?
> -------------------------------
> As Steve mentioned, we would still require winutils for running
> Hadoop on Windows. The major change here is that winutils
> need not come from a third-party repository anymore, rather it
> gets built along with the Hadoop codebase itself henceforth.
> However, I agree that we need to deprecate winutils and
> replace it with something better so that Hadoop users can have
> a smoother experience.
>
>
> What's the best way to deprecate winutils?
> --------------------------------------------------------
> Over all the time that I've spent making Hadoop cross-platform
> compatible, I've realized that the best way would be to have a
> JNI interface that wraps around a native layer. This native layer
> could be implemented majorly in C++. C++17 provides the
> std::filesystem namespace that can satisfy most of the native
> filesystem API requirements. Since std::filesystem is part of "The
> Standard Libray", these APIs will be present on most/all the C++
> compilers of the various OS platforms. For those parts that can't
> be satisfied by std::filesystem, we'll have to delve into this part
> by writing C code that makes system calls. Please note that
> these C files will need to be implemented specifically for each
> platform. I took this approach when I wrote x-platform library to
> make HDFS native client cross-platform compatible -
>
> https://github.com/apache/hadoop/tree/trunk/hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/lib/x-platform
>
>
> What am I focussing on currently?
> ------------------------------------------------
> So far, I've focussed on getting the build to work seamlessly
> on Windows. I'm now trying to protect this from breaking by
> setting up CI on Jenkins that builds Hadoop on Windows
> for the precommit validation -
> https://issues.apache.org/jira/browse/INFRA-23809
> Yes, it does involve getting
> Yetus to run on Windows. I can work on deprecating winutils
> after this.
>
> Thanks,
> --Gautham
>
> On Fri, 11 Nov 2022 at 19:51, Steve Loughran <st...@cloudera.com.invalid>
> wrote:
>
>> It's time to reach for the axe.
>>
>> We haven't shipped eight version of Apache hadoop which builds and runs on
>> windows for a long long time. I the only people trying to use the library
>> is on windows Will have been people trying to use spark on their laptops
>> with "small" dataset of only a are few tens of gigabytes at a time, the
>> kind of work where 32GB of ram and 16 cores is enough. Put differently: in
>> storage and performance of Single laptop means that it is perfectly
>> suitable for doing reasonable amounts of work and the main barrier to
>> doing
>> so is getting a copy of the winutils lib.
>>
>> I know Gautham and Inigo I trying to get windows to work as a location for
>> yarn again; not sure about hdfs. And there, yes, we have to say "they
>> likely to need an extra binary"
>>
>> But for someone wanting to count the number of rows in an avro file? do a
>> simple bit of filtering on some parquet data? Is these are the kind of
>> things that anyone with a linux/mac laptop can do with ease and it is not
>> fair to put suffering on to others. And well we could just say "why do you
>> just install Lynnox on that laptop then?", I have someone who has had a
>> Linux laptop for many years I know the written strong arguments against it
>> even beyond the "my employer demand windows with their IT software" as "a
>> latop which comes out of sleep reliably" is kind of important too.
>>
>> I how can we let the people who have to live in this world – And we have
>> someone who is clearly willing to help –Live a better life. Funnily
>> enough,
>> the fact that we have not shipped a working version of when you tails for
>> a
>> long time actually gives us an advantage: we can pick incompatible changes
>> and be confident that most people aren't going to notice.
>>
>> I think a good first step would be for Shell to work well if winutils
>> isn't
>> around -get rid of that static, WINUTILS string and path/file equivalents,
>> the ones deprecated in 2015. We can rip them out knowing no external code
>> is using them.
>>
>> Then we should look very closely at FileUtil to see how much of that is
>> needed and how can we isolate it better. If you look at the change log of
>> that file, we do have to consider that every time it execs a shell command
>> I there's a security risk and more than once we've had to fix it. Not
>> executing any external shell commands is good everywhere.
>>
>>
>>
>>
>> On Thu, 10 Nov 2022 at 19:00, Chris Nauroth <cn...@apache.org> wrote:
>>
>> > Symlink support on the local file system is still used. One example I
>> can
>> > think of is YARN container launch [1].
>> >
>> > I would welcome removal of winutils, as already described in various
>> JIRA
>> > issues. I think the biggest challenge we'll have is testing of a
>> transition
>> > from winutils to the newer Java APIs. The contract tests help, but
>> > historically there was also a tendency to break things in downstream
>> > dependent projects.
>> >
>> > I'd suggest taking this on piecemeal, transitioning small pieces of
>> > FileSystem off of winutils one at a time.
>> >
>> > [1]
>> >
>> >
>> https://github.com/apache/hadoop/blob/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/launcher/ContainerLaunch.java#L1508-L1509
>> >
>> > Chris Nauroth
>> >
>> >
>> > On Thu, Nov 10, 2022 at 10:33 AM Wei-Chiu Chuang <we...@apache.org>
>> > wrote:
>> >
>> > > >
>> > > >
>> > > >
>> > > >   * Bare Naked Local File System v0.1.0 doesn't (yet) support
>> symlinks
>> > > >     or the sticky bit.
>> > > >
>> > > ok to not support symlinks. The symlinks of HDFS are not being
>> maintained
>> > > and I am not aware of anything relying on it.
>> > > So I assume people don't need it.
>> > >
>> > > Sticky bit would be useful, I guess.
>> > >
>> > > I suppose folks working at Microsoft would be more interested in this
>> > work?
>> > > Last time I heard, Gautham and Inigo were revamping Hadoop's Windows
>> > > support.
>> > >
>> > >
>> > > >   * But the bigger issue is how to excise Winutils completely in the
>> > > >     existing Hadoop code. Winutils assumptions are hard-coded at a
>> low
>> > > >     level across various classes—even code that has nothing to do
>> with
>> > > >     the file system. The startup configuration for example calls
>> > > >     `StringUtils.equalsIgnoreCase("true", valueString)` which loads
>> the
>> > > >     `StringUtils` class, which has a static reference to `Shell`,
>> which
>> > > >     has a static block that checks for `WINUTILS_EXE`.
>> > > >   * For the most part there should no longer even be a need for
>> > anything
>> > > >     but direct Java API access for the local file system. But
>> muddling
>> > > >     things further, the existing `RawLocalFileSystem` implementation
>> > has
>> > > >     /four/ ways to access the local file system: Winutils, JNI
>> calls,
>> > > >     shell access, and a "new" approach using "stat". The "stat"
>> > approach
>> > > >     has been switched off with a hard-coded
>> `useDeprecatedFileStatus =
>> > > >     true` because of HADOOP-9652
>> > > >     <https://issues.apache.org/jira/browse/HADOOP-9652>.
>> > > >   * Local file access is not contained within `RawLocalFileSystem`
>> but
>> > > >     is scattered across other classes; `FileUtil.readLink()` for
>> > example
>> > > >     (which `RawLocalFileSystem` calls because of the deprecation
>> issue
>> > > >     above) uses the shell approach without any option to change it.
>> > > >     (This implementation-specific decision should have been
>> contained
>> > > >     within the `FileSystem` implementation itself.)
>> > > >
>> > > > In short, it's a mess that has accumulated over years and getting
>> > worse,
>> > > > charging high interest on what at first was a small, self-contained
>> > > > technical debt.
>> > > >
>> > > > I would welcome the opportunity to clean up this mess. I'm probably
>> as
>> > > > qualified as anyone to make the changes. This is one of my areas of
>> > > > expertise: I was designing a full abstract file system interface
>> (with
>> > > > pure-Java from-scratch implementations for the local file system,
>> > > > Subversion, and WebDAV—even the WebDAV HTTP implementation was from
>> > > > scratch) around the time Apache Nutch was getting off the ground.
>> Most
>> > > > recently I've worked on the Hadoop `FileSystem` API contracting for
>> > > > LinkedIn, discovering (what I consider to be) a huge bug in
>> > > > ViewFilesystem, HADOOP-18525
>> > > > <https://issues.apache.org/jira/browse/HADOOP-18525>.
>> > > >
>> > > > The cleanup should be done in several stages (e.g. consolidating
>> > > > WinUtils access; replacing code with pure Java API calls;
>> undeprecating
>> > > > the new Stat code and relegating it to a different class, etc.).
>> > > > Unfortunately it's not financially feasible for me to sit here for
>> > > > several months and revamp the Hadoop `FileSystem` subsystem for fun
>> > > > (even though I wish I could). Perhaps there is job opening at a
>> company
>> > > > related to Hadoop that would be interested in hiring me and
>> devoting a
>> > > > certain percentage of my time to fixing local `FileSystem` access.
>> If
>> > > > so, let me know where I should send my resume
>> > > > <https://www.garretwilson.com/about/resume>.
>> > > >
>> > > > Otherwise let me know if any ideas for a way forward. If there
>> proves
>> > to
>> > > > be interest in GlobalMentor Hadoop Bare Naked Local FileSystem
>> > > > <https://github.com/globalmentor/hadoop-bare-naked-local-fs> on
>> GitHub
>> > > > I'll try to maintain and improve it, but really what needs to be
>> > > > revamped is the Hadoop codebase itself. I'll be happy when Hadoop is
>> > > > fixed so that both Steve's code and my code are no longer needed.
>> > > >
>> > > > Garret
>> > > >
>> > >
>> >
>>
>

Re: way forward for Winutils excision from `FileSystem`

Posted by Gautham Banasandra <ga...@apache.org>.
Hi folks,


What have we done so far?
------------------------------------
Inigo and I have been working for quite some time now on this topic,
but our efforts have mostly been oriented towards making Hadoop
cross-platform compatible. Our focus has been on streamlining the
process of building Hadoop on Windows so that one can easily
build and run Hadoop, just like on Linux. We reached this milestone
quite recently and I've documented the steps for doing so here -
https://github.com/apache/hadoop/blob/5bb11cecea136acccac2563b37021b554e517012/BUILDING.txt#L493-L622



Is winutils still required?
-------------------------------
As Steve mentioned, we would still require winutils for running
Hadoop on Windows. The major change here is that winutils
need not come from a third-party repository anymore, rather it
gets built along with the Hadoop codebase itself henceforth.
However, I agree that we need to deprecate winutils and
replace it with something better so that Hadoop users can have
a smoother experience.


What's the best way to deprecate winutils?
--------------------------------------------------------
Over all the time that I've spent making Hadoop cross-platform
compatible, I've realized that the best way would be to have a
JNI interface that wraps around a native layer. This native layer
could be implemented majorly in C++. C++17 provides the
std::filesystem namespace that can satisfy most of the native
filesystem API requirements. Since std::filesystem is part of "The
Standard Libray", these APIs will be present on most/all the C++
compilers of the various OS platforms. For those parts that can't
be satisfied by std::filesystem, we'll have to delve into this part
by writing C code that makes system calls. Please note that
these C files will need to be implemented specifically for each
platform. I took this approach when I wrote x-platform library to
make HDFS native client cross-platform compatible -
https://github.com/apache/hadoop/tree/trunk/hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/lib/x-platform


What am I focussing on currently?
------------------------------------------------
So far, I've focussed on getting the build to work seamlessly
on Windows. I'm now trying to protect this from breaking by
setting up CI on Jenkins that builds Hadoop on Windows
for the precommit validation -
https://issues.apache.org/jira/browse/INFRA-23809
Yes, it does involve getting
Yetus to run on Windows. I can work on deprecating winutils
after this.

Thanks,
--Gautham

On Fri, 11 Nov 2022 at 19:51, Steve Loughran <st...@cloudera.com.invalid>
wrote:

> It's time to reach for the axe.
>
> We haven't shipped eight version of Apache hadoop which builds and runs on
> windows for a long long time. I the only people trying to use the library
> is on windows Will have been people trying to use spark on their laptops
> with "small" dataset of only a are few tens of gigabytes at a time, the
> kind of work where 32GB of ram and 16 cores is enough. Put differently: in
> storage and performance of Single laptop means that it is perfectly
> suitable for doing reasonable amounts of work and the main barrier to doing
> so is getting a copy of the winutils lib.
>
> I know Gautham and Inigo I trying to get windows to work as a location for
> yarn again; not sure about hdfs. And there, yes, we have to say "they
> likely to need an extra binary"
>
> But for someone wanting to count the number of rows in an avro file? do a
> simple bit of filtering on some parquet data? Is these are the kind of
> things that anyone with a linux/mac laptop can do with ease and it is not
> fair to put suffering on to others. And well we could just say "why do you
> just install Lynnox on that laptop then?", I have someone who has had a
> Linux laptop for many years I know the written strong arguments against it
> even beyond the "my employer demand windows with their IT software" as "a
> latop which comes out of sleep reliably" is kind of important too.
>
> I how can we let the people who have to live in this world – And we have
> someone who is clearly willing to help –Live a better life. Funnily enough,
> the fact that we have not shipped a working version of when you tails for a
> long time actually gives us an advantage: we can pick incompatible changes
> and be confident that most people aren't going to notice.
>
> I think a good first step would be for Shell to work well if winutils isn't
> around -get rid of that static, WINUTILS string and path/file equivalents,
> the ones deprecated in 2015. We can rip them out knowing no external code
> is using them.
>
> Then we should look very closely at FileUtil to see how much of that is
> needed and how can we isolate it better. If you look at the change log of
> that file, we do have to consider that every time it execs a shell command
> I there's a security risk and more than once we've had to fix it. Not
> executing any external shell commands is good everywhere.
>
>
>
>
> On Thu, 10 Nov 2022 at 19:00, Chris Nauroth <cn...@apache.org> wrote:
>
> > Symlink support on the local file system is still used. One example I can
> > think of is YARN container launch [1].
> >
> > I would welcome removal of winutils, as already described in various JIRA
> > issues. I think the biggest challenge we'll have is testing of a
> transition
> > from winutils to the newer Java APIs. The contract tests help, but
> > historically there was also a tendency to break things in downstream
> > dependent projects.
> >
> > I'd suggest taking this on piecemeal, transitioning small pieces of
> > FileSystem off of winutils one at a time.
> >
> > [1]
> >
> >
> https://github.com/apache/hadoop/blob/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/launcher/ContainerLaunch.java#L1508-L1509
> >
> > Chris Nauroth
> >
> >
> > On Thu, Nov 10, 2022 at 10:33 AM Wei-Chiu Chuang <we...@apache.org>
> > wrote:
> >
> > > >
> > > >
> > > >
> > > >   * Bare Naked Local File System v0.1.0 doesn't (yet) support
> symlinks
> > > >     or the sticky bit.
> > > >
> > > ok to not support symlinks. The symlinks of HDFS are not being
> maintained
> > > and I am not aware of anything relying on it.
> > > So I assume people don't need it.
> > >
> > > Sticky bit would be useful, I guess.
> > >
> > > I suppose folks working at Microsoft would be more interested in this
> > work?
> > > Last time I heard, Gautham and Inigo were revamping Hadoop's Windows
> > > support.
> > >
> > >
> > > >   * But the bigger issue is how to excise Winutils completely in the
> > > >     existing Hadoop code. Winutils assumptions are hard-coded at a
> low
> > > >     level across various classes—even code that has nothing to do
> with
> > > >     the file system. The startup configuration for example calls
> > > >     `StringUtils.equalsIgnoreCase("true", valueString)` which loads
> the
> > > >     `StringUtils` class, which has a static reference to `Shell`,
> which
> > > >     has a static block that checks for `WINUTILS_EXE`.
> > > >   * For the most part there should no longer even be a need for
> > anything
> > > >     but direct Java API access for the local file system. But
> muddling
> > > >     things further, the existing `RawLocalFileSystem` implementation
> > has
> > > >     /four/ ways to access the local file system: Winutils, JNI calls,
> > > >     shell access, and a "new" approach using "stat". The "stat"
> > approach
> > > >     has been switched off with a hard-coded `useDeprecatedFileStatus
> =
> > > >     true` because of HADOOP-9652
> > > >     <https://issues.apache.org/jira/browse/HADOOP-9652>.
> > > >   * Local file access is not contained within `RawLocalFileSystem`
> but
> > > >     is scattered across other classes; `FileUtil.readLink()` for
> > example
> > > >     (which `RawLocalFileSystem` calls because of the deprecation
> issue
> > > >     above) uses the shell approach without any option to change it.
> > > >     (This implementation-specific decision should have been contained
> > > >     within the `FileSystem` implementation itself.)
> > > >
> > > > In short, it's a mess that has accumulated over years and getting
> > worse,
> > > > charging high interest on what at first was a small, self-contained
> > > > technical debt.
> > > >
> > > > I would welcome the opportunity to clean up this mess. I'm probably
> as
> > > > qualified as anyone to make the changes. This is one of my areas of
> > > > expertise: I was designing a full abstract file system interface
> (with
> > > > pure-Java from-scratch implementations for the local file system,
> > > > Subversion, and WebDAV—even the WebDAV HTTP implementation was from
> > > > scratch) around the time Apache Nutch was getting off the ground.
> Most
> > > > recently I've worked on the Hadoop `FileSystem` API contracting for
> > > > LinkedIn, discovering (what I consider to be) a huge bug in
> > > > ViewFilesystem, HADOOP-18525
> > > > <https://issues.apache.org/jira/browse/HADOOP-18525>.
> > > >
> > > > The cleanup should be done in several stages (e.g. consolidating
> > > > WinUtils access; replacing code with pure Java API calls;
> undeprecating
> > > > the new Stat code and relegating it to a different class, etc.).
> > > > Unfortunately it's not financially feasible for me to sit here for
> > > > several months and revamp the Hadoop `FileSystem` subsystem for fun
> > > > (even though I wish I could). Perhaps there is job opening at a
> company
> > > > related to Hadoop that would be interested in hiring me and devoting
> a
> > > > certain percentage of my time to fixing local `FileSystem` access. If
> > > > so, let me know where I should send my resume
> > > > <https://www.garretwilson.com/about/resume>.
> > > >
> > > > Otherwise let me know if any ideas for a way forward. If there proves
> > to
> > > > be interest in GlobalMentor Hadoop Bare Naked Local FileSystem
> > > > <https://github.com/globalmentor/hadoop-bare-naked-local-fs> on
> GitHub
> > > > I'll try to maintain and improve it, but really what needs to be
> > > > revamped is the Hadoop codebase itself. I'll be happy when Hadoop is
> > > > fixed so that both Steve's code and my code are no longer needed.
> > > >
> > > > Garret
> > > >
> > >
> >
>

Re: way forward for Winutils excision from `FileSystem`

Posted by Steve Loughran <st...@cloudera.com.INVALID>.
It's time to reach for the axe.

We haven't shipped eight version of Apache hadoop which builds and runs on
windows for a long long time. I the only people trying to use the library
is on windows Will have been people trying to use spark on their laptops
with "small" dataset of only a are few tens of gigabytes at a time, the
kind of work where 32GB of ram and 16 cores is enough. Put differently: in
storage and performance of Single laptop means that it is perfectly
suitable for doing reasonable amounts of work and the main barrier to doing
so is getting a copy of the winutils lib.

I know Gautham and Inigo I trying to get windows to work as a location for
yarn again; not sure about hdfs. And there, yes, we have to say "they
likely to need an extra binary"

But for someone wanting to count the number of rows in an avro file? do a
simple bit of filtering on some parquet data? Is these are the kind of
things that anyone with a linux/mac laptop can do with ease and it is not
fair to put suffering on to others. And well we could just say "why do you
just install Lynnox on that laptop then?", I have someone who has had a
Linux laptop for many years I know the written strong arguments against it
even beyond the "my employer demand windows with their IT software" as "a
latop which comes out of sleep reliably" is kind of important too.

I how can we let the people who have to live in this world – And we have
someone who is clearly willing to help –Live a better life. Funnily enough,
the fact that we have not shipped a working version of when you tails for a
long time actually gives us an advantage: we can pick incompatible changes
and be confident that most people aren't going to notice.

I think a good first step would be for Shell to work well if winutils isn't
around -get rid of that static, WINUTILS string and path/file equivalents,
the ones deprecated in 2015. We can rip them out knowing no external code
is using them.

Then we should look very closely at FileUtil to see how much of that is
needed and how can we isolate it better. If you look at the change log of
that file, we do have to consider that every time it execs a shell command
I there's a security risk and more than once we've had to fix it. Not
executing any external shell commands is good everywhere.




On Thu, 10 Nov 2022 at 19:00, Chris Nauroth <cn...@apache.org> wrote:

> Symlink support on the local file system is still used. One example I can
> think of is YARN container launch [1].
>
> I would welcome removal of winutils, as already described in various JIRA
> issues. I think the biggest challenge we'll have is testing of a transition
> from winutils to the newer Java APIs. The contract tests help, but
> historically there was also a tendency to break things in downstream
> dependent projects.
>
> I'd suggest taking this on piecemeal, transitioning small pieces of
> FileSystem off of winutils one at a time.
>
> [1]
>
> https://github.com/apache/hadoop/blob/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/launcher/ContainerLaunch.java#L1508-L1509
>
> Chris Nauroth
>
>
> On Thu, Nov 10, 2022 at 10:33 AM Wei-Chiu Chuang <we...@apache.org>
> wrote:
>
> > >
> > >
> > >
> > >   * Bare Naked Local File System v0.1.0 doesn't (yet) support symlinks
> > >     or the sticky bit.
> > >
> > ok to not support symlinks. The symlinks of HDFS are not being maintained
> > and I am not aware of anything relying on it.
> > So I assume people don't need it.
> >
> > Sticky bit would be useful, I guess.
> >
> > I suppose folks working at Microsoft would be more interested in this
> work?
> > Last time I heard, Gautham and Inigo were revamping Hadoop's Windows
> > support.
> >
> >
> > >   * But the bigger issue is how to excise Winutils completely in the
> > >     existing Hadoop code. Winutils assumptions are hard-coded at a low
> > >     level across various classes—even code that has nothing to do with
> > >     the file system. The startup configuration for example calls
> > >     `StringUtils.equalsIgnoreCase("true", valueString)` which loads the
> > >     `StringUtils` class, which has a static reference to `Shell`, which
> > >     has a static block that checks for `WINUTILS_EXE`.
> > >   * For the most part there should no longer even be a need for
> anything
> > >     but direct Java API access for the local file system. But muddling
> > >     things further, the existing `RawLocalFileSystem` implementation
> has
> > >     /four/ ways to access the local file system: Winutils, JNI calls,
> > >     shell access, and a "new" approach using "stat". The "stat"
> approach
> > >     has been switched off with a hard-coded `useDeprecatedFileStatus =
> > >     true` because of HADOOP-9652
> > >     <https://issues.apache.org/jira/browse/HADOOP-9652>.
> > >   * Local file access is not contained within `RawLocalFileSystem` but
> > >     is scattered across other classes; `FileUtil.readLink()` for
> example
> > >     (which `RawLocalFileSystem` calls because of the deprecation issue
> > >     above) uses the shell approach without any option to change it.
> > >     (This implementation-specific decision should have been contained
> > >     within the `FileSystem` implementation itself.)
> > >
> > > In short, it's a mess that has accumulated over years and getting
> worse,
> > > charging high interest on what at first was a small, self-contained
> > > technical debt.
> > >
> > > I would welcome the opportunity to clean up this mess. I'm probably as
> > > qualified as anyone to make the changes. This is one of my areas of
> > > expertise: I was designing a full abstract file system interface (with
> > > pure-Java from-scratch implementations for the local file system,
> > > Subversion, and WebDAV—even the WebDAV HTTP implementation was from
> > > scratch) around the time Apache Nutch was getting off the ground. Most
> > > recently I've worked on the Hadoop `FileSystem` API contracting for
> > > LinkedIn, discovering (what I consider to be) a huge bug in
> > > ViewFilesystem, HADOOP-18525
> > > <https://issues.apache.org/jira/browse/HADOOP-18525>.
> > >
> > > The cleanup should be done in several stages (e.g. consolidating
> > > WinUtils access; replacing code with pure Java API calls; undeprecating
> > > the new Stat code and relegating it to a different class, etc.).
> > > Unfortunately it's not financially feasible for me to sit here for
> > > several months and revamp the Hadoop `FileSystem` subsystem for fun
> > > (even though I wish I could). Perhaps there is job opening at a company
> > > related to Hadoop that would be interested in hiring me and devoting a
> > > certain percentage of my time to fixing local `FileSystem` access. If
> > > so, let me know where I should send my resume
> > > <https://www.garretwilson.com/about/resume>.
> > >
> > > Otherwise let me know if any ideas for a way forward. If there proves
> to
> > > be interest in GlobalMentor Hadoop Bare Naked Local FileSystem
> > > <https://github.com/globalmentor/hadoop-bare-naked-local-fs> on GitHub
> > > I'll try to maintain and improve it, but really what needs to be
> > > revamped is the Hadoop codebase itself. I'll be happy when Hadoop is
> > > fixed so that both Steve's code and my code are no longer needed.
> > >
> > > Garret
> > >
> >
>

Re: way forward for Winutils excision from `FileSystem`

Posted by Chris Nauroth <cn...@apache.org>.
Symlink support on the local file system is still used. One example I can
think of is YARN container launch [1].

I would welcome removal of winutils, as already described in various JIRA
issues. I think the biggest challenge we'll have is testing of a transition
from winutils to the newer Java APIs. The contract tests help, but
historically there was also a tendency to break things in downstream
dependent projects.

I'd suggest taking this on piecemeal, transitioning small pieces of
FileSystem off of winutils one at a time.

[1]
https://github.com/apache/hadoop/blob/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/launcher/ContainerLaunch.java#L1508-L1509

Chris Nauroth


On Thu, Nov 10, 2022 at 10:33 AM Wei-Chiu Chuang <we...@apache.org> wrote:

> >
> >
> >
> >   * Bare Naked Local File System v0.1.0 doesn't (yet) support symlinks
> >     or the sticky bit.
> >
> ok to not support symlinks. The symlinks of HDFS are not being maintained
> and I am not aware of anything relying on it.
> So I assume people don't need it.
>
> Sticky bit would be useful, I guess.
>
> I suppose folks working at Microsoft would be more interested in this work?
> Last time I heard, Gautham and Inigo were revamping Hadoop's Windows
> support.
>
>
> >   * But the bigger issue is how to excise Winutils completely in the
> >     existing Hadoop code. Winutils assumptions are hard-coded at a low
> >     level across various classes—even code that has nothing to do with
> >     the file system. The startup configuration for example calls
> >     `StringUtils.equalsIgnoreCase("true", valueString)` which loads the
> >     `StringUtils` class, which has a static reference to `Shell`, which
> >     has a static block that checks for `WINUTILS_EXE`.
> >   * For the most part there should no longer even be a need for anything
> >     but direct Java API access for the local file system. But muddling
> >     things further, the existing `RawLocalFileSystem` implementation has
> >     /four/ ways to access the local file system: Winutils, JNI calls,
> >     shell access, and a "new" approach using "stat". The "stat" approach
> >     has been switched off with a hard-coded `useDeprecatedFileStatus =
> >     true` because of HADOOP-9652
> >     <https://issues.apache.org/jira/browse/HADOOP-9652>.
> >   * Local file access is not contained within `RawLocalFileSystem` but
> >     is scattered across other classes; `FileUtil.readLink()` for example
> >     (which `RawLocalFileSystem` calls because of the deprecation issue
> >     above) uses the shell approach without any option to change it.
> >     (This implementation-specific decision should have been contained
> >     within the `FileSystem` implementation itself.)
> >
> > In short, it's a mess that has accumulated over years and getting worse,
> > charging high interest on what at first was a small, self-contained
> > technical debt.
> >
> > I would welcome the opportunity to clean up this mess. I'm probably as
> > qualified as anyone to make the changes. This is one of my areas of
> > expertise: I was designing a full abstract file system interface (with
> > pure-Java from-scratch implementations for the local file system,
> > Subversion, and WebDAV—even the WebDAV HTTP implementation was from
> > scratch) around the time Apache Nutch was getting off the ground. Most
> > recently I've worked on the Hadoop `FileSystem` API contracting for
> > LinkedIn, discovering (what I consider to be) a huge bug in
> > ViewFilesystem, HADOOP-18525
> > <https://issues.apache.org/jira/browse/HADOOP-18525>.
> >
> > The cleanup should be done in several stages (e.g. consolidating
> > WinUtils access; replacing code with pure Java API calls; undeprecating
> > the new Stat code and relegating it to a different class, etc.).
> > Unfortunately it's not financially feasible for me to sit here for
> > several months and revamp the Hadoop `FileSystem` subsystem for fun
> > (even though I wish I could). Perhaps there is job opening at a company
> > related to Hadoop that would be interested in hiring me and devoting a
> > certain percentage of my time to fixing local `FileSystem` access. If
> > so, let me know where I should send my resume
> > <https://www.garretwilson.com/about/resume>.
> >
> > Otherwise let me know if any ideas for a way forward. If there proves to
> > be interest in GlobalMentor Hadoop Bare Naked Local FileSystem
> > <https://github.com/globalmentor/hadoop-bare-naked-local-fs> on GitHub
> > I'll try to maintain and improve it, but really what needs to be
> > revamped is the Hadoop codebase itself. I'll be happy when Hadoop is
> > fixed so that both Steve's code and my code are no longer needed.
> >
> > Garret
> >
>

Re: way forward for Winutils excision from `FileSystem`

Posted by Wei-Chiu Chuang <we...@apache.org>.
>
>
>
>   * Bare Naked Local File System v0.1.0 doesn't (yet) support symlinks
>     or the sticky bit.
>
ok to not support symlinks. The symlinks of HDFS are not being maintained
and I am not aware of anything relying on it.
So I assume people don't need it.

Sticky bit would be useful, I guess.

I suppose folks working at Microsoft would be more interested in this work?
Last time I heard, Gautham and Inigo were revamping Hadoop's Windows
support.


>   * But the bigger issue is how to excise Winutils completely in the
>     existing Hadoop code. Winutils assumptions are hard-coded at a low
>     level across various classes—even code that has nothing to do with
>     the file system. The startup configuration for example calls
>     `StringUtils.equalsIgnoreCase("true", valueString)` which loads the
>     `StringUtils` class, which has a static reference to `Shell`, which
>     has a static block that checks for `WINUTILS_EXE`.
>   * For the most part there should no longer even be a need for anything
>     but direct Java API access for the local file system. But muddling
>     things further, the existing `RawLocalFileSystem` implementation has
>     /four/ ways to access the local file system: Winutils, JNI calls,
>     shell access, and a "new" approach using "stat". The "stat" approach
>     has been switched off with a hard-coded `useDeprecatedFileStatus =
>     true` because of HADOOP-9652
>     <https://issues.apache.org/jira/browse/HADOOP-9652>.
>   * Local file access is not contained within `RawLocalFileSystem` but
>     is scattered across other classes; `FileUtil.readLink()` for example
>     (which `RawLocalFileSystem` calls because of the deprecation issue
>     above) uses the shell approach without any option to change it.
>     (This implementation-specific decision should have been contained
>     within the `FileSystem` implementation itself.)
>
> In short, it's a mess that has accumulated over years and getting worse,
> charging high interest on what at first was a small, self-contained
> technical debt.
>
> I would welcome the opportunity to clean up this mess. I'm probably as
> qualified as anyone to make the changes. This is one of my areas of
> expertise: I was designing a full abstract file system interface (with
> pure-Java from-scratch implementations for the local file system,
> Subversion, and WebDAV—even the WebDAV HTTP implementation was from
> scratch) around the time Apache Nutch was getting off the ground. Most
> recently I've worked on the Hadoop `FileSystem` API contracting for
> LinkedIn, discovering (what I consider to be) a huge bug in
> ViewFilesystem, HADOOP-18525
> <https://issues.apache.org/jira/browse/HADOOP-18525>.
>
> The cleanup should be done in several stages (e.g. consolidating
> WinUtils access; replacing code with pure Java API calls; undeprecating
> the new Stat code and relegating it to a different class, etc.).
> Unfortunately it's not financially feasible for me to sit here for
> several months and revamp the Hadoop `FileSystem` subsystem for fun
> (even though I wish I could). Perhaps there is job opening at a company
> related to Hadoop that would be interested in hiring me and devoting a
> certain percentage of my time to fixing local `FileSystem` access. If
> so, let me know where I should send my resume
> <https://www.garretwilson.com/about/resume>.
>
> Otherwise let me know if any ideas for a way forward. If there proves to
> be interest in GlobalMentor Hadoop Bare Naked Local FileSystem
> <https://github.com/globalmentor/hadoop-bare-naked-local-fs> on GitHub
> I'll try to maintain and improve it, but really what needs to be
> revamped is the Hadoop codebase itself. I'll be happy when Hadoop is
> fixed so that both Steve's code and my code are no longer needed.
>
> Garret
>