Dataflow and mounting large data sets

[Chad Dombrova <ch...@gmail.com>]

Hi all,
We have large data sets which we would like to mount over NFS within
Dataflow.  As far as I know, this is not possible.  Has anything changed
there?

The data sets that our beam pipelines operate on are too large -- about a
petabyte -- to put into a custom container or to sync from one location to
another, this is why we need the ability to mount a volume, i.e. using
docker run --mount.

In the past we used Flink and a custom build of beam to add control docker
run options: we made an MR for that here:
https://github.com/apache/beam/pull/8982/files

It was not merged into beam due to "a concern about exposing all docker run
options since it would likely be incompatible with cluster managers that
use docker".  The discussion was here
https://lists.apache.org/thread.html/fa3fede5d2967daab676d3869d2fc94f505517504c3a849db789ef54%40%3Cdev.beam.apache.org%3E

I'm happy to revisit this feature with a set of pruned down options,
including --volume and --mount, but we would need Dataflow to support this
as well, because IIRC, the docker run invocation is part of a black box in
Dataflow (i.e. not part of Beam), so as Dataflow users we rely on those
changes from Google.

Is this a feature that already exists or that we can hope to get in
Dataflow?  Or do we need to go back to Flink to support this?

thanks,
-chad

Re: Dataflow and mounting large data sets

[Andrew Pilloud via user <us...@beam.apache.org>]

I would guess that you have some existing code that expects random IO
access to the files via the Java IO or NIO interface (the common blocking
IO in a DoFn pattern), so using a Beam IO which is what we recommend and
are discussing here would be a significant rewrite?

I worked on Isilon from 6.5 - 7.2 and in those days NFS and SMB were the
high performance options, however these are complex protocols and userspace
implementations tend to be missing the features that enable that
performance. If you do go down the path of a userspace IO (either Beam or
Java NIO wrapper) you'd possibly get better results with FTP, SFTP, or
HTTP. It looks like Isilon added the ability to expose the filesystem via
the S3 protocol in 9.0
<http://doc.isilon.com/onefs/9.0.0/help/en-us/ifs_c_s3_support_intro.html>
and there
is a Beam S3 connector
<https://beam.apache.org/releases/javadoc/2.41.0/org/apache/beam/sdk/io/aws/s3/package-summary.html>
but I
would be amazed if you were on Isilon 9.0+.

On Tue, Jan 31, 2023 at 3:56 PM Luke Cwik via user <us...@beam.apache.org>
wrote:

> I would also suggest looking at NFS client implementations in Java that
> would allow you to talk to the NFS server without needing to mount it
> within the OS. A quick search yielded
> https://github.com/raisercostin/yanfs or
> https://github.com/EMCECS/nfs-client-java
>
> On Tue, Jan 31, 2023 at 3:31 PM Chad Dombrova <ch...@gmail.com> wrote:
>
>> Thanks for the info.  We are going to test this further and we'll let you
>> know how it goes.
>>
>> -chad
>>
>>
>> On Mon, Jan 30, 2023 at 2:14 PM Valentyn Tymofieiev <va...@google.com>
>> wrote:
>>
>>> It applies to custom containers as well. You can find the container
>>> manifest in the GCE VM metadata, and it should have an entry for privileged
>>> mode. The reason for this was to enable GPU accelerator support, but agree
>>> with Robert that it is not part of any contracts, so in theory this could
>>> change or perhaps be more strictly limited to accelerator support. In fact,
>>> originally, this was only enabled for pipelines using accelerators but for
>>> purely internal implementation details I believe it is currently enabled
>>> for all pipelines.
>>>
>>> So for prototyping purposes I think you could try it, but I can't make
>>> any guarantees in this thread that privileged mode will continue to work.
>>>
>>> cc: @Aaron Li <aa...@google.com> FYI
>>>
>>>
>>> On Mon, Jan 30, 2023 at 12:16 PM Robert Bradshaw <ro...@google.com>
>>> wrote:
>>>
>>>> I'm also not sure it's part of the contract that the containerization
>>>> technology we use will always have these capabilities.
>>>>
>>>> On Mon, Jan 30, 2023 at 10:53 AM Chad Dombrova <ch...@gmail.com>
>>>> wrote:
>>>> >
>>>> > Hi Valentyn,
>>>> >
>>>> >>
>>>> >> Beam SDK docker containers on Dataflow VMs are currently launched in
>>>> privileged mode.
>>>> >
>>>> >
>>>> > Does this only apply to stock sdk containers?  I'm asking because we
>>>> use a custom sdk container that we build.  We've tried various ways of
>>>> running mount from within our custom beam container in Dataflow and we
>>>> could not get it to work, while the same thing succeeds in local tests and
>>>> in our CI (gitlab).  The assessment at the time (this was maybe a year ago)
>>>> was that the container was not running in privileged mode, but if you think
>>>> that's incorrect we can revisit this and report back with some error logs.
>>>> >
>>>> > -chad
>>>> >
>>>>
>>>

Re: Dataflow and mounting large data sets

[Luke Cwik via user <us...@beam.apache.org>]

I would also suggest looking at NFS client implementations in Java that
would allow you to talk to the NFS server without needing to mount it
within the OS. A quick search yielded https://github.com/raisercostin/yanfs
or https://github.com/EMCECS/nfs-client-java

On Tue, Jan 31, 2023 at 3:31 PM Chad Dombrova <ch...@gmail.com> wrote:

> Thanks for the info.  We are going to test this further and we'll let you
> know how it goes.
>
> -chad
>
>
> On Mon, Jan 30, 2023 at 2:14 PM Valentyn Tymofieiev <va...@google.com>
> wrote:
>
>> It applies to custom containers as well. You can find the container
>> manifest in the GCE VM metadata, and it should have an entry for privileged
>> mode. The reason for this was to enable GPU accelerator support, but agree
>> with Robert that it is not part of any contracts, so in theory this could
>> change or perhaps be more strictly limited to accelerator support. In fact,
>> originally, this was only enabled for pipelines using accelerators but for
>> purely internal implementation details I believe it is currently enabled
>> for all pipelines.
>>
>> So for prototyping purposes I think you could try it, but I can't make
>> any guarantees in this thread that privileged mode will continue to work.
>>
>> cc: @Aaron Li <aa...@google.com> FYI
>>
>>
>> On Mon, Jan 30, 2023 at 12:16 PM Robert Bradshaw <ro...@google.com>
>> wrote:
>>
>>> I'm also not sure it's part of the contract that the containerization
>>> technology we use will always have these capabilities.
>>>
>>> On Mon, Jan 30, 2023 at 10:53 AM Chad Dombrova <ch...@gmail.com>
>>> wrote:
>>> >
>>> > Hi Valentyn,
>>> >
>>> >>
>>> >> Beam SDK docker containers on Dataflow VMs are currently launched in
>>> privileged mode.
>>> >
>>> >
>>> > Does this only apply to stock sdk containers?  I'm asking because we
>>> use a custom sdk container that we build.  We've tried various ways of
>>> running mount from within our custom beam container in Dataflow and we
>>> could not get it to work, while the same thing succeeds in local tests and
>>> in our CI (gitlab).  The assessment at the time (this was maybe a year ago)
>>> was that the container was not running in privileged mode, but if you think
>>> that's incorrect we can revisit this and report back with some error logs.
>>> >
>>> > -chad
>>> >
>>>
>>

Re: Dataflow and mounting large data sets

[Chad Dombrova <ch...@gmail.com>]

Thanks for the info.  We are going to test this further and we'll let you
know how it goes.

-chad


On Mon, Jan 30, 2023 at 2:14 PM Valentyn Tymofieiev <va...@google.com>
wrote:

> It applies to custom containers as well. You can find the container
> manifest in the GCE VM metadata, and it should have an entry for privileged
> mode. The reason for this was to enable GPU accelerator support, but agree
> with Robert that it is not part of any contracts, so in theory this could
> change or perhaps be more strictly limited to accelerator support. In fact,
> originally, this was only enabled for pipelines using accelerators but for
> purely internal implementation details I believe it is currently enabled
> for all pipelines.
>
> So for prototyping purposes I think you could try it, but I can't make any
> guarantees in this thread that privileged mode will continue to work.
>
> cc: @Aaron Li <aa...@google.com> FYI
>
>
> On Mon, Jan 30, 2023 at 12:16 PM Robert Bradshaw <ro...@google.com>
> wrote:
>
>> I'm also not sure it's part of the contract that the containerization
>> technology we use will always have these capabilities.
>>
>> On Mon, Jan 30, 2023 at 10:53 AM Chad Dombrova <ch...@gmail.com> wrote:
>> >
>> > Hi Valentyn,
>> >
>> >>
>> >> Beam SDK docker containers on Dataflow VMs are currently launched in
>> privileged mode.
>> >
>> >
>> > Does this only apply to stock sdk containers?  I'm asking because we
>> use a custom sdk container that we build.  We've tried various ways of
>> running mount from within our custom beam container in Dataflow and we
>> could not get it to work, while the same thing succeeds in local tests and
>> in our CI (gitlab).  The assessment at the time (this was maybe a year ago)
>> was that the container was not running in privileged mode, but if you think
>> that's incorrect we can revisit this and report back with some error logs.
>> >
>> > -chad
>> >
>>
>

Re: Dataflow and mounting large data sets

[Valentyn Tymofieiev via user <us...@beam.apache.org>]

It applies to custom containers as well. You can find the container
manifest in the GCE VM metadata, and it should have an entry for privileged
mode. The reason for this was to enable GPU accelerator support, but agree
with Robert that it is not part of any contracts, so in theory this could
change or perhaps be more strictly limited to accelerator support. In fact,
originally, this was only enabled for pipelines using accelerators but for
purely internal implementation details I believe it is currently enabled
for all pipelines.

So for prototyping purposes I think you could try it, but I can't make any
guarantees in this thread that privileged mode will continue to work.

cc: @Aaron Li <aa...@google.com> FYI


On Mon, Jan 30, 2023 at 12:16 PM Robert Bradshaw <ro...@google.com>
wrote:

> I'm also not sure it's part of the contract that the containerization
> technology we use will always have these capabilities.
>
> On Mon, Jan 30, 2023 at 10:53 AM Chad Dombrova <ch...@gmail.com> wrote:
> >
> > Hi Valentyn,
> >
> >>
> >> Beam SDK docker containers on Dataflow VMs are currently launched in
> privileged mode.
> >
> >
> > Does this only apply to stock sdk containers?  I'm asking because we use
> a custom sdk container that we build.  We've tried various ways of running
> mount from within our custom beam container in Dataflow and we could not
> get it to work, while the same thing succeeds in local tests and in our CI
> (gitlab).  The assessment at the time (this was maybe a year ago) was that
> the container was not running in privileged mode, but if you think that's
> incorrect we can revisit this and report back with some error logs.
> >
> > -chad
> >
>

Re: Dataflow and mounting large data sets

[Robert Bradshaw via user <us...@beam.apache.org>]

I'm also not sure it's part of the contract that the containerization
technology we use will always have these capabilities.

On Mon, Jan 30, 2023 at 10:53 AM Chad Dombrova <ch...@gmail.com> wrote:
>
> Hi Valentyn,
>
>>
>> Beam SDK docker containers on Dataflow VMs are currently launched in privileged mode.
>
>
> Does this only apply to stock sdk containers?  I'm asking because we use a custom sdk container that we build.  We've tried various ways of running mount from within our custom beam container in Dataflow and we could not get it to work, while the same thing succeeds in local tests and in our CI (gitlab).  The assessment at the time (this was maybe a year ago) was that the container was not running in privileged mode, but if you think that's incorrect we can revisit this and report back with some error logs.
>
> -chad
>

Re: Dataflow and mounting large data sets

[Chad Dombrova <ch...@gmail.com>]

Hi Valentyn,


> Beam SDK docker containers on Dataflow VMs are currently launched in
> privileged mode.
>

Does this only apply to stock sdk containers?  I'm asking because we use a
custom sdk container that we build.  We've tried various ways of running
mount from within our custom beam container in Dataflow and we could not
get it to work, while the same thing succeeds in local tests and in our CI
(gitlab).  The assessment at the time (this was maybe a year ago) was that
the container was not running in privileged mode, but if you think that's
incorrect we can revisit this and report back with some error logs.

-chad

Re: Dataflow and mounting large data sets

[Valentyn Tymofieiev via user <us...@beam.apache.org>]

Beam SDK docker containers on Dataflow VMs are currently launched in
privileged mode.

On Mon, Jan 30, 2023 at 9:52 AM Robert Bradshaw via user <
user@beam.apache.org> wrote:

> Different idea: is it possible to serve this data via another protocol
> (e.g. sftp) rather than requiring a mount?
>
> On Mon, Jan 30, 2023 at 9:26 AM Chad Dombrova <ch...@gmail.com> wrote:
> >
> > Hi Robert,
> > I know very little about the FileSystem classes, but I don’t think it’s
> possible for a process running in docker to create an NFS mount without
> running in privileged [1] mode, which cannot be done with Dataflow. The
> other ways of gaining access to a mount are:
> >
> > A. the node running docker has the NFS mount itself and passes it along
> using docker run --volume.
> > B. the mount is created within the container by using docker run --mount.
> >
> > Neither of these are possible with Dataflow.
> >
> > Here’s a full example of how an NFS mount can be created when running
> docker:
> >
> > docker run -it --network=host \
> >    --mount
> 'type=volume,src=pipe-nfs-test,dst=/Volumes/pipe-nfs-test,volume-driver=local,volume-opt=type=nfs,volume-opt=device=:/pipe,"volume-opt=o=addr=turbohal.luma.mel,vers=3"'
> \
> >    luma/pipe-shell -- bash
> >
> > In my ideal world, I would make a PR to add support for the docker
> --mount flag to Beam for the runners that I can control, and the Dataflow
> team would add support on their end.
> >
> > Let me know if I'm missing anything.
> >
> >
> https://docs.docker.com/engine/reference/run/#runtime-privilege-and-linux-capabilities
> >
> > thanks,
> > -chad
>

Re: Dataflow and mounting large data sets

[Robert Bradshaw via user <us...@beam.apache.org>]

Different idea: is it possible to serve this data via another protocol
(e.g. sftp) rather than requiring a mount?

On Mon, Jan 30, 2023 at 9:26 AM Chad Dombrova <ch...@gmail.com> wrote:
>
> Hi Robert,
> I know very little about the FileSystem classes, but I don’t think it’s possible for a process running in docker to create an NFS mount without running in privileged [1] mode, which cannot be done with Dataflow. The other ways of gaining access to a mount are:
>
> A. the node running docker has the NFS mount itself and passes it along using docker run --volume.
> B. the mount is created within the container by using docker run --mount.
>
> Neither of these are possible with Dataflow.
>
> Here’s a full example of how an NFS mount can be created when running docker:
>
> docker run -it --network=host \
>    --mount 'type=volume,src=pipe-nfs-test,dst=/Volumes/pipe-nfs-test,volume-driver=local,volume-opt=type=nfs,volume-opt=device=:/pipe,"volume-opt=o=addr=turbohal.luma.mel,vers=3"'  \
>    luma/pipe-shell -- bash
>
> In my ideal world, I would make a PR to add support for the docker --mount flag to Beam for the runners that I can control, and the Dataflow team would add support on their end.
>
> Let me know if I'm missing anything.
>
> https://docs.docker.com/engine/reference/run/#runtime-privilege-and-linux-capabilities
>
> thanks,
> -chad

Re: Dataflow and mounting large data sets

[Chad Dombrova <ch...@gmail.com>]

Hi Robert,
I know very little about the FileSystem classes, but I don’t think it’s
possible for a process running in docker to create an NFS mount without
running in privileged [1] mode, which cannot be done with Dataflow. The
other ways of gaining access to a mount are:

A. the node running docker has the NFS mount itself and passes it along
using docker run --volume.
B. the mount is created within the container by using docker run --mount.

Neither of these are possible with Dataflow.

Here’s a full example of how an NFS mount can be created when running
docker:

docker run -it --network=host \
   --mount 'type=volume,src=pipe-nfs-test,dst=/Volumes/pipe-nfs-test,volume-driver=local,volume-opt=type=nfs,volume-opt=device=:/pipe,"volume-opt=o=addr=turbohal.luma.mel,vers=3"'
 \
   luma/pipe-shell -- bash

In my ideal world, I would make a PR to add support for the docker --mount
flag to Beam for the runners that I can control, and the Dataflow team
would add support on their end.

Let me know if I'm missing anything.

https://docs.docker.com/engine/reference/run/#runtime-privilege-and-linux-capabilities

thanks,
-chad

Re: Dataflow and mounting large data sets

[Robert Bradshaw via user <us...@beam.apache.org>]

If it's your input/output data, presumably you could implement a
https://beam.apache.org/releases/javadoc/2.3.0/org/apache/beam/sdk/io/FileSystem.html
for nfs. (I don't know what all that would entail...)

On Mon, Jan 30, 2023 at 9:04 AM Chad Dombrova <ch...@gmail.com> wrote:
>
> Hi Israel,
> Thanks for responding.
>
>> And could not the dataset be accessed from Cloud Storage? Does it need to be specifically NFS?
>
>
> No unfortunately it can't be accessed from Cloud Storage.   Our data resides on high performance Isilon [1] servers using a posix filesystem, and NFS is the tried and true protocol for this.  This configuration cannot be changed for a multitude of reasons, not least of which is that fact that these servers outperform cloud storage at a fraction of the cost of cloud offerings (which is a very big difference for multiple petabytes of storage.  If you'd like more details on why this is not possible I'm happy to explain, but for now let's just say that it's been investigated and it's not practical).  The use of fast posix filers over NFS is fairly ubiquitous in the media and entertainment industry (if you want to know more about how we use Beam, I gave a talk at the Beam Summit a few years ago[2]).
>
> thanks!
> -chad
>
> [1] https://www.dell.com/en-hk/dt/solutions/media-entertainment.htm
> [2] https://www.youtube.com/watch?v=gvbQI3I03a8&t=644s&ab_channel=ApacheBeam
>

Re: Dataflow and mounting large data sets

[Israel Herraiz via user <us...@beam.apache.org>]

I think Isilon also supports HDFS. Years ago, I used to work with Hadoop
and Spark with data that was stored on Isilon appliances, that were
normally accessed using NFS, and we used HDFS to access that data (and to
write data back to Isilon). I did not participate myself in the
configuration of Isilon and HDFS, so I am not sure how easy/complex it is,
but it seems that there are some docs about that (e.g.
http://doc.isilon.com/onefs/hdfs/02-ifs-c-hdfs-conceptual-topics.htm).
Using HDFS from Beam would be much more straightforward than NFS.

On Mon, 30 Jan 2023 at 18:04, Chad Dombrova <ch...@gmail.com> wrote:

> Hi Israel,
> Thanks for responding.
>
> And could not the dataset be accessed from Cloud Storage? Does it need to
>> be specifically NFS?
>>
>
> No unfortunately it can't be accessed from Cloud Storage.   Our data
> resides on high performance Isilon [1] servers using a posix filesystem,
> and NFS is the tried and true protocol for this.  This configuration cannot
> be changed for a multitude of reasons, not least of which is that fact that
> these servers outperform cloud storage at a fraction of the cost of cloud
> offerings (which is a *very* *big* difference for multiple petabytes of
> storage.  If you'd like more details on why this is not possible I'm happy
> to explain, but for now let's just say that it's been investigated and it's
> not practical).  The use of fast posix filers over NFS is fairly ubiquitous
> in the media and entertainment industry (if you want to know more about how
> we use Beam, I gave a talk at the Beam Summit a few years ago[2]).
>
> thanks!
> -chad
>
> [1] https://www.dell.com/en-hk/dt/solutions/media-entertainment.htm
> [2]
> https://www.youtube.com/watch?v=gvbQI3I03a8&t=644s&ab_channel=ApacheBeam
>
>

Re: Dataflow and mounting large data sets

[Chad Dombrova <ch...@gmail.com>]

Hi Israel,
Thanks for responding.

And could not the dataset be accessed from Cloud Storage? Does it need to
> be specifically NFS?
>

No unfortunately it can't be accessed from Cloud Storage.   Our data
resides on high performance Isilon [1] servers using a posix filesystem,
and NFS is the tried and true protocol for this.  This configuration cannot
be changed for a multitude of reasons, not least of which is that fact that
these servers outperform cloud storage at a fraction of the cost of cloud
offerings (which is a *very* *big* difference for multiple petabytes of
storage.  If you'd like more details on why this is not possible I'm happy
to explain, but for now let's just say that it's been investigated and it's
not practical).  The use of fast posix filers over NFS is fairly ubiquitous
in the media and entertainment industry (if you want to know more about how
we use Beam, I gave a talk at the Beam Summit a few years ago[2]).

thanks!
-chad

[1] https://www.dell.com/en-hk/dt/solutions/media-entertainment.htm
[2] https://www.youtube.com/watch?v=gvbQI3I03a8&t=644s&ab_channel=ApacheBeam

Re: Dataflow and mounting large data sets

[Israel Herraiz via user <us...@beam.apache.org>]

And could not the dataset be accessed from Cloud Storage? Does it need to
be specifically NFS?

On Thu, 26 Jan 2023 at 18:16, Chad Dombrova <ch...@gmail.com> wrote:

> Hi all,
> We have large data sets which we would like to mount over NFS within
> Dataflow.  As far as I know, this is not possible.  Has anything changed
> there?
>
> The data sets that our beam pipelines operate on are too large -- about a
> petabyte -- to put into a custom container or to sync from one location to
> another, this is why we need the ability to mount a volume, i.e. using
> docker run --mount.
>
> In the past we used Flink and a custom build of beam to add control docker
> run options: we made an MR for that here:
> https://github.com/apache/beam/pull/8982/files
>
> It was not merged into beam due to "a concern about exposing all docker
> run options since it would likely be incompatible with cluster managers
> that use docker".  The discussion was here
> https://lists.apache.org/thread.html/fa3fede5d2967daab676d3869d2fc94f505517504c3a849db789ef54%40%3Cdev.beam.apache.org%3E
>
> I'm happy to revisit this feature with a set of pruned down options,
> including --volume and --mount, but we would need Dataflow to support this
> as well, because IIRC, the docker run invocation is part of a black box in
> Dataflow (i.e. not part of Beam), so as Dataflow users we rely on those
> changes from Google.
>
> Is this a feature that already exists or that we can hope to get in
> Dataflow?  Or do we need to go back to Flink to support this?
>
> thanks,
> -chad
>
>