Azure Blob IO for Apache Beam

[Saulo Ricci <in...@gmail.com>]

Hi,

I had posted this question
<http://stackoverflow.com/questions/41386717/azure-blob-support-in-apache-beam>
@
stack overflow. Basically I'm trying to run Apache Beam on a spark cluster
hosted in a MS Azure environment. It seems Apache Beam doesn't have support
to Azure Blobs IO, right? Should be an alternative solution the
implementation of an Azure Blob IO for this case?

Best
Saulo
-- 
Saulo

Re: Azure Blob IO for Apache Beam

[Stephen Sisk <si...@google.com.INVALID>]

If you're interested in what's involved in building a read transform for
Azure Storage Blob, folks here on the dev@ alias are happy to help. We
could help outline in a bit more detail what'd be involved in doing the
HDFS->WASB hookup that Davor mentioned in his stackoverflow answer.

S

On Fri, Dec 30, 2016 at 8:53 PM Davor Bonaci <da...@apache.org> wrote:

> Hi Saulo,
> I've responded to your Stack Overflow question, but I can give a few more
> comments here.
>
> Pei (cc'd) is working on this. See this JIRA issue [1], its sub-tasks, and
> several design documents [2, 3].
>
> It would be awesome to also have native Azure Storage Blob support in Beam
> that builds on top of current work -- we'd love that contribution!
>
> Thanks,
> Davor
>
> [1] https://issues.apache.org/jira/browse/BEAM-59
> [2]
>
> https://docs.google.com/document/d/11TdPyZ9_zmjokhNWM3Id-XJsVG3qel2lhdKTknmZ_7M
> [3]
>
> https://docs.google.com/document/d/1-7vo9nLRsEEzDGnb562PuL4q9mUiq_ZVpCAiyyJw8p8/edit#heading=h.p3gc3colc2cs
>
> On Fri, Dec 30, 2016 at 5:42 PM, Saulo Ricci <in...@gmail.com> wrote:
>
> > Hi,
> >
> > I had posted this question
> > <http://stackoverflow.com/questions/41386717/azure-blob-
> > support-in-apache-beam>
> > @
> > stack overflow. Basically I'm trying to run Apache Beam on a spark
> cluster
> > hosted in a MS Azure environment. It seems Apache Beam doesn't have
> support
> > to Azure Blobs IO, right? Should be an alternative solution the
> > implementation of an Azure Blob IO for this case?
> >
> > Best
> > Saulo
> > --
> > Saulo
> >
>

Re: Azure Blob IO for Apache Beam

[Davor Bonaci <da...@apache.org>]

Hi Saulo,
I've responded to your Stack Overflow question, but I can give a few more
comments here.

Pei (cc'd) is working on this. See this JIRA issue [1], its sub-tasks, and
several design documents [2, 3].

It would be awesome to also have native Azure Storage Blob support in Beam
that builds on top of current work -- we'd love that contribution!

Thanks,
Davor

[1] https://issues.apache.org/jira/browse/BEAM-59
[2]
https://docs.google.com/document/d/11TdPyZ9_zmjokhNWM3Id-XJsVG3qel2lhdKTknmZ_7M
[3]
https://docs.google.com/document/d/1-7vo9nLRsEEzDGnb562PuL4q9mUiq_ZVpCAiyyJw8p8/edit#heading=h.p3gc3colc2cs

On Fri, Dec 30, 2016 at 5:42 PM, Saulo Ricci <in...@gmail.com> wrote:

> Hi,
>
> I had posted this question
> <http://stackoverflow.com/questions/41386717/azure-blob-
> support-in-apache-beam>
> @
> stack overflow. Basically I'm trying to run Apache Beam on a spark cluster
> hosted in a MS Azure environment. It seems Apache Beam doesn't have support
> to Azure Blobs IO, right? Should be an alternative solution the
> implementation of an Azure Blob IO for this case?
>
> Best
> Saulo
> --
> Saulo
>