You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@beam.apache.org by "Ashwin Ramaswami (Jira)" <ji...@apache.org> on 2020/05/27 13:47:00 UTC
[jira] [Updated] (BEAM-10103) Add WasbIO / WasbFileSystem (Azure
Blob Storage)
[ https://issues.apache.org/jira/browse/BEAM-10103?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Ashwin Ramaswami updated BEAM-10103:
------------------------------------
Description:
Azure Blob Storage can be accessed by using the wasb:// and wasbs:// protocols. This should be quite similar to the hdfs:// implementations already there.
We should just be able to use it like this:
{code:python}
(
p
| ReadFromText("yourcontainer@youraccount.blob.core.windows.net/test/sample.txt")
| WriteToText("output.txt", shard_name_template="", num_shards=0)
)
{code}
See:
[1] [https://docs.microsoft.com/en-us/azure/hdinsight/hdinsight-hadoop-use-blob-storage]
[2] [https://hadoop.apache.org/docs/current/hadoop-azure/index.html]
[3] [https://gerardnico.com/azure/wasb]
was:
Azure Blob Storage can be accessed by using the wasb:// and wasbs:// protocols. This should be quite similar to the hdfs:// implementations already there.
See:
[1] [https://docs.microsoft.com/en-us/azure/hdinsight/hdinsight-hadoop-use-blob-storage]
[2] [https://hadoop.apache.org/docs/current/hadoop-azure/index.html]
[3] [https://gerardnico.com/azure/wasb]
> Add WasbIO / WasbFileSystem (Azure Blob Storage)
> ------------------------------------------------
>
> Key: BEAM-10103
> URL: https://issues.apache.org/jira/browse/BEAM-10103
> Project: Beam
> Issue Type: Bug
> Components: io-ideas
> Reporter: Ashwin Ramaswami
> Priority: P2
>
> Azure Blob Storage can be accessed by using the wasb:// and wasbs:// protocols. This should be quite similar to the hdfs:// implementations already there.
>
> We should just be able to use it like this:
>
> {code:python}
> (
> p
> | ReadFromText("yourcontainer@youraccount.blob.core.windows.net/test/sample.txt")
> | WriteToText("output.txt", shard_name_template="", num_shards=0)
> )
> {code}
>
>
> See:
> [1] [https://docs.microsoft.com/en-us/azure/hdinsight/hdinsight-hadoop-use-blob-storage]
> [2] [https://hadoop.apache.org/docs/current/hadoop-azure/index.html]
> [3] [https://gerardnico.com/azure/wasb]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)