You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "Sasikumar Venkatesh (Jira)" <ji...@apache.org> on 2020/04/17 03:39:00 UTC
[jira] [Comment Edited] (HUDI-773) Hudi On Azure Data Lake Storage
V2
[ https://issues.apache.org/jira/browse/HUDI-773?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17085380#comment-17085380 ]
Sasikumar Venkatesh edited comment on HUDI-773 at 4/17/20, 3:38 AM:
--------------------------------------------------------------------
Hi [~vinoth] and [~garyli1019]
I am experimenting Hudi on ADLS Gen 2,
{code:java}
outputPath = abfss://tpch@<account>.dfs.core.windows.net/hudi-tables
df.write.format("hudi").options(hudiOptions)...
.save(outputPath){code}
I have changed the output dir to the above path and when I write using Hudi format. It is showing the error, _*Configuration property [<<account>>.dfs.core.windows.net|http://re230bb35spstg.dfs.core.windows.net/] not found*_. I have configured ADLS FS and FSImpl class in spark using OAUTH and I can write to ADLS2 using the below code.
{code:java}
df.write.format("parquet").save(outputPath){code}
Is there any specific config related to ADLS2 in Hudi for connecting the ALDS storage account using OAuth.
Thanks and regards,
Sasikumar
was (Author: sasikumar.venkat):
Hi [~vinoth] and [~garyli1019]
I am experimenting Hudi on ADLS Gen 2,
{code:java}
outputPath = abfss://tpch@<account>.dfs.core.windows.net/hudi-tables
df.write.format("hudi").options(hudiOptions)...
.save(outputPath){code}
I have changed the output dir to the above path and when I write using Hudi format. It is showing the error, _*Configuration property [<<account>>.dfs.core.windows.net|http://re230bb35spstg.dfs.core.windows.net/] not found*_. I have configured ADLS FS and FSImpl class in spark using OAUTH and I can write to ADLS2 using the below code.
{code:java}
df.write.format("parquet").save(outputPath){code}
Is there any specific config related to ADLS2 for connecting the storage account using OAuth while using Hudi.
Thanks and regards,
Sasikumar
> Hudi On Azure Data Lake Storage V2
> ----------------------------------
>
> Key: HUDI-773
> URL: https://issues.apache.org/jira/browse/HUDI-773
> Project: Apache Hudi (incubating)
> Issue Type: New Feature
> Components: Usability
> Reporter: Yanjia Gary Li
> Assignee: Yanjia Gary Li
> Priority: Minor
> Fix For: 0.6.0
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)