[jira] [Closed] (HUDI-2747) Fix hudi cli metadata commands

["Ethan Guo (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/HUDI-2747?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Ethan Guo closed HUDI-2747.
---------------------------
    Resolution: Fixed

> Fix hudi cli metadata commands
> ------------------------------
>
>                 Key: HUDI-2747
>                 URL: https://issues.apache.org/jira/browse/HUDI-2747
>             Project: Apache Hudi
>          Issue Type: Bug
>    Affects Versions: 0.10.0
>            Reporter: sivabalan narayanan
>            Assignee: Yue Zhang
>            Priority: Blocker
>              Labels: HUDI-bug, pull-request-available
>             Fix For: 0.11.0
>
>
> Fix hudi cli metadata commands.  Currently when running hudi cli metadata commands locally, it complains about spark master, e.g., yarn master not found.  The commands need to take in spark related configs so that they can run locally and in cluster.
> {code:java}
> java.lang.reflect.UndeclaredThrowableException
> 	at org.springframework.util.ReflectionUtils.rethrowRuntimeException(ReflectionUtils.java:315)
> 	at org.springframework.util.ReflectionUtils.handleInvocationTargetException(ReflectionUtils.java:295)
> 	at org.springframework.util.ReflectionUtils.handleReflectionException(ReflectionUtils.java:279)
> 	at org.springframework.util.ReflectionUtils.invokeMethod(ReflectionUtils.java:219)
> 	at org.springframework.shell.core.SimpleExecutionStrategy.invoke(SimpleExecutionStrategy.java:68)
> 	at org.springframework.shell.core.SimpleExecutionStrategy.execute(SimpleExecutionStrategy.java:59)
> 	at org.springframework.shell.core.AbstractShell.executeCommand(AbstractShell.java:134)
> 	at org.springframework.shell.core.JLineShell.promptLoop(JLineShell.java:533)
> 	at org.springframework.shell.core.JLineShell.run(JLineShell.java:179)
> 	at java.lang.Thread.run(Thread.java:748)
> Caused by: org.apache.spark.SparkException: Could not parse Master URL: 'yarn'
> 	at org.apache.spark.SparkContext$.org$apache$spark$SparkContext$$createTaskScheduler(SparkContext.scala:2784)
> 	at org.apache.spark.SparkContext.<init>(SparkContext.scala:493)
> 	at org.apache.spark.api.java.JavaSparkContext.<init>(JavaSparkContext.scala:58)
> 	at org.apache.hudi.cli.utils.SparkUtil.initJavaSparkConf(SparkUtil.java:117)
> 	at org.apache.hudi.cli.commands.MetadataCommand.initJavaSparkContext(MetadataCommand.java:362)
> 	at org.apache.hudi.cli.commands.MetadataCommand.listPartitions(MetadataCommand.java:196)
> 	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> 	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> 	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> 	at java.lang.reflect.Method.invoke(Method.java:498)
> 	at org.springframework.util.ReflectionUtils.invokeMethod(ReflectionUtils.java:216)
> 	... 6 more {code}



--
This message was sent by Atlassian Jira
(v8.20.1#820001)