You are viewing a plain text version of this content. The canonical link for it is here.

Posted to issues@livy.apache.org by "Akshat Bordia (Jira)" <ji...@apache.org> on 2021/10/12 06:45:00 UTC

[jira] [Created] (LIVY-870) Provide Spark config via --properties-file for Batch Jobs

Akshat Bordia created LIVY-870:
----------------------------------

             Summary: Provide Spark config via --properties-file for Batch Jobs
                 Key: LIVY-870
                 URL: https://issues.apache.org/jira/browse/LIVY-870
             Project: Livy
          Issue Type: Improvement
            Reporter: Akshat Bordia


Currently for batch jobs, individual Spark configs are added as --conf parameter in Spark submit command line. If there are many Spark configurations added in Livy APIs, it might increase the length of command line and in some cases might cause issues due to limit on command line length (Windows has 8191 character length limit on the command line length.).

 

We can merge the configurations into a temporary properties file and add that as --properties-file opt in spark-submit command



--
This message was sent by Atlassian Jira
(v8.3.4#803005)