[GitHub] spark pull request: [SPARK-2678][Core] Prevents `spark-submit` fro...

[liancheng <gi...@git.apache.org>]

Github user liancheng commented on the pull request:

    https://github.com/apache/spark/pull/1699#issuecomment-50829383
  
    @vanzin @pwendell Are you suggesting something like this:
    
    ```
    if no "--" exists in CLI arguments
      treat all those options after primary resource as application options
    else:
      split them at "--"
      take those before "--" as spark-submit options
      take those after "--" as application options
    ```
    
    This logic accepts the following invocation happily:
    
    ```
    bin/spark-submit --class Foo user.jar --arg1
    ```
    
    But we still need to rewrite the following one (which is mostly used in those scripts that delegates to `spark-submit`)
    
    ```
    bin/spark-submit --class Foo user.jar --master local --arg1
    ```
    
    into
    
    ```
    bin/spark-submit --class Foo user.jar --master local -- --arg1
    ```
    
    Take `sbin/start-thriftserver.sh` as an example, the core lines of this script is:
    
    ```bash
    CLASS=org.apache.spark.sql.hive.thriftserver.HiveThriftServer2
    exec "FWDIR"/bin/spark-submit --class $CLASS spark-internal "$@"
    ```
    
    Note that `$@` may contain both `spark-submit` options as well as application options by design. We still require the *user* to add `--` after `--master local`. Although a little better, this is still an incompatible change.
    
    (And I haven't updated the documentation yet, will do that after we come to a final conclusion).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---