You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by Jiri-Kremser <gi...@git.apache.org> on 2017/11/23 16:55:03 UTC

[GitHub] spark pull request #19802: [WIP][SPARK-22594][CORE] Handling spark-submit an...

GitHub user Jiri-Kremser opened a pull request:

    https://github.com/apache/spark/pull/19802

    [WIP][SPARK-22594][CORE] Handling spark-submit and master version mismatch

    ## What changes were proposed in this pull request?
    If the `InvalidClassException` exception is thrown in the `TransportRequestHandler.processOneWayMessage` let's send the connected client a message that there is probably a version mismatch between his spark-submit and the remote spark master she is trying to contact.
    
    ## How was this patch tested?
    TODO:
    - [ ] (Please explain how this patch was tested. E.g. unit tests, integration tests, manual tests)
    - [x] Please review http://spark.apache.org/contributing.html before opening a pull request.


You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/Jiri-Kremser/spark SPARK-22594

Alternatively you can review and apply these changes as the patch at:

    https://github.com/apache/spark/pull/19802.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #19802
    
----
commit 6c48392035d6a7300a50a620e8980657304443c6
Author: Jirka Kremser <jk...@redhat.com>
Date:   2017-11-23T16:49:55Z

    [SPARK-22594][CORE] Handling spark-submit and master version mismatch

----


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #19802: [SPARK-22594][CORE] Handling spark-submit and master ver...

Posted by Jiri-Kremser <gi...@git.apache.org>.
Github user Jiri-Kremser commented on the issue:

    https://github.com/apache/spark/pull/19802
  
    > I think it's probably OK if you also re-throw that exception for now. You're just giving more info then.
    
    Hmm, In the original code the exception wasn't re-thrown. Do you really think it is a good idea? The `InvalidClassException` is a checked exception so the signature of `processOneWayMessage()` method would need to be change as well.
    
    this is the original catch-them-all code :)
    ```java
    } catch (Exception e) {
          logger.error("Error while invoking RpcHandler#receive() for one-way message.", e);
    }
    ```
    
    I think, I addressed all your inline other comments, thanks for the review.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #19802: [SPARK-22594][CORE] Handling spark-submit and master ver...

Posted by miacobv <gi...@git.apache.org>.
Github user miacobv commented on the issue:

    https://github.com/apache/spark/pull/19802
  
    I get this when I start the master and worker, without running spark-submit on both 2.2.0 and 2.2.1


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #19802: [WIP][SPARK-22594][CORE] Handling spark-submit and maste...

Posted by felixcheung <gi...@git.apache.org>.
Github user felixcheung commented on the issue:

    https://github.com/apache/spark/pull/19802
  
    I think there are some values in having a better experience with mismatch client/server versions. We discussed it might be even more common when the client was a R package (or Python perhaps down the line) @shivaram 
    
    I agree though focusing on one particular line is likely not sufficient. Perhaps we should design this out? Have this version check formally be part of the protocol handshake?



---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #19802: [SPARK-22594][CORE] Handling spark-submit and master ver...

Posted by srowen <gi...@git.apache.org>.
Github user srowen commented on the issue:

    https://github.com/apache/spark/pull/19802
  
    I see, yeah that's a fine point. Don't rethrow then.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #19802: [WIP][SPARK-22594][CORE] Handling spark-submit and maste...

Posted by Jiri-Kremser <gi...@git.apache.org>.
Github user Jiri-Kremser commented on the issue:

    https://github.com/apache/spark/pull/19802
  
    To explain better the intentions, it doesn't try to solve of somehow provide a compatibility layer between old and new versions of Spark, all it does is slightly improving the UX, because people are hitting the issue all the time (including me)
    
    couple of instances of this issue:
    
    https://stackoverflow.com/questions/32241485/apache-spark-error-local-class-incompatible-when-initiating-a-sparkcontext-clas
    https://stackoverflow.com/questions/38559597/failed-to-connect-to-spark-masterinvalidclassexception-org-apache-spark-rpc-rp
    https://issues.apache.org/jira/browse/SPARK-13956
    https://community.cloudera.com/t5/Advanced-Analytics-Apache-Spark/Spark-Standalone-error-local-class-incompatible-stream-classdesc/td-p/25909
    https://github.com/USCDataScience/sparkler/issues/56
    https://groups.google.com/a/lists.datastax.com/forum/#!topic/spark-connector-user/Z-4qSGbqhYc
    https://sparkr.atlassian.net/browse/SPARKR-72
    
    more here:
    https://www.google.com/search?ei=BxMYWt3xLJDdwQKLwr9w&q=java.io.InvalidClassException%3A+org.apache.spark.rpc.RpcEndpointRef%3B+local+class+incompatible&oq=java.io.InvalidClassException%3A+org.apache.spark.rpc.RpcEndpointRef%3B+local+class+incompatible
    
    btw. it's not only issue with `spark-submit`,  `spark-shell` connecting to a remote master has the same flaw and with this change it would get the message that something is wrong.. Otherwise, currently it just hangs without any additional error.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request #19802: [SPARK-22594][CORE] Handling spark-submit and mas...

Posted by srowen <gi...@git.apache.org>.
Github user srowen commented on a diff in the pull request:

    https://github.com/apache/spark/pull/19802#discussion_r160681021
  
    --- Diff: common/network-common/src/test/java/org/apache/spark/network/TransportRequestHandlerSuite.java ---
    @@ -100,6 +101,43 @@ public void handleFetchRequestAndStreamRequest() throws Exception {
         assert responseAndPromisePairs.size() == 3;
       }
     
    +    @Test
    +    public void handleOneWayMessageWithWrongSerialVersionUID() throws Exception {
    +        RpcHandler rpcHandler = new NoOpRpcHandler();
    +        Channel channel = mock(Channel.class);
    +        List<Pair<Object, ExtendedChannelPromise>> responseAndPromisePairs =
    +                new ArrayList<>();
    +
    +        when(channel.writeAndFlush(any()))
    +                .thenAnswer(invocationOnMock -> {
    +                    Object response = invocationOnMock.getArguments()[0];
    +                    ExtendedChannelPromise channelFuture = new ExtendedChannelPromise(channel);
    +                    responseAndPromisePairs.add(ImmutablePair.of(response, channelFuture));
    +                    return channelFuture;
    +                });
    +
    +        TransportClient reverseClient = mock(TransportClient.class);
    +        TransportRequestHandler requestHandler = new TransportRequestHandler(channel, reverseClient,
    +                rpcHandler, 2L);
    +
    +        // req.body().nioByteBuffer() is the method that throws the InvalidClassException
    +        // with wrong svUID, so let's mock it
    +        ManagedBuffer body = mock(ManagedBuffer.class);
    +        when(body.nioByteBuffer()).thenThrow(new InvalidClassException("test - wrong version"));
    +        RequestMessage msg = new OneWayMessage(body);
    +
    +        requestHandler.handle(msg);
    +
    +        assert responseAndPromisePairs.size() == 1;
    +        assert responseAndPromisePairs.get(0).getLeft() instanceof RpcFailure;
    +        assert (responseAndPromisePairs.get(0).getLeft().toString()
    +                .contains("version mismatch between client and server"));
    +        assert (((RpcFailure)responseAndPromisePairs.get(0).getLeft()).requestId
    --- End diff --
    
    Space after casts


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #19802: [SPARK-22594][CORE] Handling spark-submit and master ver...

Posted by Jiri-Kremser <gi...@git.apache.org>.
Github user Jiri-Kremser commented on the issue:

    https://github.com/apache/spark/pull/19802
  
    I've also added tests and removed "WIP" from the label? Do you guys want the change or would you like to have something more elaborated, like an initial handshake?
    
    One note about the "handshake approach". If there is new initial message that would be sent from client to server in which the client sends its version and server decides if it's backward compatible or not with this particular client version, this approach itself wouldn't solve the issue when old client (that doesn't know anything about it) tries to talk with new server. This PR solves this scenario.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #19802: [SPARK-22594][CORE] Handling spark-submit and master ver...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/19802
  
    **[Test build #4086 has finished](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/4086/testReport)** for PR 19802 at commit [`4f79632`](https://github.com/apache/spark/commit/4f79632d22b67128a6be8a285f4fc1fec0d5f12f).
     * This patch passes all tests.
     * This patch merges cleanly.
     * This patch adds no public classes.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request #19802: [WIP][SPARK-22594][CORE] Handling spark-submit an...

Posted by srowen <gi...@git.apache.org>.
Github user srowen commented on a diff in the pull request:

    https://github.com/apache/spark/pull/19802#discussion_r152959040
  
    --- Diff: common/network-common/src/main/java/org/apache/spark/network/server/TransportRequestHandler.java ---
    @@ -206,6 +207,11 @@ public void onFailure(Throwable e) {
       private void processOneWayMessage(OneWayMessage req) {
         try {
           rpcHandler.receive(reverseClient, req.body().nioByteBuffer());
    +    } catch (InvalidClassException ice) {
    --- End diff --
    
    The problem is that this won't catch all or even most errors resulting from version incompatibility. Spark has never supported or contemplated mis-matching versions internally. I don't think we should try to handle this, because it's fundamentally piecemeal and hacky.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request #19802: [SPARK-22594][CORE] Handling spark-submit and mas...

Posted by srowen <gi...@git.apache.org>.
Github user srowen commented on a diff in the pull request:

    https://github.com/apache/spark/pull/19802#discussion_r160680903
  
    --- Diff: common/network-common/src/main/java/org/apache/spark/network/protocol/RpcFailure.java ---
    @@ -24,6 +24,7 @@
     public final class RpcFailure extends AbstractMessage implements ResponseMessage {
       public final long requestId;
       public final String errorString;
    +  public static final long EMPTY_REQUEST_ID = Long.MIN_VALUE;
    --- End diff --
    
    Nit: put static members first and separated by whitespace.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #19802: [SPARK-22594][CORE] Handling spark-submit and master ver...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/19802
  
    Can one of the admins verify this patch?


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #19802: [SPARK-22594][CORE] Handling spark-submit and master ver...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/19802
  
    Can one of the admins verify this patch?


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #19802: [SPARK-22594][CORE] Handling spark-submit and master ver...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/19802
  
    **[Test build #4086 has started](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/4086/testReport)** for PR 19802 at commit [`4f79632`](https://github.com/apache/spark/commit/4f79632d22b67128a6be8a285f4fc1fec0d5f12f).


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request #19802: [SPARK-22594][CORE] Handling spark-submit and mas...

Posted by srowen <gi...@git.apache.org>.
Github user srowen commented on a diff in the pull request:

    https://github.com/apache/spark/pull/19802#discussion_r160681065
  
    --- Diff: common/network-common/src/test/java/org/apache/spark/network/TransportRequestHandlerSuite.java ---
    @@ -100,6 +101,43 @@ public void handleFetchRequestAndStreamRequest() throws Exception {
         assert responseAndPromisePairs.size() == 3;
       }
     
    +    @Test
    +    public void handleOneWayMessageWithWrongSerialVersionUID() throws Exception {
    +        RpcHandler rpcHandler = new NoOpRpcHandler();
    +        Channel channel = mock(Channel.class);
    +        List<Pair<Object, ExtendedChannelPromise>> responseAndPromisePairs =
    +                new ArrayList<>();
    +
    +        when(channel.writeAndFlush(any()))
    +                .thenAnswer(invocationOnMock -> {
    +                    Object response = invocationOnMock.getArguments()[0];
    +                    ExtendedChannelPromise channelFuture = new ExtendedChannelPromise(channel);
    +                    responseAndPromisePairs.add(ImmutablePair.of(response, channelFuture));
    +                    return channelFuture;
    +                });
    +
    +        TransportClient reverseClient = mock(TransportClient.class);
    +        TransportRequestHandler requestHandler = new TransportRequestHandler(channel, reverseClient,
    +                rpcHandler, 2L);
    +
    +        // req.body().nioByteBuffer() is the method that throws the InvalidClassException
    +        // with wrong svUID, so let's mock it
    +        ManagedBuffer body = mock(ManagedBuffer.class);
    +        when(body.nioByteBuffer()).thenThrow(new InvalidClassException("test - wrong version"));
    +        RequestMessage msg = new OneWayMessage(body);
    +
    +        requestHandler.handle(msg);
    +
    +        assert responseAndPromisePairs.size() == 1;
    +        assert responseAndPromisePairs.get(0).getLeft() instanceof RpcFailure;
    +        assert (responseAndPromisePairs.get(0).getLeft().toString()
    --- End diff --
    
    I don't think it's necessary to assert about the text because it could change


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request #19802: [WIP][SPARK-22594][CORE] Handling spark-submit an...

Posted by Jiri-Kremser <gi...@git.apache.org>.
Github user Jiri-Kremser commented on a diff in the pull request:

    https://github.com/apache/spark/pull/19802#discussion_r152960380
  
    --- Diff: common/network-common/src/main/java/org/apache/spark/network/server/TransportRequestHandler.java ---
    @@ -206,6 +207,11 @@ public void onFailure(Throwable e) {
       private void processOneWayMessage(OneWayMessage req) {
         try {
           rpcHandler.receive(reverseClient, req.body().nioByteBuffer());
    +    } catch (InvalidClassException ice) {
    --- End diff --
    
    Perhaps I should have picked different name for the PR than `Handling spark-submit and master version mismatch`. It doesn't try to solve the issue in a complex way that two different version could be able to talk to each other, all it does is saying the user. Hey, you have probably different version than spark master. I agree, it's little bit hacky, on the other hand I see no other option than to catch the `InvalidClassException`, if the version is not part of the message. Perhaps some initial handshake in which the version is sent would be cleaner.
    
    What about re-throwing the exception. This way it wouldn't change the semantics of the code, but the client would be informed. wdyt?


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #19802: [SPARK-22594][CORE] Handling spark-submit and master ver...

Posted by vanzin <gi...@git.apache.org>.
Github user vanzin commented on the issue:

    https://github.com/apache/spark/pull/19802
  
    @Jiri-Kremser if you're not planning to address the feedback you should probably close this PR.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #19802: [WIP][SPARK-22594][CORE] Handling spark-submit and maste...

Posted by Jiri-Kremser <gi...@git.apache.org>.
Github user Jiri-Kremser commented on the issue:

    https://github.com/apache/spark/pull/19802
  
    > Can you please explain more, and how to reproduce this issue?
    
    Sure,
    1. Start master and worker (in version 2.2.0 for instance):
    ```bash
    cat ./python/pyspark/version.py
    __version__='2.2.0'
    ```
    ```bash
    ./bin/spark-class org.apache.spark.deploy.worker.Worker spark://`hostname -I | cut -d' ' -f1`:7077 -c 1 -m 1G &> /dev/null &
    ./bin/spark-class org.apache.spark.deploy.master.Master 
    ```
    2. Then in another terminal run `spark-submit` from Spark in different version (be it `2.3.0-SNAPSHOT`)
    ```bash
    cat ./python/pyspark/version.py
    ...
    __version__ = "2.3.0.dev0"
    ```
    ```bash
    ./bin/spark-submit --class org.apache.spark.examples.SparkPi \
                       --master spark://`hostname -I | cut -d' ' -f1`:7077 \
                       --executor-memory 512M \
                       $PWD/examples/target/scala-2.11/jars/spark-examples_2.11-2.3.0-SNAPSHOT.jar \
                       10
    ```
    
    
    result: in the Spark Master log there is the following exception:
    ```
    java.io.InvalidClassException: org.apache.spark.rpc.RpcEndpointRef; local class incompatible: stream classdesc serialVersionUID = -1329125091869941550, local class serialVersionUID = 1835832137613908542
    ```
    but on the `spark-submit` terminal there is nothing about the possibility of running a different version, actually it tries couple of times to connect and then fails on `17/11/24 13:13:58 ERROR StandaloneSchedulerBackend: Application has been killed. Reason: All masters are unresponsive! Giving up.
    `, here is the complete log: https://pastebin.com/Wzs8vjBd
    
    > Spark's RPC is not designed for version compatible.
    
    I hear you, on the other hand PR doesn't even try to make it compatible, all it does is to translate a cryptic error to more understandable one. I think, it may be quite common to run older spark-submit against the updated spark master or at least I've hit the issue couple of times. And I had to google the exception and then on stackoverflow or elsewhere figure out that it actually means that there was a version discrepancy.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #19802: [SPARK-22594][CORE] Handling spark-submit and master ver...

Posted by vanzin <gi...@git.apache.org>.
Github user vanzin commented on the issue:

    https://github.com/apache/spark/pull/19802
  
    I think this is the wrong place for the fix, if one is desired. The transport library doesn't know anything about serialization, that's all handled by the code using it. In this case, the 'RpcEnv' stuff in core. So I don't think it's correct to handle those errors in the transport library.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #19802: [WIP][SPARK-22594][CORE] Handling spark-submit and maste...

Posted by jerryshao <gi...@git.apache.org>.
Github user jerryshao commented on the issue:

    https://github.com/apache/spark/pull/19802
  
    Can you please explain more, and how to reproduce this issue? Spark's RPC is not designed for version compatible.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request #19802: [SPARK-22594][CORE] Handling spark-submit and mas...

Posted by Jiri-Kremser <gi...@git.apache.org>.
Github user Jiri-Kremser commented on a diff in the pull request:

    https://github.com/apache/spark/pull/19802#discussion_r160688029
  
    --- Diff: common/network-common/src/test/java/org/apache/spark/network/TransportRequestHandlerSuite.java ---
    @@ -100,6 +101,43 @@ public void handleFetchRequestAndStreamRequest() throws Exception {
         assert responseAndPromisePairs.size() == 3;
       }
     
    +    @Test
    +    public void handleOneWayMessageWithWrongSerialVersionUID() throws Exception {
    +        RpcHandler rpcHandler = new NoOpRpcHandler();
    +        Channel channel = mock(Channel.class);
    +        List<Pair<Object, ExtendedChannelPromise>> responseAndPromisePairs =
    +                new ArrayList<>();
    +
    +        when(channel.writeAndFlush(any()))
    +                .thenAnswer(invocationOnMock -> {
    +                    Object response = invocationOnMock.getArguments()[0];
    +                    ExtendedChannelPromise channelFuture = new ExtendedChannelPromise(channel);
    +                    responseAndPromisePairs.add(ImmutablePair.of(response, channelFuture));
    +                    return channelFuture;
    +                });
    +
    +        TransportClient reverseClient = mock(TransportClient.class);
    +        TransportRequestHandler requestHandler = new TransportRequestHandler(channel, reverseClient,
    +                rpcHandler, 2L);
    +
    +        // req.body().nioByteBuffer() is the method that throws the InvalidClassException
    +        // with wrong svUID, so let's mock it
    +        ManagedBuffer body = mock(ManagedBuffer.class);
    +        when(body.nioByteBuffer()).thenThrow(new InvalidClassException("test - wrong version"));
    +        RequestMessage msg = new OneWayMessage(body);
    +
    +        requestHandler.handle(msg);
    +
    +        assert responseAndPromisePairs.size() == 1;
    --- End diff --
    
    :+1: 
    ah, long time no Java :man_facepalming: 


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #19802: [WIP][SPARK-22594][CORE] Handling spark-submit and maste...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/19802
  
    Can one of the admins verify this patch?


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request #19802: [SPARK-22594][CORE] Handling spark-submit and mas...

Posted by Jiri-Kremser <gi...@git.apache.org>.
Github user Jiri-Kremser closed the pull request at:

    https://github.com/apache/spark/pull/19802


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #19802: [SPARK-22594][CORE] Handling spark-submit and master ver...

Posted by Jiri-Kremser <gi...@git.apache.org>.
Github user Jiri-Kremser commented on the issue:

    https://github.com/apache/spark/pull/19802
  
    @srowen is there anything I can improve in the PR?


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #19802: [SPARK-22594][CORE] Handling spark-submit and master ver...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/19802
  
    Can one of the admins verify this patch?


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #19802: [SPARK-22594][CORE] Handling spark-submit and master ver...

Posted by srowen <gi...@git.apache.org>.
Github user srowen commented on the issue:

    https://github.com/apache/spark/pull/19802
  
    @vanzin do you have thoughts on this one?


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request #19802: [SPARK-22594][CORE] Handling spark-submit and mas...

Posted by srowen <gi...@git.apache.org>.
Github user srowen commented on a diff in the pull request:

    https://github.com/apache/spark/pull/19802#discussion_r160680946
  
    --- Diff: common/network-common/src/test/java/org/apache/spark/network/TransportRequestHandlerSuite.java ---
    @@ -100,6 +101,43 @@ public void handleFetchRequestAndStreamRequest() throws Exception {
         assert responseAndPromisePairs.size() == 3;
       }
     
    +    @Test
    +    public void handleOneWayMessageWithWrongSerialVersionUID() throws Exception {
    +        RpcHandler rpcHandler = new NoOpRpcHandler();
    --- End diff --
    
    2 space, not 4 space indent


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #19802: [SPARK-22594][CORE] Handling spark-submit and master ver...

Posted by Jiri-Kremser <gi...@git.apache.org>.
Github user Jiri-Kremser commented on the issue:

    https://github.com/apache/spark/pull/19802
  
    @srowen was Jenkins ok with the change? I can't see the results.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #19802: [WIP][SPARK-22594][CORE] Handling spark-submit and maste...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/19802
  
    Can one of the admins verify this patch?


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request #19802: [SPARK-22594][CORE] Handling spark-submit and mas...

Posted by srowen <gi...@git.apache.org>.
Github user srowen commented on a diff in the pull request:

    https://github.com/apache/spark/pull/19802#discussion_r160680820
  
    --- Diff: common/network-common/src/test/java/org/apache/spark/network/TransportRequestHandlerSuite.java ---
    @@ -100,6 +101,43 @@ public void handleFetchRequestAndStreamRequest() throws Exception {
         assert responseAndPromisePairs.size() == 3;
       }
     
    +    @Test
    +    public void handleOneWayMessageWithWrongSerialVersionUID() throws Exception {
    +        RpcHandler rpcHandler = new NoOpRpcHandler();
    +        Channel channel = mock(Channel.class);
    +        List<Pair<Object, ExtendedChannelPromise>> responseAndPromisePairs =
    +                new ArrayList<>();
    +
    +        when(channel.writeAndFlush(any()))
    +                .thenAnswer(invocationOnMock -> {
    +                    Object response = invocationOnMock.getArguments()[0];
    +                    ExtendedChannelPromise channelFuture = new ExtendedChannelPromise(channel);
    +                    responseAndPromisePairs.add(ImmutablePair.of(response, channelFuture));
    +                    return channelFuture;
    +                });
    +
    +        TransportClient reverseClient = mock(TransportClient.class);
    +        TransportRequestHandler requestHandler = new TransportRequestHandler(channel, reverseClient,
    +                rpcHandler, 2L);
    +
    +        // req.body().nioByteBuffer() is the method that throws the InvalidClassException
    +        // with wrong svUID, so let's mock it
    +        ManagedBuffer body = mock(ManagedBuffer.class);
    +        when(body.nioByteBuffer()).thenThrow(new InvalidClassException("test - wrong version"));
    +        RequestMessage msg = new OneWayMessage(body);
    +
    +        requestHandler.handle(msg);
    +
    +        assert responseAndPromisePairs.size() == 1;
    --- End diff --
    
    These are JVM asserts, not JUnit asserts.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org