You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@zeppelin.apache.org by zjffdu <gi...@git.apache.org> on 2018/02/15 02:30:40 UTC
[GitHub] zeppelin pull request #2802: ZEPPELIN-3236. Make grpc framesize configurable
GitHub user zjffdu opened a pull request:
https://github.com/apache/zeppelin/pull/2802
ZEPPELIN-3236. Make grpc framesize configurable
### What is this PR for?
A few sentences describing the overall goals of the pull request's commits.
First time? Check out the contributing guide - https://zeppelin.apache.org/contribution/contributions.html
### What type of PR is it?
[Bug Fix | Improvement | Feature | Documentation | Hot Fix | Refactoring]
### Todos
* [ ] - Task
### What is the Jira issue?
* Open an issue on Jira https://issues.apache.org/jira/browse/ZEPPELIN/
* Put link here, and add [ZEPPELIN-*Jira number*] in PR title, eg. [ZEPPELIN-533]
### How should this be tested?
* First time? Setup Travis CI as described on https://zeppelin.apache.org/contribution/contributions.html#continuous-integration
* Strongly recommended: add automated unit tests for any new or changed behavior
* Outline any manual steps to test the PR here.
### Screenshots (if appropriate)
### Questions:
* Does the licenses files need update?
* Is there breaking changes for older versions?
* Does this needs documentation?
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/zjffdu/zeppelin ZEPPELIN-3236
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/zeppelin/pull/2802.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #2802
----
commit 51431103075bfee8bc027af50ef06b7415079080
Author: Jeff Zhang <zj...@...>
Date: 2018-02-15T02:30:10Z
ZEPPELIN-3236. Make grpc framesize configurable
----
---
[GitHub] zeppelin issue #2802: ZEPPELIN-3236. Make grpc framesize configurable
Posted by ejona86 <gi...@git.apache.org>.
Github user ejona86 commented on the issue:
https://github.com/apache/zeppelin/pull/2802
I just changed them to be more clear in grpc/grpc-java#4106. Each message gets its own gRPC frame, but the very fact there are gRPC frames is more of an internal detail; users shouldn't need to know about such things.
---
[GitHub] zeppelin issue #2802: ZEPPELIN-3236. Make grpc framesize configurable
Posted by Tagar <gi...@git.apache.org>.
Github user Tagar commented on the issue:
https://github.com/apache/zeppelin/pull/2802
Thanks for the heads up, yep I figured out that I have to tune up zeppelin.ipython.grpc.framesize to a large number.
I looked over the PR. Two quick suggestions
1) Would it be possible to make spark interpreter keep and not close the stream if such an exception happens? We can see a higher limit, but I am sure users will have cases when they will try to go higher. The Spark interpreter is then in a bad state and only way to fix this is to try increase a limit again.. Not sure if this problem belongs to Zeppelin or to grpc, so provisionally opened an issue in grpc too - https://github.com/grpc/grpc-java/issues/4086
2) Should we increase the default? .. 4Mb isn't that hard to hit when ipython returns a mid-size dataset / table.
---
[GitHub] zeppelin issue #2802: ZEPPELIN-3236. Make grpc framesize configurable
Posted by Tagar <gi...@git.apache.org>.
Github user Tagar commented on the issue:
https://github.com/apache/zeppelin/pull/2802
LGTM
thanks a lot @zjffdu
---
[GitHub] zeppelin issue #2802: ZEPPELIN-3236. Make grpc framesize configurable
Posted by Tagar <gi...@git.apache.org>.
Github user Tagar commented on the issue:
https://github.com/apache/zeppelin/pull/2802
> ERROR [2018-02-14 10:39:10,922] ({grpc-default-executor-3} IPythonClient.java[onError]:138) - Fail to call IPython grpc
> io.grpc.StatusRuntimeException: RESOURCE_EXHAUSTED: io.grpc.netty.NettyClientTransport$3: Frame size 216695976 exceeds maximum: 4194304.
> at io.grpc.Status.asRuntimeException(Status.java:543)
> at io.grpc.stub.ClientCalls$StreamObserverToCallListenerAdapter.onClose(ClientCalls.java:395)
> at io.grpc.internal.ClientCallImpl.closeObserver(ClientCallImpl.java:426)
> at io.grpc.internal.ClientCallImpl.access$100(ClientCallImpl.java:76)
> at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl.close(ClientCallImpl.java:512)
> at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl.access$700(ClientCallImpl.java:429)
> at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInContext(ClientCallImpl.java:544)
> at io.grpc.internal.ContextRunnable.run(ContextRunnable.java:52)
> at io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:117)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> INFO [2018-02-14 10:39:10,923] ({grpc-default-worker-ELG-1-2} AbstractClientStream2.java[inboundDataReceived]:249) - Received data on closed stream
> INFO [2018-02-14 10:39:10,924] ({grpc-default-worker-ELG-1-2} AbstractClientStream2.java[inboundDataReceived]:249) - Received data on closed stream
> INFO [2018-02-14 10:39:10,925] ({grpc-default-worker-ELG-1-2} AbstractClientStream2.java[inboundDataReceived]:249) - Received data on closed stream
>
@zjffdu that's what I have.. is this helpful? It seems contain full exception stack.
---
[GitHub] zeppelin issue #2802: ZEPPELIN-3236. Make grpc framesize configurable
Posted by zjffdu <gi...@git.apache.org>.
Github user zjffdu commented on the issue:
https://github.com/apache/zeppelin/pull/2802
@Tagar Can you paste the full log ? I'd like to see the full stacktrace.
---
[GitHub] zeppelin issue #2802: ZEPPELIN-3236. Make grpc framesize configurable
Posted by zjffdu <gi...@git.apache.org>.
Github user zjffdu commented on the issue:
https://github.com/apache/zeppelin/pull/2802
rename the property in #2813
---
[GitHub] zeppelin issue #2802: ZEPPELIN-3236. Make grpc framesize configurable
Posted by Tagar <gi...@git.apache.org>.
Github user Tagar commented on the issue:
https://github.com/apache/zeppelin/pull/2802
@zjffdu I don't have a reproducible case yet. This error hasn't been very consistent for me so it might be also a side effect of another problem. That's why I asked above if we could capture exceptions in grpc better. Also check what you think of the suggestion made in grpc/grpc-java#4086 to make a new grpc stream for each call?
---
[GitHub] zeppelin pull request #2802: ZEPPELIN-3236. Make grpc framesize configurable
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/zeppelin/pull/2802
---
[GitHub] zeppelin issue #2802: ZEPPELIN-3236. Make grpc framesize configurable
Posted by zjffdu <gi...@git.apache.org>.
Github user zjffdu commented on the issue:
https://github.com/apache/zeppelin/pull/2802
@Tagar Do you have any sample code to reproduce this issue ? So that I can verify it and also may include it in unit test.
---
[GitHub] zeppelin issue #2802: ZEPPELIN-3236. Make grpc framesize configurable
Posted by Tagar <gi...@git.apache.org>.
Github user Tagar commented on the issue:
https://github.com/apache/zeppelin/pull/2802
1) My main point was that this exception should be thrown to the user, so he or she has a chance to increase this limit. Currently if it breaks, only way to find out about this limitation is to enable debugging and not a lot of users can do that.
2) You're right .. it's 200M not sure how that user got that much data. That wasn't from my code, but from a colleague of mine. I guess it was a larger table of data. Would you mine making default somewhere in the range 16-32M? I think a lot of folks would run into the 4M limit.
3) Also, it would be great if IPythonInterpreter would catch exceptions better. Found another problem - https://issues.apache.org/jira/browse/ZEPPELIN-3239 - unrelated to this one, but it also shows the same symptoms to the user - Spark interpreter just becomes irresponsive.
---
[GitHub] zeppelin issue #2802: ZEPPELIN-3236. Make grpc framesize configurable
Posted by zjffdu <gi...@git.apache.org>.
Github user zjffdu commented on the issue:
https://github.com/apache/zeppelin/pull/2802
@ejona86 The error from grpc is about `Frame size exceed`
---
[GitHub] zeppelin issue #2802: ZEPPELIN-3236. Make grpc framesize configurable
Posted by Tagar <gi...@git.apache.org>.
Github user Tagar commented on the issue:
https://github.com/apache/zeppelin/pull/2802
thank you @zjffdu - I will check this today.
---
[GitHub] zeppelin issue #2802: ZEPPELIN-3236. Make grpc framesize configurable
Posted by zjffdu <gi...@git.apache.org>.
Github user zjffdu commented on the issue:
https://github.com/apache/zeppelin/pull/2802
@Tagar , 4 Mb is the default value of gprc, I am fine to use another large. But from your error logs, your code will produce 200M data, I am curious to know how do you produce that, do you mind to paste your code to reproduce it ?
```
io.grpc.StatusRuntimeException: RESOURCE_EXHAUSTED: io.grpc.netty.NettyClientTransport$3: Frame size 216695976 exceeds maximum: 4194304.
```
---
[GitHub] zeppelin issue #2802: ZEPPELIN-3236. Make grpc framesize configurable
Posted by Tagar <gi...@git.apache.org>.
Github user Tagar commented on the issue:
https://github.com/apache/zeppelin/pull/2802
@zjffdu I just verified this new commit and it works great.
Glad to see exceptions in ipython backend are now being captured correctly.
I wasn't able to actually test frame size problem, as it's hard to reproduce.
As @ejona86 said today in https://github.com/grpc/grpc-java/issues/4086#issuecomment-367053465
> Note that something else is broken if you're seeing that error. You should not be receiving a frame that large. (Chances are it's not actually a frame length and we're seeing garbage.)
So I think that frame size error might be caused by some other problem.
But it's great we can make frame size configurable now.
---
[GitHub] zeppelin issue #2802: ZEPPELIN-3236. Make grpc framesize configurable
Posted by ejona86 <gi...@git.apache.org>.
Github user ejona86 commented on the issue:
https://github.com/apache/zeppelin/pull/2802
@Tagar, I mis-read the error earlier. That message is just "gRPC message too large." It'd probably be good to talk about "messages" instead of "frames" here.
---
[GitHub] zeppelin issue #2802: ZEPPELIN-3236. Make grpc framesize configurable
Posted by zjffdu <gi...@git.apache.org>.
Github user zjffdu commented on the issue:
https://github.com/apache/zeppelin/pull/2802
Thanks @ejona86
---
[GitHub] zeppelin issue #2802: ZEPPELIN-3236. Make grpc framesize configurable
Posted by zjffdu <gi...@git.apache.org>.
Github user zjffdu commented on the issue:
https://github.com/apache/zeppelin/pull/2802
@Tagar Have you tried this PR ? Does this PR solve your issue ?
---