You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by "andygrove (via GitHub)" <gi...@apache.org> on 2023/12/11 14:34:43 UTC
[I] Error, message length too large: found 7666438 bytes, the limit is: 4194304 bytes [arrow-ballista]
andygrove opened a new issue, #773:
URL: https://github.com/apache/arrow-ballista/issues/773
**Describe the bug**
I tried running some benchmarks, but some queries fail with this error:
```
2023-05-14T16:00:52.679602Z WARN tokio-runtime-worker ThreadId(47) ballista_executor::execution_loop: Executor poll work loop failed. If this continues to happen the Scheduler might be marked as dead. Error: status: OutOfRange, message: "Error, message length too large: found 7666438 bytes, the limit is: 4194304 bytes", details: [], metadata: MetadataMap { headers: {"content-type": "application/grpc", "date": "Sun, 14 May 2023 16:00:52 GMT"} }
```
**To Reproduce**
Start cluster:
```bash
./target/release/ballista-scheduler
./target/release/ballista-executor -c 24
```
Run TPC-H benchmarks
**Expected behavior**
Should not fail
**Additional context**
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Error, message length too large: found 7666438 bytes, the limit is: 4194304 bytes [arrow-ballista]
Posted by "andygrove (via GitHub)" <gi...@apache.org>.
andygrove commented on issue #773:
URL: https://github.com/apache/arrow-ballista/issues/773#issuecomment-1851202173
We set max encode/decode message size when creating the gRPC servers, but not for the clients, so I ran into this again.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-ballista] yahoNanJing commented on issue #773: Error, message length too large: found 7666438 bytes, the limit is: 4194304 bytes
Posted by "yahoNanJing (via GitHub)" <gi...@apache.org>.
yahoNanJing commented on issue #773:
URL: https://github.com/apache/arrow-ballista/issues/773#issuecomment-1564038632
Hi @andygrove, we also meet the same issue. I will propose a PR to add a config to make the maximum decoded message size configurable.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Error, message length too large: found 7666438 bytes, the limit is: 4194304 bytes [arrow-ballista]
Posted by "andygrove (via GitHub)" <gi...@apache.org>.
andygrove commented on issue #773:
URL: https://github.com/apache/arrow-ballista/issues/773#issuecomment-1850206740
I am still running into this error with the latest code.
```
2023-12-11T14:31:18.347839Z WARN task_runner ThreadId(82) ballista_executor::cpu_bound_executor: Spawned task output ignored: receiver dropped
2023-12-11T14:31:18.484649Z WARN tokio-runtime-worker ThreadId(45) ballista_executor::execution_loop: Executor poll work loop failed. If this continues to happen the Scheduler might be marked as dead. Error: status: OutOfRange, message: "Error, message length too large: found 7700152 bytes, the limit is: 4194304 bytes", details: [], metadata: MetadataMap { headers: {"content-type": "application/grpc", "date": "Mon, 11 Dec 2023 14:31:18 GMT"} }
```
I am using the default `--grpc-server-max-decoding-message-size` size of 16 MB, but the limit still appears to be 4 MB.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Error, message length too large: found 7666438 bytes, the limit is: 4194304 bytes [arrow-ballista]
Posted by "andygrove (via GitHub)" <gi...@apache.org>.
andygrove closed issue #773: Error, message length too large: found 7666438 bytes, the limit is: 4194304 bytes
URL: https://github.com/apache/arrow-ballista/issues/773
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-ballista] yahoNanJing closed issue #773: Error, message length too large: found 7666438 bytes, the limit is: 4194304 bytes
Posted by "yahoNanJing (via GitHub)" <gi...@apache.org>.
yahoNanJing closed issue #773: Error, message length too large: found 7666438 bytes, the limit is: 4194304 bytes
URL: https://github.com/apache/arrow-ballista/issues/773
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Error, message length too large: found 7666438 bytes, the limit is: 4194304 bytes [arrow-ballista]
Posted by "andygrove (via GitHub)" <gi...@apache.org>.
andygrove commented on issue #773:
URL: https://github.com/apache/arrow-ballista/issues/773#issuecomment-1850255448
I confirmed that setting the max encoding size resolves the issue for me.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Error, message length too large: found 7666438 bytes, the limit is: 4194304 bytes [arrow-ballista]
Posted by "andygrove (via GitHub)" <gi...@apache.org>.
andygrove closed issue #773: Error, message length too large: found 7666438 bytes, the limit is: 4194304 bytes
URL: https://github.com/apache/arrow-ballista/issues/773
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Error, message length too large: found 7666438 bytes, the limit is: 4194304 bytes [arrow-ballista]
Posted by "Dandandan (via GitHub)" <gi...@apache.org>.
Dandandan commented on issue #773:
URL: https://github.com/apache/arrow-ballista/issues/773#issuecomment-1850234281
We've hit some other errors related to max sizes at our end, it looks like we brought these errors back by:
* increasing max size limits (for execution plan / flight message (batches))
* reducing batch size
* enabling compression
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Error, message length too large: found 7666438 bytes, the limit is: 4194304 bytes [arrow-ballista]
Posted by "Dandandan (via GitHub)" <gi...@apache.org>.
Dandandan commented on issue #773:
URL: https://github.com/apache/arrow-ballista/issues/773#issuecomment-1850236040
AFAIK one thing we can also do and not do yet is enabling (gzip) compression on the GRPC API.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Error, message length too large: found 7666438 bytes, the limit is: 4194304 bytes [arrow-ballista]
Posted by "andygrove (via GitHub)" <gi...@apache.org>.
andygrove commented on issue #773:
URL: https://github.com/apache/arrow-ballista/issues/773#issuecomment-1850214129
We currently set the decoding max size but not the encoding max size, so perhaps that is the issue. I will test this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org