You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@accumulo.apache.org by GitBox <gi...@apache.org> on 2022/10/25 16:07:31 UTC
[GitHub] [accumulo] dlmarion opened a new issue, #3042: Thrift frame size errors
dlmarion opened a new issue, #3042:
URL: https://github.com/apache/accumulo/issues/3042
**Describe the bug**
Seeing error messages like the following in the manager, gc, and compaction coordinator logs:
```
[rpc.CustomNonBlockingServer$CustomFrameBuffer] ERROR: Read a frame size of 369295362, which is bigger than the maximum allowable frame size 16384000 for ALL connections.
[rpc.CustomNonBlockingServer$CustomFrameBuffer] ERROR: Read an invalid frame size of -2143289088. Are you using TFramedTransport on the client side?
```
**Versions (OS, Maven, Java, and others, as appropriate):**
- Affected version(s) of this project: 2.1.0-rc1
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [accumulo] ctubbsii commented on issue #3042: Thrift frame size errors
Posted by GitBox <gi...@apache.org>.
ctubbsii commented on issue #3042:
URL: https://github.com/apache/accumulo/issues/3042#issuecomment-1292194485
Go ahead and put the target branch for 2.1 anyway. That way, it will be included in 2.1.1 if there's no 2.1.0-rc3
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [accumulo] DomGarguilo commented on issue #3042: Thrift frame size errors
Posted by GitBox <gi...@apache.org>.
DomGarguilo commented on issue #3042:
URL: https://github.com/apache/accumulo/issues/3042#issuecomment-1290829769
I was seeing this when I had external compaction enabled. Now, running a cluster with **internal** compactions, I am not seeing this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [accumulo] DomGarguilo commented on issue #3042: Thrift frame size errors
Posted by GitBox <gi...@apache.org>.
DomGarguilo commented on issue #3042:
URL: https://github.com/apache/accumulo/issues/3042#issuecomment-1290873668
> I was seeing this when I had external compaction enabled. Now, running a cluster with **internal** compactions, I am not seeing this.
Nevermind. I just got this again on the manager with internal compactions. Only once though, whereas I was getting it a lot before.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [accumulo] ctubbsii commented on issue #3042: Thrift frame size errors
Posted by GitBox <gi...@apache.org>.
ctubbsii commented on issue #3042:
URL: https://github.com/apache/accumulo/issues/3042#issuecomment-1291431204
I will create a 2.1.0-RC2 without a fix for this, but I'm still tracking this for 2.1.0 because it may still end up blocking a release. If it doesn't, we can bump it off to 2.1.1 (assuming it's a problem, and not something scanning ports with pings or something like that)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [accumulo] dlmarion commented on issue #3042: Thrift frame size errors
Posted by GitBox <gi...@apache.org>.
dlmarion commented on issue #3042:
URL: https://github.com/apache/accumulo/issues/3042#issuecomment-1292204884
target branch changed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [accumulo] asfgit closed issue #3042: Thrift frame size errors
Posted by GitBox <gi...@apache.org>.
asfgit closed issue #3042: Thrift frame size errors
URL: https://github.com/apache/accumulo/issues/3042
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [accumulo] dlmarion commented on issue #3042: Thrift frame size errors
Posted by GitBox <gi...@apache.org>.
dlmarion commented on issue #3042:
URL: https://github.com/apache/accumulo/issues/3042#issuecomment-1290883024
I think for the GC process to get it, it has to come from the Monitor. From what I can tell the clients are using a TFramedTransport.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [accumulo] ctubbsii commented on issue #3042: Thrift frame size errors
Posted by GitBox <gi...@apache.org>.
ctubbsii commented on issue #3042:
URL: https://github.com/apache/accumulo/issues/3042#issuecomment-1291373312
So, one thing that can cause this is network analysis software doing port scans/probes or anti-virus checking for stuff. Thrift might try to interpret some random bytes on the wire from that kind of stuff, and fail here when it doesn't work because it parsed out some integer that won't work. Be sure you're not running into any of that.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [accumulo] ctubbsii commented on issue #3042: Thrift frame size errors
Posted by GitBox <gi...@apache.org>.
ctubbsii commented on issue #3042:
URL: https://github.com/apache/accumulo/issues/3042#issuecomment-1290827190
@dlmarion could this be caused by a bug in the multiplexed RPCs? Or possibly an outdated internal fork of Thrift client code we copied into our own project?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [accumulo] dlmarion commented on issue #3042: Thrift frame size errors
Posted by GitBox <gi...@apache.org>.
dlmarion commented on issue #3042:
URL: https://github.com/apache/accumulo/issues/3042#issuecomment-1292161018
Put up a PR for additional logging against `main`. If we need to do RC3, then we can change the target branch and merge it in.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [accumulo] dlmarion commented on issue #3042: Thrift frame size errors
Posted by GitBox <gi...@apache.org>.
dlmarion commented on issue #3042:
URL: https://github.com/apache/accumulo/issues/3042#issuecomment-1290827885
unknown. looking...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [accumulo] dlmarion commented on issue #3042: Thrift frame size errors
Posted by GitBox <gi...@apache.org>.
dlmarion commented on issue #3042:
URL: https://github.com/apache/accumulo/issues/3042#issuecomment-1292141938
These messages come from the Thrift [FrameBuffer](https://github.com/apache/thrift/blob/master/lib/java/src/main/java/org/apache/thrift/server/AbstractNonblockingServer.java#L311) class. It's possible that we could print the client address to the log to try and identify the culprit.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [accumulo] dlmarion commented on issue #3042: Thrift frame size errors
Posted by GitBox <gi...@apache.org>.
dlmarion commented on issue #3042:
URL: https://github.com/apache/accumulo/issues/3042#issuecomment-1290857759
> could this be caused by a bug in the multiplexed RPCs?
I don't think so, as the multiplexing is at the protocol level and the frame buffer is at the transport level. Still looking...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [accumulo] dlmarion commented on issue #3042: Thrift frame size errors
Posted by GitBox <gi...@apache.org>.
dlmarion commented on issue #3042:
URL: https://github.com/apache/accumulo/issues/3042#issuecomment-1297104014
I didn't see any of these errors while testing 2.1.0 RC4, so I think that maybe something external to Accumulo may have been trying to test open ports or something.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [accumulo] dlmarion commented on issue #3042: Thrift frame size errors
Posted by GitBox <gi...@apache.org>.
dlmarion commented on issue #3042:
URL: https://github.com/apache/accumulo/issues/3042#issuecomment-1290854912
But I'm seeing this in the manager, gc, and compaction coordinator logs. I'm not sure why I would be seeing it in the gc logs if it was just related to external compactions
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [accumulo] dlmarion commented on issue #3042: Thrift frame size errors
Posted by GitBox <gi...@apache.org>.
dlmarion commented on issue #3042:
URL: https://github.com/apache/accumulo/issues/3042#issuecomment-1291012301
So, I don't think this is a blocker. Everything with the test is still running - to include Accumulo processes, CI Ingest processes, and CI Agitator.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [accumulo] DomGarguilo commented on issue #3042: Thrift frame size errors
Posted by GitBox <gi...@apache.org>.
DomGarguilo commented on issue #3042:
URL: https://github.com/apache/accumulo/issues/3042#issuecomment-1292087477
> So, one thing that can cause this is network analysis software doing port scans/probes or anti-virus checking for stuff. Thrift might try to interpret some random bytes on the wire from that kind of stuff, and fail here when it doesn't work because it parsed out some integer that won't work. Be sure you're not running into any of that.
I was seeing this on a generic Centos 7 image which I assume does not ship with those sort of tools. Might be something similar though, not sure. Just commenting in case this helps narrow things down.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org