You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@accumulo.apache.org by GitBox <gi...@apache.org> on 2022/11/28 21:04:25 UTC
[GitHub] [accumulo] ivakegg opened a new issue, #3094: Thrift frame size errors
ivakegg opened a new issue, #3094:
URL: https://github.com/apache/accumulo/issues/3094
We are seeing the following message continuously on a tserver:
```
Read a frame size of 1458446612, which is bigger than the maximum allowable buffer sized for ALL connections.
```
I realized that #3042 reported the same thing against 2.1. However this is against 1.10.2, and this does not appear to be because of any port scanning or other activity as such. The reason I say that only 1 tserver is reporting this, and as soon as I drop that tserver the errors move to another tserver. Hence it is somehow tied to a tablet.
After increasing the logging on various thing, it seem correlated with opening connections to tservers that are only handling the accumulo.metadata table but I do not have absolute proof of this.
Version 1.10.2 of accumulo
CentOS 7.5
Java Corretto-8.302.,08 (1.8.0_302-b08)
Hadoop 3.3.3 (vanilla)
Zookeeper 3.7.1
The error comes from the org.apache.thrift.server.AbstractNonblockingServer which is extended by the org.apache.accumulo.server.rpc.CustomNonBlockingServer. The error is in the read() method of the internal FramBuffer class.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [accumulo] dlmarion commented on issue #3094: Thrift frame size errors
Posted by GitBox <gi...@apache.org>.
dlmarion commented on issue #3094:
URL: https://github.com/apache/accumulo/issues/3094#issuecomment-1330545845
I added #3047 in 2.1, wonder if we can apply the same in a future 1.10.x patch release (if there is one).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [accumulo] ctubbsii commented on issue #3094: Thrift frame size errors
Posted by GitBox <gi...@apache.org>.
ctubbsii commented on issue #3094:
URL: https://github.com/apache/accumulo/issues/3094#issuecomment-1386710766
Let's leave it open for now, but I removed the 2.1.1 project label from it, because we don't have anything we can do for this right now, until we get more information.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [accumulo] ctubbsii commented on issue #3094: Thrift frame size errors
Posted by GitBox <gi...@apache.org>.
ctubbsii commented on issue #3094:
URL: https://github.com/apache/accumulo/issues/3094#issuecomment-1331179822
> I added #3047 in 2.1, wonder if we can apply the same in a future 1.10.x patch release (if there is one).
There are only a couple changes currently queued for a 1.10.3. It wouldn't hurt to backport that so it could be included with them. I'm sure we'll release it, but I don't know when.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [accumulo] cshannon commented on issue #3094: Thrift frame size errors
Posted by GitBox <gi...@apache.org>.
cshannon commented on issue #3094:
URL: https://github.com/apache/accumulo/issues/3094#issuecomment-1379196440
#3134 is now merged but I assume we may want to leave this open? This issue was reported against 1.10 and not 2.1 and the bug I fixed only applies to newer thrift versions which 1.10 does not use.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Thrift frame size errors [accumulo]
Posted by "asfgit (via GitHub)" <gi...@apache.org>.
asfgit closed issue #3094: Thrift frame size errors
URL: https://github.com/apache/accumulo/issues/3094
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [accumulo] cshannon commented on issue #3094: Thrift frame size errors
Posted by GitBox <gi...@apache.org>.
cshannon commented on issue #3094:
URL: https://github.com/apache/accumulo/issues/3094#issuecomment-1335372632
The issue is that the original patch doesn't work because the client will never be set in the thread local. The thread local is set as part of ClientInfoProcessorFactory [here](https://github.com/apache/accumulo/blob/a541197cd3304910cf80dc83b7063a2bcd3eb1d6/server/base/src/main/java/org/apache/accumulo/server/rpc/ClientInfoProcessorFactory.java#L43) and this factory is configured when using TThreadPoolServer. However, TThreadPoolServer is specifically only used for blocking servers, not nonblocking. So we will need a different approach to capture/log the client when using the non blocking server.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [accumulo] ivakegg commented on issue #3094: Thrift frame size errors
Posted by GitBox <gi...@apache.org>.
ivakegg commented on issue #3094:
URL: https://github.com/apache/accumulo/issues/3094#issuecomment-1329859188
There are not exceptions being logged anywhere, and no apparent affects from these errors that I can detect.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [accumulo] ivakegg commented on issue #3094: Thrift frame size errors
Posted by GitBox <gi...@apache.org>.
ivakegg commented on issue #3094:
URL: https://github.com/apache/accumulo/issues/3094#issuecomment-1334424209
Well, I applied the patch from #3047 and restarted accumulo. Unfortunately this is what I got:
CustomFrameBuffer.read returned false when reading data from client: null
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [accumulo] ivakegg commented on issue #3094: Thrift frame size errors
Posted by GitBox <gi...@apache.org>.
ivakegg commented on issue #3094:
URL: https://github.com/apache/accumulo/issues/3094#issuecomment-1367340950
And I have sad news.... the error does not appear to be showing up anymore. Applying patches anyway to see if we can catch this some time in the future.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [accumulo] cshannon commented on issue #3094: Thrift frame size errors
Posted by GitBox <gi...@apache.org>.
cshannon commented on issue #3094:
URL: https://github.com/apache/accumulo/issues/3094#issuecomment-1335259369
This has popped up before it looks like, did a search and found:
https://lists.apache.org/thread/pdrftl1o3c1q5mcnpq7tdfffz5oxz3m1
https://issues.apache.org/jira/browse/ACCUMULO-2360
Since the general theory in the past is some non Thrift process trying to connect to the Thrift port and causing the issue, identifying the actual client using the patch from #3407 would be ideal to find out the client so we know if it's external process or something self inflicted.
I'll take a look and see if I can figure out why the client value might have been null when logging. It uses a ThreadLocal to capture the client address so for some reason the thread that was the logging doesn't have that value set.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [accumulo] ivakegg commented on issue #3094: Thrift frame size errors
Posted by GitBox <gi...@apache.org>.
ivakegg commented on issue #3094:
URL: https://github.com/apache/accumulo/issues/3094#issuecomment-1329858569
I can't tell if the tserver that is showing the error is the instigator of the communication or whether it is a separate client somewhere. At the time this system is mostly quiet (no external scans of any significance).
These errors are occurring pretty consistently every 40 seconds to a minute. The logs suggest that it may have to do with communicating with tservers handling the accumulo.metadata. In our system we have separate tservers that exclusively handle the accumulo.metadata tablets.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [accumulo] cshannon commented on issue #3094: Thrift frame size errors
Posted by GitBox <gi...@apache.org>.
cshannon commented on issue #3094:
URL: https://github.com/apache/accumulo/issues/3094#issuecomment-1335503578
https://github.com/apache/accumulo/pull/3103 addresses the null logging issue from the original patch in #3047
The problem is the server is nonblocking so different threads can process the same FrameBuffer. When I tested and increased logging I saw different threads calling read() and invoke() on the same buffer so capturing the client address in invoke() doesn't work. The new PR just captures the client address on FrameBuffer creation and stores it in a String format for easy logging as a new FrameBuffer is allocated for each request.
With this new patch the logging should now work correctly and not return null.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [accumulo] keith-turner commented on issue #3094: Thrift frame size errors
Posted by GitBox <gi...@apache.org>.
keith-turner commented on issue #3094:
URL: https://github.com/apache/accumulo/issues/3094#issuecomment-1329863587
May be worthwhile to write a giant mutation and see what the client behavior is. Once we know what the behavior might be, might know what to look for. Wonder if we could construct a multi-megabyte insert stmt that the accumulo shell would take just to see what would happen. Although, metadata writes do not always use the batch writer, sometimes they use internal code that optimized for a single mutation.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [accumulo] EdColeman commented on issue #3094: Thrift frame size errors
Posted by GitBox <gi...@apache.org>.
EdColeman commented on issue #3094:
URL: https://github.com/apache/accumulo/issues/3094#issuecomment-1329867454
When it shifts to a new tserver - is it the same tablet? Does scanning the metadata table for the range of that tablet show anything that looks odd?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [accumulo] keith-turner commented on issue #3094: Thrift frame size errors
Posted by GitBox <gi...@apache.org>.
keith-turner commented on issue #3094:
URL: https://github.com/apache/accumulo/issues/3094#issuecomment-1329796513
Do you see any client side errors anywhere? If this is happening to an accumulo client interacting w/ the metadata table I wonder if its reading or writing. Also I wonder if it throws an exception or just retries (hopefully at least logging a debug, a warn would be nice as it may be more apparent).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [accumulo] keith-turner commented on issue #3094: Thrift frame size errors
Posted by GitBox <gi...@apache.org>.
keith-turner commented on issue #3094:
URL: https://github.com/apache/accumulo/issues/3094#issuecomment-1329806408
The Accumulo GC reads a writes to the metadata table alot, did you notice anything odd in its logs around the time these errors were happening on the tservers?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [accumulo] ctubbsii commented on issue #3094: Thrift frame size errors
Posted by GitBox <gi...@apache.org>.
ctubbsii commented on issue #3094:
URL: https://github.com/apache/accumulo/issues/3094#issuecomment-1352597829
A user on the mailing list reported the same thing. I tracked down this a little bit, and noticed that many of the error messages are showing a max frame size of 16384000, which is not anything we're setting. It appears to be the default value for `new TConfiguration()` (org.apache.thrift.TConfiguration.DEFAULT_MAX_FRAME_SIZE). It's not clear to me when this is being called and affecting our RPCs, but I wonder if this is a new change in Thrift 0.17.... or perhaps related to the fact that we have a forked copy of libthrift internals in our project in order to expose the client IP address, and our fork may be diverging from libthrift in a way that exposes this. It's not clear to me, but this does appear to be a bit of a problem, at least in some cases, beyond rogue connections from network scans or other activity hitting our servers.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [accumulo] ivakegg commented on issue #3094: Thrift frame size errors
Posted by GitBox <gi...@apache.org>.
ivakegg commented on issue #3094:
URL: https://github.com/apache/accumulo/issues/3094#issuecomment-1367303598
I have great news! This error has shown up again on one of our systems. I look through the various commits here and see if I can patch the system to determine (or fix) the issue.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org