You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pinot.apache.org by "PovilasV1 (via GitHub)" <gi...@apache.org> on 2025/07/25 11:23:21 UTC
[I] BufferUnderflowException in DataTable deserialization - 1.4.0-SNAPSHOT [pinot]
PovilasV1 opened a new issue, #16426:
URL: https://github.com/apache/pinot/issues/16426
## Summary
`BufferUnderflowException` occurs during DataTable deserialization in broker-server communication when using Apache Pinot `1.4.0-SNAPSHOT`. The issue is resolved by using stable version `1.3.0`.
## Environment
- **Pinot Version**: `1.4.0-SNAPSHOT` (jar: `pinot-all-1.4.0-SNAPSHOT-jar-with-dependencies.jar`)
- **Deployment**: Kubernetes StatefulSet
- **Image**: `apachepinot/pinot:latest-21-ms-openjdk`
- **Tables**: Both OFFLINE and REALTIME tables affected
## Stack Trace
```
[pinot-broker-3 broker] 2025/07/25 10:37:59.229 ERROR [DataTableHandler] [nioEventLoopGroup-2-1] Caught exception while deserializing data table of size: 1687 from server: pinot-server-3_O
[pinot-broker-3 broker] java.nio.BufferUnderflowException
[pinot-broker-3 broker] at java.base/java.nio.ByteBuffer.get(ByteBuffer.java:826)
[pinot-broker-3 broker] at java.base/java.nio.ByteBuffer.get(ByteBuffer.java:855)
[pinot-broker-3 broker] at org.apache.pinot.common.datatable.DataTableUtils.decodeString(DataTableUtils.java:88)
[pinot-broker-3 broker] at org.apache.pinot.common.datatable.DataTableImplV4.deserializeMetadata(DataTableImplV4.java:595)
[pinot-broker-3 broker] at org.apache.pinot.common.datatable.DataTableImplV4.<init>(DataTableImplV4.java:195)
[pinot-broker-3 broker] at org.apache.pinot.common.datatable.DataTableFactory.getDataTable(DataTableFactory.java:36)
[pinot-broker-3 broker] at org.apache.pinot.core.transport.DataTableHandler.channelRead0(DataTableHandler.java:71)
[pinot-broker-3 broker] at org.apache.pinot.core.transport.DataTableHandler.channelRead0(DataTableHandler.java:38)
[pinot-broker-3 broker] at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
[pinot-broker-3 broker] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444)
[pinot-broker-3 broker] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
[pinot-broker-3 broker] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
[pinot-broker-3 broker] at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346)
[pinot-broker-3 broker] at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318)
[pinot-broker-3 broker] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444)
[pinot-broker-3 broker] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
[pinot-broker-3 broker] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
[pinot-broker-3 broker] at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1357)
[pinot-broker-3 broker] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440)
[pinot-broker-3 broker] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
[pinot-broker-3 broker] at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:868)
[pinot-broker-3 broker] at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166)
[pinot-broker-3 broker] at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:796)
[pinot-broker-3 broker] at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:732)
[pinot-broker-3 broker] at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:658)
[pinot-broker-3 broker] at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562)
[pinot-broker-3 broker] at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:998)
[pinot-broker-3 broker] at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
[pinot-broker-3 broker] at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
[pinot-broker-3 broker] at java.base/java.lang.Thread.run(Thread.java:1583)
[pinot-broker-3 broker] 2025/07/25 10:38:09.224 ERROR [PinotClientRequest] [jersey-server-managed-async-executor-3] Query processing exceptions: {427=1 servers [pinot-server-3_O] not responded}
```
## Detailed Description
### Symptoms
- `BufferUnderflowException` occurs during DataTable deserialization on broker side
- Only affects **one specific server** (`pinot-server-3`) consistently
- Error persists across pod restarts and fresh data downloads from deep storage (GCS)
- Both OFFLINE and REALTIME tables affected
### Investigation Results
#### Server-Specific Behavior
- 6 servers total, only `pinot-server-3` consistently affected
- Issue appears related to **new segments** being assigned to the problematic server
#### Resource Analysis
- Server resources adequate: 29Gi memory limit, 18Gi usage, 4 CPU limit, 367m usage
- JVM settings: `-Xms6G -Xmx16G -XX:+UseG1GC -XX:MaxGCPauseMillis=200`
- No memory pressure, GC issues, or resource constraints identified
#### Version Analysis
- All components running identical `1.4.0-SNAPSHOT` version
- Issue resolved immediately after upgrading to stable `1.3.0`
- Confirms version-specific bug in SNAPSHOT build
## Root Cause
The issue appears to be a **bug in `1.4.0-SNAPSHOT`** affecting DataTable serialization/deserialization, specifically:
1. **New segments** contain data/metadata patterns that trigger the bug
2. **DataTable metadata serialization** creates malformed byte streams
3. **Broker-side deserialization** attempts to read more bytes than available
4. **String decoding** in metadata fails with `BufferUnderflowException`
## Reproduction Steps
1. Deploy Pinot `1.4.0-SNAPSHOT` in production environment
2. Ingest data into both OFFLINE and REALTIME tables
3. Wait for new segments to be created and assigned to servers
4. Execute queries that route to servers with newly created segments
5. Observe `BufferUnderflowException` in broker logs
## Resolution
Upgrading from `1.4.0-SNAPSHOT` to `1.3.0` completely resolved the issue with no data loss or additional configuration changes required.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [I] BufferUnderflowException in DataTable deserialization - 1.4.0-SNAPSHOT [pinot]
Posted by "gortiz (via GitHub)" <gi...@apache.org>.
gortiz commented on issue #16426:
URL: https://github.com/apache/pinot/issues/16426#issuecomment-3150102396
@PovilasV1 were you using single or multi-stage queries? Did you find the issue only while the cluster was updated (so there where nodes running 1.3.0 and others running 1.4.0?) or even after all nodes were using the same version? How did you verify it only affected a single server?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [I] BufferUnderflowException in DataTable deserialization - 1.4.0-SNAPSHOT [pinot]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed issue #16426: BufferUnderflowException in DataTable deserialization - 1.4.0-SNAPSHOT
URL: https://github.com/apache/pinot/issues/16426
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [I] BufferUnderflowException in DataTable deserialization - 1.4.0-SNAPSHOT [pinot]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #16426:
URL: https://github.com/apache/pinot/issues/16426#issuecomment-4113145804
This issue has been automatically marked as stale because it has not had any activity in the last 150 days. It will be closed in 30 days if no further activity occurs. If this issue is still relevant, please leave a comment to keep it open. Thank you for your contributions!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [I] BufferUnderflowException in DataTable deserialization - 1.4.0-SNAPSHOT [pinot]
Posted by "Jackie-Jiang (via GitHub)" <gi...@apache.org>.
Jackie-Jiang commented on issue #16426:
URL: https://github.com/apache/pinot/issues/16426#issuecomment-3137419050
cc @gortiz
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [I] BufferUnderflowException in DataTable deserialization - 1.4.0-SNAPSHOT [pinot]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #16426:
URL: https://github.com/apache/pinot/issues/16426#issuecomment-4320903419
This issue has been closed because it has been inactive for 180 days. If you believe this issue is still relevant, please feel free to reopen it or create a new issue. Thank you!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [I] BufferUnderflowException in DataTable deserialization - 1.4.0-SNAPSHOT [pinot]
Posted by "Jackie-Jiang (via GitHub)" <gi...@apache.org>.
Jackie-Jiang commented on issue #16426:
URL: https://github.com/apache/pinot/issues/16426#issuecomment-3140781075
#16474 might fix your problem. Did you run into issue during upgrade when different node running different version?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [I] BufferUnderflowException in DataTable deserialization - 1.4.0-SNAPSHOT [pinot]
Posted by "PovilasV1 (via GitHub)" <gi...@apache.org>.
PovilasV1 commented on issue #16426:
URL: https://github.com/apache/pinot/issues/16426#issuecomment-3150126858
Hi, the issue only appeared on 1.4.0, all nodes were running 1.4.0. I was running single stage queries and I only ever saw errors related to one server. After downgrading to 1.3.0 the issues stopped. I upgraded to 1.4.0 after https://github.com/apache/pinot/issues/16426#issuecomment-3140781075 and so for don't see any issues.
Could it be that the node was running 1.4.0, but maybe a different commit?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [I] BufferUnderflowException in DataTable deserialization - 1.4.0-SNAPSHOT [pinot]
Posted by "Jackie-Jiang (via GitHub)" <gi...@apache.org>.
Jackie-Jiang commented on issue #16426:
URL: https://github.com/apache/pinot/issues/16426#issuecomment-3133333278
Can you double check the version deployed on each node? Given it only affects one server, and fails on every query, it is very likely a setup issue
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [I] BufferUnderflowException in DataTable deserialization - 1.4.0-SNAPSHOT [pinot]
Posted by "gortiz (via GitHub)" <gi...@apache.org>.
gortiz commented on issue #16426:
URL: https://github.com/apache/pinot/issues/16426#issuecomment-3150132604
#16474 only affects multi-stage queries, so either the issue you found is still there or there was another patch that fixed it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [I] BufferUnderflowException in DataTable deserialization - 1.4.0-SNAPSHOT [pinot]
Posted by "PovilasV1 (via GitHub)" <gi...@apache.org>.
PovilasV1 commented on issue #16426:
URL: https://github.com/apache/pinot/issues/16426#issuecomment-3130885052
I was running `latest-21-ms-openjdk` image on kubernetes.
I would see the error on every query, for example the default `select * from table limit 10` when using controller GUI.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [I] BufferUnderflowException in DataTable deserialization - 1.4.0-SNAPSHOT [pinot]
Posted by "PovilasV1 (via GitHub)" <gi...@apache.org>.
PovilasV1 commented on issue #16426:
URL: https://github.com/apache/pinot/issues/16426#issuecomment-3136611984
I've reverted back to `1.3.0`, so maybe one pod had a different commit with the same image tag?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [I] BufferUnderflowException in DataTable deserialization - 1.4.0-SNAPSHOT [pinot]
Posted by "Jackie-Jiang (via GitHub)" <gi...@apache.org>.
Jackie-Jiang commented on issue #16426:
URL: https://github.com/apache/pinot/issues/16426#issuecomment-3130152623
Are you trying out the latest master? What is the last commit has?
It would be helpful if you can also share the query
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org