You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pinot.apache.org by GitBox <gi...@apache.org> on 2022/05/04 18:24:24 UTC
[GitHub] [pinot] jasperjiaguo opened a new issue, #8638: Buffer overflow exception in `VarByteChunkSVForwardIndexWriter` of Spark data push flow
jasperjiaguo opened a new issue, #8638:
URL: https://github.com/apache/pinot/issues/8638
Our Spark push flow is failing with a table using var length dictionary, but the hadoop job succeeded on the same table.
```
03-05-2022 21:11:29 PDT validate-pinot-code-spark-avro INFO - Caused by: java.nio.BufferOverflowException
03-05-2022 21:11:29 PDT validate-pinot-code-spark-avro INFO - at java.nio.DirectByteBuffer.put(DirectByteBuffer.java:363)
03-05-2022 21:11:29 PDT validate-pinot-code-spark-avro INFO - at java.nio.ByteBuffer.put(ByteBuffer.java:859)
03-05-2022 21:11:29 PDT validate-pinot-code-spark-avro INFO - at org.apache.pinot.segment.local.io.writer.impl.VarByteChunkSVForwardIndexWriter.putBytes(VarByteChunkSVForwardIndexWriter.java:101)
03-05-2022 21:11:29 PDT validate-pinot-code-spark-avro INFO - at org.apache.pinot.segment.local.io.writer.impl.VarByteChunkSVForwardIndexWriter.putString(VarByteChunkSVForwardIndexWriter.java:92)
03-05-2022 21:11:29 PDT validate-pinot-code-spark-avro INFO - at org.apache.pinot.segment.local.segment.creator.impl.fwd.SingleValueVarByteRawIndexCreator.putString(SingleValueVarByteRawIndexCreator.java:109)
03-05-2022 21:11:29 PDT validate-pinot-code-spark-avro INFO - at org.apache.pinot.segment.local.segment.creator.impl.SegmentColumnarIndexCreator.indexRow(SegmentColumnarIndexCreator.java:377)
03-05-2022 21:11:29 PDT validate-pinot-code-spark-avro INFO - at org.apache.pinot.segment.local.segment.creator.impl.SegmentIndexCreationDriverImpl.build(SegmentIndexCreationDriverImpl.java:244)
03-05-2022 21:11:29 PDT validate-pinot-code-spark-avro INFO - at <LinkedIn Spark push job>
03-05-2022 21:11:29 PDT validate-pinot-code-spark-avro INFO - at org.apache.spark.api.java.JavaRDDLike$$anonfun$foreachPartition$1.apply(JavaRDDLike.scala:219)
03-05-2022 21:11:29 PDT validate-pinot-code-spark-avro INFO - at org.apache.spark.api.java.JavaRDDLike$$anonfun$foreachPartition$1.apply(JavaRDDLike.scala:219)
03-05-2022 21:11:29 PDT validate-pinot-code-spark-avro INFO - at org.apache.spark.rdd.RDD$$anonfun$foreachPartition$1$$anonfun$apply$29.apply(RDD.scala:929)
03-05-2022 21:11:29 PDT validate-pinot-code-spark-avro INFO - at org.apache.spark.rdd.RDD$$anonfun$foreachPartition$1$$anonfun$apply$29.apply(RDD.scala:929)
03-05-2022 21:11:29 PDT validate-pinot-code-spark-avro INFO - at org.apache.spark.SparkContext$$anonfun$runJob$5.apply(SparkContext.scala:2094)
03-05-2022 21:11:29 PDT validate-pinot-code-spark-avro INFO - at org.apache.spark.SparkContext$$anonfun$runJob$5.apply(SparkContext.scala:2094)
03-05-2022 21:11:29 PDT validate-pinot-code-spark-avro INFO - at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:87)
03-05-2022 21:11:29 PDT validate-pinot-code-spark-avro INFO - at org.apache.spark.scheduler.Task.run(Task.scala:109)
03-05-2022 21:11:29 PDT validate-pinot-code-spark-avro INFO - at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:429)
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] Jackie-Jiang commented on issue #8638: Buffer overflow exception in `VarByteChunkSVForwardIndexWriter` of Spark data push flow
Posted by GitBox <gi...@apache.org>.
Jackie-Jiang commented on issue #8638:
URL: https://github.com/apache/pinot/issues/8638#issuecomment-1117687224
It think this issue is fixed in #8286. Can you please try the same job with the latest version?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] jasperjiaguo commented on issue #8638: Buffer overflow exception in `VarByteChunkSVForwardIndexWriter` of Spark data push flow
Posted by GitBox <gi...@apache.org>.
jasperjiaguo commented on issue #8638:
URL: https://github.com/apache/pinot/issues/8638#issuecomment-1130569805
@npawar No. I tested with the latest version having this commit, but still giving me the same error.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] npawar commented on issue #8638: Buffer overflow exception in `VarByteChunkSVForwardIndexWriter` of Spark data push flow
Posted by GitBox <gi...@apache.org>.
npawar commented on issue #8638:
URL: https://github.com/apache/pinot/issues/8638#issuecomment-1129396574
@jasperjiaguo were you able to try with the fix? is this resolved for you?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] jasperjiaguo commented on issue #8638: Buffer overflow exception in `VarByteChunkSVForwardIndexWriter` of Spark data push flow
Posted by GitBox <gi...@apache.org>.
jasperjiaguo commented on issue #8638:
URL: https://github.com/apache/pinot/issues/8638#issuecomment-1117665375
cc @Jackie-Jiang @siddharthteotia
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org