You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@arrow.apache.org by "laskoviymishka (via GitHub)" <gi...@apache.org> on 2025/03/31 09:42:31 UTC
[I] Wierd panic on write [arrow-go]
laskoviymishka opened a new issue, #337:
URL: https://github.com/apache/arrow-go/issues/337
### Describe the bug, including details regarding any error messages, version, and platform.
i'm implementing streaming (https://github.com/transferia/iceberg/pull/3) sink from kafka-like sources into iceberg, and utilize iceberg to create parquet files, I expirience some weird panic in pq.Write method:
```
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x18 pc=0x3e2f7b4]
goroutine 415 [running]:
github.com/apache/arrow-go/v18/arrow/memory.(*Buffer).Bytes(...)
/home/runner/go/pkg/mod/github.com/apache/arrow-go/v18@v18.2.0/arrow/memory/buffer.go:106
github.com/apache/arrow-go/v18/parquet/file.(*page).Data(...)
/home/runner/go/pkg/mod/github.com/apache/arrow-go/v18@v18.2.0/parquet/file/page_reader.go:90
github.com/apache/arrow-go/v18/parquet/file.(*columnWriter).TotalBytesWritten(...)
/home/runner/go/pkg/mod/github.com/apache/arrow-go/v18@v18.2.0/parquet/file/column_writer.go:203
github.com/apache/arrow-go/v18/parquet/file.(*rowGroupWriter).Close(0xc0015267e0)
/home/runner/go/pkg/mod/github.com/apache/arrow-go/v18@v18.2.0/parquet/file/row_group_writer.go:237 +0x8b
github.com/apache/arrow-go/v18/parquet/pqarrow.(*FileWriter).Close(0xc000612690)
/home/runner/go/pkg/mod/github.com/apache/arrow-go/v18@v18.2.0/parquet/pqarrow/file_writer.go:303 +0x38
github.com/apache/arrow-go/v18/parquet/pqarrow.(*FileWriter).Write(0xc000612690, {0x699b5c0, 0xc001928360})
/home/runner/go/pkg/mod/github.com/apache/arrow-go/v18@v18.2.0/parquet/pqarrow/file_writer.go:243 +0x2a5
github.com/transferia/iceberg.writeFile({0xc00170bab0?, 0x61?}, 0xc00060c2a0, {0xc001553b00, 0x1, 0x1})
/home/runner/work/iceberg/iceberg/s3_writer.go:59 +0x43b
github.com/transferia/iceberg.(*SinkStreaming).writeBatch(0xc00184e1b0, 0xc00060c2a0, {0xc001553b00, 0x1, 0x1})
/home/runner/work/iceberg/iceberg/sink_streaming.go:173 +0xde
github.com/transferia/iceberg.(*SinkStreaming).writeDataToTable(...)
/home/runner/work/iceberg/iceberg/sink_streaming.go:162
github.com/transferia/iceberg.(*SinkStreaming).processTable(0xc00184e1b0, {0xc001553b00, 0x1, 0x1})
/home/runner/work/iceberg/iceberg/sink_streaming.go:105 +0x195
github.com/transferia/iceberg.(*SinkStreaming).Push(0xc00184e1b0, {0xc0015539e0, 0x1, 0x4aeb8e?})
/home/runner/work/iceberg/iceberg/sink_streaming.go:80 +0x327
github.com/transferia/transferia/pkg/middlewares.(*errorTracker).Push(0xc0010837d0, {0xc0015539e0?, 0xc000af16f0?, 0xc0015539e0?})
/home/runner/go/pkg/mod/github.com/transferia/transferia@v0.0.2/pkg/middlewares/error_tracker.go:35 +0x29
github.com/transferia/transferia/pkg/middlewares.(*outputDataMetering).Push(0xc000567810?, {0xc0015539e0, 0x1, 0x1})
/home/runner/go/pkg/mod/github.com/transferia/transferia@v0.0.2/pkg/middlewares/metering.go:65 +0x2a
github.com/transferia/transferia/pkg/middlewares.(*statistician).Push(0xc001ce63c0, {0xc0015539e0, 0x1, 0x1})
/home/runner/go/pkg/mod/github.com/transferia/transferia@v0.0.2/pkg/middlewares/statistician.go:58 +0x92
github.com/transferia/transferia/pkg/middlewares.(*filter).Push(0xc001852810, {0xc0015539e0, 0x1, 0x1})
/home/runner/go/pkg/mod/github.com/transferia/transferia@v0.0.2/pkg/middlewares/filter.go:76 +0xa3
github.com/transferia/transferia/pkg/middlewares.(*nonRowSeparator).Push(0xc000583e30, {0xc0015539e0, 0x1, 0x1})
/home/runner/go/pkg/mod/github.com/transferia/transferia@v0.0.2/pkg/middlewares/nonrow_separator.go:50 +0x37f
github.com/transferia/transferia/pkg/middlewares.(*inputDataMetering).Push(0xc000af18f0?, {0xc0015539e0, 0x1, 0x1})
/home/runner/go/pkg/mod/github.com/transferia/transferia@v0.0.2/pkg/middlewares/metering.go:43 +0x2a
github.com/transferia/transferia/pkg/middlewares/async.(*synchronizer).AsyncPush(0xc001852840, {0xc0015539e0?, 0x1, 0xc0001bc060?})
/home/runner/go/pkg/mod/github.com/transferia/transferia@v0.0.2/pkg/middlewares/async/synchronizer.go:61 +0xe5
github.com/transferia/transferia/pkg/middlewares/async.(*measurer).AsyncPush(0xc000c8b820, {0xc0015539e0, 0x1, 0x1})
/home/runner/go/pkg/mod/github.com/transferia/transferia@v0.0.2/pkg/middlewares/async/measurer.go:59 +0x146
github.com/transferia/transferia/pkg/parsequeue.(*ParseQueue[...]).pushLoop(0x69c73e0)
/home/runner/go/pkg/mod/github.com/transferia/transferia@v0.0.2/pkg/parsequeue/parsequeue.go:88 +0x13a
created by github.com/transferia/transferia/pkg/parsequeue.New[...] in goroutine 405
/home/runner/go/pkg/mod/github.com/transferia/transferia@v0.0.2/pkg/parsequeue/parsequeue.go:161 +0x1f7
```
Here is some diagnostic that I did collect:
```
Converting 1 items to Arrow Record with schema: schema:
fields: 8
- id: type=int32, nullable
- level: type=utf8, nullable
- caller: type=utf8, nullable
- msg: type=utf8, nullable
- _timestamp: type=timestamp[us, tz=UTC]
- _partition: type=binary
- _offset: type=int64
- _idx: type=int64
Processing field 0: id (type: int32)
Item 0, Field id: Value type is int32
Processing field 1: level (type: utf8)
Item 0, Field level: Value type is string
Processing field 2: caller (type: utf8)
Item 0, Field caller: Value type is string
Processing field 3: msg (type: utf8)
Item 0, Field msg: Value type is string
Processing field 4: _timestamp (type: timestamp[us, tz=UTC])
Item 0, Field _timestamp: Value type is time.Time
Processing field 5: _partition (type: binary)
Item 0, Field _partition: Value type is string
Processing field 6: _offset (type: int64)
Item 0, Field _offset: Value type is uint64
Processing field 7: _idx (type: int64)
Item 0, Field _idx: Value type is uint32
Writing record with 1 rows and 8 columns to s3://warehouse/streaming/topic1/data/00000-0-7711175e-7cbe-48a0-a534-4142d4bacede-0-00003.parquet
Recovered from panic in Write:
runtime error: invalid memory address or nil pointer dereferenceRecord details:
NumRows: 1
NumCols: 8
Column 0: id (type: int32)
Column 1: level (type: utf8)
Column 2: caller (type: utf8)
Column 3: msg (type: utf8)
Column 4: _timestamp (type: timestamp[us, tz=UTC])
Column 5: _partition (type: binary)
Column 6: _offset (type: int64)
Column 7: _idx (type: int64)
```
### Component(s)
Parquet
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@arrow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Wierd panic on write [arrow-go]
Posted by "laskoviymishka (via GitHub)" <gi...@apache.org>.
laskoviymishka closed issue #337: Wierd panic on write
URL: https://github.com/apache/arrow-go/issues/337
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Wierd panic on write [arrow-go]
Posted by "zeroshade (via GitHub)" <gi...@apache.org>.
zeroshade commented on issue #337:
URL: https://github.com/apache/arrow-go/issues/337#issuecomment-2767087215
Hmm, I haven't been able to reproduce this crash yet. Given where it's crashing, is it possible that the batch you're trying to write is empty? I'll see if attempting to write an empty batch can reproduce this crash
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Wierd panic on write [arrow-go]
Posted by "laskoviymishka (via GitHub)" <gi...@apache.org>.
laskoviymishka commented on issue #337:
URL: https://github.com/apache/arrow-go/issues/337#issuecomment-2844548868
Wasn't able to repro on latest, will close for now.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org