You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@beam.apache.org by GitBox <gi...@apache.org> on 2022/06/04 22:47:15 UTC
[GitHub] [beam] damccorm opened a new issue, #21354: [Go SDK] Decoding a schema row doesn't respect field names from struct tags
damccorm opened a new issue, #21354:
URL: https://github.com/apache/beam/issues/21354
Will attempt to create a reproducible code snippet soon.
For now, the basics of the bug are that when an element encoded as a Row via schemas gets decoded to a Go struct, it doesn't respect struct tags somewhere in the conversion process. More specifically, if I have an external transform that outputs a Row, and I take that as input to a native Go transform that accepts some Go struct "Foo", it will fail if the field names are different, even if the struct tags on Foo match the external row's field names.
Example error message. The following error is caused by the WordCount and CorpusDate fields in the native struct not matching the field names "word_count" and "corpus_date" from the raw Row. The row gets decoded into a struct with the field names Word_count and Corpus_date, and ignoring the struct tags of the struct it's attempting to match:
```
panic: reflect: Call using struct { Word string "beam:\"word\""; Word_count int64 "beam:\"word_count\"";
Corpus string "beam:\"corpus\""; Corpus_date int64 "beam:\"corpus_date\"" } as type struct { Word string
"beam:\"word\""; WordCount int64 "beam:\"word_count\""; Corpus string "beam:\"corpus\""; CorpusDate
int64 "beam:\"corpus_date\"" }
Full error:
while executing Process for Plan[process-bundle-descriptor-291]:
2:
DataSink[S[ptransform-289@localhost:12371]] Coder:W;coder-315<KV;coder-316<int[varintz;c3];coder-317,string;coder-318>>!GWC
3:
PCollection[pcollection-304] Out:[2]
4: ParDo[beam.addFixedKeyFn] Out:[2]
5: PCollection[pcollection-300]
Out:[4]
6: ParDo[main.main.func1] Out:[5]
1: DataSource[S[ptransform-288@localhost:12371], 0] Coder:W;coder-310<LP;coder-311<R[struct
{ Word string "beam:\"word\""; Word_count int64 "beam:\"word_count\""; Corpus string "beam:\"corpus\"";
Corpus_date int64 "beam:\"corpus_date\"" }]>>!GWC Out:6
caused by:
panic: reflect: Call using struct
{ Word string "beam:\"word\""; Word_count int64 "beam:\"word_count\""; Corpus string "beam:\"corpus\"";
Corpus_date int64 "beam:\"corpus_date\"" } as type struct { Word string "beam:\"word\""; WordCount int64
"beam:\"word_count\""; Corpus string "beam:\"corpus\""; CorpusDate int64 "beam:\"corpus_date\"" } goroutine
39 [running]:
runtime/debug.Stack()
/usr/lib/google-golang/src/runtime/debug/stack.go:24 +0x65
github.com/apache/beam/sdks/v2/go/pkg/beam/core/runtime/exec.callNoPanic.func1()
/usr/local/google/home/danoliveira/repos/beam/sdks/go/pkg/beam/core/runtime/exec/util.go:58
+0xa5
panic({0xe8f160, 0xc0003149e0})
/usr/lib/google-golang/src/runtime/panic.go:1038 +0x215
reflect.Value.call({0xc0003267e0,
0xc0003b3e10, 0x5be732}, {0x102e0e5, 0x4}, {0xc0000fe3d8, 0x1, 0x679031})
/usr/lib/google-golang/src/reflect/value.go:411
+0x1965
reflect.Value.Call({0xc0003267e0, 0xc0003b3e10, 0x1}, {0xc0000fe3d8, 0x1, 0x1})
/usr/lib/google-golang/src/reflect/value.go:339
+0xc5
github.com/apache/beam/sdks/v2/go/pkg/beam/core/util/reflectx.(*reflectFunc).Call(0xc000426060,
{0xc0003326f0, 0x0, 0xf5ef00})
/usr/local/google/home/danoliveira/repos/beam/sdks/go/pkg/beam/core/util/reflectx/call.go:87
+0x59
github.com/apache/beam/sdks/v2/go/pkg/beam/core/runtime/exec.(*invoker).initCall.func33({0x18e7a90,
0x1, 0x1}, 0xffdf3b645a1cac09)
/usr/local/google/home/danoliveira/repos/beam/sdks/go/pkg/beam/core/runtime/exec/fn_arity.go:229
+0x7b
github.com/apache/beam/sdks/v2/go/pkg/beam/core/runtime/exec.(*invoker).Invoke(0xc000477860,
{0x1168a50, 0xc00033cdc0}, {0x18e7a90, 0x0, 0x1}, 0x203000, 0xc00015d660, {0x1934830, 0x0, ...})
/usr/local/google/home/danoliveira/repos/beam/sdks/go/pkg/beam/core/runtime/exec/fn.go:186
+0x7a2
github.com/apache/beam/sdks/v2/go/pkg/beam/core/runtime/exec.(*ParDo).invokeProcessFn(0xc0004420e0,
{0x1168a50, 0xc00033cdc0}, {0x18e7a90, 0x1, 0x1}, 0x30, 0x1904be0)
/usr/local/google/home/danoliveira/repos/beam/sdks/go/pkg/beam/core/runtime/exec/pardo.go:316
+0x146
github.com/apache/beam/sdks/v2/go/pkg/beam/core/runtime/exec.(*ParDo).processSingleWindow(0xc0004420e0,
0xc00015d660)
/usr/local/google/home/danoliveira/repos/beam/sdks/go/pkg/beam/core/runtime/exec/pardo.go:166
+0x4b
github.com/apache/beam/sdks/v2/go/pkg/beam/core/runtime/exec.(*ParDo).processMainInput(0xc000348fc0,
0x1168a50)
/usr/local/google/home/danoliveira/repos/beam/sdks/go/pkg/beam/core/runtime/exec/pardo.go:146
+0x9c
github.com/apache/beam/sdks/v2/go/pkg/beam/core/runtime/exec.(*ParDo).ProcessElement(0xc0004420e0,
{0x114d8a0, 0xc000426300}, 0xc0001f2540, {0x0, 0x0, 0x0})
/usr/local/google/home/danoliveira/repos/beam/sdks/go/pkg/beam/core/runtime/exec/pardo.go:132
+0x1a5
github.com/apache/beam/sdks/v2/go/pkg/beam/core/runtime/exec.(*DataSource).Process(0xc0000ec8c0,
{0x1168a50, 0xc00033cd00})
/usr/local/google/home/danoliveira/repos/beam/sdks/go/pkg/beam/core/runtime/exec/datasource.go:174
+0x4ab
github.com/apache/beam/sdks/v2/go/pkg/beam/core/runtime/exec.callNoPanic({0x1168a50, 0xc00033cd00},
0xc0002d6240)
/usr/local/google/home/danoliveira/repos/beam/sdks/go/pkg/beam/core/runtime/exec/util.go:62
+0x6c
github.com/apache/beam/sdks/v2/go/pkg/beam/core/runtime/exec.(*Plan).Execute(0xc000457030, {0x1168a50,
0xc00033cd00}, {0xc0003f07b0, 0x24}, {{0x115ee60, 0xc0002d61b0}, {0x11631d8, 0xc0002d61e0}})
/usr/local/google/home/danoliveira/repos/beam/sdks/go/pkg/beam/core/runtime/exec/plan.go:118
+0x3a5
github.com/apache/beam/sdks/v2/go/pkg/beam/core/runtime/harness.(*control).handleInstruction(0xc000452000,
{0x1168938, 0xc00033e0f0}, 0xc0003fe550)
/usr/local/google/home/danoliveira/repos/beam/sdks/go/pkg/beam/core/runtime/harness/harness.go:331
+0xb6f
github.com/apache/beam/sdks/v2/go/pkg/beam/core/runtime/harness.Main.func3({0x1168938, 0xc00033e0f0},
0xc0003fe550)
/usr/local/google/home/danoliveira/repos/beam/sdks/go/pkg/beam/core/runtime/harness/harness.go:144
+0x19d
created by github.com/apache/beam/sdks/v2/go/pkg/beam/core/runtime/harness.Main
/usr/local/google/home/danoliveira/repos/beam/sdks/go/pkg/beam/core/runtime/harness/harness.go:163
+0xa8f
```
Imported from Jira [BEAM-13420](https://issues.apache.org/jira/browse/BEAM-13420). Original Jira may contain additional context.
Reported by: danoliveira.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org