You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@beam.apache.org by "Eliaaazzz (via GitHub)" <gi...@apache.org> on 2026/10/07 22:53:27 UTC

[PR] [Go SDK] Add pass through registration for the fixed and variable length string and bytes logical types [beam]

Eliaaazzz opened a new pull request, #40156:
URL: https://github.com/apache/beam/pull/40156

   Part of #39684, the Go side of #23526. Depends on #40155 (the first four commits); review the last commit.
   
   `beam:logical_type:fixed_char:v1`, `var_char:v1`, `fixed_bytes:v1` and `var_bytes:v1` are STRING and BYTES on the wire; their INT32 argument is the fixed or maximum length. The Go row coder is built from the Go type alone, so a plain `string` or `[]byte` field is the natural Go representation, and it must keep producing an atomic STRING or BYTES schema field.
   
   - `schema.RegisterPassThroughLogicalType(urn, goType)` maps a URN to the Go type of its representation without mapping that Go type back. A schema field with the URN decodes with the plain row encoding of the Go type, and the argument is ignored (the length is not enforced). Any argument matches because the arguments PR falls back to the registration without an argument.
   - The four standard types are registered this way with `string` and `[]byte`, so a JDBC or SQL row from Java with CHAR, VARCHAR, BINARY or VARBINARY columns maps to `string` and `[]byte` fields (nullable columns map to pointers).
   - The char, varchar, binary and varbinary case of standard_coders.yaml now runs in the Go regression test.
   
   Tests: `ToType` of the yaml schema gives `*string` and `*[]byte` fields, decoding the yaml example with nulls, and plain `string` and `[]byte` fields still produce atomic schema fields.
   
   ------------------------
   
   Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:
   
    - [x] Mention the appropriate issue in your description (for example: `addresses #123`), if applicable. This will automatically add a link to the pull request in the issue. If you would like the issue to automatically close on merging the pull request, comment `fixes #<ISSUE NUMBER>` instead.
    - [x] Update `CHANGES.md` with noteworthy changes.
    - [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).
   
   See the [Contributor Guide](https://beam.apache.org/contribute) for more tips on [how to make review process smoother](https://github.com/apache/beam/blob/master/CONTRIBUTING.md#make-the-reviewers-job-easier).
   
   To check the build health, please visit [https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md](https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md)
   
   GitHub Actions Tests Status (on master branch)
   ------------------------------------------------------------------------------------------------
   [![Build python source distribution and wheels](https://github.com/apache/beam/actions/workflows/build_wheels.yml/badge.svg?event=schedule&&?branch=master)](https://github.com/apache/beam/actions?query=workflow%3A%22Build+python+source+distribution+and+wheels%22+branch%3Amaster+event%3Aschedule)
   [![Python tests](https://github.com/apache/beam/actions/workflows/python_tests.yml/badge.svg?event=schedule&&?branch=master)](https://github.com/apache/beam/actions?query=workflow%3A%22Python+Tests%22+branch%3Amaster+event%3Aschedule)
   [![Java tests](https://github.com/apache/beam/actions/workflows/java_tests.yml/badge.svg?event=schedule&&?branch=master)](https://github.com/apache/beam/actions?query=workflow%3A%22Java+Tests%22+branch%3Amaster+event%3Aschedule)
   [![Go tests](https://github.com/apache/beam/actions/workflows/go_tests.yml/badge.svg?event=schedule&&?branch=master)](https://github.com/apache/beam/actions?query=workflow%3A%22Go+tests%22+branch%3Amaster+event%3Aschedule)
   
   See [CI.md](https://github.com/apache/beam/blob/master/CI.md) for more information about GitHub Actions CI or the [workflows README](https://github.com/apache/beam/blob/master/.github/workflows/README.md) to see a list of phrases to trigger workflows.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [Go SDK] Add pass through registration for the fixed and variable length string and bytes logical types [beam]

Posted by "codecov[bot] (via GitHub)" <gi...@apache.org>.
codecov[bot] commented on PR #40156:
URL: https://github.com/apache/beam/pull/40156#issuecomment-6357662307

   ## [Codecov](https://app.codecov.io/gh/apache/beam/pull/40156?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
   :x: Patch coverage is `60.00000%` with `6 lines` in your changes missing coverage. Please review.
   :white_check_mark: Project coverage is 28.91%. Comparing base ([`a8d6afd`](https://app.codecov.io/gh/apache/beam/commit/a8d6afdf5cbc8c4635732ad6dd8b702d16d869ec?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)) to head ([`ef7ab94`](https://app.codecov.io/gh/apache/beam/commit/ef7ab9408d9931e4dce7d57ecf91346442dda35c?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)).
   :warning: Report is 53 commits behind head on master.
   
   | [Files with missing lines](https://app.codecov.io/gh/apache/beam/pull/40156?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Patch % | Lines |
   |---|---|---|
   | [...kg/beam/core/runtime/graphx/schema/logicaltypes.go](https://app.codecov.io/gh/apache/beam/pull/40156?src=pr&el=tree&filepath=sdks%2Fgo%2Fpkg%2Fbeam%2Fcore%2Fruntime%2Fgraphx%2Fschema%2Flogicaltypes.go&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-c2Rrcy9nby9wa2cvYmVhbS9jb3JlL3J1bnRpbWUvZ3JhcGh4L3NjaGVtYS9sb2dpY2FsdHlwZXMuZ28=) | 45.45% | [4 Missing and 2 partials :warning: ](https://app.codecov.io/gh/apache/beam/pull/40156?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
   
   <details><summary>Additional details and impacted files</summary>
   
   
   
   ```diff
   @@             Coverage Diff             @@
   ##           master   #40156       +/-   ##
   ===========================================
   - Coverage   80.04%   28.91%   -51.13%     
   ===========================================
     Files         490      381      -109     
     Lines       82916    71227    -11689     
   ===========================================
   - Hits        66373    20598    -45775     
   - Misses      16543    48771    +32228     
   - Partials        0     1858     +1858     
   ```
   
   | [Flag](https://app.codecov.io/gh/apache/beam/pull/40156/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
   |---|---|---|
   | [go](https://app.codecov.io/gh/apache/beam/pull/40156/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `28.91% <60.00%> (?)` | |
   | [python](https://app.codecov.io/gh/apache/beam/pull/40156/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
   
   Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#carryforward-flags-in-the-pull-request-comment) to find out more.
   </details>
   
   [:umbrella: View full report in Codecov by Harness](https://app.codecov.io/gh/apache/beam/pull/40156?dropdown=coverage&src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).   
   :loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
   <details><summary> :rocket: New features to boost your workflow: </summary>
   
   - :snowflake: [Test Analytics](https://docs.codecov.com/docs/test-analytics): Detect flaky tests, report on failures, and find test suite problems.
   - :package: [JS Bundle Analysis](https://docs.codecov.com/docs/javascript-bundle-analysis): Save yourself from yourself by tracking and limiting bundle sizes in JS merges.
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [Go SDK] Add pass through registration for the fixed and variable length string and bytes logical types [beam]

Posted by "Eliaaazzz (via GitHub) via github" <gi...@beam.apache.org>.
Eliaaazzz commented on PR #40156:
URL: https://github.com/apache/beam/pull/40156#issuecomment-6679348830

   Thanks @shunping for reviewing and merging #40155! I merged master into this branch and resolved the conflicts, so the diff shows only this PR's own changes again. The release note is now in the 2.79.0 section.
   
   Locally the schema, graphx, coder and exec package tests pass, and the char, varchar, binary and varbinary case of standard_coders.yaml passes in the fromyaml regression test.
   
   @lostluck or @shunping, I would appreciate a review when you have time. Happy to adjust anything you spot.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [Go SDK] Add pass through registration for the fixed and variable length string and bytes logical types [beam]

Posted by "lostluck (via GitHub) via github" <gi...@beam.apache.org>.
lostluck commented on code in PR #40156:
URL: https://github.com/apache/beam/pull/40156#discussion_r4497243215


##########
sdks/go/pkg/beam/core/runtime/graphx/schema/standard_logical_types_test.go:
##########
@@ -499,3 +500,265 @@ func TestMillisInstant_Range(t *testing.T) {
 		t.Error("truncated millis_instant decoded successfully")
 	}
 }
+
+func TestPassThroughLogicalTypes(t *testing.T) {
+	nullable := func(urn string, representation pipepb.AtomicType, length int32) *pipepb.FieldType {
+		ft := logicalFieldType(urn, atomicType(representation))
+		ft.Nullable = true
+		ft.GetLogicalType().ArgumentType = int32FieldType()
+		ft.GetLogicalType().Argument = int32FieldValue(length)
+		return ft
+	}
+	// The schema of the char, varchar, binary and varbinary case of standard_coders.yaml.
+	s := &pipepb.Schema{
+		Id: "pass-through-logical-types-test",
+		Fields: []*pipepb.Field{
+			{Name: "f_char", Type: nullable(URNFixedChar, pipepb.AtomicType_STRING, 5)},
+			{Name: "f_varchar", Type: nullable(URNVarChar, pipepb.AtomicType_STRING, 10)},
+			{Name: "f_bytes", Type: nullable(URNFixedBytes, pipepb.AtomicType_BYTES, 5)},
+			{Name: "f_varbytes", Type: nullable(URNVarBytes, pipepb.AtomicType_BYTES, 10)},
+		},
+	}
+	rt, err := ToType(s)
+	if err != nil {
+		t.Fatalf("ToType(%v) = %v, want nil error", prototext.Format(s), err)
+	}
+	stringPtr := reflect.PointerTo(typeOf[string]())
+	bytesPtr := reflect.PointerTo(typeOf[[]byte]())
+	for i, want := range []reflect.Type{stringPtr, stringPtr, bytesPtr, bytesPtr} {
+		if got := rt.Field(i).Type; got != want {
+			t.Errorf("ToType field %d type = %v, want %v", i, got, want)
+		}
+	}
+	// Decoding uses the plain string and bytes encodings, and pads nothing.
+	dec, err := coder.RowDecoderForStruct(rt)
+	if err != nil {
+		t.Fatalf("RowDecoderForStruct(%v) = %v, want nil error", rt, err)
+	}
+	got, err := dec(bytes.NewBufferString("\x04\x01\x06\x05null?\x04null"))
+	if err != nil {
+		t.Fatalf("dec = %v, want nil error", err)
+	}
+	rv := reflect.ValueOf(got)
+	if s := rv.Field(0).Elem().String(); s != "null?" {
+		t.Errorf("f_char = %q, want %q", s, "null?")
+	}
+	if !rv.Field(1).IsNil() || !rv.Field(2).IsNil() {
+		t.Errorf("f_varchar and f_bytes = %v, %v, want nil", rv.Field(1), rv.Field(2))
+	}
+	if b := rv.Field(3).Elem().Bytes(); string(b) != "null" {
+		t.Errorf("f_varbytes = %q, want %q", b, "null")
+	}
+	// Plain string and bytes fields keep their atomic schema types.
+	schm, err := FromType(reflect.TypeOf(struct {
+		S string
+		B []byte
+	}{}))
+	if err != nil {
+		t.Fatalf("FromType = %v, want nil error", err)
+	}
+	for i, want := range []pipepb.AtomicType{pipepb.AtomicType_STRING, pipepb.AtomicType_BYTES} {
+		if got := schm.GetFields()[i].GetType(); got.GetLogicalType() != nil || got.GetAtomicType() != want {
+			t.Errorf("FromType field %d = %v, want atomic %v", i, got, want)
+		}
+	}
+}
+
+func TestRegisterPassThroughLogicalType(t *testing.T) {
+	r := NewRegistry()
+	const urn = "test:pass_through:v1"
+	r.RegisterPassThroughLogicalType(urn, typeOf[string]())
+	r.RegisterPassThroughLogicalType(urn, typeOf[string]())
+	for _, length := range []int32{1, 100} {
+		ft := logicalFieldType(urn, atomicType(pipepb.AtomicType_STRING))
+		ft.GetLogicalType().ArgumentType = int32FieldType()
+		ft.GetLogicalType().Argument = int32FieldValue(length)
+		rt, err := r.ToType(&pipepb.Schema{Fields: []*pipepb.Field{{Name: "value", Type: ft}}})
+		if err != nil {
+			t.Fatalf("ToType(length=%v): %v", length, err)
+		}
+		if got := rt.Field(0).Type; got != typeOf[string]() {
+			t.Errorf("field type = %v, want string", got)
+		}
+	}
+	t.Run("ConflictingRegistration", func(t *testing.T) {
+		defer func() {
+			if recover() == nil {
+				t.Error("conflicting registration succeeded")

Review Comment:
   It's trite, but for testing best practices don't just say what happened, say waht should have happened. In this case, "wanted it to fail". 
   
   See https://google.github.io/styleguide/go/decisions#useful-test-failures for guidance.



##########
sdks/go/pkg/beam/core/runtime/graphx/schema/standard_logical_types_test.go:
##########
@@ -499,3 +500,265 @@ func TestMillisInstant_Range(t *testing.T) {
 		t.Error("truncated millis_instant decoded successfully")
 	}
 }
+
+func TestPassThroughLogicalTypes(t *testing.T) {
+	nullable := func(urn string, representation pipepb.AtomicType, length int32) *pipepb.FieldType {
+		ft := logicalFieldType(urn, atomicType(representation))
+		ft.Nullable = true
+		ft.GetLogicalType().ArgumentType = int32FieldType()
+		ft.GetLogicalType().Argument = int32FieldValue(length)
+		return ft
+	}
+	// The schema of the char, varchar, binary and varbinary case of standard_coders.yaml.
+	s := &pipepb.Schema{
+		Id: "pass-through-logical-types-test",
+		Fields: []*pipepb.Field{
+			{Name: "f_char", Type: nullable(URNFixedChar, pipepb.AtomicType_STRING, 5)},
+			{Name: "f_varchar", Type: nullable(URNVarChar, pipepb.AtomicType_STRING, 10)},
+			{Name: "f_bytes", Type: nullable(URNFixedBytes, pipepb.AtomicType_BYTES, 5)},
+			{Name: "f_varbytes", Type: nullable(URNVarBytes, pipepb.AtomicType_BYTES, 10)},
+		},
+	}
+	rt, err := ToType(s)
+	if err != nil {
+		t.Fatalf("ToType(%v) = %v, want nil error", prototext.Format(s), err)
+	}
+	stringPtr := reflect.PointerTo(typeOf[string]())
+	bytesPtr := reflect.PointerTo(typeOf[[]byte]())
+	for i, want := range []reflect.Type{stringPtr, stringPtr, bytesPtr, bytesPtr} {
+		if got := rt.Field(i).Type; got != want {
+			t.Errorf("ToType field %d type = %v, want %v", i, got, want)
+		}
+	}
+	// Decoding uses the plain string and bytes encodings, and pads nothing.
+	dec, err := coder.RowDecoderForStruct(rt)
+	if err != nil {
+		t.Fatalf("RowDecoderForStruct(%v) = %v, want nil error", rt, err)
+	}
+	got, err := dec(bytes.NewBufferString("\x04\x01\x06\x05null?\x04null"))
+	if err != nil {
+		t.Fatalf("dec = %v, want nil error", err)
+	}
+	rv := reflect.ValueOf(got)
+	if s := rv.Field(0).Elem().String(); s != "null?" {
+		t.Errorf("f_char = %q, want %q", s, "null?")
+	}
+	if !rv.Field(1).IsNil() || !rv.Field(2).IsNil() {
+		t.Errorf("f_varchar and f_bytes = %v, %v, want nil", rv.Field(1), rv.Field(2))
+	}
+	if b := rv.Field(3).Elem().Bytes(); string(b) != "null" {
+		t.Errorf("f_varbytes = %q, want %q", b, "null")
+	}
+	// Plain string and bytes fields keep their atomic schema types.
+	schm, err := FromType(reflect.TypeOf(struct {
+		S string
+		B []byte
+	}{}))
+	if err != nil {
+		t.Fatalf("FromType = %v, want nil error", err)
+	}
+	for i, want := range []pipepb.AtomicType{pipepb.AtomicType_STRING, pipepb.AtomicType_BYTES} {
+		if got := schm.GetFields()[i].GetType(); got.GetLogicalType() != nil || got.GetAtomicType() != want {
+			t.Errorf("FromType field %d = %v, want atomic %v", i, got, want)
+		}
+	}
+}
+
+func TestRegisterPassThroughLogicalType(t *testing.T) {
+	r := NewRegistry()
+	const urn = "test:pass_through:v1"
+	r.RegisterPassThroughLogicalType(urn, typeOf[string]())
+	r.RegisterPassThroughLogicalType(urn, typeOf[string]())

Review Comment:
   Why is it called twice here? My read is that it just overwrites the first one. If the intent is to validate that it overwrites/doesn't panic, please add a comment to that effect.



##########
sdks/go/pkg/beam/core/runtime/graphx/schema/standard_logical_types_test.go:
##########
@@ -499,3 +500,265 @@ func TestMillisInstant_Range(t *testing.T) {
 		t.Error("truncated millis_instant decoded successfully")
 	}
 }
+
+func TestPassThroughLogicalTypes(t *testing.T) {
+	nullable := func(urn string, representation pipepb.AtomicType, length int32) *pipepb.FieldType {
+		ft := logicalFieldType(urn, atomicType(representation))
+		ft.Nullable = true
+		ft.GetLogicalType().ArgumentType = int32FieldType()
+		ft.GetLogicalType().Argument = int32FieldValue(length)
+		return ft
+	}
+	// The schema of the char, varchar, binary and varbinary case of standard_coders.yaml.
+	s := &pipepb.Schema{
+		Id: "pass-through-logical-types-test",
+		Fields: []*pipepb.Field{
+			{Name: "f_char", Type: nullable(URNFixedChar, pipepb.AtomicType_STRING, 5)},
+			{Name: "f_varchar", Type: nullable(URNVarChar, pipepb.AtomicType_STRING, 10)},
+			{Name: "f_bytes", Type: nullable(URNFixedBytes, pipepb.AtomicType_BYTES, 5)},
+			{Name: "f_varbytes", Type: nullable(URNVarBytes, pipepb.AtomicType_BYTES, 10)},
+		},
+	}
+	rt, err := ToType(s)
+	if err != nil {
+		t.Fatalf("ToType(%v) = %v, want nil error", prototext.Format(s), err)
+	}
+	stringPtr := reflect.PointerTo(typeOf[string]())
+	bytesPtr := reflect.PointerTo(typeOf[[]byte]())
+	for i, want := range []reflect.Type{stringPtr, stringPtr, bytesPtr, bytesPtr} {
+		if got := rt.Field(i).Type; got != want {
+			t.Errorf("ToType field %d type = %v, want %v", i, got, want)
+		}
+	}
+	// Decoding uses the plain string and bytes encodings, and pads nothing.
+	dec, err := coder.RowDecoderForStruct(rt)
+	if err != nil {
+		t.Fatalf("RowDecoderForStruct(%v) = %v, want nil error", rt, err)
+	}
+	got, err := dec(bytes.NewBufferString("\x04\x01\x06\x05null?\x04null"))
+	if err != nil {
+		t.Fatalf("dec = %v, want nil error", err)
+	}
+	rv := reflect.ValueOf(got)
+	if s := rv.Field(0).Elem().String(); s != "null?" {
+		t.Errorf("f_char = %q, want %q", s, "null?")
+	}
+	if !rv.Field(1).IsNil() || !rv.Field(2).IsNil() {
+		t.Errorf("f_varchar and f_bytes = %v, %v, want nil", rv.Field(1), rv.Field(2))
+	}
+	if b := rv.Field(3).Elem().Bytes(); string(b) != "null" {
+		t.Errorf("f_varbytes = %q, want %q", b, "null")
+	}
+	// Plain string and bytes fields keep their atomic schema types.
+	schm, err := FromType(reflect.TypeOf(struct {
+		S string
+		B []byte
+	}{}))
+	if err != nil {
+		t.Fatalf("FromType = %v, want nil error", err)
+	}
+	for i, want := range []pipepb.AtomicType{pipepb.AtomicType_STRING, pipepb.AtomicType_BYTES} {
+		if got := schm.GetFields()[i].GetType(); got.GetLogicalType() != nil || got.GetAtomicType() != want {
+			t.Errorf("FromType field %d = %v, want atomic %v", i, got, want)
+		}
+	}
+}
+
+func TestRegisterPassThroughLogicalType(t *testing.T) {
+	r := NewRegistry()
+	const urn = "test:pass_through:v1"
+	r.RegisterPassThroughLogicalType(urn, typeOf[string]())
+	r.RegisterPassThroughLogicalType(urn, typeOf[string]())
+	for _, length := range []int32{1, 100} {
+		ft := logicalFieldType(urn, atomicType(pipepb.AtomicType_STRING))
+		ft.GetLogicalType().ArgumentType = int32FieldType()
+		ft.GetLogicalType().Argument = int32FieldValue(length)
+		rt, err := r.ToType(&pipepb.Schema{Fields: []*pipepb.Field{{Name: "value", Type: ft}}})
+		if err != nil {
+			t.Fatalf("ToType(length=%v): %v", length, err)
+		}
+		if got := rt.Field(0).Type; got != typeOf[string]() {
+			t.Errorf("field type = %v, want string", got)
+		}
+	}
+	t.Run("ConflictingRegistration", func(t *testing.T) {
+		defer func() {
+			if recover() == nil {
+				t.Error("conflicting registration succeeded")
+			}
+		}()
+		r.RegisterPassThroughLogicalType(urn, typeOf[[]byte]())
+	})
+	lt, ok := r.lookupLogicalType(urn, reflect.Value{})
+	if !ok || lt.GoType() != typeOf[string]() {
+		t.Errorf("registration after conflict = %v, %v", lt, ok)
+	}
+}
+
+func TestRegisterPassThroughLogicalType_PublicAPI(t *testing.T) {
+	const urn = "test:public_pass_through:v1"
+	t.Cleanup(func() { delete(defaultRegistry.logicalTypes, logicalTypeKey{identifier: urn}) })
+	RegisterPassThroughLogicalType(urn, typeOf[string]())
+	incoming := &pipepb.Schema{Fields: []*pipepb.Field{{
+		Name: "value", Type: logicalFieldType(urn, atomicType(pipepb.AtomicType_STRING)),
+	}}}
+	rt, err := ToType(incoming)
+	if err != nil {
+		t.Fatalf("ToType: %v", err)
+	}
+	if got := rt.Field(0).Type; got != typeOf[string]() {
+		t.Fatalf("field type = %v, want string", got)
+	}
+	dec, err := coder.RowDecoderForStruct(rt)
+	if err != nil {
+		t.Fatalf("RowDecoderForStruct: %v", err)
+	}
+	wire := []byte("\x01\x00\x05hello")
+	row, err := dec(bytes.NewReader(wire))
+	if err != nil {
+		t.Fatalf("decode: %v", err)
+	}
+	if got := reflect.ValueOf(row).Field(0).String(); got != "hello" {
+		t.Errorf("decoded field = %q, want hello", got)
+	}
+	enc, err := coder.RowEncoderForStruct(rt)
+	if err != nil {
+		t.Fatalf("RowEncoderForStruct: %v", err)
+	}
+	var buf bytes.Buffer
+	if err := enc(row, &buf); err != nil {
+		t.Fatalf("encode: %v", err)
+	}
+	if got := buf.Bytes(); !bytes.Equal(got, wire) {
+		t.Errorf("encoded row = %x, want %x", got, wire)
+	}
+	outgoing, err := FromType(rt)
+	if err != nil {
+		t.Fatalf("FromType: %v", err)
+	}
+	if got := outgoing.GetFields()[0].GetType(); got.GetLogicalType() != nil || got.GetAtomicType() != pipepb.AtomicType_STRING {
+		t.Errorf("outgoing field = %v, want atomic STRING", got)
+	}
+}

Review Comment:
   Nothing wrong with the test, but please do use a blank line or two to visually distinguish sections of the code to improve readability. There's no benefit for humans or robots to pack everything in like this. 
   
   For example, this test has a sequence of "call API -> check results" sections so they can be separated that way, which makes reading easier.
   
   eg:
   ```suggestion
   func TestRegisterPassThroughLogicalType_PublicAPI(t *testing.T) {
   	const urn = "test:public_pass_through:v1"
   	t.Cleanup(func() { delete(defaultRegistry.logicalTypes, logicalTypeKey{identifier: urn}) })
   	RegisterPassThroughLogicalType(urn, typeOf[string]())
   	incoming := &pipepb.Schema{Fields: []*pipepb.Field{{
   		Name: "value", Type: logicalFieldType(urn, atomicType(pipepb.AtomicType_STRING)),
   	}}}
   
   	rt, err := ToType(incoming)
   	if err != nil {
   		t.Fatalf("ToType: %v", err)
   	}
   	if got := rt.Field(0).Type; got != typeOf[string]() {
   		t.Fatalf("field type = %v, want string", got)
   	}
   
   	dec, err := coder.RowDecoderForStruct(rt)
   	if err != nil {
   		t.Fatalf("RowDecoderForStruct: %v", err)
   	}
   	wire := []byte("\x01\x00\x05hello")
   	row, err := dec(bytes.NewReader(wire))
   	if err != nil {
   		t.Fatalf("decode: %v", err)
   	}
   	if got := reflect.ValueOf(row).Field(0).String(); got != "hello" {
   		t.Errorf("decoded field = %q, want hello", got)
   	}
   
   	enc, err := coder.RowEncoderForStruct(rt)
   	if err != nil {
   		t.Fatalf("RowEncoderForStruct: %v", err)
   	}
   	var buf bytes.Buffer
   	if err := enc(row, &buf); err != nil {
   		t.Fatalf("encode: %v", err)
   	}
   	if got := buf.Bytes(); !bytes.Equal(got, wire) {
   		t.Errorf("encoded row = %x, want %x", got, wire)
   	}
   
   	outgoing, err := FromType(rt)
   	if err != nil {
   		t.Fatalf("FromType: %v", err)
   	}
   	if got := outgoing.GetFields()[0].GetType(); got.GetLogicalType() != nil || got.GetAtomicType() != pipepb.AtomicType_STRING {
   		t.Errorf("outgoing field = %v, want atomic STRING", got)
   	}
   }
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [Go SDK] Add pass through registration for the fixed and variable length string and bytes logical types [beam]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on code in PR #40156:
URL: https://github.com/apache/beam/pull/40156#discussion_r4356159210


##########
sdks/go/pkg/beam/core/runtime/graphx/schema/logicaltypes.go:
##########
@@ -361,3 +361,34 @@ func checkLogicalTypeKinds(lt LogicalType) {
 		panic(fmt.Sprintf("LogicalType[%v] has the pointer storage type %v, storage types must not be pointers", lt.ID(), lt.StorageType()))
 	}
 }
+
+// RegisterPassThroughLogicalType registers a logical type identified by urn
+// whose values are used as their representation type. A schema field with
+// the urn maps to goType and uses the row encoding of goType, and any
+// argument of the logical type is ignored. Go values of goType are not mapped
+// back to the logical type, so a struct with a goType field produces a schema
+// field of the representation type. This suits the standard fixed and
+// variable length string and bytes logical types, whose argument only
+// constrains the length.
+//
+// RegisterPassThroughLogicalType must be called before beam.Init(), and
+// conventionally is called in a package init() function.
+func RegisterPassThroughLogicalType(urn string, goType reflect.Type) {
+	defaultRegistry.RegisterPassThroughLogicalType(urn, goType)
+}
+
+// RegisterPassThroughLogicalType registers a logical type whose values are
+// used as their representation type. See the package level function.
+func (r *Registry) RegisterPassThroughLogicalType(urn string, goType reflect.Type) {
+	if len(urn) == 0 {
+		panic(fmt.Sprintf("invalid pass through logical type, bad urn for %v", goType))
+	}
+	if _, err := r.reflectTypeToFieldType(goType); err != nil {
+		panic(fmt.Sprintf("pass through LogicalType[%v] has an invalid type %v: %v", urn, goType, err))
+	}
+	lt := ToLogicalType(urn, goType, goType)
+	if prev, ok := r.logicalTypes[lt.key()]; ok && !prev.equals(lt) {

Review Comment:
   **🟡 Medium**: `RegisterPassThroughLogicalType` currently accepts pointer types (e.g., `*string`) because `r.reflectTypeToFieldType` succeeds on pointers (setting `Nullable: true`). However, `fieldTypeToReflectType` wraps `lt.GoType()` in `reflect.PtrTo` only when `sft.GetNullable()` is true, and the row coder determines field nullability from whether the struct field type is a pointer. If a pointer type is registered as `goType`, a non-nullable logical type field would map to `*T` (causing the row coder to expect a nullability bit not present in the wire format), and a nullable field would map to `**T`.
   
   Call `checkLogicalTypeKinds(lt)` (or explicitly disallow `goType.Kind() == reflect.Ptr`) before registering `lt`.
   
   ```suggestion
   	lt := ToLogicalType(urn, goType, goType)
   	checkLogicalTypeKinds(lt)
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [Go SDK] Add pass through registration for the fixed and variable length string and bytes logical types [beam]

Posted by "lostluck (via GitHub) via github" <gi...@beam.apache.org>.
lostluck merged PR #40156:
URL: https://github.com/apache/beam/pull/40156


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [Go SDK] Add pass through registration for the fixed and variable length string and bytes logical types [beam]

Posted by "github-actions[bot] (via GitHub) via github" <gi...@beam.apache.org>.
github-actions[bot] commented on PR #40156:
URL: https://github.com/apache/beam/pull/40156#issuecomment-6630432677

   Reminder, please take a look at this pr: @lostluck 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [Go SDK] Add pass through registration for the fixed and variable length string and bytes logical types [beam]

Posted by "lostluck (via GitHub) via github" <gi...@beam.apache.org>.
lostluck commented on PR #40156:
URL: https://github.com/apache/beam/pull/40156#issuecomment-6712272317

   LGTM thanks for the changes and contribution!


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [Go SDK] Add pass through registration for the fixed and variable length string and bytes logical types [beam]

Posted by "Eliaaazzz (via GitHub)" <gi...@apache.org>.
Eliaaazzz closed pull request #40156: [Go SDK] Add pass through registration for the fixed and variable length string and bytes logical types
URL: https://github.com/apache/beam/pull/40156


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [Go SDK] Add pass through registration for the fixed and variable length string and bytes logical types [beam]

Posted by "Eliaaazzz (via GitHub) via github" <gi...@beam.apache.org>.
Eliaaazzz commented on PR #40156:
URL: https://github.com/apache/beam/pull/40156#issuecomment-6702600026

   Thanks @lostluck for the review and the patient guidance. I learned from your feedback. Fixed in 783fa9bae06: a comment now explains the repeated identical registration, the flagged failure message says it wanted a panic and I went through the other new checks with the same guide, and the call and check sections are separated with blank lines. I also merged master again to resolve the CHANGES.md conflict.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [Go SDK] Add pass through registration for the fixed and variable length string and bytes logical types [beam]

Posted by "Eliaaazzz (via GitHub)" <gi...@apache.org>.
Eliaaazzz commented on PR #40156:
URL: https://github.com/apache/beam/pull/40156#issuecomment-5715121961

   Closing this for now to keep the series at three open PRs while #40152, #40153 and #40154 are reviewed. The branch stays in my fork.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [Go SDK] Add pass through registration for the fixed and variable length string and bytes logical types [beam]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #40156:
URL: https://github.com/apache/beam/pull/40156#issuecomment-6048756542

   Assigning reviewers:
   
   R: @lostluck for label go.
   
   This pull request likely touches a core component ("core" label). Please review with scrutiny.
   
   
   Note: If you would like to opt out of this review, comment `assign to next reviewer`.
   
   Available commands:
   - `stop reviewer notifications` - opt out of the automated review tooling
   - `remind me after tests pass` - tag the comment author after tests pass
   - `waiting on author` - shift the attention set back to the author (any comment or push by the author will return the attention set to the reviewers)
   
   The PR bot will only process comments in the main thread (not review comments).


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [Go SDK] Add pass through registration for the fixed and variable length string and bytes logical types [beam]

Posted by "Eliaaazzz (via GitHub)" <gi...@apache.org>.
Eliaaazzz commented on code in PR #40156:
URL: https://github.com/apache/beam/pull/40156#discussion_r4356677204


##########
sdks/go/pkg/beam/core/runtime/graphx/schema/logicaltypes.go:
##########
@@ -361,3 +361,34 @@ func checkLogicalTypeKinds(lt LogicalType) {
 		panic(fmt.Sprintf("LogicalType[%v] has the pointer storage type %v, storage types must not be pointers", lt.ID(), lt.StorageType()))
 	}
 }
+
+// RegisterPassThroughLogicalType registers a logical type identified by urn
+// whose values are used as their representation type. A schema field with
+// the urn maps to goType and uses the row encoding of goType, and any
+// argument of the logical type is ignored. Go values of goType are not mapped
+// back to the logical type, so a struct with a goType field produces a schema
+// field of the representation type. This suits the standard fixed and
+// variable length string and bytes logical types, whose argument only
+// constrains the length.
+//
+// RegisterPassThroughLogicalType must be called before beam.Init(), and
+// conventionally is called in a package init() function.
+func RegisterPassThroughLogicalType(urn string, goType reflect.Type) {
+	defaultRegistry.RegisterPassThroughLogicalType(urn, goType)
+}
+
+// RegisterPassThroughLogicalType registers a logical type whose values are
+// used as their representation type. See the package level function.
+func (r *Registry) RegisterPassThroughLogicalType(urn string, goType reflect.Type) {
+	if len(urn) == 0 {
+		panic(fmt.Sprintf("invalid pass through logical type, bad urn for %v", goType))
+	}
+	if _, err := r.reflectTypeToFieldType(goType); err != nil {
+		panic(fmt.Sprintf("pass through LogicalType[%v] has an invalid type %v: %v", urn, goType, err))
+	}
+	lt := ToLogicalType(urn, goType, goType)
+	if prev, ok := r.logicalTypes[lt.key()]; ok && !prev.equals(lt) {

Review Comment:
   Fixed in 2933f26f483. Pass-through registration now calls checkLogicalTypeKinds before schema conversion or registry insertion. Tests reject pointer and interface types, verify failed registrations leave no mapping, and cover required, nullable-value and null string/bytes rows against exact wire bytes.
   
   The pointer regressions fail on 5a9be29 and pass with the fix. Codex ran the four focused Go packages, schema coverage, repeated shuffled registration tests and the diff check; all passed. Both registration functions have 100% local statement coverage.
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [Go SDK] Add pass through registration for the fixed and variable length string and bytes logical types [beam]

Posted by "Eliaaazzz (via GitHub) via github" <gi...@beam.apache.org>.
Eliaaazzz commented on code in PR #40156:
URL: https://github.com/apache/beam/pull/40156#discussion_r4356677204


##########
sdks/go/pkg/beam/core/runtime/graphx/schema/logicaltypes.go:
##########
@@ -361,3 +361,34 @@ func checkLogicalTypeKinds(lt LogicalType) {
 		panic(fmt.Sprintf("LogicalType[%v] has the pointer storage type %v, storage types must not be pointers", lt.ID(), lt.StorageType()))
 	}
 }
+
+// RegisterPassThroughLogicalType registers a logical type identified by urn
+// whose values are used as their representation type. A schema field with
+// the urn maps to goType and uses the row encoding of goType, and any
+// argument of the logical type is ignored. Go values of goType are not mapped
+// back to the logical type, so a struct with a goType field produces a schema
+// field of the representation type. This suits the standard fixed and
+// variable length string and bytes logical types, whose argument only
+// constrains the length.
+//
+// RegisterPassThroughLogicalType must be called before beam.Init(), and
+// conventionally is called in a package init() function.
+func RegisterPassThroughLogicalType(urn string, goType reflect.Type) {
+	defaultRegistry.RegisterPassThroughLogicalType(urn, goType)
+}
+
+// RegisterPassThroughLogicalType registers a logical type whose values are
+// used as their representation type. See the package level function.
+func (r *Registry) RegisterPassThroughLogicalType(urn string, goType reflect.Type) {
+	if len(urn) == 0 {
+		panic(fmt.Sprintf("invalid pass through logical type, bad urn for %v", goType))
+	}
+	if _, err := r.reflectTypeToFieldType(goType); err != nil {
+		panic(fmt.Sprintf("pass through LogicalType[%v] has an invalid type %v: %v", urn, goType, err))
+	}
+	lt := ToLogicalType(urn, goType, goType)
+	if prev, ok := r.logicalTypes[lt.key()]; ok && !prev.equals(lt) {

Review Comment:
   Fixed in 2933f26f483. Pass-through registration now calls checkLogicalTypeKinds before schema conversion or registry insertion. Tests reject pointer and interface types, verify failed registrations leave no mapping, and cover required, nullable-value and null string/bytes rows against exact wire bytes.
   
   The pointer regressions fail on 5a9be29 and pass with the fix. I ran the four focused Go packages, schema coverage, repeated shuffled registration tests and the diff check, and all passed. Both registration functions have 100% local statement coverage.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org