You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@fory.apache.org by "stevenschlansker (via GitHub)" <gi...@apache.org> on 2026/10/16 23:43:01 UTC
[I] xlang: the snake_case conversion of field names is not specified, and the runtimes do not agree [fory]
stevenschlansker opened a new issue, #4187:
URL: https://github.com/apache/fory/issues/4187
### Search before asking
- [x] I had searched in the [issues](https://github.com/apache/fory/issues) and found no similar issues.
### Version
All
### Component(s)
Other
### Minimal reproduce step
## Problem
The xlang specification says that field names are "converted to `snake_case`" (docs/specification/xlang_serialization_spec.md, lines 882 and 1732), and line 1800 refers to "the snake_case field name". It does not define the conversion. Each runtime has its own conversion, and the results are different for acronyms, an acronym followed by a digit, a leading capital letter, and non-ASCII capital letters. Rust and Python write the declared field name without a conversion.
Java, Kotlin, Scala, JS, Dart, C#, Swift, and C++ agree for lowerCamelCase names that have only single ASCII capital letters: they all write `createdAt` as `created_at`. A Go field must be exported, so Go writes `created_at` for `CreatedAt`. Rust and Python write the declared name, so they match only for a field declared as `created_at`.
## Field name written to TypeDef/TypeMeta (apache/main at a11d257b4)
Every cell comes from a probe program that ran the runtime code at a11d257b4 and, where possible, encoded and decoded a real TypeDef/TypeMeta.
| Input | Java, Kotlin, Scala, JS | Go | Dart | C#, Swift | C++ | Rust | Python |
|---|---|---|---|---|---|---|---|
| `createdAt` | `created_at` | (1) | `created_at` | `created_at` | `created_at` | `createdAt` (2) | `createdAt` |
| `fooURL` | `foo_u_r_l` | (1) | `foo_u_r_l` | `foo_url` | `foo_url` | `fooURL` (2) | `fooURL` |
| `URLValue` | `_u_r_l_value` | `u_r_l_value` | `u_r_l_value` | `url_value` | `url_value` | `URLValue` (2) | `URLValue` |
| `HTTP2Server` | `_h_t_t_p2_server` | `h_t_t_p2_server` | `h_t_t_p2_server` | `htt_p2_server` | `http2_server` | `HTTP2Server` (2) | `HTTP2Server` |
| `CreatedAt` | `_created_at` | `created_at` | `created_at` | `created_at` | `created_at` | `CreatedAt` (2) | `CreatedAt` |
| `x2Y` | `x2_y` | (1) | `x2_y` | `x2_y` | `x2_y` | `x2Y` (2) | `x2Y` |
| `fooÖl` | `fooÖl` | (1) | (3) | `foo_öl` | `fooÖl` | (4) | (4) |
| `already_snake` | `already_snake` | (1) | `already_snake` | `already_snake` | `already_snake` | `already_snake` | `already_snake` |
1. Go writes only exported fields, so a name that starts with a lowercase letter does not reach a Go TypeDef. The `SnakeCase` helper gives `created_at`, `foo_u_r_l`, `x2_y`, and `fooÖl` for these names.
2. The Rust runtime writes the raw name (probe of `TypeMeta::new` with `FieldInfo::new`), but `#[derive(ForyStruct)]` does not compile for a field name with an ASCII capital letter (error E0063, "missing field"). The derive converts identifiers to snake_case at `object/util.rs:410` and `:422`.
3. `fooÖl` is not a valid Dart identifier.
4. Rust and Python stop with "Unsupported character for LOWER_UPPER_DIGIT_SPECIAL encoding". Java writes the name as UTF-8.
Some runtimes use a second conversion for the field sort order and the struct fingerprint, and for some inputs it gives a different result from their written name:
| Input | JS sort and fingerprint | Rust sort and fingerprint | Python sort and fingerprint |
|---|---|---|---|
| `fooURL` | `foo_url` | `foo_url` | `foo_url` |
| `URLValue` | `url_value` | `url_value` | `url_value` |
| `HTTP2Server` | `htt_p2_server` | `htt_p2_server` | `http2_server` |
| `CreatedAt` | `created_at` | `created_at` | `created_at` |
| `fooÖl` | `fooÖl` | `fooÖl` | `foo_öl` |
In C#, Swift, and C++, the second conversion gave the same result as the written name for every input.
## Conversion functions
- Java: `StringUtils.lowerCamelToLowerUnderscore` (`StringUtils.java:259`), applied at `TypeDefEncoder.java:277`. The probe built an xlang `TypeDef`, wrote each field with `TypeDefEncoder.writeFieldsInfo`, and found the decoded names in `TypeDef.getEncoded()`. Kotlin and Scala use this fory-core path and were not probed separately.
- JavaScript: `lowerCamelToLowerUnderscore` (`TypeMeta.ts:979`) writes the name (`TypeMeta.ts:917`). `toSnakeCase` (`TypeMeta.ts:1059`) gives the sort key, the fingerprint, and the read match. The probe encoded `TypeMeta.fromTypeInfo(...).toBytes()` and decoded the names.
- Go: `SnakeCase` (`go/fory/util.go:60`), applied at `field_spec.go:427` and written at `type_def.go:509`. Unexported fields are skipped at `type_def.go:494`. The probe ran `buildTypeDef` and `decodeTypeDef`.
- Dart: `_toSnakeCase` (`fory_generator.dart:5739`, called at `:1984`). The runtime writes the generated identifier unchanged (`type_resolver.dart:1181`). The probe ran `build_runner` and read `TypeDef.encoded`.
- C#: the generator's `ToSnakeCase` (`ForyModelGenerator.Analysis.cs:2803`, called at `:1490`) and the runtime's `LowerCamelToLowerUnderscore` (`TypeMeta.cs:81`, applied at `:333`). Both use `char.IsUpper`, so they convert non-ASCII capital letters. The probe compiled a `[ForyStruct]` class with the source generator and round-tripped `TypeMeta.Encode` and `Decode`. A type with both `createdAt` and `CreatedAt` fails the build with FORY016 ("duplicate structural field identity 'created_at'").
- Swift: `lowerCamelToLowerUnderscore` (`TypeMeta.swift:1123`, applied at `:267`). The probe compiled `Sources/Fory` and round-tripped `TypeMeta.encode` and `decode`. The macro's `toSnakeCase` (`ForyObjectMacro.swift:3698`) gives the sort order; the probe ran a verbatim copy of it, because the macro build needs swift-syntax.
- C++: `to_snake_case` (`cpp/fory/util/string_util.h:287`), applied at `type_resolver.h:1183`. The probe built `fory_serialization`, ran `FieldInfoBuilder::build()`, and round-tripped `TypeMeta::to_bytes` and `from_bytes`. The sort order uses `normalize_field_name` (`type_resolver.cc:1540`), which calls a second, file-local `to_snake_case` (`type_resolver.cc:1479`).
- Rust: the derive passes the raw identifier to `FieldInfo::new_with_id` (`rust/fory-derive/src/object/field_codec.rs:201`). The probe ran `TypeMeta::new` with `FieldInfo::new` and decoded the result. `to_snake_case` (`string_util.rs:110`) gives the sort order and the fingerprint.
- Python: the raw dataclass attribute (`meta/typedef.py:1302`), written by `typedef_encoder.write_field_info` (`typedef_encoder.py:247`), which the probe ran with a recording buffer. `_to_snake_case` (`struct.py:1183`) gives the sort order and the fingerprint. The probe did not run `build_field_infos` or the full `encode_typedef`, because pyfory does not import without its native extension.
## Proposal
Define one exact conversion in the xlang specification, apply it in every runtime for the written name, the sort order, and the fingerprint, and add known-answer cases (for example, the names in the table above) to the cross-language tests. A change to the conversion changes the wire names of some fields, so the specification must also say how existing data with such names is handled.
## Related
We found no existing issue for this problem. The row-format schema-evolution work in #3714 uses the Java field-name conversion for its wire names and documents the portable subset, ordinary lowerCamelCase names, until this issue is resolved.
### What did you expect to see?
xlang schema should always produce the same normalized names
### What did you see instead?
different languages produce different names for the same schema
### Anything Else?
Issue diagnosed with Claude Opus 5.5
### Are you willing to submit a PR?
- [ ] I'm willing to submit a PR!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@fory.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@fory.apache.org
For additional commands, e-mail: commits-help@fory.apache.org
Re: [I] xlang: the snake_case conversion of field names is not specified, and the runtimes do not agree [fory]
Posted by "dhruv-dagar (via GitHub)" <gi...@apache.org>.
dhruv-dagar commented on issue #4187:
URL: https://github.com/apache/fory/issues/4187#issuecomment-6271786906
.take
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@fory.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@fory.apache.org
For additional commands, e-mail: commits-help@fory.apache.org
Re: [I] xlang: the snake_case conversion of field names is not specified, and the runtimes do not agree [fory]
Posted by "dhruv-dagar (via GitHub)" <gi...@apache.org>.
dhruv-dagar commented on issue #4187:
URL: https://github.com/apache/fory/issues/4187#issuecomment-6262896917
Hi @stevenschlansker , I would like to contribute to this issue
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@fory.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@fory.apache.org
For additional commands, e-mail: commits-help@fory.apache.org