You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@beam.apache.org by "mkhludnev (via GitHub)" <gi...@apache.org> on 2023/07/08 14:10:55 UTC
[GitHub] [beam] mkhludnev opened a new pull request, #27406: Reproducing the compatibility between JdbcIO.readRows() and AvroUtils.toAvroSchema() : (#27380)
mkhludnev opened a new pull request, #27406:
URL: https://github.com/apache/beam/pull/27406
- AvroUtils can't create schema from Beam schema made by JdbcIO.readRows() for NUMERIC and DECIMAL types.
- reproducer for (#27380)
Results seem really sad. AvroUtils.toAvroSchema() fails on NUMERIC and DECIMAL (here it pass for INT but I'm gust verifying the test approach). Failure looks like:
```
java.lang.RuntimeException: Unhandled logical type beam:logical_type:fixed_decimal:v1
at org.apache.beam.sdk.extensions.avro.schemas.utils.AvroUtils.getFieldSchema(AvroUtils.java:1095)
at org.apache.beam.sdk.extensions.avro.schemas.utils.AvroUtils.toAvroField(AvroUtils.java:453)
at org.apache.beam.sdk.extensions.avro.schemas.utils.AvroUtils.toAvroSchema(AvroUtils.java:498)
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] mkhludnev commented on pull request #27406: Reproducing the compatibility problem between JdbcIO.readRows() and AvroUtils.toAvroSchema() : (#27380)
Posted by "mkhludnev (via GitHub)" <gi...@apache.org>.
mkhludnev commented on PR #27406:
URL: https://github.com/apache/beam/pull/27406#issuecomment-1642698384
> Could you elaborate a bit about that and what are the next steps from your point of view and what help do you need?
when Postgres `NUMERIC|DECIMAL` (no precision, no scale) writes value with fractional part `1.23` into Avro.
JdbcIOToAvroIT fails with
```
Caused by: org.apache.avro.AvroTypeException: Cannot encode decimal with scale 2 as scale 0
at app//org.apache.avro.Conversions$DecimalConversion.toBytes(Conversions.java:92)
at app//org.apache.beam.sdk.extensions.avro.schemas.utils.AvroUtils.genericFromBeamField(AvroUtils.java:1162)
```
The problem here is that Avro schema should cary two values for this types: bigdecimal bytes (we have it now) and length of fractional part ie scale (since it might be arbitrary for SQL `NUMERIC`). Or we already have an approach to cary SQL `NUMERIC` in Avro.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] mkhludnev commented on pull request #27406: Reproducing the compatibility problem between JdbcIO.readRows() and AvroUtils.toAvroSchema() : (#27380)
Posted by "mkhludnev (via GitHub)" <gi...@apache.org>.
mkhludnev commented on PR #27406:
URL: https://github.com/apache/beam/pull/27406#issuecomment-1630313700
After checking doc about types:
- NUMERIC/DECIMAL(10)/(10,2) should be represented via FIXED in Avro, however number of digits should be converted into number of bytes in the precision field
- NUMERIC/DECIMAL which is reported by Postgres JDBC as precision=0 (not -1 nor MAX_INT!) it should be represented as BYTES. Since every value may have a variable number of digits, however it should also carry a scale field.
WDYT?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] mkhludnev commented on pull request #27406: Reproducing the compatibility problem between JdbcIO.readRows() and AvroUtils.toAvroSchema() : (#27380)
Posted by "mkhludnev (via GitHub)" <gi...@apache.org>.
mkhludnev commented on PR #27406:
URL: https://github.com/apache/beam/pull/27406#issuecomment-1631152000
> . There is slightly difference between SQL NUMERIC and DECIMAL. Probably check that did this difference get preserved in the old implementation?
Hm.. I don't know. Here's what I've got https://www.postgresql.org/docs/current/datatype-numeric.html#DATATYPE-INT
> The types `decimal` and `numeric` are equivalent. Both types are part of the SQL standard.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] Abacn commented on pull request #27406: Reproducing the compatibility problem between JdbcIO.readRows() and AvroUtils.toAvroSchema() : (#27380)
Posted by "Abacn (via GitHub)" <gi...@apache.org>.
Abacn commented on PR #27406:
URL: https://github.com/apache/beam/pull/27406#issuecomment-1627896592
Thanks, if I understood correctly, the problem is on AvroUtil side; JdbcIO can still read from jdbc and send correct elements (Decimal with correct unit) downstream; but if try to write to Avro files, avro schema cannot be created from Beam schema that Beam Row provides. Is that the issue?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] github-actions[bot] commented on pull request #27406: Reproducing the compatibility problem between JdbcIO.readRows() and AvroUtils.toAvroSchema() : (#27380)
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #27406:
URL: https://github.com/apache/beam/pull/27406#issuecomment-1737320673
This pull request has been closed due to lack of activity. If you think that is incorrect, or the pull request requires review, you can revive the PR at any time.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] mkhludnev commented on pull request #27406: Reproducing the compatibility problem between JdbcIO.readRows() and AvroUtils.toAvroSchema() : (#27380)
Posted by "mkhludnev (via GitHub)" <gi...@apache.org>.
mkhludnev commented on PR #27406:
URL: https://github.com/apache/beam/pull/27406#issuecomment-1627354746
My IT hasn't picked by CI, I don't know why. But the failure looks like https://paste.apache.org/wms63
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] mkhludnev commented on pull request #27406: Reproducing the compatibility problem between JdbcIO.readRows() and AvroUtils.toAvroSchema() : (#27380)
Posted by "mkhludnev (via GitHub)" <gi...@apache.org>.
mkhludnev commented on PR #27406:
URL: https://github.com/apache/beam/pull/27406#issuecomment-1627359996
> Failure looks like:
>
> ```
> java.lang.RuntimeException: Unhandled logical type beam:logical_type:fixed_decimal:v1
> at org.apache.beam.sdk.extensions.avro.schemas.utils.AvroUtils.getFieldSchema(AvroUtils.java:1095)
> at org.apache.beam.sdk.extensions.avro.schemas.utils.AvroUtils.toAvroField(AvroUtils.java:453)
> at org.apache.beam.sdk.extensions.avro.schemas.utils.AvroUtils.toAvroSchema(AvroUtils.java:498)
> ```
This particular one can be fixed via https://github.com/apache/beam/blame/2d75860eb83feb167e5f909a2fa004d68649100c/sdks/java/extensions/avro/src/main/java/org/apache/beam/sdk/extensions/avro/schemas/utils/AvroUtils.java#L1195 but it's just a top of the hill
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] mkhludnev commented on pull request #27406: Reproducing the compatibility problem between JdbcIO.readRows() and AvroUtils.toAvroSchema() : (#27380)
Posted by "mkhludnev (via GitHub)" <gi...@apache.org>.
mkhludnev commented on PR #27406:
URL: https://github.com/apache/beam/pull/27406#issuecomment-1642760135
> Could you elaborate a bit about that and what are the next steps from your point of view and what help do you need?
When JdbcIOToAvroIT `NUMERIC/DECIMAL(10,2)` (with precision and scale) writes value into Avro. It fails with
```
Caused by: java.lang.RuntimeException: Unhandled logical type beam:logical_type:fixed_decimal:v1
at org.apache.beam.sdk.extensions.avro.schemas.utils.AvroUtils.genericFromBeamField(AvroUtils.java:1228)
```
`BigDecimal` arrives into `AU.genericFromBeamField` with FixedPrecisionNumber logical type. I'm able to capture it via
`if (FixedPrecisionNumber.IDENTIFIER.equals(identifier)) {` but what's the GenericData in this case. Should we just yield same BigDecimal or I should somehow construct bytes of FixedSchema?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] github-actions[bot] closed pull request #27406: Reproducing the compatibility problem between JdbcIO.readRows() and AvroUtils.toAvroSchema() : (#27380)
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed pull request #27406: Reproducing the compatibility problem between JdbcIO.readRows() and AvroUtils.toAvroSchema() : (#27380)
URL: https://github.com/apache/beam/pull/27406
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] github-actions[bot] commented on pull request #27406: Reproducing the compatibility problem between JdbcIO.readRows() and AvroUtils.toAvroSchema() : (#27380)
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #27406:
URL: https://github.com/apache/beam/pull/27406#issuecomment-1727647509
This pull request has been marked as stale due to 60 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think that’s incorrect or this pull request requires a review, please simply write any comment. If closed, you can revive the PR at any time and @mention a reviewer or discuss it on the dev@beam.apache.org list. Thank you for your contributions.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] Abacn commented on pull request #27406: Reproducing the compatibility problem between JdbcIO.readRows() and AvroUtils.toAvroSchema() : (#27380)
Posted by "Abacn (via GitHub)" <gi...@apache.org>.
Abacn commented on PR #27406:
URL: https://github.com/apache/beam/pull/27406#issuecomment-1630878351
> I suppose to add a condition for capturing FixedPrecisionNumeric into AvroUtils.getFieldSchema(AvroUtils.java:1095).
Thanks, these sounds good.
> JdbcIO.readRows() doesn't pass the scale when converts DECIMAL(10,2).
From the change of this PR, is it due to Jdbc's SchemaUtil treat NUMERIC and DECIMAL differently (NUMERIC -> fixed_numeric logical type; DECIMAL -> DECIMAL field type). There is slightly difference between SQL NUMERIC and DECIMAL. Probably check that did this difference get preserved in the old implementation?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] mkhludnev commented on a diff in pull request #27406: Reproducing the compatibility problem between JdbcIO.readRows() and AvroUtils.toAvroSchema() : (#27380)
Posted by "mkhludnev (via GitHub)" <gi...@apache.org>.
mkhludnev commented on code in PR #27406:
URL: https://github.com/apache/beam/pull/27406#discussion_r1259699225
##########
sdks/java/extensions/avro/src/main/java/org/apache/beam/sdk/extensions/avro/schemas/utils/AvroUtils.java:
##########
@@ -1055,45 +1049,7 @@ private static org.apache.avro.Schema getFieldSchema(
break;
case LOGICAL_TYPE:
- String identifier = fieldType.getLogicalType().getIdentifier();
- if (FixedBytes.IDENTIFIER.equals(identifier)) {
- FixedBytesField fixedBytesField =
- checkNotNull(FixedBytesField.fromBeamFieldType(fieldType));
- baseType = fixedBytesField.toAvroType("fixed", namespace + "." + fieldName);
- } else if (VariableBytes.IDENTIFIER.equals(identifier)) {
- // treat VARBINARY as bytes as that is what avro supports
- baseType = org.apache.avro.Schema.create(Type.BYTES);
- } else if (FixedString.IDENTIFIER.equals(identifier)
- || "CHAR".equals(identifier)
- || "NCHAR".equals(identifier)) {
- baseType =
- buildHiveLogicalTypeSchema("char", (int) fieldType.getLogicalType().getArgument());
- } else if (VariableString.IDENTIFIER.equals(identifier)
- || "NVARCHAR".equals(identifier)
- || "VARCHAR".equals(identifier)
- || "LONGNVARCHAR".equals(identifier)
- || "LONGVARCHAR".equals(identifier)) {
- baseType =
- buildHiveLogicalTypeSchema("varchar", (int) fieldType.getLogicalType().getArgument());
- } else if (EnumerationType.IDENTIFIER.equals(identifier)) {
- EnumerationType enumerationType = fieldType.getLogicalType(EnumerationType.class);
- baseType =
- org.apache.avro.Schema.createEnum(fieldName, "", "", enumerationType.getValues());
- } else if (OneOfType.IDENTIFIER.equals(identifier)) {
- OneOfType oneOfType = fieldType.getLogicalType(OneOfType.class);
- baseType =
- org.apache.avro.Schema.createUnion(
- oneOfType.getOneOfSchema().getFields().stream()
- .map(x -> getFieldSchema(x.getType(), x.getName(), namespace))
- .collect(Collectors.toList()));
- } else if ("DATE".equals(identifier) || SqlTypes.DATE.getIdentifier().equals(identifier)) {
- baseType = LogicalTypes.date().addToSchema(org.apache.avro.Schema.create(Type.INT));
- } else if ("TIME".equals(identifier)) {
- baseType = LogicalTypes.timeMillis().addToSchema(org.apache.avro.Schema.create(Type.INT));
- } else {
- throw new RuntimeException(
- "Unhandled logical type " + fieldType.getLogicalType().getIdentifier());
- }
+ baseType = getLogicalTypeFieldSchema(fieldType, fieldName, namespace);
Review Comment:
Extracted logical type handling before adding FixedPrecisionNumeric case.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] mkhludnev commented on a diff in pull request #27406: Reproducing the compatibility problem between JdbcIO.readRows() and AvroUtils.toAvroSchema() : (#27380)
Posted by "mkhludnev (via GitHub)" <gi...@apache.org>.
mkhludnev commented on code in PR #27406:
URL: https://github.com/apache/beam/pull/27406#discussion_r1259990781
##########
sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/logicaltypes/FixedPrecisionNumeric.java:
##########
@@ -113,4 +115,13 @@ public BigDecimal toInputType(BigDecimal base) {
}
return base;
}
+
+ /**
+ * @TODO is byte or fixed?
+ * @TODO should is pass namespace/field name, etc?
+ * */
+ public org.apache.avro.Schema toAvroType() {
Review Comment:
Fair. It needs to access `precision, scale)` which are private, yet, with no getters.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] Abacn commented on a diff in pull request #27406: Reproducing the compatibility problem between JdbcIO.readRows() and AvroUtils.toAvroSchema() : (#27380)
Posted by "Abacn (via GitHub)" <gi...@apache.org>.
Abacn commented on code in PR #27406:
URL: https://github.com/apache/beam/pull/27406#discussion_r1259763372
##########
sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/logicaltypes/FixedPrecisionNumeric.java:
##########
@@ -113,4 +115,13 @@ public BigDecimal toInputType(BigDecimal base) {
}
return base;
}
+
+ /**
+ * @TODO is byte or fixed?
+ * @TODO should is pass namespace/field name, etc?
+ * */
+ public org.apache.avro.Schema toAvroType() {
Review Comment:
We are in the process of getting rid of avro dependency in java core (CC: @aromanenko-dev) this function probably should goes to AvroUtils
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] mkhludnev commented on a diff in pull request #27406: Reproducing the compatibility problem between JdbcIO.readRows() and AvroUtils.toAvroSchema() : (#27380)
Posted by "mkhludnev (via GitHub)" <gi...@apache.org>.
mkhludnev commented on code in PR #27406:
URL: https://github.com/apache/beam/pull/27406#discussion_r1259516781
##########
sdks/java/extensions/avro/src/main/java/org/apache/beam/sdk/extensions/avro/schemas/utils/AvroUtils.java:
##########
@@ -1124,6 +1080,53 @@ private static org.apache.avro.Schema getFieldSchema(
return fieldType.getNullable() ? ReflectData.makeNullable(baseType) : baseType;
}
+ private static org.apache.avro.Schema getLogicalTypeFieldSchema(FieldType fieldType, String fieldName, String namespace) {
+ org.apache.avro.Schema baseType;
+ String identifier = fieldType.getLogicalType().getIdentifier();
+ if (FixedBytes.IDENTIFIER.equals(identifier)) {
+ FixedBytesField fixedBytesField =
+ checkNotNull(FixedBytesField.fromBeamFieldType(fieldType));
+ baseType = fixedBytesField.toAvroType("fixed", namespace + "." + fieldName);
+ } else if (FixedPrecisionNumeric.IDENTIFIER.equals(identifier)) {
+ FixedPrecisionNumeric decimalType = fieldType.getLogicalType(FixedPrecisionNumeric.class);
Review Comment:
here's what been added
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] mkhludnev commented on pull request #27406: Reproducing the compatibility problem between JdbcIO.readRows() and AvroUtils.toAvroSchema() : (#27380)
Posted by "mkhludnev (via GitHub)" <gi...@apache.org>.
mkhludnev commented on PR #27406:
URL: https://github.com/apache/beam/pull/27406#issuecomment-1630544772
pushed https://github.com/apache/beam/pull/27406/commits/dd74e7a8aa5ac393fbeeda36d782ff9dd04230ab which just let to convert beam schema to Avro's. Now I'm checking writing Avro for these types.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] aromanenko-dev commented on pull request #27406: Reproducing the compatibility problem between JdbcIO.readRows() and AvroUtils.toAvroSchema() : (#27380)
Posted by "aromanenko-dev (via GitHub)" <gi...@apache.org>.
aromanenko-dev commented on PR #27406:
URL: https://github.com/apache/beam/pull/27406#issuecomment-1645747240
@mkhludnev Thanks! Sorry, I didn't have a time this week to take a look more deeply on this isse and the next week I'll be mostly off. So, we can come back to this only after.
In the same time, since seems there is no obvious solution, I'd recommend to start a discussion on `dev@beam.apache.org` and explain this issue in details there to have more eyes in this problem.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] aromanenko-dev commented on pull request #27406: Reproducing the compatibility problem between JdbcIO.readRows() and AvroUtils.toAvroSchema() : (#27380)
Posted by "aromanenko-dev (via GitHub)" <gi...@apache.org>.
aromanenko-dev commented on PR #27406:
URL: https://github.com/apache/beam/pull/27406#issuecomment-1642375464
@mkhludnev Thanks for you contribution!
I'm sorry but I'm a bit lost with the current state of your JdbcIO/Avro changes.
Could you elaborate a bit about that and what are the next steps from your point of view and what help do you need?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] mkhludnev commented on pull request #27406: Reproducing the compatibility problem between JdbcIO.readRows() and AvroUtils.toAvroSchema() : (#27380)
Posted by "mkhludnev (via GitHub)" <gi...@apache.org>.
mkhludnev commented on PR #27406:
URL: https://github.com/apache/beam/pull/27406#issuecomment-1643957342
@aromanenko-dev, another principal question: What's the proper Avro type for SQL type with declared precision `NUMERIC/DECIMAL(10)`:
- is it `fixed` with explicitly defined length since we know max number of decimal digits it carries. btw is there a conversion for number of digits to the length of byte array?
- or it is `bytes` of variable length since a particular value might not have all digits it's declared.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] mkhludnev commented on pull request #27406: Reproducing the compatibility problem between JdbcIO.readRows() and AvroUtils.toAvroSchema() : (#27380)
Posted by "mkhludnev (via GitHub)" <gi...@apache.org>.
mkhludnev commented on PR #27406:
URL: https://github.com/apache/beam/pull/27406#issuecomment-1628323082
> Thanks, if I understood correctly, the problem is on AvroUtil side; JdbcIO can still read from jdbc and send correct elements (Decimal with correct unit) downstream; but if try to write to Avro files, avro schema cannot be created from Beam schema that Beam Row provides. Is that the issue?
1. @Abacn, thanks for looking into.
2. Your interpretation of end-user problem is correct.
3. This problem/failure also hides another the smaller one - JdbcIO.readRows() doesn't pass the scale when converts DECIMAL(10,2).
4. Can you give a clue where start to tackle this problem? I suppose to add a condition for capturing FixedPrecisionNumeric into `AvroUtils.getFieldSchema(AvroUtils.java:1095)`. WDYT?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] mkhludnev commented on pull request #27406: Reproducing the compatibility problem between JdbcIO.readRows() and AvroUtils.toAvroSchema() : (#27380)
Posted by "mkhludnev (via GitHub)" <gi...@apache.org>.
mkhludnev commented on PR #27406:
URL: https://github.com/apache/beam/pull/27406#issuecomment-1631462476
[7a80a24](https://github.com/apache/beam/pull/27406/commits/7a80a2403b9310260f22f33539dfe4f78ebb8f8f) Adding toGeneric conversion into test pipeline. I have no idea how to fix it.
I really appreciate if you give me a clue!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org