You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@beam.apache.org by "barunkumaracharya (via GitHub)" <gi...@apache.org> on 2026/02/05 19:47:51 UTC
[I] [Bug]: JavaFieldSchema does not support for Beam Logical Types [beam]
barunkumaracharya opened a new issue, #37524:
URL: https://github.com/apache/beam/issues/37524
### What happened?
I had a Iceberg Row which had a column called "eventTime". The eventTime column datatype was [org.apache.beam.sdk.schemas.logicaltypes.DateTime](https://github.com/apache/beam/blob/master/sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/logicaltypes/DateTime.java).
I wanted to convert the Row DataType to my java class and for this purpose, i was trying to use the [Convert.FromRow](https://github.com/apache/beam/blob/master/sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/transforms/Convert.java) transform
There is no test case that converts a logical beam type to java type in ConvertTest.java
I have tried many data types and options but I was not able to convert it to a Java DataType. I have tried Java Instant, Java LocalDateTime, Joda Instant / LocalDateTime but was still not able to convert it. One of the main reasons that i observed was that the Static Schema Inference class that JavaFieldSchema schemaProvider class uses to determine the Beam Type from a Java Type does not have support for Beam Logical Type. Checkout function **fieldFromType** in this [file](https://github.com/apache/beam/blob/master/sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/utils/StaticSchemaInference.java)
This was the error that i was continuously facing when i was trying to convert it -
`"Cannot convert between types that don't have equivalent schemas."`
which is present in the ConvertHelpers.java class.
### Issue Priority
Priority: 2 (default / most bugs should be filed as P2)
### Issue Components
- [ ] Component: Python SDK
- [x] Component: Java SDK
- [ ] Component: Go SDK
- [ ] Component: Typescript SDK
- [ ] Component: IO connector
- [ ] Component: Beam YAML
- [ ] Component: Beam examples
- [ ] Component: Beam playground
- [ ] Component: Beam katas
- [ ] Component: Website
- [ ] Component: Infrastructure
- [ ] Component: Spark Runner
- [ ] Component: Flink Runner
- [ ] Component: Samza Runner
- [ ] Component: Twister2 Runner
- [ ] Component: Hazelcast Jet Runner
- [ ] Component: Google Cloud Dataflow Runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug]: JavaFieldSchema does not support for Beam Logical Types [beam]
Posted by "barunkumaracharya (via GitHub)" <gi...@apache.org>.
barunkumaracharya commented on issue #37524:
URL: https://github.com/apache/beam/issues/37524#issuecomment-4374025367
I had tried all such combinations, it had not worked
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug]: JavaFieldSchema does not support for Beam Logical Types [beam]
Posted by "ahmedabu98 (via GitHub)" <gi...@apache.org>.
ahmedabu98 commented on issue #37524:
URL: https://github.com/apache/beam/issues/37524#issuecomment-4253411924
Hey @barunkumaracharya, this just came back up in my notifications. Sorry for the late reply.
If you take a look at the error:
```
Exception in thread "main" java.lang.RuntimeException:
Cannot convert between types that don't have equivalent schemas.
input schema: Fields:
Field{name=time, description=, type=LOGICAL_TYPE<beam:logical_type:datetime:v1>, options={{}}}
Field{name=name, description=, type=STRING, options={{}}}
output schema: Fields:
Field{name=time, description=, type=DATETIME, options={{}}}
Field{name=name, description=, type=STRING, options={{}}}
```
The table is returning "name" as a [logical type DateTime](https://github.com/apache/beam/blob/master/sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/logicaltypes/DateTime.java) field (maps to a Java LocalDateTime), but your Pojo is defined with a Joda Instant, which maps to a primitive [Schema.FieldType.DATETIME](https://github.com/apache/beam/blob/4ee32cddb4c10dcf738558e4ffcafc50255a1bd0/sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/Schema.java#L797-L798) (not a logical type).
Try re-defining your Pojo to use a Java LocalDateTime and see if it works?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug]: JavaFieldSchema does not support for Beam Logical Types [beam]
Posted by "ahmedabu98 (via GitHub)" <gi...@apache.org>.
ahmedabu98 closed issue #37524: [Bug]: JavaFieldSchema does not support for Beam Logical Types
URL: https://github.com/apache/beam/issues/37524
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug]: JavaFieldSchema does not support for Beam Logical Types [beam]
Posted by "tvalentyn (via GitHub)" <gi...@apache.org>.
tvalentyn commented on issue #37524:
URL: https://github.com/apache/beam/issues/37524#issuecomment-3874970169
cc: @ahmedabu98 who might have some thoughts.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug]: JavaFieldSchema does not support for Beam Logical Types [beam]
Posted by "ahmedabu98 (via GitHub)" <gi...@apache.org>.
ahmedabu98 commented on issue #37524:
URL: https://github.com/apache/beam/issues/37524#issuecomment-3880732964
Hey @barunkumaracharya, can you please provide a simple piece of code to reproduce this error? We can try taking a look
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug]: JavaFieldSchema does not support for Beam Logical Types [beam]
Posted by "barunkumaracharya (via GitHub)" <gi...@apache.org>.
barunkumaracharya commented on issue #37524:
URL: https://github.com/apache/beam/issues/37524#issuecomment-3885280175
I created a table using spark sql - this was my table creation command -
**CREATE TABLE catalog.default.table1 (time TIMESTAMP_NTZ, name STRING)**
Then, i inserted 1 row using spark sql.
**INSERT INTO catalog.default.table1 SELECT CAST(CAST(1770816992L AS TIMESTAMP) AS TIMESTAMP_NTZ), 'abdcd'**
Then, i tried reading the data via apache beam using this example -
`package com.sample.iceberg.job;
import lombok.Data;
import lombok.extern.slf4j.Slf4j;
import org.apache.beam.runners.direct.DirectRunner;
import org.apache.beam.sdk.Pipeline;
import org.apache.beam.sdk.managed.Managed;
import org.apache.beam.sdk.options.PipelineOptions;
import org.apache.beam.sdk.options.PipelineOptionsFactory;
import org.apache.beam.sdk.schemas.JavaFieldSchema;
import org.apache.beam.sdk.schemas.annotations.DefaultSchema;
import org.apache.beam.sdk.schemas.transforms.Convert;
import org.apache.beam.sdk.values.PCollection;
import org.apache.beam.sdk.values.Row;
import org.joda.time.Instant;
import javax.annotation.Nullable;
import java.util.HashMap;
import java.util.Map;
@Slf4j
public class IcebergToFile {
@Data
@DefaultSchema(JavaFieldSchema.class)
private static class myPojo {
@Nullable Instant time;
@Nullable String name;
}
public IcebergToFile() {
}
public static void main(String[] args) {
Map<String, Object> tableConfig = new HashMap<>();
Map<String, Object> catalogProps = new HashMap<>();
catalogProps.put("type", "hadoop");
catalogProps.put("warehouse", "gs://gs-bucket/warehouse/");
Map<String, Object> configProps = new HashMap<>();
configProps.put("fs.gs.auth.service.account.enable", "true");
configProps.put("fs.gs.impl", "com.google.cloud.hadoop.fs.gcs.GoogleHadoopFileSystem");
configProps.put("fs.gs.auth.type", "SERVICE_ACCOUNT_JSON_KEYFILE");
configProps.put("fs.AbstractFileSystem.gs.impl", "com.google.cloud.hadoop.fs.gcs.GoogleHadoopFS");
configProps.put("fs.gs.auth.service.account.json.keyfile", "/fileToJsonKeyFile.json");
tableConfig.put("catalog_name", "catalog");
tableConfig.put("table", "default.table1");
tableConfig.put("catalog_properties", catalogProps);
tableConfig.put("config_properties", configProps);
PipelineOptions options = PipelineOptionsFactory.create();
options.setRunner(DirectRunner.class);
options.setJobName("sample iceberg job");
Pipeline pipeline = Pipeline.create(options);
PCollection<Row> tableData = pipeline.apply(Managed.read(Managed.ICEBERG).withConfig(tableConfig)).getSinglePCollection();
PCollection<myPojo> myPojoCollection = tableData.apply("convert", Convert.fromRows(myPojo.class));
}
}`
The error that i get while running this program is this -
`[main] WARN org.apache.hadoop.util.NativeCodeLoader - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
[main] INFO org.apache.iceberg.CatalogUtil - Loading custom FileIO implementation: org.apache.iceberg.hadoop.HadoopFileIO
Feb 11, 2026 9:04:20 PM com.google.cloud.hadoop.fs.gcs.GhfsStorageStatistics updateMinMaxStats
INFO: Detected potential high latency for operation op_open. latencyMs=622; previousMaxLatencyMs=0; operationCount=1; context=gs://gs-bucket/warehouse/default/table1/metadata/version-hint.text
[main] INFO org.apache.iceberg.BaseMetastoreCatalog - Table loaded by catalog: catalog.default.table1
[main] INFO org.apache.iceberg.CatalogUtil - Loading custom FileIO implementation: org.apache.iceberg.hadoop.HadoopFileIO
[main] INFO org.apache.iceberg.BaseMetastoreCatalog - Table loaded by catalog: catalog.default.table1
Exception in thread "main" java.lang.RuntimeException: Cannot convert between types that don't have equivalent schemas. input schema: Fields:
Field{name=time, description=, type=LOGICAL_TYPE<beam:logical_type:datetime:v1>, options={{}}}
Field{name=name, description=, type=STRING, options={{}}}
Encoding positions:
{name=1, time=0}
Options:{{}}UUID: 07b4b693-67c4-4bf4-aa9b-02bab33de9f2 output schema: Fields:
Field{name=time, description=, type=DATETIME, options={{}}}
Field{name=name, description=, type=STRING, options={{}}}
Encoding positions:
{name=1, time=0}
Options:{{}}UUID: 2f7c9d70-ac9b-4516-bab8-c9546b259883
at org.apache.beam.sdk.schemas.utils.ConvertHelpers.getConvertedSchemaInformation(ConvertHelpers.java:137)
at org.apache.beam.sdk.schemas.transforms.Convert$ConvertTransform.expand(Convert.java:118)
at org.apache.beam.sdk.schemas.transforms.Convert$ConvertTransform.expand(Convert.java:97)
at org.apache.beam.sdk.Pipeline.applyInternal(Pipeline.java:559)
at org.apache.beam.sdk.Pipeline.applyTransform(Pipeline.java:507)
at org.apache.beam.sdk.values.PCollection.apply(PCollection.java:365)
at com.sample.iceberg.job.IcebergToFile.main(IcebergToFile.java:60)
`
@ahmedabu98 For your reference
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug]: JavaFieldSchema does not support for Beam Logical Types [beam]
Posted by "sachinnn99 (via GitHub)" <gi...@apache.org>.
sachinnn99 commented on issue #37524:
URL: https://github.com/apache/beam/issues/37524#issuecomment-4246104654
.take-issue
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug]: JavaFieldSchema does not support for Beam Logical Types [beam]
Posted by "sachinnn99 (via GitHub)" <gi...@apache.org>.
sachinnn99 commented on issue #37524:
URL: https://github.com/apache/beam/issues/37524#issuecomment-4182293781
Is this still open? I'd like to work on it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug]: JavaFieldSchema does not support for Beam Logical Types [beam]
Posted by "barunkumaracharya (via GitHub)" <gi...@apache.org>.
barunkumaracharya commented on issue #37524:
URL: https://github.com/apache/beam/issues/37524#issuecomment-4190523200
Yes, this is still open. You can work on it, if you want
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org