You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by "Artem (Jira)" <ji...@apache.org> on 2024/03/07 15:04:00 UTC
[jira] [Updated] (FLINK-34620) Process recursive protobuf schemas in PbToRowTypeUtil
[ https://issues.apache.org/jira/browse/FLINK-34620?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Artem updated FLINK-34620:
--------------------------
Description:
The current version of _PbToRowTypeUtil.generateRowType_ doesn't support recursive protobuf schemas like
{code:java}
message RecursiveMessage {
int32 id = 1;
RecursiveMessage message = 2;
} {code}
It throws _java.lang.StackOverflowError._
This could be implemented by trimming the recursion. Same approach is used in Apache Spark, for example. ([link)|#L178])]
then output Table schemas for the above proto-schema will be like this:
{code:java}
recursiveFieldMaxDepth=0: message: ROW< id: INTEGER >
recursiveFieldMaxDepth=1: message: ROW< id: INTEGER, message: ROW< id: INTEGER > > recursiveFieldMaxDepth=2: message: ROW< id: INTEGER, message: ROW< id: INTEGER, message: ROW< id: INTEGER > > > {code}
was:
The current version of _PbToRowTypeUtil.generateRowType_ doesn't support recursive protobuf schemas like
message RecursiveMessage \{
int32 id = 1;
RecursiveMessage message = 2;
}
It throws _java.lang.StackOverflowError._
This could be implemented by trimming the recursion. Same approach is used in Apache Spark, for example. ([link|[https://github.com/apache/spark/blob/master/connector/protobuf/src/main/scala/org/apache/spark/sql/protobuf/utils/SchemaConverters.scala#L178])]
then output Table schemas for the above proto-schema will be like this:
recursiveFieldMaxDepth=0: message: ROW< id: INTEGER >
recursiveFieldMaxDepth=1: message: ROW< id: INTEGER, message: ROW< id: INTEGER > > recursiveFieldMaxDepth=2: message: ROW< id: INTEGER, message: ROW< id: INTEGER, message: ROW< id: INTEGER > > >
> Process recursive protobuf schemas in PbToRowTypeUtil
> -----------------------------------------------------
>
> Key: FLINK-34620
> URL: https://issues.apache.org/jira/browse/FLINK-34620
> Project: Flink
> Issue Type: Improvement
> Components: Formats (JSON, Avro, Parquet, ORC, SequenceFile)
> Reporter: Artem
> Priority: Major
>
> The current version of _PbToRowTypeUtil.generateRowType_ doesn't support recursive protobuf schemas like
> {code:java}
> message RecursiveMessage {
> int32 id = 1;
> RecursiveMessage message = 2;
> } {code}
> It throws _java.lang.StackOverflowError._
> This could be implemented by trimming the recursion. Same approach is used in Apache Spark, for example. ([link)|#L178])]
>
> then output Table schemas for the above proto-schema will be like this:
> {code:java}
> recursiveFieldMaxDepth=0: message: ROW< id: INTEGER >
> recursiveFieldMaxDepth=1: message: ROW< id: INTEGER, message: ROW< id: INTEGER > > recursiveFieldMaxDepth=2: message: ROW< id: INTEGER, message: ROW< id: INTEGER, message: ROW< id: INTEGER > > > {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)