You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@parquet.apache.org by "Nils Broman (Jira)" <ji...@apache.org> on 2022/06/07 14:08:00 UTC
[jira] [Updated] (PARQUET-2153) Cannot read schema from parquet file
[ https://issues.apache.org/jira/browse/PARQUET-2153?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Nils Broman updated PARQUET-2153:
---------------------------------
Description:
I'm trying to generate a Avro schema from a parquet file. I get the issue both when using [https://github.com/benwatson528/intellij-avro-parquet-plugin] in Intellij as well as when I'm using my own implementation of generating a schema.
The parquet file contains nested entries and arrays: \{"a": a, "b": [{"c": c}]}
I get the following error message:
{code:java}
org.apache.avro.SchemaParseException: Can't redefine: element
at org.apache.avro.Schema$Names.put(Schema.java:1547)
at org.apache.avro.Schema$NamedSchema.writeNameRef(Schema.java:810)
at org.apache.avro.Schema$RecordSchema.toJson(Schema.java:972)
at org.apache.avro.Schema$UnionSchema.toJson(Schema.java:1239)
at org.apache.avro.Schema$RecordSchema.fieldsToJson(Schema.java:1000)
at org.apache.avro.Schema$RecordSchema.toJson(Schema.java:984)
at org.apache.avro.Schema$ArraySchema.toJson(Schema.java:1134)
at org.apache.avro.Schema$UnionSchema.toJson(Schema.java:1239)
at org.apache.avro.Schema$RecordSchema.fieldsToJson(Schema.java:1000)
at org.apache.avro.Schema$RecordSchema.toJson(Schema.java:984)
at org.apache.avro.Schema$UnionSchema.toJson(Schema.java:1239)
at org.apache.avro.Schema$RecordSchema.fieldsToJson(Schema.java:1000)
at org.apache.avro.Schema$RecordSchema.toJson(Schema.java:984)
at org.apache.avro.Schema.toString(Schema.java:424)
at org.apache.avro.Schema.toString(Schema.java:396)
at uk.co.hadoopathome.intellij.viewer.fileformat.ParquetFileReader.getSchema(ParquetFileReader.java:65)
at uk.co.hadoopathome.intellij.viewer.FileViewerToolWindow$2.doInBackground(FileViewerToolWindow.java:196)
at uk.co.hadoopathome.intellij.viewer.FileViewerToolWindow$2.doInBackground(FileViewerToolWindow.java:184)
at java.desktop/javax.swing.SwingWorker$1.call(SwingWorker.java:304)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at java.desktop/javax.swing.SwingWorker.run(SwingWorker.java:343)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:829) {code}
It appears to be an issue similar to [PARQUET-1441|https://issues.apache.org/jira/browse/PARQUET-1441] or [PARQUET-1409|https://issues.apache.org/jira/browse/PARQUET-1409].
Or could it possibly be something wrong in my parquet file?
was:
I'm trying to generate a Avro schema from a parquet file. I get the issue both when using [https://github.com/benwatson528/intellij-avro-parquet-plugin] in Intellij as well as when I'm using my own implementation of generating a schema.
The parquet file contains nested entries and arrays: \{"a": a, "b": [{"c": c}]}
I get the following error message:
{code:java}
org.apache.avro.SchemaParseException: Can't redefine: element
at org.apache.avro.Schema$Names.put(Schema.java:1547)
at org.apache.avro.Schema$NamedSchema.writeNameRef(Schema.java:810)
at org.apache.avro.Schema$RecordSchema.toJson(Schema.java:972)
at org.apache.avro.Schema$UnionSchema.toJson(Schema.java:1239)
at org.apache.avro.Schema$RecordSchema.fieldsToJson(Schema.java:1000)
at org.apache.avro.Schema$RecordSchema.toJson(Schema.java:984)
at org.apache.avro.Schema$ArraySchema.toJson(Schema.java:1134)
at org.apache.avro.Schema$UnionSchema.toJson(Schema.java:1239)
at org.apache.avro.Schema$RecordSchema.fieldsToJson(Schema.java:1000)
at org.apache.avro.Schema$RecordSchema.toJson(Schema.java:984)
at org.apache.avro.Schema$UnionSchema.toJson(Schema.java:1239)
at org.apache.avro.Schema$RecordSchema.fieldsToJson(Schema.java:1000)
at org.apache.avro.Schema$RecordSchema.toJson(Schema.java:984)
at org.apache.avro.Schema.toString(Schema.java:424)
at org.apache.avro.Schema.toString(Schema.java:396)
at uk.co.hadoopathome.intellij.viewer.fileformat.ParquetFileReader.getSchema(ParquetFileReader.java:65)
at uk.co.hadoopathome.intellij.viewer.FileViewerToolWindow$2.doInBackground(FileViewerToolWindow.java:196)
at uk.co.hadoopathome.intellij.viewer.FileViewerToolWindow$2.doInBackground(FileViewerToolWindow.java:184)
at java.desktop/javax.swing.SwingWorker$1.call(SwingWorker.java:304)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at java.desktop/javax.swing.SwingWorker.run(SwingWorker.java:343)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:829) {code}
It appears to be an issue similar to [link title|https://issues.apache.org/jira/browse/PARQUET-1441] or [link title|https://issues.apache.org/jira/browse/PARQUET-1409].
Or could it possibly be something wrong in my parquet file?
> Cannot read schema from parquet file
> ------------------------------------
>
> Key: PARQUET-2153
> URL: https://issues.apache.org/jira/browse/PARQUET-2153
> Project: Parquet
> Issue Type: Bug
> Components: parquet-avro
> Affects Versions: 1.12.3
> Reporter: Nils Broman
> Priority: Major
>
> I'm trying to generate a Avro schema from a parquet file. I get the issue both when using [https://github.com/benwatson528/intellij-avro-parquet-plugin] in Intellij as well as when I'm using my own implementation of generating a schema.
> The parquet file contains nested entries and arrays: \{"a": a, "b": [{"c": c}]}
> I get the following error message:
> {code:java}
> org.apache.avro.SchemaParseException: Can't redefine: element
> at org.apache.avro.Schema$Names.put(Schema.java:1547)
> at org.apache.avro.Schema$NamedSchema.writeNameRef(Schema.java:810)
> at org.apache.avro.Schema$RecordSchema.toJson(Schema.java:972)
> at org.apache.avro.Schema$UnionSchema.toJson(Schema.java:1239)
> at org.apache.avro.Schema$RecordSchema.fieldsToJson(Schema.java:1000)
> at org.apache.avro.Schema$RecordSchema.toJson(Schema.java:984)
> at org.apache.avro.Schema$ArraySchema.toJson(Schema.java:1134)
> at org.apache.avro.Schema$UnionSchema.toJson(Schema.java:1239)
> at org.apache.avro.Schema$RecordSchema.fieldsToJson(Schema.java:1000)
> at org.apache.avro.Schema$RecordSchema.toJson(Schema.java:984)
> at org.apache.avro.Schema$UnionSchema.toJson(Schema.java:1239)
> at org.apache.avro.Schema$RecordSchema.fieldsToJson(Schema.java:1000)
> at org.apache.avro.Schema$RecordSchema.toJson(Schema.java:984)
> at org.apache.avro.Schema.toString(Schema.java:424)
> at org.apache.avro.Schema.toString(Schema.java:396)
> at uk.co.hadoopathome.intellij.viewer.fileformat.ParquetFileReader.getSchema(ParquetFileReader.java:65)
> at uk.co.hadoopathome.intellij.viewer.FileViewerToolWindow$2.doInBackground(FileViewerToolWindow.java:196)
> at uk.co.hadoopathome.intellij.viewer.FileViewerToolWindow$2.doInBackground(FileViewerToolWindow.java:184)
> at java.desktop/javax.swing.SwingWorker$1.call(SwingWorker.java:304)
> at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
> at java.desktop/javax.swing.SwingWorker.run(SwingWorker.java:343)
> at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
> at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
> at java.base/java.lang.Thread.run(Thread.java:829) {code}
> It appears to be an issue similar to [PARQUET-1441|https://issues.apache.org/jira/browse/PARQUET-1441] or [PARQUET-1409|https://issues.apache.org/jira/browse/PARQUET-1409].
> Or could it possibly be something wrong in my parquet file?
--
This message was sent by Atlassian Jira
(v8.20.7#820007)