[Anuj Jain <an...@gmail.com>]

Hi Community,


I am trying to use flink-parquet for reading and writing parquet files from
the Flink filesystem connectors.

In File source, I would be decoding parquet files and converting them to
avro records and similarly in file sink i would be encoding avro records to
parquet files.


For collection i am using

BulkFormat<Object, FileSourceSplit> bulkFormat =
                new
StreamFormatAdapter<>(AvroParquetReaders.forSpecificRecord(recordClazz));
FileSource<Object> source = FileSource.forBulkFileFormat(bulkFormat,
path).build();


and for sinking i am using

FileSink<Object> sink = FileSink.forBulkFormat(path,
AvroParquetWriters.forSpecificRecord(recordClazz)).build()


Query: The StreamFormatAdapter class is marked @Internal and,
AvroParquetReaders
and AvroParquetWriters classes are marked @Experimental – does it mean that
in future flink releases these classes can be changed in a non-backward
compatible way like plugging of any other 3PP rather than “parquet-avro” or
changing the API structure thus impacting the application code ?

Would it be safe to use the code as specified above ?


Thanks and Regards

Anuj

Re: Query on flink-parquet

[Hang Ruan <ru...@gmail.com>]

Hi, Anuj,

Classes annotated with @Internal or @Experimental can be changed across any
two releases. Classes annotated with @PublicEvolving only can be changed
across minor releases (1.17.0 and 1.18.0).
So the classes you mentioned may be changed. If the API changed in a new
release, you have to modify your code to use the new release.

Best,
Hang

Anuj Jain <an...@gmail.com> 于2023年5月8日周一 13:28写道：

>
>> Hi Community,
>>
>>
>> I am trying to use flink-parquet for reading and writing parquet files
>> from the Flink filesystem connectors.
>>
>> In File source, I would be decoding parquet files and converting them to
>> avro records and similarly in file sink i would be encoding avro records to
>> parquet files.
>>
>>
>> For collection i am using
>>
>> BulkFormat<Object, FileSourceSplit> bulkFormat =
>>                 new
>> StreamFormatAdapter<>(AvroParquetReaders.forSpecificRecord(recordClazz));
>> FileSource<Object> source = FileSource.forBulkFileFormat(bulkFormat,
>> path).build();
>>
>>
>> and for sinking i am using
>>
>> FileSink<Object> sink = FileSink.forBulkFormat(path,
>> AvroParquetWriters.forSpecificRecord(recordClazz)).build()
>>
>>
>> Query: The StreamFormatAdapter class is marked @Internal and, AvroParquetReaders
>> and AvroParquetWriters classes are marked @Experimental – does it mean
>> that in future flink releases these classes can be changed in a
>> non-backward compatible way like plugging of any other 3PP rather than
>> “parquet-avro” or changing the API structure thus impacting the application
>> code ?
>>
>> Would it be safe to use the code as specified above ?
>>
>>
>> Thanks and Regards
>>
>> Anuj
>>
>

Query on flink-parquet

[Anuj Jain <an...@gmail.com>]

>
>
> Hi Community,
>
>
> I am trying to use flink-parquet for reading and writing parquet files
> from the Flink filesystem connectors.
>
> In File source, I would be decoding parquet files and converting them to
> avro records and similarly in file sink i would be encoding avro records to
> parquet files.
>
>
> For collection i am using
>
> BulkFormat<Object, FileSourceSplit> bulkFormat =
>                 new
> StreamFormatAdapter<>(AvroParquetReaders.forSpecificRecord(recordClazz));
> FileSource<Object> source = FileSource.forBulkFileFormat(bulkFormat,
> path).build();
>
>
> and for sinking i am using
>
> FileSink<Object> sink = FileSink.forBulkFormat(path,
> AvroParquetWriters.forSpecificRecord(recordClazz)).build()
>
>
> Query: The StreamFormatAdapter class is marked @Internal and, AvroParquetReaders
> and AvroParquetWriters classes are marked @Experimental – does it mean
> that in future flink releases these classes can be changed in a
> non-backward compatible way like plugging of any other 3PP rather than
> “parquet-avro” or changing the API structure thus impacting the application
> code ?
>
> Would it be safe to use the code as specified above ?
>
>
> Thanks and Regards
>
> Anuj
>