You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@flink.apache.org by 殿李 <li...@gmail.com> on 2020/07/10 09:28:07 UTC
Does Flink support TFRecordFileOutputFormat?
Hi,
Does Flink support TFRecordFileOutputFormat? I can't find the relevant information in the document.
As far as I know, spark is supportive.
Best regards
Peidian Li
Re: Does Flink support TFRecordFileOutputFormat?
Posted by Peidian Li <li...@gmail.com>.
Thanks, I'll check it out.
Jingsong Li <ji...@gmail.com> 于2020年7月13日周一 下午2:50写道：
> Hi,
>
> Flink also has `HadoopOutputFormat`, it can wrap hadoop OutputFormat to
> Flink sink.
> You can have a try.
>
> Best,
> Jingsong
>
> On Mon, Jul 13, 2020 at 2:34 PM 殿李 <li...@gmail.com> wrote:
>
>> Hi,
>>
>> Yes, TF means TensorFlow.
>>
>> This class may not be in the spark package, but spark supports writing
>> this file format to HDFS.
>>
>> tfRDD.saveAsNewAPIHadoopFile(output, "org.tensorflow.hadoop.io
>> .TFRecordFileOutputFormat",
>>
>> keyClass="org.apache.hadoop.io.BytesWritable",
>>
>> valueClass="org.apache.hadoop.io.NullWritable”)
>>
>>
>>
>> > 2020年7月13日 下午2:21，Danny Chan <yu...@gmail.com> 写道：
>> >
>> > I didn’t see any class named TFRecordFileOutputFormat in Spark, for TF
>> do you mean TensorFlow ?
>> >
>> > Best,
>> > Danny Chan
>> > 在 2020年7月10日 +0800 PM5:28，殿李 <li...@gmail.com>，写道：
>> >> Hi,
>> >>
>> >> Does Flink support TFRecordFileOutputFormat? I can't find the relevant
>> information in the document.
>> >>
>> >> As far as I know, spark is supportive.
>> >>
>> >>
>> >> Best regards
>> >> Peidian Li
>>
>>
>
> --
> Best, Jingsong Lee
>
--
-----------
Best Regards
Peidian Li
Re: Does Flink support TFRecordFileOutputFormat?
Posted by Jingsong Li <ji...@gmail.com>.
Hi,
Flink also has `HadoopOutputFormat`, it can wrap hadoop OutputFormat to
Flink sink.
You can have a try.
Best,
Jingsong
On Mon, Jul 13, 2020 at 2:34 PM 殿李 <li...@gmail.com> wrote:
> Hi,
>
> Yes, TF means TensorFlow.
>
> This class may not be in the spark package, but spark supports writing
> this file format to HDFS.
>
> tfRDD.saveAsNewAPIHadoopFile(output, "org.tensorflow.hadoop.io
> .TFRecordFileOutputFormat",
>
> keyClass="org.apache.hadoop.io.BytesWritable",
>
> valueClass="org.apache.hadoop.io.NullWritable”)
>
>
>
> > 2020年7月13日 下午2:21，Danny Chan <yu...@gmail.com> 写道：
> >
> > I didn’t see any class named TFRecordFileOutputFormat in Spark, for TF
> do you mean TensorFlow ?
> >
> > Best,
> > Danny Chan
> > 在 2020年7月10日 +0800 PM5:28，殿李 <li...@gmail.com>，写道：
> >> Hi,
> >>
> >> Does Flink support TFRecordFileOutputFormat? I can't find the relevant
> information in the document.
> >>
> >> As far as I know, spark is supportive.
> >>
> >>
> >> Best regards
> >> Peidian Li
>
>
--
Best, Jingsong Lee
Re: Does Flink support TFRecordFileOutputFormat?
Posted by 殿李 <li...@gmail.com>.
Hi,
Yes, TF means TensorFlow.
This class may not be in the spark package, but spark supports writing this file format to HDFS.
tfRDD.saveAsNewAPIHadoopFile(output, "org.tensorflow.hadoop.io.TFRecordFileOutputFormat",
keyClass="org.apache.hadoop.io.BytesWritable",
valueClass="org.apache.hadoop.io.NullWritable”)
> 2020年7月13日 下午2:21，Danny Chan <yu...@gmail.com> 写道：
>
> I didn’t see any class named TFRecordFileOutputFormat in Spark, for TF do you mean TensorFlow ?
>
> Best,
> Danny Chan
> 在 2020年7月10日 +0800 PM5:28，殿李 <li...@gmail.com>，写道：
>> Hi,
>>
>> Does Flink support TFRecordFileOutputFormat? I can't find the relevant information in the document.
>>
>> As far as I know, spark is supportive.
>>
>>
>> Best regards
>> Peidian Li
Re: Does Flink support TFRecordFileOutputFormat?
Posted by Danny Chan <yu...@gmail.com>.
I didn’t see any class named TFRecordFileOutputFormat in Spark, for TF do you mean TensorFlow ?
Best,
Danny Chan
在 2020年7月10日 +0800 PM5:28，殿李 <li...@gmail.com>，写道：
> Hi,
>
> Does Flink support TFRecordFileOutputFormat? I can't find the relevant information in the document.
>
> As far as I know, spark is supportive.
>
>
> Best regards
> Peidian Li