You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@flink.apache.org by yuxia <lu...@alumni.sjtu.edu.cn> on 2022/05/06 08:36:24 UTC
Re: How to return JSON Object from UDF
Does the DatatypeHint with bridgedTo can meet your requirements?
For example:
'
public @DataTypeHint(
value = "RAW",
bridgedTo = JSONObject .class,
rawSerializer = JSONObject Serializer.class) JSONObject eval(String str) {
return JSONObject .parse(str);
}
'
You may need to provide a class like JSONObject Serializer that extends TypeSerializerSingleton.
Best regards,
Yuxia
发件人: "Surendra Lalwani" <su...@swiggy.in>
收件人: "User" <us...@flink.apache.org>
发送时间: 星期五, 2022年 5 月 06日 下午 4:40:19
主题: How to return JSON Object from UDF
Hi Team,
I am using Flink 1.13.6 and I have created a UDF and I want to return JSONObject from that UDF or basically an Object but it doesn't seems to work as there is no datatype hint compatible to Object. in earlier flink versions when DataTypeHint wasn't there, it used to work. Any help would be appreciated.
Thanks and Regards ,
Surendra Lalwani
IMPORTANT NOTICE: This e-mail, including any attachments, may contain confidential information and is intended only for the addressee(s) named above. If you are not the intended recipient(s), you should not disseminate, distribute, or copy this e-mail. Please notify the sender by reply e-mail immediately if you have received this e-mail in error and permanently delete all copies of the original message from your system. E-mail transmission cannot be guaranteed to be secure as it could be intercepted, corrupted, lost, destroyed, arrive late or incomplete, or contain viruses. Company accepts no liability for any damage or loss of confidential information caused by this email or due to any virus transmitted by this email or otherwise.
Re: How to return JSON Object from UDF
Posted by Surendra Lalwani <su...@swiggy.in>.
Hi Timo,
Please find the example:
@DataTypeHint(value = "RAW", bridgedTo = JsonObject.class)
public Object eval(String jsonString) {
//Logic to parse
}
Thanks and Regards ,
Surendra Lalwani
On Fri, May 6, 2022 at 3:13 PM Timo Walther <tw...@apache.org> wrote:
> Can you show the full example? It looks like there is still a JSONObject
> without a @DataTypeHint next to it.
>
>
>
> Am 06.05.22 um 11:18 schrieb Surendra Lalwani:
>
> Hi Timo,
>
> I tried this but still getting error:
>
> Caused by: org.apache.flink.table.api.ValidationException: Could not
> extract a data type from 'class net.minidev.json.JSONObject'. Interpreting
> it as a structured type was also not successful. at
> org.apache.flink.table.types.extraction.ExtractionUtils.extractionError(ExtractionUtils.java:361)
> at
> org.apache.flink.table.types.extraction.DataTypeExtractor.extractDataTypeOrError(DataTypeExtractor.java:291)
> at
> org.apache.flink.table.types.extraction.DataTypeExtractor.extractDataTypeOrRawWithTemplate(DataTypeExtractor.java:233)
> ... 83 common frames omittedCaused by:
> org.apache.flink.table.api.ValidationException: Field 'threshold' of class
> 'net.minidev.json.JSONObject' is neither publicly accessible nor does it
> have a corresponding getter method.
>
>
> Thanks and Regards ,
> Surendra Lalwani
>
>
> On Fri, May 6, 2022 at 2:43 PM Timo Walther <tw...@apache.org> wrote:
>
>> Hi Surendra,
>>
>> in general we would like to encourage users to use the SQL type system
>> classes instead of RAW types. Otherwise they are simply black boxes in the
>> SQL engine. A STRING or ROW type might be more appropriate.
>>
>> You can use
>>
>> @DataTypeHint(value = "RAW") // defaults to Object.class
>>
>> @DataTypeHint(value = "RAW", bridgedTo =JSONObject.class) // more
>> precise class information
>>
>> the rawSerializer is usually not required and Kyro will be used in this
>> case.
>>
>> Regards,
>> Timo
>>
>>
>> Am 06.05.22 um 10:36 schrieb yuxia:
>>
>> Does the DatatypeHint with bridgedTo can meet your requirements?
>> For example:
>> '
>> public @DataTypeHint(
>> value = "RAW",
>> bridgedTo =JSONObject.class,
>> rawSerializer =JSONObjectSerializer.class) JSONObject eval(String
>> str) {
>> return JSONObject.parse(str);
>> }
>> '
>> You may need to provide a class likeJSONObjectSerializer that
>> extends TypeSerializerSingleton.
>>
>>
>> Best regards,
>> Yuxia
>>
>> ------------------------------
>> *发件人: *"Surendra Lalwani" <su...@swiggy.in>
>> <su...@swiggy.in>
>> *收件人: *"User" <us...@flink.apache.org> <us...@flink.apache.org>
>> *发送时间: *星期五, 2022年 5 月 06日 下午 4:40:19
>> *主题: *How to return JSON Object from UDF
>>
>> Hi Team,
>>
>> I am using Flink 1.13.6 and I have created a UDF and I want to return
>> JSONObject from that UDF or basically an Object but it doesn't seems to
>> work as there is no datatype hint compatible to Object. in earlier flink
>> versions when DataTypeHint wasn't there, it used to work. Any help would be
>> appreciated.
>>
>> Thanks and Regards ,
>> Surendra Lalwani
>>
>>
>> ------------------------------
>> IMPORTANT NOTICE: This e-mail, including any attachments, may contain
>> confidential information and is intended only for the addressee(s) named
>> above. If you are not the intended recipient(s), you should not
>> disseminate, distribute, or copy this e-mail. Please notify the sender by
>> reply e-mail immediately if you have received this e-mail in error and
>> permanently delete all copies of the original message from your system.
>> E-mail transmission cannot be guaranteed to be secure as it could be
>> intercepted, corrupted, lost, destroyed, arrive late or incomplete, or
>> contain viruses. Company accepts no liability for any damage or loss of
>> confidential information caused by this email or due to any virus
>> transmitted by this email or otherwise.
>>
>>
>>
>
> ------------------------------
> IMPORTANT NOTICE: This e-mail, including any attachments, may contain
> confidential information and is intended only for the addressee(s) named
> above. If you are not the intended recipient(s), you should not
> disseminate, distribute, or copy this e-mail. Please notify the sender by
> reply e-mail immediately if you have received this e-mail in error and
> permanently delete all copies of the original message from your system.
> E-mail transmission cannot be guaranteed to be secure as it could be
> intercepted, corrupted, lost, destroyed, arrive late or incomplete, or
> contain viruses. Company accepts no liability for any damage or loss of
> confidential information caused by this email or due to any virus
> transmitted by this email or otherwise.
>
>
>
--
IMPORTANT NOTICE: This e-mail, including any attachments, may contain
confidential information and is intended only for the addressee(s) named
above. If you are not the intended recipient(s), you should not
disseminate, distribute, or copy this e-mail. Please notify the sender by
reply e-mail immediately if you have received this e-mail in error and
permanently delete all copies of the original message from your system.
E-mail transmission cannot be guaranteed to be secure as it could be
intercepted, corrupted, lost, destroyed, arrive late or incomplete, or
contain viruses. Company accepts no liability for any damage or loss of
confidential information caused by this email or due to any virus
transmitted by this email or otherwise.
Re: How to return JSON Object from UDF
Posted by Timo Walther <tw...@apache.org>.
Can you show the full example? It looks like there is still a JSONObject
without a @DataTypeHint next to it.
Am 06.05.22 um 11:18 schrieb Surendra Lalwani:
> Hi Timo,
>
> I tried this but still getting error:
>
> Caused by: org.apache.flink.table.api.ValidationException: Could not
> extract a data type from 'class net.minidev.json.JSONObject'.
> Interpreting it as a structured type was also not successful.
> at
> org.apache.flink.table.types.extraction.ExtractionUtils.extractionError(ExtractionUtils.java:361)
> at
> org.apache.flink.table.types.extraction.DataTypeExtractor.extractDataTypeOrError(DataTypeExtractor.java:291)
> at
> org.apache.flink.table.types.extraction.DataTypeExtractor.extractDataTypeOrRawWithTemplate(DataTypeExtractor.java:233)
> ... 83 common frames omittedCaused by:
> org.apache.flink.table.api.ValidationException: Field 'threshold' of
> class 'net.minidev.json.JSONObject' is neither publicly accessible nor
> does it have a corresponding getter method.
>
>
> Thanks and Regards ,
> Surendra Lalwani
>
>
> On Fri, May 6, 2022 at 2:43 PM Timo Walther <tw...@apache.org> wrote:
>
> Hi Surendra,
>
> in general we would like to encourage users to use the SQL type
> system classes instead of RAW types. Otherwise they are simply
> black boxes in the SQL engine. A STRING or ROW type might be more
> appropriate.
>
> You can use
>
> @DataTypeHint(value = "RAW") // defaults to Object.class
>
> @DataTypeHint(value = "RAW", bridgedTo =JSONObject.class) // more
> precise class information
>
> the rawSerializer is usually not required and Kyro will be used in
> this case.
>
> Regards,
> Timo
>
>
> Am 06.05.22 um 10:36 schrieb yuxia:
>> Does the DatatypeHint with bridgedTo can meet your requirements?
>> For example:
>> '
>> public @DataTypeHint(
>> value = "RAW",
>> bridgedTo =JSONObject.class,
>> rawSerializer =JSONObjectSerializer.class)
>> JSONObject eval(String str) {
>> return JSONObject.parse(str);
>> }
>> '
>> You may need to provide a class likeJSONObjectSerializer that
>> extends TypeSerializerSingleton.
>>
>>
>> Best regards,
>> Yuxia
>>
>> ------------------------------------------------------------------------
>> *发件人: *"Surendra Lalwani" <su...@swiggy.in>
>> <ma...@swiggy.in>
>> *收件人: *"User" <us...@flink.apache.org>
>> <ma...@flink.apache.org>
>> *发送时间: *星期五, 2022年 5 月 06日 下午 4:40:19
>> *主题: *How to return JSON Object from UDF
>>
>> Hi Team,
>>
>> I am using Flink 1.13.6 and I have created a UDF and I want to
>> return JSONObject from that UDF or basically an Object but it
>> doesn't seems to work as there is no datatype hint compatible to
>> Object. in earlier flink versions when DataTypeHint wasn't there,
>> it used to work. Any help would be appreciated.
>>
>> Thanks and Regards ,
>> Surendra Lalwani
>>
>>
>> ------------------------------------------------------------------------
>> IMPORTANT NOTICE: This e-mail, including any attachments, may
>> contain confidential information and is intended only for the
>> addressee(s) named above. If you are not the intended
>> recipient(s), you should not disseminate, distribute, or copy
>> this e-mail. Please notify the sender by reply e-mail immediately
>> if you have received this e-mail in error and permanently delete
>> all copies of the original message from your system. E-mail
>> transmission cannot be guaranteed to be secure as it could be
>> intercepted, corrupted, lost, destroyed, arrive late or
>> incomplete, or contain viruses. Company accepts no liability for
>> any damage or loss of confidential information caused by this
>> email or due to any virus transmitted by this email or otherwise.
>>
>
>
>
> ------------------------------------------------------------------------
> IMPORTANT NOTICE: This e-mail, including any attachments, may contain
> confidential information and is intended only for the addressee(s)
> named above. If you are not the intended recipient(s), you should not
> disseminate, distribute, or copy this e-mail. Please notify the sender
> by reply e-mail immediately if you have received this e-mail in error
> and permanently delete all copies of the original message from your
> system. E-mail transmission cannot be guaranteed to be secure as it
> could be intercepted, corrupted, lost, destroyed, arrive late or
> incomplete, or contain viruses. Company accepts no liability for any
> damage or loss of confidential information caused by this email or due
> to any virus transmitted by this email or otherwise.
Re: How to return JSON Object from UDF
Posted by Surendra Lalwani <su...@swiggy.in>.
Hi Timo,
I tried this but still getting error:
Caused by: org.apache.flink.table.api.ValidationException: Could not
extract a data type from 'class net.minidev.json.JSONObject'. Interpreting
it as a structured type was also not successful. at
org.apache.flink.table.types.extraction.ExtractionUtils.extractionError(ExtractionUtils.java:361)
at
org.apache.flink.table.types.extraction.DataTypeExtractor.extractDataTypeOrError(DataTypeExtractor.java:291)
at
org.apache.flink.table.types.extraction.DataTypeExtractor.extractDataTypeOrRawWithTemplate(DataTypeExtractor.java:233)
... 83 common frames omittedCaused by:
org.apache.flink.table.api.ValidationException: Field 'threshold' of class
'net.minidev.json.JSONObject' is neither publicly accessible nor does it
have a corresponding getter method.
Thanks and Regards ,
Surendra Lalwani
On Fri, May 6, 2022 at 2:43 PM Timo Walther <tw...@apache.org> wrote:
> Hi Surendra,
>
> in general we would like to encourage users to use the SQL type system
> classes instead of RAW types. Otherwise they are simply black boxes in the
> SQL engine. A STRING or ROW type might be more appropriate.
>
> You can use
>
> @DataTypeHint(value = "RAW") // defaults to Object.class
>
> @DataTypeHint(value = "RAW", bridgedTo =JSONObject.class) // more
> precise class information
>
> the rawSerializer is usually not required and Kyro will be used in this
> case.
>
> Regards,
> Timo
>
>
> Am 06.05.22 um 10:36 schrieb yuxia:
>
> Does the DatatypeHint with bridgedTo can meet your requirements?
> For example:
> '
> public @DataTypeHint(
> value = "RAW",
> bridgedTo =JSONObject.class,
> rawSerializer =JSONObjectSerializer.class) JSONObject eval(String
> str) {
> return JSONObject.parse(str);
> }
> '
> You may need to provide a class likeJSONObjectSerializer that
> extends TypeSerializerSingleton.
>
>
> Best regards,
> Yuxia
>
> ------------------------------
> *发件人: *"Surendra Lalwani" <su...@swiggy.in>
> <su...@swiggy.in>
> *收件人: *"User" <us...@flink.apache.org> <us...@flink.apache.org>
> *发送时间: *星期五, 2022年 5 月 06日 下午 4:40:19
> *主题: *How to return JSON Object from UDF
>
> Hi Team,
>
> I am using Flink 1.13.6 and I have created a UDF and I want to return
> JSONObject from that UDF or basically an Object but it doesn't seems to
> work as there is no datatype hint compatible to Object. in earlier flink
> versions when DataTypeHint wasn't there, it used to work. Any help would be
> appreciated.
>
> Thanks and Regards ,
> Surendra Lalwani
>
>
> ------------------------------
> IMPORTANT NOTICE: This e-mail, including any attachments, may contain
> confidential information and is intended only for the addressee(s) named
> above. If you are not the intended recipient(s), you should not
> disseminate, distribute, or copy this e-mail. Please notify the sender by
> reply e-mail immediately if you have received this e-mail in error and
> permanently delete all copies of the original message from your system.
> E-mail transmission cannot be guaranteed to be secure as it could be
> intercepted, corrupted, lost, destroyed, arrive late or incomplete, or
> contain viruses. Company accepts no liability for any damage or loss of
> confidential information caused by this email or due to any virus
> transmitted by this email or otherwise.
>
>
>
--
IMPORTANT NOTICE: This e-mail, including any attachments, may contain
confidential information and is intended only for the addressee(s) named
above. If you are not the intended recipient(s), you should not
disseminate, distribute, or copy this e-mail. Please notify the sender by
reply e-mail immediately if you have received this e-mail in error and
permanently delete all copies of the original message from your system.
E-mail transmission cannot be guaranteed to be secure as it could be
intercepted, corrupted, lost, destroyed, arrive late or incomplete, or
contain viruses. Company accepts no liability for any damage or loss of
confidential information caused by this email or due to any virus
transmitted by this email or otherwise.
Re: How to return JSON Object from UDF
Posted by Timo Walther <tw...@apache.org>.
Hi Surendra,
in general we would like to encourage users to use the SQL type system
classes instead of RAW types. Otherwise they are simply black boxes in
the SQL engine. A STRING or ROW type might be more appropriate.
You can use
@DataTypeHint(value = "RAW") // defaults to Object.class
@DataTypeHint(value = "RAW", bridgedTo =JSONObject.class) // more
precise class information
the rawSerializer is usually not required and Kyro will be used in this
case.
Regards,
Timo
Am 06.05.22 um 10:36 schrieb yuxia:
> Does the DatatypeHint with bridgedTo can meet your requirements?
> For example:
> '
> public @DataTypeHint(
> value = "RAW",
> bridgedTo =JSONObject.class,
> rawSerializer =JSONObjectSerializer.class) JSONObject
> eval(String str) {
> return JSONObject.parse(str);
> }
> '
> You may need to provide a class likeJSONObjectSerializer that
> extends TypeSerializerSingleton.
>
>
> Best regards,
> Yuxia
>
> ------------------------------------------------------------------------
> *发件人: *"Surendra Lalwani" <su...@swiggy.in>
> *收件人: *"User" <us...@flink.apache.org>
> *发送时间: *星期五, 2022年 5 月 06日 下午 4:40:19
> *主题: *How to return JSON Object from UDF
>
> Hi Team,
>
> I am using Flink 1.13.6 and I have created a UDF and I want to return
> JSONObject from that UDF or basically an Object but it doesn't seems
> to work as there is no datatype hint compatible to Object. in earlier
> flink versions when DataTypeHint wasn't there, it used to work. Any
> help would be appreciated.
>
> Thanks and Regards ,
> Surendra Lalwani
>
>
> ------------------------------------------------------------------------
> IMPORTANT NOTICE: This e-mail, including any attachments, may contain
> confidential information and is intended only for the addressee(s)
> named above. If you are not the intended recipient(s), you should not
> disseminate, distribute, or copy this e-mail. Please notify the sender
> by reply e-mail immediately if you have received this e-mail in error
> and permanently delete all copies of the original message from your
> system. E-mail transmission cannot be guaranteed to be secure as it
> could be intercepted, corrupted, lost, destroyed, arrive late or
> incomplete, or contain viruses. Company accepts no liability for any
> damage or loss of confidential information caused by this email or due
> to any virus transmitted by this email or otherwise.
>