You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@nifi.apache.org by KhajaAsmath Mohammed <md...@gmail.com> on 2020/10/13 21:20:35 UTC
Putdatabase- non closeable input stream- json tree reader
Hi,
I am seeing lot of issues with our data when using putdatabase with
jsontree reader. JSON is valid when I checked actual data but NIFI is
complaining about non closeable input stream. May I know how to resolve
this?
Here is sample json and error message
error = 'Unrecognized character escape '>' (code 62) at [Source:
org.apache.nifi.stream.io.NonCloseableInputStream@44fe2a37; line: 1,
column: 7162]'
JSON DATA:
{"sensor":"http://canvas.xxxx.edu/
","sendTime":"2020-07-11T00:33:43.719Z","dataVersion":"
http://purl.imsglobal.org/ctx/caliper/v1p1","context":"
http://purl.imsglobal.org/ctx/caliper/v1p1","id":"urn:uuid:71965033-0e7d-4a76-b9f8-a19c2723b574","type":"AssignableEvent","actor_id":"urn:instructure:canvas:user:131710000000053817","actor_type":"Person","actor_extensions_user_login":"ckho","actor_extensions_user_sis_id":"A16061867","actor_extensions_root_account_id":"131710000000000001","actor_extensions_root_account_lti_guid":"YYzs3up2ikmNIyNSel3biohOBqVtaOJCALBEKdzq:canvas-lms","actor_extensions_root_account_uuid":"YYzs3up2ikmNIyNSel3biohOBqVtaOJCALBEKdzq","actor_extensions_entity_id":"131710000000053817","action":"Submitted","object_id":"urn:instructure:canvas:submission:131710000009353003","object_type":"Attempt","object_dateCreated":"2020-07-11T00:33:42.000Z","object_extensions_submission_type":"online_text_entry","object_extensions_entity_id":"131710000009353003","object_assignee_id":"urn:instructure:canvas:user:131710000000053817","object_assignee_type":"Person","object_assignable_id":"urn:instructure:canvas:assignment:131710000000149907","object_assignable_type":"AssignableDigitalResource","object_count":1,"object_body":"<p><span
style=\"font-weight: 400;\">Tuesday’s discussion centered on the topic of
phonemes, and how they connect to various writing systems that are found
across numerous languages. These relationships between phonemes and writing
systems were bounded by the various types and classifications of writing
systems, such as abjads, abugidas, alphabets, syllabaries, and logographic
systems. In addition, the writing systems were also placed on a spectrum of
mapping between letters and sounds, with one end of the scale being
opaque/inconsistent and the other being
transparent/consistent. </span></p>\r\n<p><span style=\"font-weight:
400;\"> </span><span style=\"font-weight: 400;\">The first round of
discussions created several examples of different types of writing systems
along with their characteristics. For example, romantic languages such as
Italian and Spanish were found to be generally transparent, whereas a
language like Japanese was relatively more opaque.</span></p>\r\n<p><span
style=\"font-weight: 400;\">Students also discussed how the teaching of
writing systems influenced language learning, particularly within opaque
examples of language such as English. For example, the English plural
marker, s, is phonologically found to be expressed as /z/. However, the s
continues to be the prevailing marker of plurality within English. So, when
learning a language and being exposed to its writing system early on, such
as a second or third language, these inconsistencies can make it more
difficult to obtain a proper understanding of the word within a lexicon.
Going the other way around, the understanding of phonemes in one language
could affect the learning of another language’s writing system, which could
represent the same phoneme in different ways. Tania gives the example of
English to Korean, where Hangul represents what would be the same phonemes
in English separately. </span></p>\r\n<p><span style=\"font-weight:
400;\">For me, the relationship between phonemes of a particular language
and writing systems represents the absolute complexity found in studying
human language. Through this brief discussion, students were able to
generate ideas spanning across the globe that all inherently dealt with the
same concepts of language. The discussion also generated a specific
difficulty in learning the English language, which is something that I have
experienced growing up as the child of immigrants, and seeing family
members continue to improve their English. </span></p>\r\n<p><span
style=\"font-weight: 400;\">I also think that the different types of
writing systems create different emphasis on individual words throughout
different cultures. For example, the logographic systems of language
generate meaning through a single symbol, which has carried over to be a
popular concept within America through logos and
tattoos. </span></p>\r\n<p><span style=\"font-weight: 400;\">Finally, one
particular oddity that I noticed was how the Korean alphabet contrasts with
the systems of neighboring countries such as Japan and China. For other
Asian countries such as Vietnam, I know that their alphabet comes from the
French, so I am now more curious to research how the Korean alphabet came
to be.
</span></p>","EVENT_ID":"f30c6a2c-7df9-49a4-b6c2-5c6b9ead6683","SOURCE_SYSTEM_MODULE":"CANVAS","SOURCE_SYSTEM":"CANVAS","INTEGRATION_TIMESTAMP":"","NIFI_TIMESTAMP":"","NIFI_EVENT_ID":"51bb8d09-a493-4983-9c4d-ab5386c210f2","UNIQUE_ID":"d81fa382-1271-426c-8cfe-3949ea2f0896","EVENT_TYPE":"AssignableEvent","EVENT_TIMESTAMP":"2020-07-11
00:33:42.677","eventTime":"2020-07-11 00:33:42.677","referrer":"
https://canvas.xxxx.edu/courses/16047/assignments/149907?module_item_id=401323
","edApp_id":"http://canvas.xxxx.edu/
","edApp_type":"SoftwareApplication","group_id":"urn:instructure:canvas:course:131710000000016047","group_type":"CourseOffering","group_extensions_context_type":"Course","group_extensions_entity_id":"131710000000016047","membership_id":"urn:instructure:canvas:course:131710000000016047:Learner:131710000000053817","membership_type":"Membership","membership_member_id":"urn:instructure:canvas:user:131710000000053817","membership_member_type":"Person","membership_organization_id":"urn:instructure:canvas:course:131710000000016047","membership_organization_type":"CourseOffering","membership_roles[0]":"Learner","session_id":"urn:instructure:canvas:session:e314e8b7eeadec3c6ea78e5849e74691","session_type":"Session","extensions_hostname":"
canvas.xxxx.edu","extensions_request_id":"df347080-a01e-4837-b764-df8c6df7b736","extensions_user_agent":"Mozilla/5.0
(Windows NT 10.0; Win64; x64; rv:78.0) Gecko/20100101
Firefox/78.0","extensions_client_ip":"67.164.12.32","extensions_request_url":"
https://canvas.xxxx.edu/courses/16047/assignments/149907/submissions
","extensions_version":"1.0.0","KAFKA_PARTITION":"1","EVENT_BAG":"{sensor:
http://canvas.xxxx.edu/,sendTime:2020-07-11T00:33:43.719Z,dataVersion:http://purl.imsglobal.org/ctx/caliper/v1p1,data:[{@context:http://purl.imsglobal.org/ctx/caliper/v1p1,id:urn:uuid:71965033-0e7d-4a76-b9f8-a19c2723b574,type:AssignableEvent,actor:{id:urn:instructure:canvas:user:131710000000053817,type:Person,extensions:{com.instructure.canvas:{user_login:ckho,user_sis_id:A16061867,root_account_id:131710000000000001,root_account_lti_guid:YYzs3up2ikmNIyNSel3biohOBqVtaOJCALBEKdzq:canvas-lms,root_account_uuid:YYzs3up2ikmNIyNSel3biohOBqVtaOJCALBEKdzq,entity_id:131710000000053817}}},action:Submitted,object:{id:urn:instructure:canvas:submission:131710000009353003,type:Attempt,dateCreated:2020-07-11T00:33:42.000Z,extensions:{com.instructure.canvas:{submission_type:online_text_entry,entity_id:131710000009353003}},assignee:{id:urn:instructure:canvas:user:131710000000053817,type:Person},assignable:{id:urn:instructure:canvas:assignment:131710000000149907,type:AssignableDigitalResource},count:1,body:<p><span
style=\font-weight: 400;\>Tuesday’s discussion centered on the topic of
phonemes, and how they connect to various writing systems that are found
across numerous languages. These relationships between phonemes and writing
systems were bounded by the various types and classifications of writing
systems, such as abjads, abugidas, alphabets, syllabaries, and logographic
systems. In addition, the writing systems were also placed on a spectrum of
mapping between letters and sounds, with one end of the scale being
opaque/inconsistent and the other being
transparent/consistent. </span></p>\r\n<p><span style=\font-weight:
400;\> </span><span style=\font-weight: 400;\>The first round of
discussions created several examples of different types of writing systems
along with their characteristics. For example, romantic languages such as
Italian and Spanish were found to be generally transparent, whereas a
language like Japanese was relatively more opaque.</span></p>\r\n<p><span
style=\font-weight: 400;\>Students also discussed how the teaching of
writing systems influenced language learning, particularly within opaque
examples of language such as English. For example, the English plural
marker, s, is phonologically found to be expressed as /z/. However, the s
continues to be the prevailing marker of plurality within English. So, when
learning a language and being exposed to its writing system early on, such
as a second or third language, these inconsistencies can make it more
difficult to obtain a proper understanding of the word within a lexicon.
Going the other way around, the understanding of phonemes in one language
could affect the learning of another language’s writing system, which could
represent the same phoneme in different ways. Tania gives the example of
English to Korean, where Hangul represents what would be the same phonemes
in English separately. </span></p>\r\n<p><span style=\font-weight:
400;\>For me, the relationship between phonemes of a particular language
and writing systems represents the absolute complexity found in studying
human language. Through this brief discussion, students were able to
generate ideas spanning across the globe that all inherently dealt with the
same concepts of language. The discussion also generated a specific
difficulty in learning the English language, which is something that I have
experienced growing up as the child of immigrants, and seeing family
members continue to improve their English. </span></p>\r\n<p><span
style=\font-weight: 400;\>I also think that the different types of writing
systems create different emphasis on individual words throughout different
cultures. For example, the logographic systems of language generate meaning
through a single symbol, which has carried over to be a popular concept
within America through logos and tattoos. </span></p>\r\n<p><span
style=\font-weight: 400;\>Finally, one particular oddity that I noticed was
how the Korean alphabet contrasts with the systems of neighboring countries
such as Japan and China. For other Asian countries such as Vietnam, I know
that their alphabet comes from the French, so I am now more curious to
research how the Korean alphabet came to be.
</span></p>},eventTime:2020-07-11T00:33:42.677Z,referrer:
https://canvas.xxxx.edu/courses/16047/assignments/149907?module_item_id=401323,edApp:{id:http://canvas.xxxx.edu/,type:SoftwareApplication},group:{id:urn:instructure:canvas:course:131710000000016047,type:CourseOffering,extensions:{com.instructure.canvas:{context_type:Course,entity_id:131710000000016047}}},membership:{id:urn:instructure:canvas:course:131710000000016047:Learner:131710000000053817,type:Membership,member:{id:urn:instructure:canvas:user:131710000000053817,type:Person},organization:{id:urn:instructure:canvas:course:131710000000016047,type:CourseOffering},roles:[Learner]},session:{id:urn:instructure:canvas:session:e314e8b7eeadec3c6ea78e5849e74691,type:Session},extensions:{com.instructure.canvas:{hostname:canvas.xxxx.edu,request_id:df347080-a01e-4837-b764-df8c6df7b736,user_agent:Mozilla/5.0
(Windows NT 10.0; Win64; x64; rv:78.0) Gecko/20100101
Firefox/78.0,client_ip:67.164.12.32,request_url:
https://canvas.xxxx.edu/courses/16047/assignments/149907/submissions,version:1.0.0}}}]}
}","KAFKA_TOPIC":"LMS_CANVAS_OPENEDX_DATA_PROD","KAFKA_OFFSET":"40960518"}
Thanks,
Asmath
Re: Putdatabase- non closeable input stream- json tree reader
Posted by Pierre Villard <pi...@gmail.com>.
Hi,
Could you provide a flow template reproducing the issue?
Thanks,
Pierre
Le mar. 13 oct. 2020 à 23:20, KhajaAsmath Mohammed <md...@gmail.com>
a écrit :
> Hi,
>
> I am seeing lot of issues with our data when using putdatabase with
> jsontree reader. JSON is valid when I checked actual data but NIFI is
> complaining about non closeable input stream. May I know how to resolve
> this?
>
> Here is sample json and error message
>
> error = 'Unrecognized character escape '>' (code 62) at [Source:
> org.apache.nifi.stream.io.NonCloseableInputStream@44fe2a37; line: 1,
> column: 7162]'
>
>
>
> JSON DATA:
>
> {"sensor":"http://canvas.xxxx.edu/
> ","sendTime":"2020-07-11T00:33:43.719Z","dataVersion":"
> http://purl.imsglobal.org/ctx/caliper/v1p1","context":"
> http://purl.imsglobal.org/ctx/caliper/v1p1","id":"urn:uuid:71965033-0e7d-4a76-b9f8-a19c2723b574","type":"AssignableEvent","actor_id":"urn:instructure:canvas:user:131710000000053817","actor_type":"Person","actor_extensions_user_login":"ckho","actor_extensions_user_sis_id":"A16061867","actor_extensions_root_account_id":"131710000000000001","actor_extensions_root_account_lti_guid":"YYzs3up2ikmNIyNSel3biohOBqVtaOJCALBEKdzq:canvas-lms","actor_extensions_root_account_uuid":"YYzs3up2ikmNIyNSel3biohOBqVtaOJCALBEKdzq","actor_extensions_entity_id":"131710000000053817","action":"Submitted","object_id":"urn:instructure:canvas:submission:131710000009353003","object_type":"Attempt","object_dateCreated":"2020-07-11T00:33:42.000Z","object_extensions_submission_type":"online_text_entry","object_extensions_entity_id":"131710000009353003","object_assignee_id":"urn:instructure:canvas:user:131710000000053817","object_assignee_type":"Person","object_assignable_id":"urn:instructure:canvas:assignment:131710000000149907","object_assignable_type":"AssignableDigitalResource","object_count":1,"object_body":"<p><span
> style=\"font-weight: 400;\">Tuesday’s discussion centered on the topic of
> phonemes, and how they connect to various writing systems that are found
> across numerous languages. These relationships between phonemes and writing
> systems were bounded by the various types and classifications of writing
> systems, such as abjads, abugidas, alphabets, syllabaries, and logographic
> systems. In addition, the writing systems were also placed on a spectrum of
> mapping between letters and sounds, with one end of the scale being
> opaque/inconsistent and the other being
> transparent/consistent. </span></p>\r\n<p><span style=\"font-weight:
> 400;\"> </span><span style=\"font-weight: 400;\">The first round of
> discussions created several examples of different types of writing systems
> along with their characteristics. For example, romantic languages such as
> Italian and Spanish were found to be generally transparent, whereas a
> language like Japanese was relatively more opaque.</span></p>\r\n<p><span
> style=\"font-weight: 400;\">Students also discussed how the teaching of
> writing systems influenced language learning, particularly within opaque
> examples of language such as English. For example, the English plural
> marker, s, is phonologically found to be expressed as /z/. However, the s
> continues to be the prevailing marker of plurality within English. So, when
> learning a language and being exposed to its writing system early on, such
> as a second or third language, these inconsistencies can make it more
> difficult to obtain a proper understanding of the word within a lexicon.
> Going the other way around, the understanding of phonemes in one language
> could affect the learning of another language’s writing system, which could
> represent the same phoneme in different ways. Tania gives the example of
> English to Korean, where Hangul represents what would be the same phonemes
> in English separately. </span></p>\r\n<p><span style=\"font-weight:
> 400;\">For me, the relationship between phonemes of a particular language
> and writing systems represents the absolute complexity found in studying
> human language. Through this brief discussion, students were able to
> generate ideas spanning across the globe that all inherently dealt with the
> same concepts of language. The discussion also generated a specific
> difficulty in learning the English language, which is something that I have
> experienced growing up as the child of immigrants, and seeing family
> members continue to improve their English. </span></p>\r\n<p><span
> style=\"font-weight: 400;\">I also think that the different types of
> writing systems create different emphasis on individual words throughout
> different cultures. For example, the logographic systems of language
> generate meaning through a single symbol, which has carried over to be a
> popular concept within America through logos and
> tattoos. </span></p>\r\n<p><span style=\"font-weight: 400;\">Finally, one
> particular oddity that I noticed was how the Korean alphabet contrasts with
> the systems of neighboring countries such as Japan and China. For other
> Asian countries such as Vietnam, I know that their alphabet comes from the
> French, so I am now more curious to research how the Korean alphabet came
> to be.
> </span></p>","EVENT_ID":"f30c6a2c-7df9-49a4-b6c2-5c6b9ead6683","SOURCE_SYSTEM_MODULE":"CANVAS","SOURCE_SYSTEM":"CANVAS","INTEGRATION_TIMESTAMP":"","NIFI_TIMESTAMP":"","NIFI_EVENT_ID":"51bb8d09-a493-4983-9c4d-ab5386c210f2","UNIQUE_ID":"d81fa382-1271-426c-8cfe-3949ea2f0896","EVENT_TYPE":"AssignableEvent","EVENT_TIMESTAMP":"2020-07-11
> 00:33:42.677","eventTime":"2020-07-11 00:33:42.677","referrer":"
> https://canvas.xxxx.edu/courses/16047/assignments/149907?module_item_id=401323
> ","edApp_id":"http://canvas.xxxx.edu/
> ","edApp_type":"SoftwareApplication","group_id":"urn:instructure:canvas:course:131710000000016047","group_type":"CourseOffering","group_extensions_context_type":"Course","group_extensions_entity_id":"131710000000016047","membership_id":"urn:instructure:canvas:course:131710000000016047:Learner:131710000000053817","membership_type":"Membership","membership_member_id":"urn:instructure:canvas:user:131710000000053817","membership_member_type":"Person","membership_organization_id":"urn:instructure:canvas:course:131710000000016047","membership_organization_type":"CourseOffering","membership_roles[0]":"Learner","session_id":"urn:instructure:canvas:session:e314e8b7eeadec3c6ea78e5849e74691","session_type":"Session","extensions_hostname":"
> canvas.xxxx.edu","extensions_request_id":"df347080-a01e-4837-b764-df8c6df7b736","extensions_user_agent":"Mozilla/5.0
> (Windows NT 10.0; Win64; x64; rv:78.0) Gecko/20100101
> Firefox/78.0","extensions_client_ip":"67.164.12.32","extensions_request_url":"
> https://canvas.xxxx.edu/courses/16047/assignments/149907/submissions
> ","extensions_version":"1.0.0","KAFKA_PARTITION":"1","EVENT_BAG":"{sensor:
> http://canvas.xxxx.edu/,sendTime:2020-07-11T00:33:43.719Z,dataVersion:http://purl.imsglobal.org/ctx/caliper/v1p1,data:[{@context:http://purl.imsglobal.org/ctx/caliper/v1p1,id:urn:uuid:71965033-0e7d-4a76-b9f8-a19c2723b574,type:AssignableEvent,actor:{id:urn:instructure:canvas:user:131710000000053817,type:Person,extensions:{com.instructure.canvas:{user_login:ckho,user_sis_id:A16061867,root_account_id:131710000000000001,root_account_lti_guid:YYzs3up2ikmNIyNSel3biohOBqVtaOJCALBEKdzq:canvas-lms,root_account_uuid:YYzs3up2ikmNIyNSel3biohOBqVtaOJCALBEKdzq,entity_id:131710000000053817}}},action:Submitted,object:{id:urn:instructure:canvas:submission:131710000009353003,type:Attempt,dateCreated:2020-07-11T00:33:42.000Z,extensions:{com.instructure.canvas:{submission_type:online_text_entry,entity_id:131710000009353003}},assignee:{id:urn:instructure:canvas:user:131710000000053817,type:Person},assignable:{id:urn:instructure:canvas:assignment:131710000000149907,type:AssignableDigitalResource},count:1,body:<p><span
> style=\font-weight: 400;\>Tuesday’s discussion centered on the topic of
> phonemes, and how they connect to various writing systems that are found
> across numerous languages. These relationships between phonemes and writing
> systems were bounded by the various types and classifications of writing
> systems, such as abjads, abugidas, alphabets, syllabaries, and logographic
> systems. In addition, the writing systems were also placed on a spectrum of
> mapping between letters and sounds, with one end of the scale being
> opaque/inconsistent and the other being
> transparent/consistent. </span></p>\r\n<p><span style=\font-weight:
> 400;\> </span><span style=\font-weight: 400;\>The first round of
> discussions created several examples of different types of writing systems
> along with their characteristics. For example, romantic languages such as
> Italian and Spanish were found to be generally transparent, whereas a
> language like Japanese was relatively more opaque.</span></p>\r\n<p><span
> style=\font-weight: 400;\>Students also discussed how the teaching of
> writing systems influenced language learning, particularly within opaque
> examples of language such as English. For example, the English plural
> marker, s, is phonologically found to be expressed as /z/. However, the s
> continues to be the prevailing marker of plurality within English. So, when
> learning a language and being exposed to its writing system early on, such
> as a second or third language, these inconsistencies can make it more
> difficult to obtain a proper understanding of the word within a lexicon.
> Going the other way around, the understanding of phonemes in one language
> could affect the learning of another language’s writing system, which could
> represent the same phoneme in different ways. Tania gives the example of
> English to Korean, where Hangul represents what would be the same phonemes
> in English separately. </span></p>\r\n<p><span style=\font-weight:
> 400;\>For me, the relationship between phonemes of a particular language
> and writing systems represents the absolute complexity found in studying
> human language. Through this brief discussion, students were able to
> generate ideas spanning across the globe that all inherently dealt with the
> same concepts of language. The discussion also generated a specific
> difficulty in learning the English language, which is something that I have
> experienced growing up as the child of immigrants, and seeing family
> members continue to improve their English. </span></p>\r\n<p><span
> style=\font-weight: 400;\>I also think that the different types of writing
> systems create different emphasis on individual words throughout different
> cultures. For example, the logographic systems of language generate meaning
> through a single symbol, which has carried over to be a popular concept
> within America through logos and tattoos. </span></p>\r\n<p><span
> style=\font-weight: 400;\>Finally, one particular oddity that I noticed was
> how the Korean alphabet contrasts with the systems of neighboring countries
> such as Japan and China. For other Asian countries such as Vietnam, I know
> that their alphabet comes from the French, so I am now more curious to
> research how the Korean alphabet came to be.
> </span></p>},eventTime:2020-07-11T00:33:42.677Z,referrer:
> https://canvas.xxxx.edu/courses/16047/assignments/149907?module_item_id=401323,edApp:{id:http://canvas.xxxx.edu/,type:SoftwareApplication},group:{id:urn:instructure:canvas:course:131710000000016047,type:CourseOffering,extensions:{com.instructure.canvas:{context_type:Course,entity_id:131710000000016047}}},membership:{id:urn:instructure:canvas:course:131710000000016047:Learner:131710000000053817,type:Membership,member:{id:urn:instructure:canvas:user:131710000000053817,type:Person},organization:{id:urn:instructure:canvas:course:131710000000016047,type:CourseOffering},roles:[Learner]},session:{id:urn:instructure:canvas:session:e314e8b7eeadec3c6ea78e5849e74691,type:Session},extensions:{com.instructure.canvas:{hostname:canvas.xxxx.edu,request_id:df347080-a01e-4837-b764-df8c6df7b736,user_agent:Mozilla/5.0
> (Windows NT 10.0; Win64; x64; rv:78.0) Gecko/20100101
> Firefox/78.0,client_ip:67.164.12.32,request_url:
> https://canvas.xxxx.edu/courses/16047/assignments/149907/submissions,version:1.0.0}}}]}
> }","KAFKA_TOPIC":"LMS_CANVAS_OPENEDX_DATA_PROD","KAFKA_OFFSET":"40960518"}
>
>
>
> Thanks,
>
> Asmath
>