You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pinot.apache.org by GitBox <gi...@apache.org> on 2022/02/17 20:45:49 UTC
[GitHub] [pinot] fritzb opened a new issue #8220: Unable to ingest long JSON string with JSON indexing enabled
fritzb opened a new issue #8220:
URL: https://github.com/apache/pinot/issues/8220
I have a long JSON string, and it is rejected during ingestion. Can the JSON string limit be lifted higher ?
"jsonData": "{\"BOUNDARY_LINE_LANE\":\"aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaabbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbasdljfhlasjdhfklasjdhflkashdfklajshdflkjhasdfkjhasdlkfjhasldkjfhaklsdjhflkasjdhfklashdflkjashdfkljhasdlfkjhasdklfhaskldjfhlaksjdhfklasjhdfklajshdflkjashdfklashdflkjhasdklfjhaslkdfjhasldjhflaskjdhflkasjdhflkajshdfljashdflkahsdflkjhasdlkfjhaskldjfhlkasdjfhlkajsdhfklasjhdflkajshdfklasjhdflkjhasdflkhasdlkfhaslkdjfhlaksdhflkasjdhflkasjdhflkashdfklahsdfkljhasdlkfjhaskdljfhlaksjdfhlkasjdhflkasjdhflkasjdhflkjahsdfkljashdfkljashdflkjhasdlkfjhaslkdjfhklasdjfhklasjdhfkljashdfkljsahdfkljashdfkljhasdfkljhaslkdfhklasjdhflkajshdflkashdflkjashdflkjhasdfkljhasdlkfhaskldjfhlkasjdhflkasjdhflkjashdflkjashdflkjahsdflkjhasdlkfjhaslkdjfhlaskjdhfklajsdhflksajdhflkasjdhfklajhsdfkljashdfkljashdfkljashdfkljhasdfkjhasdkljfhaslkjdfhlkasjdhflkjasdhfkljashdflkjashdfkljahsdflkj
hasdklfjhalksdjfhlakjsdfhlkajsdhfkajsdhflkajsdflkjhasdflkjh\"}"
shaded.com.fasterxml.jackson.core.io.JsonEOFException: Unexpected end-of-input: was expecting closing quote for a string value
at [Source: (String)"{"BOUNDARY_LINE_LANE":"aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaabbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbasdljfhlasjdhfklasjdhflkashdfklajshdflkjhasdfkjhasdlkfjhasldkjfhaklsdjhflkasjdhfklashdflkjashdfkljhasdlfkjhasdklfhaskldjfhlaksjdhfklasjhdfklajshdflkjashdfklashdflkjhasdklfjhaslkdfjhasldjhflaskjdhflkasjdhflkajshdfljashdflkahsdflkjhasdlkfjhaskldjfhlkasdjfhlkajsdhfklasjhdflkajshd"[truncated 12 chars]; line: 1, column: 1025]
at shaded.com.fasterxml.jackson.core.base.ParserMinimalBase._reportInvalidEOF(ParserMinimalBase.java:664) ~[pinot-all-0.9.3-jar-with-dependencies.jar:0.9.3-e23f213cf0d16b1e9e086174d734a4db868542cb]
at shaded.com.fasterxml.jackson.core.json.ReaderBasedJsonParser._finishString2(ReaderBasedJsonParser.java:2051) ~[pinot-all-0.9.3-jar-with-dependencies.jar:0.9.3-e23f213cf0d16b1e9e086174d734a4db868542cb]
at shaded.com.fasterxml.jackson.core.json.ReaderBasedJsonParser._finishString(ReaderBasedJsonParser.java:2038) ~[pinot-all-0.9.3-jar-with-dependencies.jar:0.9.3-e23f213cf0d16b1e9e086174d734a4db868542cb]
at shaded.com.fasterxml.jackson.core.json.ReaderBasedJsonParser.getText(ReaderBasedJsonParser.java:293) ~[pinot-all-0.9.3-jar-with-dependencies.jar:0.9.3-e23f213cf0d16b1e9e086174d734a4db868542cb]
at shaded.com.fasterxml.jackson.databind.deser.std.BaseNodeDeserializer.deserializeObject(JsonNodeDeserializer.java:267) ~[pinot-all-0.9.3-jar-with-dependencies.jar:0.9.3-e23f213cf0d16b1e9e086174d734a4db868542cb]
at shaded.com.fasterxml.jackson.databind.deser.std.JsonNodeDeserializer.deserialize(JsonNodeDeserializer.java:68) ~[pinot-all-0.9.3-jar-with-dependencies.jar:0.9.3-e23f213cf0d16b1e9e086174d734a4db868542cb]
at shaded.com.fasterxml.jackson.databind.deser.std.JsonNodeDeserializer.deserialize(JsonNodeDeserializer.java:15) ~[pinot-all-0.9.3-jar-with-dependencies.jar:0.9.3-e23f213cf0d16b1e9e086174d734a4db868542cb]
at shaded.com.fasterxml.jackson.databind.ObjectReader._bindAsTree(ObjectReader.java:1770) ~[pinot-all-0.9.3-jar-with-dependencies.jar:0.9.3-e23f213cf0d16b1e9e086174d734a4db868542cb]
at shaded.com.fasterxml.jackson.databind.ObjectReader._bindAndCloseAsTree(ObjectReader.java:1735) ~[pinot-all-0.9.3-jar-with-dependencies.jar:0.9.3-e23f213cf0d16b1e9e086174d734a4db868542cb]
at shaded.com.fasterxml.jackson.databind.ObjectReader.readTree(ObjectReader.java:1422) ~[pinot-all-0.9.3-jar-with-dependencies.jar:0.9.3-e23f213cf0d16b1e9e086174d734a4db868542cb]
at org.apache.pinot.spi.utils.JsonUtils.stringToJsonNode(JsonUtils.java:87) ~[pinot-all-0.9.3-jar-with-dependencies.jar:0.9.3-e23f213cf0d16b1e9e086174d734a4db868542cb]
at org.apache.pinot.segment.local.realtime.impl.json.MutableJsonIndex.add(MutableJsonIndex.java:76) ~[pinot-all-0.9.3-jar-with-dependencies.jar:0.9.3-e23f213cf0d16b1e9e086174d734a4db868542cb]
at org.apache.pinot.segment.local.indexsegment.mutable.MutableSegmentImpl.addNewRow(MutableSegmentImpl.java:643) ~[pinot-all-0.9.3-jar-with-dependencies.jar:0.9.3-e23f213cf0d16b1e9e086174d734a4db868542cb]
at org.apache.pinot.segment.local.indexsegment.mutable.MutableSegmentImpl.index(MutableSegmentImpl.java:491) ~[pinot-all-0.9.3-jar-with-dependencies.jar:0.9.3-e23f213cf0d16b1e9e086174d734a4db868542cb]
at org.apache.pinot.core.data.manager.realtime.LLRealtimeSegmentDataManager.processStreamEvents(LLRealtimeSegmentDataManager.java:524) [pinot-all-0.9.3-jar-with-dependencies.jar:0.9.3-e23f213cf0d16b1e9e086174d734a4db868542cb]
at org.apache.pinot.core.data.manager.realtime.LLRealtimeSegmentDataManager.consumeLoop(LLRealtimeSegmentDataManager.java:420) [pinot-all-0.9.3-jar-with-dependencies.jar:0.9.3-e23f213cf0d16b1e9e086174d734a4db868542cb]
at org.apache.pinot.core.data.manager.realtime.LLRealtimeSegmentDataManager$PartitionConsumer.run(LLRealtimeSegmentDataManager.java:568) [pinot-all-0.9.3-jar-with-dependencies.jar:0.9.3-e23f213cf0d16b1e9e086174d734a4db868542cb]
at java.lang.Thread.run(Thread.java:829) [?:?]
Consumed 0 events from (rate:0.0/s), currentOffset=8215, numRowsConsumedSoFar=0, numRowsIndexedSoFar=0
Consumed 0 events from (rate:0.0/s), currentOffset=11734, numRowsConsumedSoFar=0, numRowsIndexedSoFar=0
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] Jackie-Jiang commented on issue #8220: Unable to ingest long JSON string with JSON indexing enabled
Posted by GitBox <gi...@apache.org>.
Jackie-Jiang commented on issue #8220:
URL: https://github.com/apache/pinot/issues/8220#issuecomment-1043643390
For `STRING` data type, by default we limit the length to 512 (configurable in schema with key `maxLength`). There is no such limit for `JSON` data type.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] richardstartin commented on issue #8220: Unable to ingest long JSON string with JSON indexing enabled
Posted by GitBox <gi...@apache.org>.
richardstartin commented on issue #8220:
URL: https://github.com/apache/pinot/issues/8220#issuecomment-1044284076
The general solution to this problem is to use the JSON data type for JSON columns. Please reopen if there are still issues.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] richardstartin commented on issue #8220: Unable to ingest long JSON string with JSON indexing enabled
Posted by GitBox <gi...@apache.org>.
richardstartin commented on issue #8220:
URL: https://github.com/apache/pinot/issues/8220#issuecomment-1043420256
Have you set your column data type to JSON? For string columns you can configure the maxLength - please see here: https://docs.pinot.apache.org/configuration-reference/schema#advanced-fields
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] fritzb commented on issue #8220: Unable to ingest long JSON string with JSON indexing enabled
Posted by GitBox <gi...@apache.org>.
fritzb commented on issue #8220:
URL: https://github.com/apache/pinot/issues/8220#issuecomment-1043512472
I have the dataType set to STRING. This is the first time I've heard about data type Json.
```
Schema:
{
"name": "json_data",
"defaultNullValue": "",
"dataType": "STRING"
}
Table Config:
"tableIndexConfig": {
"jsonIndexColumns": [
"json_data"
]
}
```
Let me change it to the following and see if this is still an issue.
```
Schema:
{
"name": "json_data",
"defaultNullValue": "",
"dataType": "JSON"
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] richardstartin closed issue #8220: Unable to ingest long JSON string with JSON indexing enabled
Posted by GitBox <gi...@apache.org>.
richardstartin closed issue #8220:
URL: https://github.com/apache/pinot/issues/8220
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org