You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@kylin.apache.org by "Hélder Hugo Ferreira (Jira)" <ji...@apache.org> on 2021/01/08 16:25:00 UTC
[jira] [Created] (KYLIN-4866) Streaming from Kafka with nested
timestamp column
Hélder Hugo Ferreira created KYLIN-4866:
-------------------------------------------
Summary: Streaming from Kafka with nested timestamp column
Key: KYLIN-4866
URL: https://issues.apache.org/jira/browse/KYLIN-4866
Project: Kylin
Issue Type: Bug
Components: Real-time Streaming
Affects Versions: v3.1.0
Environment: Docker setup
Reporter: Hélder Hugo Ferreira
Attachments: image-2021-01-08-16-16-51-615.png
We have an IOT Data Platform which saves in Kafka Json messages with the following format:
{code:json}
{
"SysProperties": {
"EventId": "1610120564321723276",
"EnqueueTime": "2021-01-08T15:42:44.045Z",
"UserName": "dummyuser"
},
"AppProperties": {
"EventDefinition": "EquipmentData",
"EventTime": "2021-01-08T15:42:43.26+00:00",
"ApplicationName": "ConnectIoTSimulator",
"ApplicationContext": null
},
"Data": {
"Equipment": "DUMMY-01",
"Material": "DUMMYZA9983884",
"MeasurementData": {
"Temp": 57,
"Pressure": 48.379,
"Energy": 47,
"Humidity": 31.727
}
}
{code}
From these messages, we are trying to build a cube in which the "AppProperties.EventTime" shall be the timestamp column (using format yyyy-MM-dd'T'HH:mm:ss.SSSZZ), "Data.MeasurementData.Temp", "Data.MeasurementData.Pressure", "Data.MeasurementData.Energy" and "Data.MeasurementData.Humidity" shall be the metrics and all other fields dimensions.
We can setup everything fine in the kylin portal like in the tutorial, however on runtime we have the following exceptions logged into the kylin_streaming_receiver.log:
!image-2021-01-08-16-16-51-615.png!
After some investigation, we figured out it only works if the timestamp field within the Json messages is not nested inside inner objects, like in the following example:
{code:json}
{
"timestamp": 1610038228492,
"nestedData": {
"Equipment": "EPFTES-01",
"Material": "ZA9983884",
"Temp": 10,
"Pressure": 20.459,
"Energy": 30,
"Humidity": 80.967
}
}{code}
Therefore, we believe there is a bug handling the timestamp column whenever it is nested within Json messages.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)