You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@kylin.apache.org by "Hélder Hugo Ferreira (Jira)" <ji...@apache.org> on 2021/01/11 09:52:00 UTC
[jira] [Updated] (KYLIN-4866) Streaming from Kafka with nested
timestamp column
[ https://issues.apache.org/jira/browse/KYLIN-4866?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Hélder Hugo Ferreira updated KYLIN-4866:
----------------------------------------
Description:
We have an IOT Data Platform which saves in Kafka Json messages with the following format:
{code:json}
{
"SysProperties": {
"EventId": "1610120564321723276",
"EnqueueTime": "2021-01-08T15:42:44.045Z",
"UserName": "dummyuser"
},
"AppProperties": {
"EventDefinition": "EquipmentData",
"EventTime": "2021-01-08T15:42:43.26+00:00",
"ApplicationName": "ConnectIoTSimulator",
"ApplicationContext": null
},
"Data": {
"Equipment": "DUMMY-01",
"Material": "DUMMYZA9983884",
"MeasurementData": {
"Temp": 57,
"Pressure": 48.379,
"Energy": 47,
"Humidity": 31.727
}
}
{code}
From these messages, we are trying to build a cube in which the "AppProperties.EventTime" shall be the timestamp column (using format yyyy-MM-dd'T'HH:mm:ss.SSSZZ), "Data.MeasurementData.Temp", "Data.MeasurementData.Pressure", "Data.MeasurementData.Energy" and "Data.MeasurementData.Humidity" shall be the metrics and all other fields dimensions.
We can setup everything fine in the kylin portal (like in the tutorial), however on runtime we have the following exceptions logged by the streaming receiver component:
!image-2021-01-08-16-16-51-615.png!
After some investigation, we figured out it only works if the timestamp field within the Json messages is not nested inside inner objects, like in the following example:
{code:json}
{
"timestamp": 1610038228492,
"nestedData": {
"Equipment": "EPFTES-01",
"Material": "ZA9983884",
"Temp": 10,
"Pressure": 20.459,
"Energy": 30,
"Humidity": 80.967
}
}{code}
Therefore, we believe there is a bug handling the timestamp column when it is nested within Json messages.
was:
We have an IOT Data Platform which saves in Kafka Json messages with the following format:
{code:json}
{
"SysProperties": {
"EventId": "1610120564321723276",
"EnqueueTime": "2021-01-08T15:42:44.045Z",
"UserName": "dummyuser"
},
"AppProperties": {
"EventDefinition": "EquipmentData",
"EventTime": "2021-01-08T15:42:43.26+00:00",
"ApplicationName": "ConnectIoTSimulator",
"ApplicationContext": null
},
"Data": {
"Equipment": "DUMMY-01",
"Material": "DUMMYZA9983884",
"MeasurementData": {
"Temp": 57,
"Pressure": 48.379,
"Energy": 47,
"Humidity": 31.727
}
}
{code}
From these messages, we are trying to build a cube in which the "AppProperties.EventTime" shall be the timestamp column (using format yyyy-MM-dd'T'HH:mm:ss.SSSZZ), "Data.MeasurementData.Temp", "Data.MeasurementData.Pressure", "Data.MeasurementData.Energy" and "Data.MeasurementData.Humidity" shall be the metrics and all other fields dimensions.
We can setup everything fine in the kylin portal like in the tutorial, however on runtime we have the following exceptions logged into the kylin_streaming_receiver.log:
!image-2021-01-08-16-16-51-615.png!
After some investigation, we figured out it only works if the timestamp field within the Json messages is not nested inside inner objects, like in the following example:
{code:json}
{
"timestamp": 1610038228492,
"nestedData": {
"Equipment": "EPFTES-01",
"Material": "ZA9983884",
"Temp": 10,
"Pressure": 20.459,
"Energy": 30,
"Humidity": 80.967
}
}{code}
Therefore, we believe there is a bug handling the timestamp column whenever it is nested within Json messages.
> Streaming from Kafka with nested timestamp column
> -------------------------------------------------
>
> Key: KYLIN-4866
> URL: https://issues.apache.org/jira/browse/KYLIN-4866
> Project: Kylin
> Issue Type: Bug
> Components: Real-time Streaming
> Affects Versions: v3.1.0
> Environment: Docker setup
> Reporter: Hélder Hugo Ferreira
> Priority: Major
> Attachments: image-2021-01-08-16-16-51-615.png
>
>
> We have an IOT Data Platform which saves in Kafka Json messages with the following format:
> {code:json}
> {
> "SysProperties": {
> "EventId": "1610120564321723276",
> "EnqueueTime": "2021-01-08T15:42:44.045Z",
> "UserName": "dummyuser"
> },
> "AppProperties": {
> "EventDefinition": "EquipmentData",
> "EventTime": "2021-01-08T15:42:43.26+00:00",
> "ApplicationName": "ConnectIoTSimulator",
> "ApplicationContext": null
> },
> "Data": {
> "Equipment": "DUMMY-01",
> "Material": "DUMMYZA9983884",
> "MeasurementData": {
> "Temp": 57,
> "Pressure": 48.379,
> "Energy": 47,
> "Humidity": 31.727
> }
> }
> {code}
> From these messages, we are trying to build a cube in which the "AppProperties.EventTime" shall be the timestamp column (using format yyyy-MM-dd'T'HH:mm:ss.SSSZZ), "Data.MeasurementData.Temp", "Data.MeasurementData.Pressure", "Data.MeasurementData.Energy" and "Data.MeasurementData.Humidity" shall be the metrics and all other fields dimensions.
> We can setup everything fine in the kylin portal (like in the tutorial), however on runtime we have the following exceptions logged by the streaming receiver component:
> !image-2021-01-08-16-16-51-615.png!
> After some investigation, we figured out it only works if the timestamp field within the Json messages is not nested inside inner objects, like in the following example:
> {code:json}
> {
> "timestamp": 1610038228492,
> "nestedData": {
> "Equipment": "EPFTES-01",
> "Material": "ZA9983884",
> "Temp": 10,
> "Pressure": 20.459,
> "Energy": 30,
> "Humidity": 80.967
> }
> }{code}
> Therefore, we believe there is a bug handling the timestamp column when it is nested within Json messages.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)