[jira] [Assigned] (KYLIN-4866) Streaming from Kafka with nested timestamp column

["Xiaoxiang Yu (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/KYLIN-4866?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Xiaoxiang Yu reassigned KYLIN-4866:
-----------------------------------

    Assignee: Xiaoxiang Yu

> Streaming from Kafka with nested timestamp column
> -------------------------------------------------
>
>                 Key: KYLIN-4866
>                 URL: https://issues.apache.org/jira/browse/KYLIN-4866
>             Project: Kylin
>          Issue Type: Bug
>          Components: Real-time Streaming
>    Affects Versions: v3.1.0
>         Environment: Docker setup
>            Reporter: Hélder Hugo Ferreira
>            Assignee: Xiaoxiang Yu
>            Priority: Major
>             Fix For: v3.1.2
>
>         Attachments: image-2021-01-08-16-16-51-615.png
>
>
> We have an IOT Data Platform which saves in Kafka Json messages with the following format: 
> {code:json}
> {
>  "SysProperties": {
>  "EventId": "1610120564321723276",
>  "EnqueueTime": "2021-01-08T15:42:44.045Z",
>  "UserName": "dummyuser"
>  },
>  "AppProperties": {
>  "EventDefinition": "EquipmentData",
>  "EventTime": "2021-01-08T15:42:43.26+00:00",
>  "ApplicationName": "ConnectIoTSimulator",
>  "ApplicationContext": null
>  },
>  "Data": {
>  "Equipment": "DUMMY-01",
>  "Material": "DUMMYZA9983884",
>  "MeasurementData": {
>  "Temp": 57,
>  "Pressure": 48.379,
>  "Energy": 47,
>  "Humidity": 31.727
>  }
> }
> {code}
> From these messages, we are trying to build a cube in which the "AppProperties.EventTime" shall be the timestamp column (using format yyyy-MM-dd'T'HH:mm:ss.SSSZZ), "Data.MeasurementData.Temp", "Data.MeasurementData.Pressure", "Data.MeasurementData.Energy" and "Data.MeasurementData.Humidity" shall be the metrics and all other fields dimensions.
> We can setup everything fine in the kylin portal (like in the tutorial), however on runtime we have the following exceptions logged by the streaming receiver component:
> !image-2021-01-08-16-16-51-615.png!
> After some investigation, we figured out it only works if the timestamp field within the Json messages is not nested inside inner objects, like in the following example:
> {code:json}
> {
>  "timestamp": 1610038228492,
>  "nestedData": {
>  "Equipment": "EPFTES-01",
>  "Material": "ZA9983884",
>  "Temp": 10,
>  "Pressure": 20.459,
>  "Energy": 30,
>  "Humidity": 80.967
>  }
> }{code}
> Therefore, we believe there is a bug handling the timestamp column when it is nested within Json messages.  



--
This message was sent by Atlassian Jira
(v8.3.4#803005)