[GitHub] [incubator-kyuubi] ulysses-you commented on pull request #936: Add date info for json event path

[GitBox <gi...@apache.org>]

ulysses-you commented on pull request #936:
URL: https://github.com/apache/incubator-kyuubi/pull/936#issuecomment-899929942


   The origin requirements is to collect some days event. Let's say we need to collect last 7 days statement event, after this PR, we can index those files easily using date partition path.
   
   > Why need we duplicate the date info we already have in all kinds of events？
   
   yeah, We can use `sc.startTime` instead of `CurrentDate` here.
   
   > How does it possible to let event be the sub partition column of date，while they contains totally different schemas
   
   It depends on how to use. I didn't try to create a table on all events since they have different schema, but it can help index one event files with the certain date.
   
   > Why does it only affect the json writer？
   
   Currently, We only support write metrics using json format(exclude the Spark Event logger), it can be a quick start. 
   
   > How a static date handle session or statement events that might be created in the same engine but in different days？
   
   I think it's not a problem, all events for the certain engine can be written in one date partition and it also help to index event log.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@kyuubi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org