[jira] [Commented] (DRILL-2362) Drill should manage Query Profiling archiving

["ASF GitHub Bot (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/DRILL-2362?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17608728#comment-17608728 ] 

ASF GitHub Bot commented on DRILL-2362:
---------------------------------------

jnturton commented on PR #1750:
URL: https://github.com/apache/drill/pull/1750#issuecomment-1256153990

   Hi Kunal! Thank you for this contribution, I'd like to help to move it forward. 
   
   > However, I am not sure if we should automatically index any new profile that has been copy-pasted into the root dir. For e.g., we might get a profile from a JIRA and would like to view it. Should we leave it there (and try to render it) or should we index it ASAP ?
   
   I share your concern here.  I think we should consider having Drill only write _new_ profiles to partitioned directories. Any partitioning of historical profiles can be done externally by admins, in my opinion, and we can add examples of "housekeeping" scripts for doing that to the Drill documentation.
   
   Would you like to do any of the following?
   
   - Resume the work on this PR, I volunteer to be a reviewer.
   - Receive a PR from me to your fork here that's rebased and has some changes I think we want.
   - You're too busy now, so I'll pull your commits here into a new branch of my own and open a new PR.
   
   Thanks
   James




> Drill should manage Query Profiling archiving
> ---------------------------------------------
>
>                 Key: DRILL-2362
>                 URL: https://issues.apache.org/jira/browse/DRILL-2362
>             Project: Apache Drill
>          Issue Type: New Feature
>          Components: Storage - Other
>    Affects Versions: 0.7.0
>            Reporter: Chris Westin
>            Priority: Major
>
> We collect query profile information for analysis purposes, but we keep it forever. At this time, for a few queries, it isn't a problem. But as users start putting Drill into production, automated use via other applications will make this grow quickly. We need to come up with a retention policy mechanism, with suitable settings administrators can use, and implement it so that this data can be cleaned up.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)