You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@drill.apache.org by François Méthot <fm...@gmail.com> on 2016/03/01 17:25:06 UTC

Rolling Window

Hi,

  We need to manage a rolling window of parquet data within drill.

Our parquet files are partitioned by hour,
Once hdfs reach a certain usage threshold, we want to delete the oldest
partition folder.

A simple approach would be to run a cron job that check the hdfs usage and
delete the oldest partition folder if necessary, would that cause issue if
this operation occurs while a query is  running on those files?

Would you recommend instead writing a script/app that submit a "drop table"
on the oldest partition folder using odbc interface?

Any other ideas are welcome.

Thanks a lot!
François

Re: Rolling Window

Posted by François Méthot <fm...@gmail.com>.
I found this:

https://drill.apache.org/docs/drop-table/ :

"Currently, Drill does not have a mechanism in place, such as read locks on
files, to address concurrency issues. For example, if one user runs a query
that references a table that another user simultaneously issues the DROP
TABLE command against, there is no mechanism in place to prevent a
collision of the two processes. In such a scenario, Drill may return
partial query results or a system error to the user running the query when
the table is dropped."

A solution would be to perform a delete operation in HDFS when no queries
are in "running" state or schedule a delete outside business hours.



On Tue, Mar 1, 2016 at 11:25 AM, François Méthot <fm...@gmail.com>
wrote:

> Hi,
>
>   We need to manage a rolling window of parquet data within drill.
>
> Our parquet files are partitioned by hour,
> Once hdfs reach a certain usage threshold, we want to delete the oldest
> partition folder.
>
> A simple approach would be to run a cron job that check the hdfs usage and
> delete the oldest partition folder if necessary, would that cause issue if
> this operation occurs while a query is  running on those files?
>
> Would you recommend instead writing a script/app that submit a "drop
> table" on the oldest partition folder using odbc interface?
>
> Any other ideas are welcome.
>
> Thanks a lot!
> François
>
>
>
>
>
>
>
>