You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@metron.apache.org by Farrukh Naveed Anjum <an...@gmail.com> on 2018/12/26 07:06:33 UTC
Data Rotation and Deletion Scripts for Metron
Hi,
How can write a data retention script for metron. So that safe hard disk
space. Like I want to Store only last N Days (Say 30 days) data in HDFS,
Elasticssearch and Logs in Metron.
Right now, Metron eats up my all hard drive space on server.
Any Idea / Help will be appreciated.
--
With Regards
Farrukh Naveed Anjum
Re: Data Rotation and Deletion Scripts for Metron
Posted by "Zeolla@GMail.com" <ze...@gmail.com>.
This is what we used to have in a very old version of Metron, and is what I
was referring to above. Not sure how applicable it still is.
https://github.com/apache/metron/blob/Metron_0.3.1/metron-deployment/roles/metron_streaming/tasks/hdfs_purge.yml
Jon
On Mon, Dec 31, 2018 at 12:33 AM Farrukh Naveed Anjum <
anjum.farrukh@gmail.com> wrote:
> Any suggestions ? to get ride of classic data and save up some space.
>
> On Wed, Dec 26, 2018 at 8:09 PM Farrukh Naveed Anjum <
> anjum.farrukh@gmail.com> wrote:
>
>> Any exact command / example ? I am using BRO Tool for Log Collection
>>
>> On Wed, Dec 26, 2018 at 6:03 PM Zeolla@GMail.com <ze...@gmail.com>
>> wrote:
>>
>>> Aren't the logs stored in date-based directories? I thought we used to
>>> deploy a daily cron with a hadoop fs -rmr based on today's date minus 30
>>> days or so.
>>>
>>> Jon
>>>
>>> On Wed, Dec 26, 2018 at 6:20 AM Farrukh Naveed Anjum <
>>> anjum.farrukh@gmail.com> wrote:
>>>
>>>> what about data in HDFS and Other Logs ? Any idea to purne them ?
>>>>
>>>> On Wed, Dec 26, 2018 at 3:51 PM Zeolla@GMail.com <ze...@gmail.com>
>>>> wrote:
>>>>
>>>>> There used to be something that was automatically deployed as a cron
>>>>> that did that. For ES I know it moved to curator
>>>>> https://github.com/apache/metron/blob/master/metron-platform/metron-data-management/README.md#pruning-data-from-elasticsearch
>>>>>
>>>>> I don't know if the HDFS job still exists or not, but it used to.
>>>>>
>>>>> Jon
>>>>>
>>>>> On Wed, Dec 26, 2018, 2:06 AM Farrukh Naveed Anjum <
>>>>> anjum.farrukh@gmail.com> wrote:
>>>>>
>>>>>> Hi,
>>>>>>
>>>>>> How can write a data retention script for metron. So that safe hard
>>>>>> disk space. Like I want to Store only last N Days (Say 30 days) data in
>>>>>> HDFS, Elasticssearch and Logs in Metron.
>>>>>>
>>>>>> Right now, Metron eats up my all hard drive space on server.
>>>>>>
>>>>>> Any Idea / Help will be appreciated.
>>>>>>
>>>>>>
>>>>>> --
>>>>>> With Regards
>>>>>> Farrukh Naveed Anjum
>>>>>>
>>>>> --
>>>>>
>>>>> Jon Zeolla
>>>>>
>>>>
>>>>
>>>> --
>>>> With Regards
>>>> Farrukh Naveed Anjum
>>>>
>>> --
>>>
>>> Jon Zeolla
>>>
>>
>>
>> --
>> With Regards
>> Farrukh Naveed Anjum
>>
>
>
> --
> With Regards
> Farrukh Naveed Anjum
>
--
Jon Zeolla
Re: Data Rotation and Deletion Scripts for Metron
Posted by Farrukh Naveed Anjum <an...@gmail.com>.
Any suggestions ? to get ride of classic data and save up some space.
On Wed, Dec 26, 2018 at 8:09 PM Farrukh Naveed Anjum <
anjum.farrukh@gmail.com> wrote:
> Any exact command / example ? I am using BRO Tool for Log Collection
>
> On Wed, Dec 26, 2018 at 6:03 PM Zeolla@GMail.com <ze...@gmail.com> wrote:
>
>> Aren't the logs stored in date-based directories? I thought we used to
>> deploy a daily cron with a hadoop fs -rmr based on today's date minus 30
>> days or so.
>>
>> Jon
>>
>> On Wed, Dec 26, 2018 at 6:20 AM Farrukh Naveed Anjum <
>> anjum.farrukh@gmail.com> wrote:
>>
>>> what about data in HDFS and Other Logs ? Any idea to purne them ?
>>>
>>> On Wed, Dec 26, 2018 at 3:51 PM Zeolla@GMail.com <ze...@gmail.com>
>>> wrote:
>>>
>>>> There used to be something that was automatically deployed as a cron
>>>> that did that. For ES I know it moved to curator
>>>> https://github.com/apache/metron/blob/master/metron-platform/metron-data-management/README.md#pruning-data-from-elasticsearch
>>>>
>>>> I don't know if the HDFS job still exists or not, but it used to.
>>>>
>>>> Jon
>>>>
>>>> On Wed, Dec 26, 2018, 2:06 AM Farrukh Naveed Anjum <
>>>> anjum.farrukh@gmail.com> wrote:
>>>>
>>>>> Hi,
>>>>>
>>>>> How can write a data retention script for metron. So that safe hard
>>>>> disk space. Like I want to Store only last N Days (Say 30 days) data in
>>>>> HDFS, Elasticssearch and Logs in Metron.
>>>>>
>>>>> Right now, Metron eats up my all hard drive space on server.
>>>>>
>>>>> Any Idea / Help will be appreciated.
>>>>>
>>>>>
>>>>> --
>>>>> With Regards
>>>>> Farrukh Naveed Anjum
>>>>>
>>>> --
>>>>
>>>> Jon Zeolla
>>>>
>>>
>>>
>>> --
>>> With Regards
>>> Farrukh Naveed Anjum
>>>
>> --
>>
>> Jon Zeolla
>>
>
>
> --
> With Regards
> Farrukh Naveed Anjum
>
--
With Regards
Farrukh Naveed Anjum
Re: Data Rotation and Deletion Scripts for Metron
Posted by Farrukh Naveed Anjum <an...@gmail.com>.
Any exact command / example ? I am using BRO Tool for Log Collection
On Wed, Dec 26, 2018 at 6:03 PM Zeolla@GMail.com <ze...@gmail.com> wrote:
> Aren't the logs stored in date-based directories? I thought we used to
> deploy a daily cron with a hadoop fs -rmr based on today's date minus 30
> days or so.
>
> Jon
>
> On Wed, Dec 26, 2018 at 6:20 AM Farrukh Naveed Anjum <
> anjum.farrukh@gmail.com> wrote:
>
>> what about data in HDFS and Other Logs ? Any idea to purne them ?
>>
>> On Wed, Dec 26, 2018 at 3:51 PM Zeolla@GMail.com <ze...@gmail.com>
>> wrote:
>>
>>> There used to be something that was automatically deployed as a cron
>>> that did that. For ES I know it moved to curator
>>> https://github.com/apache/metron/blob/master/metron-platform/metron-data-management/README.md#pruning-data-from-elasticsearch
>>>
>>> I don't know if the HDFS job still exists or not, but it used to.
>>>
>>> Jon
>>>
>>> On Wed, Dec 26, 2018, 2:06 AM Farrukh Naveed Anjum <
>>> anjum.farrukh@gmail.com> wrote:
>>>
>>>> Hi,
>>>>
>>>> How can write a data retention script for metron. So that safe hard
>>>> disk space. Like I want to Store only last N Days (Say 30 days) data in
>>>> HDFS, Elasticssearch and Logs in Metron.
>>>>
>>>> Right now, Metron eats up my all hard drive space on server.
>>>>
>>>> Any Idea / Help will be appreciated.
>>>>
>>>>
>>>> --
>>>> With Regards
>>>> Farrukh Naveed Anjum
>>>>
>>> --
>>>
>>> Jon Zeolla
>>>
>>
>>
>> --
>> With Regards
>> Farrukh Naveed Anjum
>>
> --
>
> Jon Zeolla
>
--
With Regards
Farrukh Naveed Anjum
Re: Data Rotation and Deletion Scripts for Metron
Posted by "Zeolla@GMail.com" <ze...@gmail.com>.
Aren't the logs stored in date-based directories? I thought we used to
deploy a daily cron with a hadoop fs -rmr based on today's date minus 30
days or so.
Jon
On Wed, Dec 26, 2018 at 6:20 AM Farrukh Naveed Anjum <
anjum.farrukh@gmail.com> wrote:
> what about data in HDFS and Other Logs ? Any idea to purne them ?
>
> On Wed, Dec 26, 2018 at 3:51 PM Zeolla@GMail.com <ze...@gmail.com> wrote:
>
>> There used to be something that was automatically deployed as a cron that
>> did that. For ES I know it moved to curator
>> https://github.com/apache/metron/blob/master/metron-platform/metron-data-management/README.md#pruning-data-from-elasticsearch
>>
>> I don't know if the HDFS job still exists or not, but it used to.
>>
>> Jon
>>
>> On Wed, Dec 26, 2018, 2:06 AM Farrukh Naveed Anjum <
>> anjum.farrukh@gmail.com> wrote:
>>
>>> Hi,
>>>
>>> How can write a data retention script for metron. So that safe hard disk
>>> space. Like I want to Store only last N Days (Say 30 days) data in HDFS,
>>> Elasticssearch and Logs in Metron.
>>>
>>> Right now, Metron eats up my all hard drive space on server.
>>>
>>> Any Idea / Help will be appreciated.
>>>
>>>
>>> --
>>> With Regards
>>> Farrukh Naveed Anjum
>>>
>> --
>>
>> Jon Zeolla
>>
>
>
> --
> With Regards
> Farrukh Naveed Anjum
>
--
Jon Zeolla
Re: Data Rotation and Deletion Scripts for Metron
Posted by Farrukh Naveed Anjum <an...@gmail.com>.
what about data in HDFS and Other Logs ? Any idea to purne them ?
On Wed, Dec 26, 2018 at 3:51 PM Zeolla@GMail.com <ze...@gmail.com> wrote:
> There used to be something that was automatically deployed as a cron that
> did that. For ES I know it moved to curator
> https://github.com/apache/metron/blob/master/metron-platform/metron-data-management/README.md#pruning-data-from-elasticsearch
>
> I don't know if the HDFS job still exists or not, but it used to.
>
> Jon
>
> On Wed, Dec 26, 2018, 2:06 AM Farrukh Naveed Anjum <
> anjum.farrukh@gmail.com> wrote:
>
>> Hi,
>>
>> How can write a data retention script for metron. So that safe hard disk
>> space. Like I want to Store only last N Days (Say 30 days) data in HDFS,
>> Elasticssearch and Logs in Metron.
>>
>> Right now, Metron eats up my all hard drive space on server.
>>
>> Any Idea / Help will be appreciated.
>>
>>
>> --
>> With Regards
>> Farrukh Naveed Anjum
>>
> --
>
> Jon Zeolla
>
--
With Regards
Farrukh Naveed Anjum
Re: Data Rotation and Deletion Scripts for Metron
Posted by "Zeolla@GMail.com" <ze...@gmail.com>.
There used to be something that was automatically deployed as a cron that
did that. For ES I know it moved to curator
https://github.com/apache/metron/blob/master/metron-platform/metron-data-management/README.md#pruning-data-from-elasticsearch
I don't know if the HDFS job still exists or not, but it used to.
Jon
On Wed, Dec 26, 2018, 2:06 AM Farrukh Naveed Anjum <an...@gmail.com>
wrote:
> Hi,
>
> How can write a data retention script for metron. So that safe hard disk
> space. Like I want to Store only last N Days (Say 30 days) data in HDFS,
> Elasticssearch and Logs in Metron.
>
> Right now, Metron eats up my all hard drive space on server.
>
> Any Idea / Help will be appreciated.
>
>
> --
> With Regards
> Farrukh Naveed Anjum
>
--
Jon Zeolla