You are viewing a plain text version of this content. The canonical link for it is here.
Posted to mapreduce-user@hadoop.apache.org by Winston Lin <wi...@gmail.com> on 2013/05/08 02:49:51 UTC
get recent changed files in hadoop
Any idea to get recent changed file in hadoop? e.g. files created yesterday?
fs -ls will only give us all the files.
Thanks
Winston
Re: get recent changed files in hadoop
Posted by Jean-Marc Spaggiari <je...@spaggiari.org>.
You can still parse the hadoop ls ouput with bash and sort it (revert,
cut, sort, etc.), but that will read all the entries, just just the x
first one...
2013/5/7 Winston Lin <wi...@gmail.com>:
> look like we cannot even sort the output of ls by date with fs command?
>
> In *ux system, we can do ls -t ...to sort by modification time, newest
> first
>
> Winston
>
>
>
>
> On Wed, May 8, 2013 at 1:47 PM, Mohammad Tariq <do...@gmail.com> wrote:
>>
>> I don't think any such thing is available OOTB.
>>
>> Warm Regards,
>> Tariq
>> https://mtariq.jux.com/
>> cloudfront.blogspot.com
>>
>>
>> On Wed, May 8, 2013 at 8:51 AM, Rahul Bhattacharjee
>> <ra...@gmail.com> wrote:
>>>
>>> Is any such option available in other posix shells?
>>>
>>>
>>> On Wednesday, May 8, 2013, Winston Lin wrote:
>>>>
>>>> Any idea to get recent changed file in hadoop? e.g. files created
>>>> yesterday?
>>>>
>>>> fs -ls will only give us all the files.
>>>>
>>>> Thanks
>>>> Winston
>>>
>>>
>>>
>>> --
>>> Sent from Gmail Mobile
>>
>>
>
Re: get recent changed files in hadoop
Posted by Jean-Marc Spaggiari <je...@spaggiari.org>.
You can still parse the hadoop ls ouput with bash and sort it (revert,
cut, sort, etc.), but that will read all the entries, just just the x
first one...
2013/5/7 Winston Lin <wi...@gmail.com>:
> look like we cannot even sort the output of ls by date with fs command?
>
> In *ux system, we can do ls -t ...to sort by modification time, newest
> first
>
> Winston
>
>
>
>
> On Wed, May 8, 2013 at 1:47 PM, Mohammad Tariq <do...@gmail.com> wrote:
>>
>> I don't think any such thing is available OOTB.
>>
>> Warm Regards,
>> Tariq
>> https://mtariq.jux.com/
>> cloudfront.blogspot.com
>>
>>
>> On Wed, May 8, 2013 at 8:51 AM, Rahul Bhattacharjee
>> <ra...@gmail.com> wrote:
>>>
>>> Is any such option available in other posix shells?
>>>
>>>
>>> On Wednesday, May 8, 2013, Winston Lin wrote:
>>>>
>>>> Any idea to get recent changed file in hadoop? e.g. files created
>>>> yesterday?
>>>>
>>>> fs -ls will only give us all the files.
>>>>
>>>> Thanks
>>>> Winston
>>>
>>>
>>>
>>> --
>>> Sent from Gmail Mobile
>>
>>
>
Re: get recent changed files in hadoop
Posted by Jean-Marc Spaggiari <je...@spaggiari.org>.
You can still parse the hadoop ls ouput with bash and sort it (revert,
cut, sort, etc.), but that will read all the entries, just just the x
first one...
2013/5/7 Winston Lin <wi...@gmail.com>:
> look like we cannot even sort the output of ls by date with fs command?
>
> In *ux system, we can do ls -t ...to sort by modification time, newest
> first
>
> Winston
>
>
>
>
> On Wed, May 8, 2013 at 1:47 PM, Mohammad Tariq <do...@gmail.com> wrote:
>>
>> I don't think any such thing is available OOTB.
>>
>> Warm Regards,
>> Tariq
>> https://mtariq.jux.com/
>> cloudfront.blogspot.com
>>
>>
>> On Wed, May 8, 2013 at 8:51 AM, Rahul Bhattacharjee
>> <ra...@gmail.com> wrote:
>>>
>>> Is any such option available in other posix shells?
>>>
>>>
>>> On Wednesday, May 8, 2013, Winston Lin wrote:
>>>>
>>>> Any idea to get recent changed file in hadoop? e.g. files created
>>>> yesterday?
>>>>
>>>> fs -ls will only give us all the files.
>>>>
>>>> Thanks
>>>> Winston
>>>
>>>
>>>
>>> --
>>> Sent from Gmail Mobile
>>
>>
>
Re: get recent changed files in hadoop
Posted by Jean-Marc Spaggiari <je...@spaggiari.org>.
You can still parse the hadoop ls ouput with bash and sort it (revert,
cut, sort, etc.), but that will read all the entries, just just the x
first one...
2013/5/7 Winston Lin <wi...@gmail.com>:
> look like we cannot even sort the output of ls by date with fs command?
>
> In *ux system, we can do ls -t ...to sort by modification time, newest
> first
>
> Winston
>
>
>
>
> On Wed, May 8, 2013 at 1:47 PM, Mohammad Tariq <do...@gmail.com> wrote:
>>
>> I don't think any such thing is available OOTB.
>>
>> Warm Regards,
>> Tariq
>> https://mtariq.jux.com/
>> cloudfront.blogspot.com
>>
>>
>> On Wed, May 8, 2013 at 8:51 AM, Rahul Bhattacharjee
>> <ra...@gmail.com> wrote:
>>>
>>> Is any such option available in other posix shells?
>>>
>>>
>>> On Wednesday, May 8, 2013, Winston Lin wrote:
>>>>
>>>> Any idea to get recent changed file in hadoop? e.g. files created
>>>> yesterday?
>>>>
>>>> fs -ls will only give us all the files.
>>>>
>>>> Thanks
>>>> Winston
>>>
>>>
>>>
>>> --
>>> Sent from Gmail Mobile
>>
>>
>
Re: get recent changed files in hadoop
Posted by Winston Lin <wi...@gmail.com>.
look like we cannot even sort the output of ls by date with fs command?
In *ux system, we can do ls -t ...to sort by modification time, newest
first
Winston
On Wed, May 8, 2013 at 1:47 PM, Mohammad Tariq <do...@gmail.com> wrote:
> I don't think any such thing is available OOTB.
>
> Warm Regards,
> Tariq
> https://mtariq.jux.com/
> cloudfront.blogspot.com
>
>
> On Wed, May 8, 2013 at 8:51 AM, Rahul Bhattacharjee <
> rahul.rec.dgp@gmail.com> wrote:
>
>> Is any such option available in other posix shells?
>>
>>
>> On Wednesday, May 8, 2013, Winston Lin wrote:
>>
>>> Any idea to get recent changed file in hadoop? e.g. files created
>>> yesterday?
>>>
>>> fs -ls will only give us all the files.
>>>
>>> Thanks
>>> Winston
>>>
>>
>>
>> --
>> Sent from Gmail Mobile
>>
>
>
Re: get recent changed files in hadoop
Posted by Winston Lin <wi...@gmail.com>.
look like we cannot even sort the output of ls by date with fs command?
In *ux system, we can do ls -t ...to sort by modification time, newest
first
Winston
On Wed, May 8, 2013 at 1:47 PM, Mohammad Tariq <do...@gmail.com> wrote:
> I don't think any such thing is available OOTB.
>
> Warm Regards,
> Tariq
> https://mtariq.jux.com/
> cloudfront.blogspot.com
>
>
> On Wed, May 8, 2013 at 8:51 AM, Rahul Bhattacharjee <
> rahul.rec.dgp@gmail.com> wrote:
>
>> Is any such option available in other posix shells?
>>
>>
>> On Wednesday, May 8, 2013, Winston Lin wrote:
>>
>>> Any idea to get recent changed file in hadoop? e.g. files created
>>> yesterday?
>>>
>>> fs -ls will only give us all the files.
>>>
>>> Thanks
>>> Winston
>>>
>>
>>
>> --
>> Sent from Gmail Mobile
>>
>
>
Re: get recent changed files in hadoop
Posted by Winston Lin <wi...@gmail.com>.
look like we cannot even sort the output of ls by date with fs command?
In *ux system, we can do ls -t ...to sort by modification time, newest
first
Winston
On Wed, May 8, 2013 at 1:47 PM, Mohammad Tariq <do...@gmail.com> wrote:
> I don't think any such thing is available OOTB.
>
> Warm Regards,
> Tariq
> https://mtariq.jux.com/
> cloudfront.blogspot.com
>
>
> On Wed, May 8, 2013 at 8:51 AM, Rahul Bhattacharjee <
> rahul.rec.dgp@gmail.com> wrote:
>
>> Is any such option available in other posix shells?
>>
>>
>> On Wednesday, May 8, 2013, Winston Lin wrote:
>>
>>> Any idea to get recent changed file in hadoop? e.g. files created
>>> yesterday?
>>>
>>> fs -ls will only give us all the files.
>>>
>>> Thanks
>>> Winston
>>>
>>
>>
>> --
>> Sent from Gmail Mobile
>>
>
>
Re: get recent changed files in hadoop
Posted by Winston Lin <wi...@gmail.com>.
look like we cannot even sort the output of ls by date with fs command?
In *ux system, we can do ls -t ...to sort by modification time, newest
first
Winston
On Wed, May 8, 2013 at 1:47 PM, Mohammad Tariq <do...@gmail.com> wrote:
> I don't think any such thing is available OOTB.
>
> Warm Regards,
> Tariq
> https://mtariq.jux.com/
> cloudfront.blogspot.com
>
>
> On Wed, May 8, 2013 at 8:51 AM, Rahul Bhattacharjee <
> rahul.rec.dgp@gmail.com> wrote:
>
>> Is any such option available in other posix shells?
>>
>>
>> On Wednesday, May 8, 2013, Winston Lin wrote:
>>
>>> Any idea to get recent changed file in hadoop? e.g. files created
>>> yesterday?
>>>
>>> fs -ls will only give us all the files.
>>>
>>> Thanks
>>> Winston
>>>
>>
>>
>> --
>> Sent from Gmail Mobile
>>
>
>
Re: get recent changed files in hadoop
Posted by Mohammad Tariq <do...@gmail.com>.
I don't think any such thing is available OOTB.
Warm Regards,
Tariq
https://mtariq.jux.com/
cloudfront.blogspot.com
On Wed, May 8, 2013 at 8:51 AM, Rahul Bhattacharjee <rahul.rec.dgp@gmail.com
> wrote:
> Is any such option available in other posix shells?
>
>
> On Wednesday, May 8, 2013, Winston Lin wrote:
>
>> Any idea to get recent changed file in hadoop? e.g. files created
>> yesterday?
>>
>> fs -ls will only give us all the files.
>>
>> Thanks
>> Winston
>>
>
>
> --
> Sent from Gmail Mobile
>
Re: get recent changed files in hadoop
Posted by Mohammad Tariq <do...@gmail.com>.
I don't think any such thing is available OOTB.
Warm Regards,
Tariq
https://mtariq.jux.com/
cloudfront.blogspot.com
On Wed, May 8, 2013 at 8:51 AM, Rahul Bhattacharjee <rahul.rec.dgp@gmail.com
> wrote:
> Is any such option available in other posix shells?
>
>
> On Wednesday, May 8, 2013, Winston Lin wrote:
>
>> Any idea to get recent changed file in hadoop? e.g. files created
>> yesterday?
>>
>> fs -ls will only give us all the files.
>>
>> Thanks
>> Winston
>>
>
>
> --
> Sent from Gmail Mobile
>
Re: get recent changed files in hadoop
Posted by Mohammad Tariq <do...@gmail.com>.
I don't think any such thing is available OOTB.
Warm Regards,
Tariq
https://mtariq.jux.com/
cloudfront.blogspot.com
On Wed, May 8, 2013 at 8:51 AM, Rahul Bhattacharjee <rahul.rec.dgp@gmail.com
> wrote:
> Is any such option available in other posix shells?
>
>
> On Wednesday, May 8, 2013, Winston Lin wrote:
>
>> Any idea to get recent changed file in hadoop? e.g. files created
>> yesterday?
>>
>> fs -ls will only give us all the files.
>>
>> Thanks
>> Winston
>>
>
>
> --
> Sent from Gmail Mobile
>
Re: get recent changed files in hadoop
Posted by Mohammad Tariq <do...@gmail.com>.
I don't think any such thing is available OOTB.
Warm Regards,
Tariq
https://mtariq.jux.com/
cloudfront.blogspot.com
On Wed, May 8, 2013 at 8:51 AM, Rahul Bhattacharjee <rahul.rec.dgp@gmail.com
> wrote:
> Is any such option available in other posix shells?
>
>
> On Wednesday, May 8, 2013, Winston Lin wrote:
>
>> Any idea to get recent changed file in hadoop? e.g. files created
>> yesterday?
>>
>> fs -ls will only give us all the files.
>>
>> Thanks
>> Winston
>>
>
>
> --
> Sent from Gmail Mobile
>
Re: get recent changed files in hadoop
Posted by Rahul Bhattacharjee <ra...@gmail.com>.
Is any such option available in other posix shells?
On Wednesday, May 8, 2013, Winston Lin wrote:
> Any idea to get recent changed file in hadoop? e.g. files created
> yesterday?
>
> fs -ls will only give us all the files.
>
> Thanks
> Winston
>
--
Sent from Gmail Mobile
Re: get recent changed files in hadoop
Posted by Winston Lin <wi...@gmail.com>.
Thanks so much for all your help. I think I would temporally use ls to
get the list and then grep or awk to parse it.
Winston
On Wed, May 8, 2013 at 9:29 PM, shashwat shriparv <dwivedishashwat@gmail.com
> wrote:
> hadoop dfs -ls|grep "date you want to see"
>
> you can use sort on that what ever you like ..
>
> *Thanks & Regards *
>
> ∞
> Shashwat Shriparv
>
>
>
> On Wed, May 8, 2013 at 6:19 AM, Winston Lin <wi...@gmail.com> wrote:
>
>> Any idea to get recent changed file in hadoop? e.g. files created
>> yesterday?
>>
>> fs -ls will only give us all the files.
>>
>> Thanks
>> Winston
>>
>
>
Re: get recent changed files in hadoop
Posted by Chris Nauroth <cn...@hortonworks.com>.
It sounds like you're interested in something equivalent to the Unix find
command for HDFS. HADOOP-8989 tracks adding this feature. There is a
patch available, but it's still under review, so the feature isn't
available in a release yet.
https://issues.apache.org/jira/browse/HADOOP-8989
Chris Nauroth
Hortonworks
http://hortonworks.com/
On Wed, May 8, 2013 at 4:29 AM, shashwat shriparv <dwivedishashwat@gmail.com
> wrote:
> hadoop dfs -ls|grep "date you want to see"
>
> you can use sort on that what ever you like ..
>
> *Thanks & Regards *
>
> ∞
> Shashwat Shriparv
>
>
>
> On Wed, May 8, 2013 at 6:19 AM, Winston Lin <wi...@gmail.com> wrote:
>
>> Any idea to get recent changed file in hadoop? e.g. files created
>> yesterday?
>>
>> fs -ls will only give us all the files.
>>
>> Thanks
>> Winston
>>
>
>
Re: get recent changed files in hadoop
Posted by Chris Nauroth <cn...@hortonworks.com>.
It sounds like you're interested in something equivalent to the Unix find
command for HDFS. HADOOP-8989 tracks adding this feature. There is a
patch available, but it's still under review, so the feature isn't
available in a release yet.
https://issues.apache.org/jira/browse/HADOOP-8989
Chris Nauroth
Hortonworks
http://hortonworks.com/
On Wed, May 8, 2013 at 4:29 AM, shashwat shriparv <dwivedishashwat@gmail.com
> wrote:
> hadoop dfs -ls|grep "date you want to see"
>
> you can use sort on that what ever you like ..
>
> *Thanks & Regards *
>
> ∞
> Shashwat Shriparv
>
>
>
> On Wed, May 8, 2013 at 6:19 AM, Winston Lin <wi...@gmail.com> wrote:
>
>> Any idea to get recent changed file in hadoop? e.g. files created
>> yesterday?
>>
>> fs -ls will only give us all the files.
>>
>> Thanks
>> Winston
>>
>
>
Re: get recent changed files in hadoop
Posted by Winston Lin <wi...@gmail.com>.
Thanks so much for all your help. I think I would temporally use ls to
get the list and then grep or awk to parse it.
Winston
On Wed, May 8, 2013 at 9:29 PM, shashwat shriparv <dwivedishashwat@gmail.com
> wrote:
> hadoop dfs -ls|grep "date you want to see"
>
> you can use sort on that what ever you like ..
>
> *Thanks & Regards *
>
> ∞
> Shashwat Shriparv
>
>
>
> On Wed, May 8, 2013 at 6:19 AM, Winston Lin <wi...@gmail.com> wrote:
>
>> Any idea to get recent changed file in hadoop? e.g. files created
>> yesterday?
>>
>> fs -ls will only give us all the files.
>>
>> Thanks
>> Winston
>>
>
>
Re: get recent changed files in hadoop
Posted by Winston Lin <wi...@gmail.com>.
Thanks so much for all your help. I think I would temporally use ls to
get the list and then grep or awk to parse it.
Winston
On Wed, May 8, 2013 at 9:29 PM, shashwat shriparv <dwivedishashwat@gmail.com
> wrote:
> hadoop dfs -ls|grep "date you want to see"
>
> you can use sort on that what ever you like ..
>
> *Thanks & Regards *
>
> ∞
> Shashwat Shriparv
>
>
>
> On Wed, May 8, 2013 at 6:19 AM, Winston Lin <wi...@gmail.com> wrote:
>
>> Any idea to get recent changed file in hadoop? e.g. files created
>> yesterday?
>>
>> fs -ls will only give us all the files.
>>
>> Thanks
>> Winston
>>
>
>
Re: get recent changed files in hadoop
Posted by Winston Lin <wi...@gmail.com>.
Thanks so much for all your help. I think I would temporally use ls to
get the list and then grep or awk to parse it.
Winston
On Wed, May 8, 2013 at 9:29 PM, shashwat shriparv <dwivedishashwat@gmail.com
> wrote:
> hadoop dfs -ls|grep "date you want to see"
>
> you can use sort on that what ever you like ..
>
> *Thanks & Regards *
>
> ∞
> Shashwat Shriparv
>
>
>
> On Wed, May 8, 2013 at 6:19 AM, Winston Lin <wi...@gmail.com> wrote:
>
>> Any idea to get recent changed file in hadoop? e.g. files created
>> yesterday?
>>
>> fs -ls will only give us all the files.
>>
>> Thanks
>> Winston
>>
>
>
Re: get recent changed files in hadoop
Posted by Chris Nauroth <cn...@hortonworks.com>.
It sounds like you're interested in something equivalent to the Unix find
command for HDFS. HADOOP-8989 tracks adding this feature. There is a
patch available, but it's still under review, so the feature isn't
available in a release yet.
https://issues.apache.org/jira/browse/HADOOP-8989
Chris Nauroth
Hortonworks
http://hortonworks.com/
On Wed, May 8, 2013 at 4:29 AM, shashwat shriparv <dwivedishashwat@gmail.com
> wrote:
> hadoop dfs -ls|grep "date you want to see"
>
> you can use sort on that what ever you like ..
>
> *Thanks & Regards *
>
> ∞
> Shashwat Shriparv
>
>
>
> On Wed, May 8, 2013 at 6:19 AM, Winston Lin <wi...@gmail.com> wrote:
>
>> Any idea to get recent changed file in hadoop? e.g. files created
>> yesterday?
>>
>> fs -ls will only give us all the files.
>>
>> Thanks
>> Winston
>>
>
>
Re: get recent changed files in hadoop
Posted by Chris Nauroth <cn...@hortonworks.com>.
It sounds like you're interested in something equivalent to the Unix find
command for HDFS. HADOOP-8989 tracks adding this feature. There is a
patch available, but it's still under review, so the feature isn't
available in a release yet.
https://issues.apache.org/jira/browse/HADOOP-8989
Chris Nauroth
Hortonworks
http://hortonworks.com/
On Wed, May 8, 2013 at 4:29 AM, shashwat shriparv <dwivedishashwat@gmail.com
> wrote:
> hadoop dfs -ls|grep "date you want to see"
>
> you can use sort on that what ever you like ..
>
> *Thanks & Regards *
>
> ∞
> Shashwat Shriparv
>
>
>
> On Wed, May 8, 2013 at 6:19 AM, Winston Lin <wi...@gmail.com> wrote:
>
>> Any idea to get recent changed file in hadoop? e.g. files created
>> yesterday?
>>
>> fs -ls will only give us all the files.
>>
>> Thanks
>> Winston
>>
>
>
Re: get recent changed files in hadoop
Posted by shashwat shriparv <dw...@gmail.com>.
hadoop dfs -ls|grep "date you want to see"
you can use sort on that what ever you like ..
*Thanks & Regards *
∞
Shashwat Shriparv
On Wed, May 8, 2013 at 6:19 AM, Winston Lin <wi...@gmail.com> wrote:
> Any idea to get recent changed file in hadoop? e.g. files created
> yesterday?
>
> fs -ls will only give us all the files.
>
> Thanks
> Winston
>
Re: get recent changed files in hadoop
Posted by Rahul Bhattacharjee <ra...@gmail.com>.
Is any such option available in other posix shells?
On Wednesday, May 8, 2013, Winston Lin wrote:
> Any idea to get recent changed file in hadoop? e.g. files created
> yesterday?
>
> fs -ls will only give us all the files.
>
> Thanks
> Winston
>
--
Sent from Gmail Mobile
Re: get recent changed files in hadoop
Posted by shashwat shriparv <dw...@gmail.com>.
hadoop dfs -ls|grep "date you want to see"
you can use sort on that what ever you like ..
*Thanks & Regards *
∞
Shashwat Shriparv
On Wed, May 8, 2013 at 6:19 AM, Winston Lin <wi...@gmail.com> wrote:
> Any idea to get recent changed file in hadoop? e.g. files created
> yesterday?
>
> fs -ls will only give us all the files.
>
> Thanks
> Winston
>
Re: get recent changed files in hadoop
Posted by Rahul Bhattacharjee <ra...@gmail.com>.
Is any such option available in other posix shells?
On Wednesday, May 8, 2013, Winston Lin wrote:
> Any idea to get recent changed file in hadoop? e.g. files created
> yesterday?
>
> fs -ls will only give us all the files.
>
> Thanks
> Winston
>
--
Sent from Gmail Mobile
Re: get recent changed files in hadoop
Posted by shashwat shriparv <dw...@gmail.com>.
hadoop dfs -ls|grep "date you want to see"
you can use sort on that what ever you like ..
*Thanks & Regards *
∞
Shashwat Shriparv
On Wed, May 8, 2013 at 6:19 AM, Winston Lin <wi...@gmail.com> wrote:
> Any idea to get recent changed file in hadoop? e.g. files created
> yesterday?
>
> fs -ls will only give us all the files.
>
> Thanks
> Winston
>
Re: get recent changed files in hadoop
Posted by shashwat shriparv <dw...@gmail.com>.
hadoop dfs -ls|grep "date you want to see"
you can use sort on that what ever you like ..
*Thanks & Regards *
∞
Shashwat Shriparv
On Wed, May 8, 2013 at 6:19 AM, Winston Lin <wi...@gmail.com> wrote:
> Any idea to get recent changed file in hadoop? e.g. files created
> yesterday?
>
> fs -ls will only give us all the files.
>
> Thanks
> Winston
>
Re: get recent changed files in hadoop
Posted by Rahul Bhattacharjee <ra...@gmail.com>.
Is any such option available in other posix shells?
On Wednesday, May 8, 2013, Winston Lin wrote:
> Any idea to get recent changed file in hadoop? e.g. files created
> yesterday?
>
> fs -ls will only give us all the files.
>
> Thanks
> Winston
>
--
Sent from Gmail Mobile