You are viewing a plain text version of this content. The canonical link for it is here.
Posted to mapreduce-user@hadoop.apache.org by Sanjay Subramanian <Sa...@wizecommerce.com> on 2013/07/10 17:19:22 UTC
EBADF: Bad file descriptor
2013-07-10 07:11:50,131 WARN [Readahead Thread #1] org.apache.hadoop.io.ReadaheadPool: Failed readahead on ifile
EBADF: Bad file descriptor
at org.apache.hadoop.io.nativeio.NativeIO.posix_fadvise(Native Method)
at org.apache.hadoop.io.nativeio.NativeIO.posixFadviseIfPossible(NativeIO.java:145)
at org.apache.hadoop.io.ReadaheadPool$ReadaheadRequestImpl.run(ReadaheadPool.java:205)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
Hi
I have a Oozie workflow that runs a MR job and I have started getting this error past two days in one of the MR jobs that is being processed.
However if I run it again , it succeeds :-( but about 1 hr is wasted in the process.
Any clues ?
Or should I post this issue in the Oozie postings ?
Thanks
sanjay
Configuration
Name Value
impression.log.record.cached.tag cached=
impression.log.record.end.tag [end
impressions.mapreduce.conf.file.full.path /workflows/impressions/config/aggregations.conf<http://thv01:8888/filebrowser/view/workflows/impressions/config/aggregations.conf>
mapred.job.queue.name default
mapred.mapper.new-api true
mapred.reducer.new-api true
mapreduce.input.fileinputformat.inputdir /data/input/impressionlogs/outpdirlogs/9999-99-99<http://thv01:8888/filebrowser/view/data/input/impressionlogs/outpdirlogs/9999-99-99>
mapreduce.job.inputformat.class com.wizecommerce.utils.mapred.ZipMultipleLineRecordInputFormat
mapreduce.job.map.class com.wizecommerce.parser.mapred.OutpdirImpressionLogMapper
mapreduce.job.maps 500
mapreduce.job.name OutpdirImpressions_0000475-130611151004460-oozie-oozi-W
mapreduce.job.output.value.class org.apache.hadoop.io.Text
mapreduce.job.outputformat.class com.wizecommerce.utils.mapred.NextagTextOutputFormat
mapreduce.job.reduce.class com.wizecommerce.parser.mapred.OutpdirImpressionLogReducer
mapreduce.job.reduces 8
mapreduce.map.output.compress true
mapreduce.map.output.compress.codec org.apache.hadoop.io.compress.SnappyCodec
mapreduce.map.output.key.class org.apache.hadoop.io.Textorg.apache.hadoop.io.Text
mapreduce.map.output.value.class com.wizecommerce.parser.dao.OutpdirLogRecord
mapreduce.output.fileoutputformat.compress true
mapreduce.output.fileoutputformat.compress.codec com.hadoop.compression.lzo.LzopCodec
mapreduce.output.fileoutputformat.outputdir /data/output/impressions/outpdir/9999-99-99/0000475-130611151004460-oozie-oozi-W/outpdir_impressions_ptitle<http://thv01:8888/filebrowser/view/data/output/impressions/outpdir/9999-99-99/0000475-130611151004460-oozie-oozi-W/outpdir_impressions_ptitle>
mapreduce.tasktracker.map.tasks.maximum 12
mapreduce.tasktracker.reduce.tasks.maximum 8
outpdir.log.exclude.processing.datatypes header,sellerhidden
CONFIDENTIALITY NOTICE
======================
This email message and any attachments are for the exclusive use of the intended recipient(s) and may contain confidential and privileged information. Any unauthorized review, use, disclosure or distribution is prohibited. If you are not the intended recipient, please contact the sender by reply email and destroy all copies of the original message along with any attachments, from your computer system. If you are the intended recipient, please be advised that the content of this message is subject to access, review and disclosure by the sender's Email System Administrator.
Re: EBADF: Bad file descriptor
Posted by Sanjay Subramanian <Sa...@wizecommerce.com>.
Thanks
I will look into logs to see if I see anything else…
sanjay
From: Colin McCabe <cm...@alumni.cmu.edu>>
Reply-To: "user@hadoop.apache.org<ma...@hadoop.apache.org>" <us...@hadoop.apache.org>>
Date: Wednesday, July 10, 2013 11:52 AM
To: "user@hadoop.apache.org<ma...@hadoop.apache.org>" <us...@hadoop.apache.org>>
Subject: Re: EBADF: Bad file descriptor
To clarify a little bit, the readahead pool can sometimes spit out this message if you close a file while a readahead request is in flight. It's not an error and just reflects the fact that the file was closed hastily, probably because of some other bug which is the real problem.
Colin
On Wed, Jul 10, 2013 at 11:50 AM, Colin McCabe <cm...@alumni.cmu.edu>> wrote:
That's just a warning message. It's not causing your problem-- it's just a symptom.
You will have to find out why the MR job failed.
best,
Colin
On Wed, Jul 10, 2013 at 8:19 AM, Sanjay Subramanian <Sa...@wizecommerce.com>> wrote:
2013-07-10 07:11:50,131 WARN [Readahead Thread #1] org.apache.hadoop.io.ReadaheadPool: Failed readahead on ifile
EBADF: Bad file descriptor
at org.apache.hadoop.io.nativeio.NativeIO.posix_fadvise(Native Method)
at org.apache.hadoop.io.nativeio.NativeIO.posixFadviseIfPossible(NativeIO.java:145)
at org.apache.hadoop.io.ReadaheadPool$ReadaheadRequestImpl.run(ReadaheadPool.java:205)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
Hi
I have a Oozie workflow that runs a MR job and I have started getting this error past two days in one of the MR jobs that is being processed.
However if I run it again , it succeeds :-( but about 1 hr is wasted in the process.
Any clues ?
Or should I post this issue in the Oozie postings ?
Thanks
sanjay
Configuration
Name Value
impression.log.record.cached.tag cached=
impression.log.record.end.tag [end
impressions.mapreduce.conf.file.full.path /workflows/impressions/config/aggregations.conf<http://thv01:8888/filebrowser/view/workflows/impressions/config/aggregations.conf>
mapred.job.queue.name<http://mapred.job.queue.name> default
mapred.mapper.new-api true
mapred.reducer.new-api true
mapreduce.input.fileinputformat.inputdir /data/input/impressionlogs/outpdirlogs/9999-99-99<http://thv01:8888/filebrowser/view/data/input/impressionlogs/outpdirlogs/9999-99-99>
mapreduce.job.inputformat.class com.wizecommerce.utils.mapred.ZipMultipleLineRecordInputFormat
mapreduce.job.map.class com.wizecommerce.parser.mapred.OutpdirImpressionLogMapper
mapreduce.job.maps 500
mapreduce.job.name<http://mapreduce.job.name> OutpdirImpressions_0000475-130611151004460-oozie-oozi-W
mapreduce.job.output.value.class org.apache.hadoop.io.Text
mapreduce.job.outputformat.class com.wizecommerce.utils.mapred.NextagTextOutputFormat
mapreduce.job.reduce.class com.wizecommerce.parser.mapred.OutpdirImpressionLogReducer
mapreduce.job.reduces 8
mapreduce.map.output.compress true
mapreduce.map.output.compress.codec org.apache.hadoop.io.compress.SnappyCodec
mapreduce.map.output.key.class org.apache.hadoop.io.Textorg.apache.hadoop.io.Text
mapreduce.map.output.value.class com.wizecommerce.parser.dao.OutpdirLogRecord
mapreduce.output.fileoutputformat.compress true
mapreduce.output.fileoutputformat.compress.codec com.hadoop.compression.lzo.LzopCodec
mapreduce.output.fileoutputformat.outputdir /data/output/impressions/outpdir/9999-99-99/0000475-130611151004460-oozie-oozi-W/outpdir_impressions_ptitle<http://thv01:8888/filebrowser/view/data/output/impressions/outpdir/9999-99-99/0000475-130611151004460-oozie-oozi-W/outpdir_impressions_ptitle>
mapreduce.tasktracker.map.tasks.maximum 12
mapreduce.tasktracker.reduce.tasks.maximum 8
outpdir.log.exclude.processing.datatypes header,sellerhidden
CONFIDENTIALITY NOTICE
======================
This email message and any attachments are for the exclusive use of the intended recipient(s) and may contain confidential and privileged information. Any unauthorized review, use, disclosure or distribution is prohibited. If you are not the intended recipient, please contact the sender by reply email and destroy all copies of the original message along with any attachments, from your computer system. If you are the intended recipient, please be advised that the content of this message is subject to access, review and disclosure by the sender's Email System Administrator.
CONFIDENTIALITY NOTICE
======================
This email message and any attachments are for the exclusive use of the intended recipient(s) and may contain confidential and privileged information. Any unauthorized review, use, disclosure or distribution is prohibited. If you are not the intended recipient, please contact the sender by reply email and destroy all copies of the original message along with any attachments, from your computer system. If you are the intended recipient, please be advised that the content of this message is subject to access, review and disclosure by the sender's Email System Administrator.
Re: EBADF: Bad file descriptor
Posted by Sanjay Subramanian <Sa...@wizecommerce.com>.
Thanks
I will look into logs to see if I see anything else…
sanjay
From: Colin McCabe <cm...@alumni.cmu.edu>>
Reply-To: "user@hadoop.apache.org<ma...@hadoop.apache.org>" <us...@hadoop.apache.org>>
Date: Wednesday, July 10, 2013 11:52 AM
To: "user@hadoop.apache.org<ma...@hadoop.apache.org>" <us...@hadoop.apache.org>>
Subject: Re: EBADF: Bad file descriptor
To clarify a little bit, the readahead pool can sometimes spit out this message if you close a file while a readahead request is in flight. It's not an error and just reflects the fact that the file was closed hastily, probably because of some other bug which is the real problem.
Colin
On Wed, Jul 10, 2013 at 11:50 AM, Colin McCabe <cm...@alumni.cmu.edu>> wrote:
That's just a warning message. It's not causing your problem-- it's just a symptom.
You will have to find out why the MR job failed.
best,
Colin
On Wed, Jul 10, 2013 at 8:19 AM, Sanjay Subramanian <Sa...@wizecommerce.com>> wrote:
2013-07-10 07:11:50,131 WARN [Readahead Thread #1] org.apache.hadoop.io.ReadaheadPool: Failed readahead on ifile
EBADF: Bad file descriptor
at org.apache.hadoop.io.nativeio.NativeIO.posix_fadvise(Native Method)
at org.apache.hadoop.io.nativeio.NativeIO.posixFadviseIfPossible(NativeIO.java:145)
at org.apache.hadoop.io.ReadaheadPool$ReadaheadRequestImpl.run(ReadaheadPool.java:205)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
Hi
I have a Oozie workflow that runs a MR job and I have started getting this error past two days in one of the MR jobs that is being processed.
However if I run it again , it succeeds :-( but about 1 hr is wasted in the process.
Any clues ?
Or should I post this issue in the Oozie postings ?
Thanks
sanjay
Configuration
Name Value
impression.log.record.cached.tag cached=
impression.log.record.end.tag [end
impressions.mapreduce.conf.file.full.path /workflows/impressions/config/aggregations.conf<http://thv01:8888/filebrowser/view/workflows/impressions/config/aggregations.conf>
mapred.job.queue.name<http://mapred.job.queue.name> default
mapred.mapper.new-api true
mapred.reducer.new-api true
mapreduce.input.fileinputformat.inputdir /data/input/impressionlogs/outpdirlogs/9999-99-99<http://thv01:8888/filebrowser/view/data/input/impressionlogs/outpdirlogs/9999-99-99>
mapreduce.job.inputformat.class com.wizecommerce.utils.mapred.ZipMultipleLineRecordInputFormat
mapreduce.job.map.class com.wizecommerce.parser.mapred.OutpdirImpressionLogMapper
mapreduce.job.maps 500
mapreduce.job.name<http://mapreduce.job.name> OutpdirImpressions_0000475-130611151004460-oozie-oozi-W
mapreduce.job.output.value.class org.apache.hadoop.io.Text
mapreduce.job.outputformat.class com.wizecommerce.utils.mapred.NextagTextOutputFormat
mapreduce.job.reduce.class com.wizecommerce.parser.mapred.OutpdirImpressionLogReducer
mapreduce.job.reduces 8
mapreduce.map.output.compress true
mapreduce.map.output.compress.codec org.apache.hadoop.io.compress.SnappyCodec
mapreduce.map.output.key.class org.apache.hadoop.io.Textorg.apache.hadoop.io.Text
mapreduce.map.output.value.class com.wizecommerce.parser.dao.OutpdirLogRecord
mapreduce.output.fileoutputformat.compress true
mapreduce.output.fileoutputformat.compress.codec com.hadoop.compression.lzo.LzopCodec
mapreduce.output.fileoutputformat.outputdir /data/output/impressions/outpdir/9999-99-99/0000475-130611151004460-oozie-oozi-W/outpdir_impressions_ptitle<http://thv01:8888/filebrowser/view/data/output/impressions/outpdir/9999-99-99/0000475-130611151004460-oozie-oozi-W/outpdir_impressions_ptitle>
mapreduce.tasktracker.map.tasks.maximum 12
mapreduce.tasktracker.reduce.tasks.maximum 8
outpdir.log.exclude.processing.datatypes header,sellerhidden
CONFIDENTIALITY NOTICE
======================
This email message and any attachments are for the exclusive use of the intended recipient(s) and may contain confidential and privileged information. Any unauthorized review, use, disclosure or distribution is prohibited. If you are not the intended recipient, please contact the sender by reply email and destroy all copies of the original message along with any attachments, from your computer system. If you are the intended recipient, please be advised that the content of this message is subject to access, review and disclosure by the sender's Email System Administrator.
CONFIDENTIALITY NOTICE
======================
This email message and any attachments are for the exclusive use of the intended recipient(s) and may contain confidential and privileged information. Any unauthorized review, use, disclosure or distribution is prohibited. If you are not the intended recipient, please contact the sender by reply email and destroy all copies of the original message along with any attachments, from your computer system. If you are the intended recipient, please be advised that the content of this message is subject to access, review and disclosure by the sender's Email System Administrator.
Re: EBADF: Bad file descriptor
Posted by Sanjay Subramanian <Sa...@wizecommerce.com>.
Thanks
I will look into logs to see if I see anything else…
sanjay
From: Colin McCabe <cm...@alumni.cmu.edu>>
Reply-To: "user@hadoop.apache.org<ma...@hadoop.apache.org>" <us...@hadoop.apache.org>>
Date: Wednesday, July 10, 2013 11:52 AM
To: "user@hadoop.apache.org<ma...@hadoop.apache.org>" <us...@hadoop.apache.org>>
Subject: Re: EBADF: Bad file descriptor
To clarify a little bit, the readahead pool can sometimes spit out this message if you close a file while a readahead request is in flight. It's not an error and just reflects the fact that the file was closed hastily, probably because of some other bug which is the real problem.
Colin
On Wed, Jul 10, 2013 at 11:50 AM, Colin McCabe <cm...@alumni.cmu.edu>> wrote:
That's just a warning message. It's not causing your problem-- it's just a symptom.
You will have to find out why the MR job failed.
best,
Colin
On Wed, Jul 10, 2013 at 8:19 AM, Sanjay Subramanian <Sa...@wizecommerce.com>> wrote:
2013-07-10 07:11:50,131 WARN [Readahead Thread #1] org.apache.hadoop.io.ReadaheadPool: Failed readahead on ifile
EBADF: Bad file descriptor
at org.apache.hadoop.io.nativeio.NativeIO.posix_fadvise(Native Method)
at org.apache.hadoop.io.nativeio.NativeIO.posixFadviseIfPossible(NativeIO.java:145)
at org.apache.hadoop.io.ReadaheadPool$ReadaheadRequestImpl.run(ReadaheadPool.java:205)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
Hi
I have a Oozie workflow that runs a MR job and I have started getting this error past two days in one of the MR jobs that is being processed.
However if I run it again , it succeeds :-( but about 1 hr is wasted in the process.
Any clues ?
Or should I post this issue in the Oozie postings ?
Thanks
sanjay
Configuration
Name Value
impression.log.record.cached.tag cached=
impression.log.record.end.tag [end
impressions.mapreduce.conf.file.full.path /workflows/impressions/config/aggregations.conf<http://thv01:8888/filebrowser/view/workflows/impressions/config/aggregations.conf>
mapred.job.queue.name<http://mapred.job.queue.name> default
mapred.mapper.new-api true
mapred.reducer.new-api true
mapreduce.input.fileinputformat.inputdir /data/input/impressionlogs/outpdirlogs/9999-99-99<http://thv01:8888/filebrowser/view/data/input/impressionlogs/outpdirlogs/9999-99-99>
mapreduce.job.inputformat.class com.wizecommerce.utils.mapred.ZipMultipleLineRecordInputFormat
mapreduce.job.map.class com.wizecommerce.parser.mapred.OutpdirImpressionLogMapper
mapreduce.job.maps 500
mapreduce.job.name<http://mapreduce.job.name> OutpdirImpressions_0000475-130611151004460-oozie-oozi-W
mapreduce.job.output.value.class org.apache.hadoop.io.Text
mapreduce.job.outputformat.class com.wizecommerce.utils.mapred.NextagTextOutputFormat
mapreduce.job.reduce.class com.wizecommerce.parser.mapred.OutpdirImpressionLogReducer
mapreduce.job.reduces 8
mapreduce.map.output.compress true
mapreduce.map.output.compress.codec org.apache.hadoop.io.compress.SnappyCodec
mapreduce.map.output.key.class org.apache.hadoop.io.Textorg.apache.hadoop.io.Text
mapreduce.map.output.value.class com.wizecommerce.parser.dao.OutpdirLogRecord
mapreduce.output.fileoutputformat.compress true
mapreduce.output.fileoutputformat.compress.codec com.hadoop.compression.lzo.LzopCodec
mapreduce.output.fileoutputformat.outputdir /data/output/impressions/outpdir/9999-99-99/0000475-130611151004460-oozie-oozi-W/outpdir_impressions_ptitle<http://thv01:8888/filebrowser/view/data/output/impressions/outpdir/9999-99-99/0000475-130611151004460-oozie-oozi-W/outpdir_impressions_ptitle>
mapreduce.tasktracker.map.tasks.maximum 12
mapreduce.tasktracker.reduce.tasks.maximum 8
outpdir.log.exclude.processing.datatypes header,sellerhidden
CONFIDENTIALITY NOTICE
======================
This email message and any attachments are for the exclusive use of the intended recipient(s) and may contain confidential and privileged information. Any unauthorized review, use, disclosure or distribution is prohibited. If you are not the intended recipient, please contact the sender by reply email and destroy all copies of the original message along with any attachments, from your computer system. If you are the intended recipient, please be advised that the content of this message is subject to access, review and disclosure by the sender's Email System Administrator.
CONFIDENTIALITY NOTICE
======================
This email message and any attachments are for the exclusive use of the intended recipient(s) and may contain confidential and privileged information. Any unauthorized review, use, disclosure or distribution is prohibited. If you are not the intended recipient, please contact the sender by reply email and destroy all copies of the original message along with any attachments, from your computer system. If you are the intended recipient, please be advised that the content of this message is subject to access, review and disclosure by the sender's Email System Administrator.
Re: EBADF: Bad file descriptor
Posted by Sanjay Subramanian <Sa...@wizecommerce.com>.
Thanks
I will look into logs to see if I see anything else…
sanjay
From: Colin McCabe <cm...@alumni.cmu.edu>>
Reply-To: "user@hadoop.apache.org<ma...@hadoop.apache.org>" <us...@hadoop.apache.org>>
Date: Wednesday, July 10, 2013 11:52 AM
To: "user@hadoop.apache.org<ma...@hadoop.apache.org>" <us...@hadoop.apache.org>>
Subject: Re: EBADF: Bad file descriptor
To clarify a little bit, the readahead pool can sometimes spit out this message if you close a file while a readahead request is in flight. It's not an error and just reflects the fact that the file was closed hastily, probably because of some other bug which is the real problem.
Colin
On Wed, Jul 10, 2013 at 11:50 AM, Colin McCabe <cm...@alumni.cmu.edu>> wrote:
That's just a warning message. It's not causing your problem-- it's just a symptom.
You will have to find out why the MR job failed.
best,
Colin
On Wed, Jul 10, 2013 at 8:19 AM, Sanjay Subramanian <Sa...@wizecommerce.com>> wrote:
2013-07-10 07:11:50,131 WARN [Readahead Thread #1] org.apache.hadoop.io.ReadaheadPool: Failed readahead on ifile
EBADF: Bad file descriptor
at org.apache.hadoop.io.nativeio.NativeIO.posix_fadvise(Native Method)
at org.apache.hadoop.io.nativeio.NativeIO.posixFadviseIfPossible(NativeIO.java:145)
at org.apache.hadoop.io.ReadaheadPool$ReadaheadRequestImpl.run(ReadaheadPool.java:205)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
Hi
I have a Oozie workflow that runs a MR job and I have started getting this error past two days in one of the MR jobs that is being processed.
However if I run it again , it succeeds :-( but about 1 hr is wasted in the process.
Any clues ?
Or should I post this issue in the Oozie postings ?
Thanks
sanjay
Configuration
Name Value
impression.log.record.cached.tag cached=
impression.log.record.end.tag [end
impressions.mapreduce.conf.file.full.path /workflows/impressions/config/aggregations.conf<http://thv01:8888/filebrowser/view/workflows/impressions/config/aggregations.conf>
mapred.job.queue.name<http://mapred.job.queue.name> default
mapred.mapper.new-api true
mapred.reducer.new-api true
mapreduce.input.fileinputformat.inputdir /data/input/impressionlogs/outpdirlogs/9999-99-99<http://thv01:8888/filebrowser/view/data/input/impressionlogs/outpdirlogs/9999-99-99>
mapreduce.job.inputformat.class com.wizecommerce.utils.mapred.ZipMultipleLineRecordInputFormat
mapreduce.job.map.class com.wizecommerce.parser.mapred.OutpdirImpressionLogMapper
mapreduce.job.maps 500
mapreduce.job.name<http://mapreduce.job.name> OutpdirImpressions_0000475-130611151004460-oozie-oozi-W
mapreduce.job.output.value.class org.apache.hadoop.io.Text
mapreduce.job.outputformat.class com.wizecommerce.utils.mapred.NextagTextOutputFormat
mapreduce.job.reduce.class com.wizecommerce.parser.mapred.OutpdirImpressionLogReducer
mapreduce.job.reduces 8
mapreduce.map.output.compress true
mapreduce.map.output.compress.codec org.apache.hadoop.io.compress.SnappyCodec
mapreduce.map.output.key.class org.apache.hadoop.io.Textorg.apache.hadoop.io.Text
mapreduce.map.output.value.class com.wizecommerce.parser.dao.OutpdirLogRecord
mapreduce.output.fileoutputformat.compress true
mapreduce.output.fileoutputformat.compress.codec com.hadoop.compression.lzo.LzopCodec
mapreduce.output.fileoutputformat.outputdir /data/output/impressions/outpdir/9999-99-99/0000475-130611151004460-oozie-oozi-W/outpdir_impressions_ptitle<http://thv01:8888/filebrowser/view/data/output/impressions/outpdir/9999-99-99/0000475-130611151004460-oozie-oozi-W/outpdir_impressions_ptitle>
mapreduce.tasktracker.map.tasks.maximum 12
mapreduce.tasktracker.reduce.tasks.maximum 8
outpdir.log.exclude.processing.datatypes header,sellerhidden
CONFIDENTIALITY NOTICE
======================
This email message and any attachments are for the exclusive use of the intended recipient(s) and may contain confidential and privileged information. Any unauthorized review, use, disclosure or distribution is prohibited. If you are not the intended recipient, please contact the sender by reply email and destroy all copies of the original message along with any attachments, from your computer system. If you are the intended recipient, please be advised that the content of this message is subject to access, review and disclosure by the sender's Email System Administrator.
CONFIDENTIALITY NOTICE
======================
This email message and any attachments are for the exclusive use of the intended recipient(s) and may contain confidential and privileged information. Any unauthorized review, use, disclosure or distribution is prohibited. If you are not the intended recipient, please contact the sender by reply email and destroy all copies of the original message along with any attachments, from your computer system. If you are the intended recipient, please be advised that the content of this message is subject to access, review and disclosure by the sender's Email System Administrator.
Re: EBADF: Bad file descriptor
Posted by Colin McCabe <cm...@alumni.cmu.edu>.
To clarify a little bit, the readahead pool can sometimes spit out this
message if you close a file while a readahead request is in flight. It's
not an error and just reflects the fact that the file was closed hastily,
probably because of some other bug which is the real problem.
Colin
On Wed, Jul 10, 2013 at 11:50 AM, Colin McCabe <cm...@alumni.cmu.edu>wrote:
> That's just a warning message. It's not causing your problem-- it's just
> a symptom.
>
> You will have to find out why the MR job failed.
>
> best,
> Colin
>
>
> On Wed, Jul 10, 2013 at 8:19 AM, Sanjay Subramanian <
> Sanjay.Subramanian@wizecommerce.com> wrote:
>
>> 2013-07-10 07:11:50,131 WARN [Readahead Thread #1]
>> org.apache.hadoop.io.ReadaheadPool: Failed readahead on ifile
>> EBADF: Bad file descriptor
>> at org.apache.hadoop.io.nativeio.NativeIO.posix_fadvise(Native Method)
>> at
>> org.apache.hadoop.io.nativeio.NativeIO.posixFadviseIfPossible(NativeIO.java:145)
>> at
>> org.apache.hadoop.io.ReadaheadPool$ReadaheadRequestImpl.run(ReadaheadPool.java:205)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>> at java.lang.Thread.run(Thread.java:662)
>>
>> Hi
>>
>> I have a Oozie workflow that runs a MR job and I have started getting
>> this error past two days in one of the MR jobs that is being processed.
>> However if I run it again , it succeeds :-( but about 1 hr is wasted in
>> the process.
>>
>> Any clues ?
>>
>> Or should I post this issue in the Oozie postings ?
>>
>> Thanks
>>
>> sanjay
>>
>> Configuration Name Value impression.log.record.cached.tag cached=
>> impression.log.record.end.tag [end
>> impressions.mapreduce.conf.file.full.path
>> /workflows/impressions/config/aggregations.conf<http://thv01:8888/filebrowser/view/workflows/impressions/config/aggregations.conf>
>> mapred.job.queue.name default mapred.mapper.new-api true
>> mapred.reducer.new-api true mapreduce.input.fileinputformat.inputdir
>> /data/input/impressionlogs/outpdirlogs/9999-99-99<http://thv01:8888/filebrowser/view/data/input/impressionlogs/outpdirlogs/9999-99-99>
>> mapreduce.job.inputformat.class
>> com.wizecommerce.utils.mapred.ZipMultipleLineRecordInputFormat
>> mapreduce.job.map.class
>> com.wizecommerce.parser.mapred.OutpdirImpressionLogMapper
>> mapreduce.job.maps 500 mapreduce.job.name
>> OutpdirImpressions_0000475-130611151004460-oozie-oozi-W
>> mapreduce.job.output.value.class org.apache.hadoop.io.Text
>> mapreduce.job.outputformat.class
>> com.wizecommerce.utils.mapred.NextagTextOutputFormat
>> mapreduce.job.reduce.class
>> com.wizecommerce.parser.mapred.OutpdirImpressionLogReducer
>> mapreduce.job.reduces 8 mapreduce.map.output.compress true
>> mapreduce.map.output.compress.codec
>> org.apache.hadoop.io.compress.SnappyCodec mapreduce.map.output.key.class
>> org.apache.hadoop.io.Textorg.apache.hadoop.io.Text
>> mapreduce.map.output.value.class
>> com.wizecommerce.parser.dao.OutpdirLogRecord
>> mapreduce.output.fileoutputformat.compress true
>> mapreduce.output.fileoutputformat.compress.codec
>> com.hadoop.compression.lzo.LzopCodec
>> mapreduce.output.fileoutputformat.outputdir
>> /data/output/impressions/outpdir/9999-99-99/0000475-130611151004460-oozie-oozi-W/outpdir_impressions_ptitle<http://thv01:8888/filebrowser/view/data/output/impressions/outpdir/9999-99-99/0000475-130611151004460-oozie-oozi-W/outpdir_impressions_ptitle>
>> mapreduce.tasktracker.map.tasks.maximum 12
>> mapreduce.tasktracker.reduce.tasks.maximum 8
>> outpdir.log.exclude.processing.datatypes header,sellerhidden
>>
>>
>>
>> CONFIDENTIALITY NOTICE
>> ======================
>> This email message and any attachments are for the exclusive use of the
>> intended recipient(s) and may contain confidential and privileged
>> information. Any unauthorized review, use, disclosure or distribution is
>> prohibited. If you are not the intended recipient, please contact the
>> sender by reply email and destroy all copies of the original message along
>> with any attachments, from your computer system. If you are the intended
>> recipient, please be advised that the content of this message is subject to
>> access, review and disclosure by the sender's Email System Administrator.
>>
>
>
Re: EBADF: Bad file descriptor
Posted by Colin McCabe <cm...@alumni.cmu.edu>.
To clarify a little bit, the readahead pool can sometimes spit out this
message if you close a file while a readahead request is in flight. It's
not an error and just reflects the fact that the file was closed hastily,
probably because of some other bug which is the real problem.
Colin
On Wed, Jul 10, 2013 at 11:50 AM, Colin McCabe <cm...@alumni.cmu.edu>wrote:
> That's just a warning message. It's not causing your problem-- it's just
> a symptom.
>
> You will have to find out why the MR job failed.
>
> best,
> Colin
>
>
> On Wed, Jul 10, 2013 at 8:19 AM, Sanjay Subramanian <
> Sanjay.Subramanian@wizecommerce.com> wrote:
>
>> 2013-07-10 07:11:50,131 WARN [Readahead Thread #1]
>> org.apache.hadoop.io.ReadaheadPool: Failed readahead on ifile
>> EBADF: Bad file descriptor
>> at org.apache.hadoop.io.nativeio.NativeIO.posix_fadvise(Native Method)
>> at
>> org.apache.hadoop.io.nativeio.NativeIO.posixFadviseIfPossible(NativeIO.java:145)
>> at
>> org.apache.hadoop.io.ReadaheadPool$ReadaheadRequestImpl.run(ReadaheadPool.java:205)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>> at java.lang.Thread.run(Thread.java:662)
>>
>> Hi
>>
>> I have a Oozie workflow that runs a MR job and I have started getting
>> this error past two days in one of the MR jobs that is being processed.
>> However if I run it again , it succeeds :-( but about 1 hr is wasted in
>> the process.
>>
>> Any clues ?
>>
>> Or should I post this issue in the Oozie postings ?
>>
>> Thanks
>>
>> sanjay
>>
>> Configuration Name Value impression.log.record.cached.tag cached=
>> impression.log.record.end.tag [end
>> impressions.mapreduce.conf.file.full.path
>> /workflows/impressions/config/aggregations.conf<http://thv01:8888/filebrowser/view/workflows/impressions/config/aggregations.conf>
>> mapred.job.queue.name default mapred.mapper.new-api true
>> mapred.reducer.new-api true mapreduce.input.fileinputformat.inputdir
>> /data/input/impressionlogs/outpdirlogs/9999-99-99<http://thv01:8888/filebrowser/view/data/input/impressionlogs/outpdirlogs/9999-99-99>
>> mapreduce.job.inputformat.class
>> com.wizecommerce.utils.mapred.ZipMultipleLineRecordInputFormat
>> mapreduce.job.map.class
>> com.wizecommerce.parser.mapred.OutpdirImpressionLogMapper
>> mapreduce.job.maps 500 mapreduce.job.name
>> OutpdirImpressions_0000475-130611151004460-oozie-oozi-W
>> mapreduce.job.output.value.class org.apache.hadoop.io.Text
>> mapreduce.job.outputformat.class
>> com.wizecommerce.utils.mapred.NextagTextOutputFormat
>> mapreduce.job.reduce.class
>> com.wizecommerce.parser.mapred.OutpdirImpressionLogReducer
>> mapreduce.job.reduces 8 mapreduce.map.output.compress true
>> mapreduce.map.output.compress.codec
>> org.apache.hadoop.io.compress.SnappyCodec mapreduce.map.output.key.class
>> org.apache.hadoop.io.Textorg.apache.hadoop.io.Text
>> mapreduce.map.output.value.class
>> com.wizecommerce.parser.dao.OutpdirLogRecord
>> mapreduce.output.fileoutputformat.compress true
>> mapreduce.output.fileoutputformat.compress.codec
>> com.hadoop.compression.lzo.LzopCodec
>> mapreduce.output.fileoutputformat.outputdir
>> /data/output/impressions/outpdir/9999-99-99/0000475-130611151004460-oozie-oozi-W/outpdir_impressions_ptitle<http://thv01:8888/filebrowser/view/data/output/impressions/outpdir/9999-99-99/0000475-130611151004460-oozie-oozi-W/outpdir_impressions_ptitle>
>> mapreduce.tasktracker.map.tasks.maximum 12
>> mapreduce.tasktracker.reduce.tasks.maximum 8
>> outpdir.log.exclude.processing.datatypes header,sellerhidden
>>
>>
>>
>> CONFIDENTIALITY NOTICE
>> ======================
>> This email message and any attachments are for the exclusive use of the
>> intended recipient(s) and may contain confidential and privileged
>> information. Any unauthorized review, use, disclosure or distribution is
>> prohibited. If you are not the intended recipient, please contact the
>> sender by reply email and destroy all copies of the original message along
>> with any attachments, from your computer system. If you are the intended
>> recipient, please be advised that the content of this message is subject to
>> access, review and disclosure by the sender's Email System Administrator.
>>
>
>
Re: EBADF: Bad file descriptor
Posted by Colin McCabe <cm...@alumni.cmu.edu>.
To clarify a little bit, the readahead pool can sometimes spit out this
message if you close a file while a readahead request is in flight. It's
not an error and just reflects the fact that the file was closed hastily,
probably because of some other bug which is the real problem.
Colin
On Wed, Jul 10, 2013 at 11:50 AM, Colin McCabe <cm...@alumni.cmu.edu>wrote:
> That's just a warning message. It's not causing your problem-- it's just
> a symptom.
>
> You will have to find out why the MR job failed.
>
> best,
> Colin
>
>
> On Wed, Jul 10, 2013 at 8:19 AM, Sanjay Subramanian <
> Sanjay.Subramanian@wizecommerce.com> wrote:
>
>> 2013-07-10 07:11:50,131 WARN [Readahead Thread #1]
>> org.apache.hadoop.io.ReadaheadPool: Failed readahead on ifile
>> EBADF: Bad file descriptor
>> at org.apache.hadoop.io.nativeio.NativeIO.posix_fadvise(Native Method)
>> at
>> org.apache.hadoop.io.nativeio.NativeIO.posixFadviseIfPossible(NativeIO.java:145)
>> at
>> org.apache.hadoop.io.ReadaheadPool$ReadaheadRequestImpl.run(ReadaheadPool.java:205)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>> at java.lang.Thread.run(Thread.java:662)
>>
>> Hi
>>
>> I have a Oozie workflow that runs a MR job and I have started getting
>> this error past two days in one of the MR jobs that is being processed.
>> However if I run it again , it succeeds :-( but about 1 hr is wasted in
>> the process.
>>
>> Any clues ?
>>
>> Or should I post this issue in the Oozie postings ?
>>
>> Thanks
>>
>> sanjay
>>
>> Configuration Name Value impression.log.record.cached.tag cached=
>> impression.log.record.end.tag [end
>> impressions.mapreduce.conf.file.full.path
>> /workflows/impressions/config/aggregations.conf<http://thv01:8888/filebrowser/view/workflows/impressions/config/aggregations.conf>
>> mapred.job.queue.name default mapred.mapper.new-api true
>> mapred.reducer.new-api true mapreduce.input.fileinputformat.inputdir
>> /data/input/impressionlogs/outpdirlogs/9999-99-99<http://thv01:8888/filebrowser/view/data/input/impressionlogs/outpdirlogs/9999-99-99>
>> mapreduce.job.inputformat.class
>> com.wizecommerce.utils.mapred.ZipMultipleLineRecordInputFormat
>> mapreduce.job.map.class
>> com.wizecommerce.parser.mapred.OutpdirImpressionLogMapper
>> mapreduce.job.maps 500 mapreduce.job.name
>> OutpdirImpressions_0000475-130611151004460-oozie-oozi-W
>> mapreduce.job.output.value.class org.apache.hadoop.io.Text
>> mapreduce.job.outputformat.class
>> com.wizecommerce.utils.mapred.NextagTextOutputFormat
>> mapreduce.job.reduce.class
>> com.wizecommerce.parser.mapred.OutpdirImpressionLogReducer
>> mapreduce.job.reduces 8 mapreduce.map.output.compress true
>> mapreduce.map.output.compress.codec
>> org.apache.hadoop.io.compress.SnappyCodec mapreduce.map.output.key.class
>> org.apache.hadoop.io.Textorg.apache.hadoop.io.Text
>> mapreduce.map.output.value.class
>> com.wizecommerce.parser.dao.OutpdirLogRecord
>> mapreduce.output.fileoutputformat.compress true
>> mapreduce.output.fileoutputformat.compress.codec
>> com.hadoop.compression.lzo.LzopCodec
>> mapreduce.output.fileoutputformat.outputdir
>> /data/output/impressions/outpdir/9999-99-99/0000475-130611151004460-oozie-oozi-W/outpdir_impressions_ptitle<http://thv01:8888/filebrowser/view/data/output/impressions/outpdir/9999-99-99/0000475-130611151004460-oozie-oozi-W/outpdir_impressions_ptitle>
>> mapreduce.tasktracker.map.tasks.maximum 12
>> mapreduce.tasktracker.reduce.tasks.maximum 8
>> outpdir.log.exclude.processing.datatypes header,sellerhidden
>>
>>
>>
>> CONFIDENTIALITY NOTICE
>> ======================
>> This email message and any attachments are for the exclusive use of the
>> intended recipient(s) and may contain confidential and privileged
>> information. Any unauthorized review, use, disclosure or distribution is
>> prohibited. If you are not the intended recipient, please contact the
>> sender by reply email and destroy all copies of the original message along
>> with any attachments, from your computer system. If you are the intended
>> recipient, please be advised that the content of this message is subject to
>> access, review and disclosure by the sender's Email System Administrator.
>>
>
>
Re: EBADF: Bad file descriptor
Posted by Colin McCabe <cm...@alumni.cmu.edu>.
To clarify a little bit, the readahead pool can sometimes spit out this
message if you close a file while a readahead request is in flight. It's
not an error and just reflects the fact that the file was closed hastily,
probably because of some other bug which is the real problem.
Colin
On Wed, Jul 10, 2013 at 11:50 AM, Colin McCabe <cm...@alumni.cmu.edu>wrote:
> That's just a warning message. It's not causing your problem-- it's just
> a symptom.
>
> You will have to find out why the MR job failed.
>
> best,
> Colin
>
>
> On Wed, Jul 10, 2013 at 8:19 AM, Sanjay Subramanian <
> Sanjay.Subramanian@wizecommerce.com> wrote:
>
>> 2013-07-10 07:11:50,131 WARN [Readahead Thread #1]
>> org.apache.hadoop.io.ReadaheadPool: Failed readahead on ifile
>> EBADF: Bad file descriptor
>> at org.apache.hadoop.io.nativeio.NativeIO.posix_fadvise(Native Method)
>> at
>> org.apache.hadoop.io.nativeio.NativeIO.posixFadviseIfPossible(NativeIO.java:145)
>> at
>> org.apache.hadoop.io.ReadaheadPool$ReadaheadRequestImpl.run(ReadaheadPool.java:205)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>> at java.lang.Thread.run(Thread.java:662)
>>
>> Hi
>>
>> I have a Oozie workflow that runs a MR job and I have started getting
>> this error past two days in one of the MR jobs that is being processed.
>> However if I run it again , it succeeds :-( but about 1 hr is wasted in
>> the process.
>>
>> Any clues ?
>>
>> Or should I post this issue in the Oozie postings ?
>>
>> Thanks
>>
>> sanjay
>>
>> Configuration Name Value impression.log.record.cached.tag cached=
>> impression.log.record.end.tag [end
>> impressions.mapreduce.conf.file.full.path
>> /workflows/impressions/config/aggregations.conf<http://thv01:8888/filebrowser/view/workflows/impressions/config/aggregations.conf>
>> mapred.job.queue.name default mapred.mapper.new-api true
>> mapred.reducer.new-api true mapreduce.input.fileinputformat.inputdir
>> /data/input/impressionlogs/outpdirlogs/9999-99-99<http://thv01:8888/filebrowser/view/data/input/impressionlogs/outpdirlogs/9999-99-99>
>> mapreduce.job.inputformat.class
>> com.wizecommerce.utils.mapred.ZipMultipleLineRecordInputFormat
>> mapreduce.job.map.class
>> com.wizecommerce.parser.mapred.OutpdirImpressionLogMapper
>> mapreduce.job.maps 500 mapreduce.job.name
>> OutpdirImpressions_0000475-130611151004460-oozie-oozi-W
>> mapreduce.job.output.value.class org.apache.hadoop.io.Text
>> mapreduce.job.outputformat.class
>> com.wizecommerce.utils.mapred.NextagTextOutputFormat
>> mapreduce.job.reduce.class
>> com.wizecommerce.parser.mapred.OutpdirImpressionLogReducer
>> mapreduce.job.reduces 8 mapreduce.map.output.compress true
>> mapreduce.map.output.compress.codec
>> org.apache.hadoop.io.compress.SnappyCodec mapreduce.map.output.key.class
>> org.apache.hadoop.io.Textorg.apache.hadoop.io.Text
>> mapreduce.map.output.value.class
>> com.wizecommerce.parser.dao.OutpdirLogRecord
>> mapreduce.output.fileoutputformat.compress true
>> mapreduce.output.fileoutputformat.compress.codec
>> com.hadoop.compression.lzo.LzopCodec
>> mapreduce.output.fileoutputformat.outputdir
>> /data/output/impressions/outpdir/9999-99-99/0000475-130611151004460-oozie-oozi-W/outpdir_impressions_ptitle<http://thv01:8888/filebrowser/view/data/output/impressions/outpdir/9999-99-99/0000475-130611151004460-oozie-oozi-W/outpdir_impressions_ptitle>
>> mapreduce.tasktracker.map.tasks.maximum 12
>> mapreduce.tasktracker.reduce.tasks.maximum 8
>> outpdir.log.exclude.processing.datatypes header,sellerhidden
>>
>>
>>
>> CONFIDENTIALITY NOTICE
>> ======================
>> This email message and any attachments are for the exclusive use of the
>> intended recipient(s) and may contain confidential and privileged
>> information. Any unauthorized review, use, disclosure or distribution is
>> prohibited. If you are not the intended recipient, please contact the
>> sender by reply email and destroy all copies of the original message along
>> with any attachments, from your computer system. If you are the intended
>> recipient, please be advised that the content of this message is subject to
>> access, review and disclosure by the sender's Email System Administrator.
>>
>
>
Re: EBADF: Bad file descriptor
Posted by Colin McCabe <cm...@alumni.cmu.edu>.
That's just a warning message. It's not causing your problem-- it's just a
symptom.
You will have to find out why the MR job failed.
best,
Colin
On Wed, Jul 10, 2013 at 8:19 AM, Sanjay Subramanian <
Sanjay.Subramanian@wizecommerce.com> wrote:
> 2013-07-10 07:11:50,131 WARN [Readahead Thread #1]
> org.apache.hadoop.io.ReadaheadPool: Failed readahead on ifile
> EBADF: Bad file descriptor
> at org.apache.hadoop.io.nativeio.NativeIO.posix_fadvise(Native Method)
> at
> org.apache.hadoop.io.nativeio.NativeIO.posixFadviseIfPossible(NativeIO.java:145)
> at
> org.apache.hadoop.io.ReadaheadPool$ReadaheadRequestImpl.run(ReadaheadPool.java:205)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
>
> Hi
>
> I have a Oozie workflow that runs a MR job and I have started getting
> this error past two days in one of the MR jobs that is being processed.
> However if I run it again , it succeeds :-( but about 1 hr is wasted in
> the process.
>
> Any clues ?
>
> Or should I post this issue in the Oozie postings ?
>
> Thanks
>
> sanjay
>
> Configuration Name Value impression.log.record.cached.tag cached=
> impression.log.record.end.tag [end
> impressions.mapreduce.conf.file.full.path
> /workflows/impressions/config/aggregations.conf<http://thv01:8888/filebrowser/view/workflows/impressions/config/aggregations.conf>
> mapred.job.queue.name default mapred.mapper.new-api true
> mapred.reducer.new-api true mapreduce.input.fileinputformat.inputdir
> /data/input/impressionlogs/outpdirlogs/9999-99-99<http://thv01:8888/filebrowser/view/data/input/impressionlogs/outpdirlogs/9999-99-99>
> mapreduce.job.inputformat.class
> com.wizecommerce.utils.mapred.ZipMultipleLineRecordInputFormat
> mapreduce.job.map.class
> com.wizecommerce.parser.mapred.OutpdirImpressionLogMapper
> mapreduce.job.maps 500 mapreduce.job.name
> OutpdirImpressions_0000475-130611151004460-oozie-oozi-W
> mapreduce.job.output.value.class org.apache.hadoop.io.Text
> mapreduce.job.outputformat.class
> com.wizecommerce.utils.mapred.NextagTextOutputFormat
> mapreduce.job.reduce.class
> com.wizecommerce.parser.mapred.OutpdirImpressionLogReducer
> mapreduce.job.reduces 8 mapreduce.map.output.compress true
> mapreduce.map.output.compress.codec
> org.apache.hadoop.io.compress.SnappyCodec mapreduce.map.output.key.class
> org.apache.hadoop.io.Textorg.apache.hadoop.io.Text
> mapreduce.map.output.value.class
> com.wizecommerce.parser.dao.OutpdirLogRecord
> mapreduce.output.fileoutputformat.compress true
> mapreduce.output.fileoutputformat.compress.codec
> com.hadoop.compression.lzo.LzopCodec
> mapreduce.output.fileoutputformat.outputdir
> /data/output/impressions/outpdir/9999-99-99/0000475-130611151004460-oozie-oozi-W/outpdir_impressions_ptitle<http://thv01:8888/filebrowser/view/data/output/impressions/outpdir/9999-99-99/0000475-130611151004460-oozie-oozi-W/outpdir_impressions_ptitle>
> mapreduce.tasktracker.map.tasks.maximum 12
> mapreduce.tasktracker.reduce.tasks.maximum 8
> outpdir.log.exclude.processing.datatypes header,sellerhidden
>
>
>
> CONFIDENTIALITY NOTICE
> ======================
> This email message and any attachments are for the exclusive use of the
> intended recipient(s) and may contain confidential and privileged
> information. Any unauthorized review, use, disclosure or distribution is
> prohibited. If you are not the intended recipient, please contact the
> sender by reply email and destroy all copies of the original message along
> with any attachments, from your computer system. If you are the intended
> recipient, please be advised that the content of this message is subject to
> access, review and disclosure by the sender's Email System Administrator.
>
Re: EBADF: Bad file descriptor
Posted by Colin McCabe <cm...@alumni.cmu.edu>.
That's just a warning message. It's not causing your problem-- it's just a
symptom.
You will have to find out why the MR job failed.
best,
Colin
On Wed, Jul 10, 2013 at 8:19 AM, Sanjay Subramanian <
Sanjay.Subramanian@wizecommerce.com> wrote:
> 2013-07-10 07:11:50,131 WARN [Readahead Thread #1]
> org.apache.hadoop.io.ReadaheadPool: Failed readahead on ifile
> EBADF: Bad file descriptor
> at org.apache.hadoop.io.nativeio.NativeIO.posix_fadvise(Native Method)
> at
> org.apache.hadoop.io.nativeio.NativeIO.posixFadviseIfPossible(NativeIO.java:145)
> at
> org.apache.hadoop.io.ReadaheadPool$ReadaheadRequestImpl.run(ReadaheadPool.java:205)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
>
> Hi
>
> I have a Oozie workflow that runs a MR job and I have started getting
> this error past two days in one of the MR jobs that is being processed.
> However if I run it again , it succeeds :-( but about 1 hr is wasted in
> the process.
>
> Any clues ?
>
> Or should I post this issue in the Oozie postings ?
>
> Thanks
>
> sanjay
>
> Configuration Name Value impression.log.record.cached.tag cached=
> impression.log.record.end.tag [end
> impressions.mapreduce.conf.file.full.path
> /workflows/impressions/config/aggregations.conf<http://thv01:8888/filebrowser/view/workflows/impressions/config/aggregations.conf>
> mapred.job.queue.name default mapred.mapper.new-api true
> mapred.reducer.new-api true mapreduce.input.fileinputformat.inputdir
> /data/input/impressionlogs/outpdirlogs/9999-99-99<http://thv01:8888/filebrowser/view/data/input/impressionlogs/outpdirlogs/9999-99-99>
> mapreduce.job.inputformat.class
> com.wizecommerce.utils.mapred.ZipMultipleLineRecordInputFormat
> mapreduce.job.map.class
> com.wizecommerce.parser.mapred.OutpdirImpressionLogMapper
> mapreduce.job.maps 500 mapreduce.job.name
> OutpdirImpressions_0000475-130611151004460-oozie-oozi-W
> mapreduce.job.output.value.class org.apache.hadoop.io.Text
> mapreduce.job.outputformat.class
> com.wizecommerce.utils.mapred.NextagTextOutputFormat
> mapreduce.job.reduce.class
> com.wizecommerce.parser.mapred.OutpdirImpressionLogReducer
> mapreduce.job.reduces 8 mapreduce.map.output.compress true
> mapreduce.map.output.compress.codec
> org.apache.hadoop.io.compress.SnappyCodec mapreduce.map.output.key.class
> org.apache.hadoop.io.Textorg.apache.hadoop.io.Text
> mapreduce.map.output.value.class
> com.wizecommerce.parser.dao.OutpdirLogRecord
> mapreduce.output.fileoutputformat.compress true
> mapreduce.output.fileoutputformat.compress.codec
> com.hadoop.compression.lzo.LzopCodec
> mapreduce.output.fileoutputformat.outputdir
> /data/output/impressions/outpdir/9999-99-99/0000475-130611151004460-oozie-oozi-W/outpdir_impressions_ptitle<http://thv01:8888/filebrowser/view/data/output/impressions/outpdir/9999-99-99/0000475-130611151004460-oozie-oozi-W/outpdir_impressions_ptitle>
> mapreduce.tasktracker.map.tasks.maximum 12
> mapreduce.tasktracker.reduce.tasks.maximum 8
> outpdir.log.exclude.processing.datatypes header,sellerhidden
>
>
>
> CONFIDENTIALITY NOTICE
> ======================
> This email message and any attachments are for the exclusive use of the
> intended recipient(s) and may contain confidential and privileged
> information. Any unauthorized review, use, disclosure or distribution is
> prohibited. If you are not the intended recipient, please contact the
> sender by reply email and destroy all copies of the original message along
> with any attachments, from your computer system. If you are the intended
> recipient, please be advised that the content of this message is subject to
> access, review and disclosure by the sender's Email System Administrator.
>
Re: EBADF: Bad file descriptor
Posted by Colin McCabe <cm...@alumni.cmu.edu>.
That's just a warning message. It's not causing your problem-- it's just a
symptom.
You will have to find out why the MR job failed.
best,
Colin
On Wed, Jul 10, 2013 at 8:19 AM, Sanjay Subramanian <
Sanjay.Subramanian@wizecommerce.com> wrote:
> 2013-07-10 07:11:50,131 WARN [Readahead Thread #1]
> org.apache.hadoop.io.ReadaheadPool: Failed readahead on ifile
> EBADF: Bad file descriptor
> at org.apache.hadoop.io.nativeio.NativeIO.posix_fadvise(Native Method)
> at
> org.apache.hadoop.io.nativeio.NativeIO.posixFadviseIfPossible(NativeIO.java:145)
> at
> org.apache.hadoop.io.ReadaheadPool$ReadaheadRequestImpl.run(ReadaheadPool.java:205)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
>
> Hi
>
> I have a Oozie workflow that runs a MR job and I have started getting
> this error past two days in one of the MR jobs that is being processed.
> However if I run it again , it succeeds :-( but about 1 hr is wasted in
> the process.
>
> Any clues ?
>
> Or should I post this issue in the Oozie postings ?
>
> Thanks
>
> sanjay
>
> Configuration Name Value impression.log.record.cached.tag cached=
> impression.log.record.end.tag [end
> impressions.mapreduce.conf.file.full.path
> /workflows/impressions/config/aggregations.conf<http://thv01:8888/filebrowser/view/workflows/impressions/config/aggregations.conf>
> mapred.job.queue.name default mapred.mapper.new-api true
> mapred.reducer.new-api true mapreduce.input.fileinputformat.inputdir
> /data/input/impressionlogs/outpdirlogs/9999-99-99<http://thv01:8888/filebrowser/view/data/input/impressionlogs/outpdirlogs/9999-99-99>
> mapreduce.job.inputformat.class
> com.wizecommerce.utils.mapred.ZipMultipleLineRecordInputFormat
> mapreduce.job.map.class
> com.wizecommerce.parser.mapred.OutpdirImpressionLogMapper
> mapreduce.job.maps 500 mapreduce.job.name
> OutpdirImpressions_0000475-130611151004460-oozie-oozi-W
> mapreduce.job.output.value.class org.apache.hadoop.io.Text
> mapreduce.job.outputformat.class
> com.wizecommerce.utils.mapred.NextagTextOutputFormat
> mapreduce.job.reduce.class
> com.wizecommerce.parser.mapred.OutpdirImpressionLogReducer
> mapreduce.job.reduces 8 mapreduce.map.output.compress true
> mapreduce.map.output.compress.codec
> org.apache.hadoop.io.compress.SnappyCodec mapreduce.map.output.key.class
> org.apache.hadoop.io.Textorg.apache.hadoop.io.Text
> mapreduce.map.output.value.class
> com.wizecommerce.parser.dao.OutpdirLogRecord
> mapreduce.output.fileoutputformat.compress true
> mapreduce.output.fileoutputformat.compress.codec
> com.hadoop.compression.lzo.LzopCodec
> mapreduce.output.fileoutputformat.outputdir
> /data/output/impressions/outpdir/9999-99-99/0000475-130611151004460-oozie-oozi-W/outpdir_impressions_ptitle<http://thv01:8888/filebrowser/view/data/output/impressions/outpdir/9999-99-99/0000475-130611151004460-oozie-oozi-W/outpdir_impressions_ptitle>
> mapreduce.tasktracker.map.tasks.maximum 12
> mapreduce.tasktracker.reduce.tasks.maximum 8
> outpdir.log.exclude.processing.datatypes header,sellerhidden
>
>
>
> CONFIDENTIALITY NOTICE
> ======================
> This email message and any attachments are for the exclusive use of the
> intended recipient(s) and may contain confidential and privileged
> information. Any unauthorized review, use, disclosure or distribution is
> prohibited. If you are not the intended recipient, please contact the
> sender by reply email and destroy all copies of the original message along
> with any attachments, from your computer system. If you are the intended
> recipient, please be advised that the content of this message is subject to
> access, review and disclosure by the sender's Email System Administrator.
>
Re: EBADF: Bad file descriptor
Posted by Colin McCabe <cm...@alumni.cmu.edu>.
That's just a warning message. It's not causing your problem-- it's just a
symptom.
You will have to find out why the MR job failed.
best,
Colin
On Wed, Jul 10, 2013 at 8:19 AM, Sanjay Subramanian <
Sanjay.Subramanian@wizecommerce.com> wrote:
> 2013-07-10 07:11:50,131 WARN [Readahead Thread #1]
> org.apache.hadoop.io.ReadaheadPool: Failed readahead on ifile
> EBADF: Bad file descriptor
> at org.apache.hadoop.io.nativeio.NativeIO.posix_fadvise(Native Method)
> at
> org.apache.hadoop.io.nativeio.NativeIO.posixFadviseIfPossible(NativeIO.java:145)
> at
> org.apache.hadoop.io.ReadaheadPool$ReadaheadRequestImpl.run(ReadaheadPool.java:205)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
>
> Hi
>
> I have a Oozie workflow that runs a MR job and I have started getting
> this error past two days in one of the MR jobs that is being processed.
> However if I run it again , it succeeds :-( but about 1 hr is wasted in
> the process.
>
> Any clues ?
>
> Or should I post this issue in the Oozie postings ?
>
> Thanks
>
> sanjay
>
> Configuration Name Value impression.log.record.cached.tag cached=
> impression.log.record.end.tag [end
> impressions.mapreduce.conf.file.full.path
> /workflows/impressions/config/aggregations.conf<http://thv01:8888/filebrowser/view/workflows/impressions/config/aggregations.conf>
> mapred.job.queue.name default mapred.mapper.new-api true
> mapred.reducer.new-api true mapreduce.input.fileinputformat.inputdir
> /data/input/impressionlogs/outpdirlogs/9999-99-99<http://thv01:8888/filebrowser/view/data/input/impressionlogs/outpdirlogs/9999-99-99>
> mapreduce.job.inputformat.class
> com.wizecommerce.utils.mapred.ZipMultipleLineRecordInputFormat
> mapreduce.job.map.class
> com.wizecommerce.parser.mapred.OutpdirImpressionLogMapper
> mapreduce.job.maps 500 mapreduce.job.name
> OutpdirImpressions_0000475-130611151004460-oozie-oozi-W
> mapreduce.job.output.value.class org.apache.hadoop.io.Text
> mapreduce.job.outputformat.class
> com.wizecommerce.utils.mapred.NextagTextOutputFormat
> mapreduce.job.reduce.class
> com.wizecommerce.parser.mapred.OutpdirImpressionLogReducer
> mapreduce.job.reduces 8 mapreduce.map.output.compress true
> mapreduce.map.output.compress.codec
> org.apache.hadoop.io.compress.SnappyCodec mapreduce.map.output.key.class
> org.apache.hadoop.io.Textorg.apache.hadoop.io.Text
> mapreduce.map.output.value.class
> com.wizecommerce.parser.dao.OutpdirLogRecord
> mapreduce.output.fileoutputformat.compress true
> mapreduce.output.fileoutputformat.compress.codec
> com.hadoop.compression.lzo.LzopCodec
> mapreduce.output.fileoutputformat.outputdir
> /data/output/impressions/outpdir/9999-99-99/0000475-130611151004460-oozie-oozi-W/outpdir_impressions_ptitle<http://thv01:8888/filebrowser/view/data/output/impressions/outpdir/9999-99-99/0000475-130611151004460-oozie-oozi-W/outpdir_impressions_ptitle>
> mapreduce.tasktracker.map.tasks.maximum 12
> mapreduce.tasktracker.reduce.tasks.maximum 8
> outpdir.log.exclude.processing.datatypes header,sellerhidden
>
>
>
> CONFIDENTIALITY NOTICE
> ======================
> This email message and any attachments are for the exclusive use of the
> intended recipient(s) and may contain confidential and privileged
> information. Any unauthorized review, use, disclosure or distribution is
> prohibited. If you are not the intended recipient, please contact the
> sender by reply email and destroy all copies of the original message along
> with any attachments, from your computer system. If you are the intended
> recipient, please be advised that the content of this message is subject to
> access, review and disclosure by the sender's Email System Administrator.
>