You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hive.apache.org by "Johan Oskarsson (JIRA)" <ji...@apache.org> on 2009/01/07 20:06:44 UTC
[jira] Created: (HIVE-217) Stream closed exception
Stream closed exception
-----------------------
Key: HIVE-217
URL: https://issues.apache.org/jira/browse/HIVE-217
Project: Hadoop Hive
Issue Type: Bug
Components: Serializers/Deserializers
Environment: Hive from trunk, hadoop 0.18.2, ~20 machines
Reporter: Johan Oskarsson
Priority: Critical
When running a query similar to the following:
"insert overwrite table outputtable select a, b, cast(sum(counter) as INT) from tablea join tableb on (tablea.username=tableb.username) join tablec on (tablec.userid = tablea.userid) join tabled on (tablec.id=tabled.id) where insertdate >= 'somedate' and insertdate <= 'someotherdate' group by a, b;"
Where one table is ~40gb or so and the others are a couple of hundred mb. The error happens in the first mapred job that processes the 40gb.
I get the following exception (see attached file for full stack trace):
Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: java.io.IOException: Stream closed.
at org.apache.hadoop.hive.ql.exec.FileSinkOperator.process(FileSinkOperator.java:162)
It happens in one reduce task and is reproducible, running the same query gives the error.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HIVE-217) Stream closed exception
Posted by "Joydeep Sen Sarma (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-217?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12663788#action_12663788 ]
Joydeep Sen Sarma commented on HIVE-217:
----------------------------------------
+1
thanks Johan!
> Stream closed exception
> -----------------------
>
> Key: HIVE-217
> URL: https://issues.apache.org/jira/browse/HIVE-217
> Project: Hadoop Hive
> Issue Type: Bug
> Components: Query Processor
> Environment: Hive from trunk, hadoop 0.18.2, ~20 machines
> Reporter: Johan Oskarsson
> Priority: Critical
> Fix For: 0.2.0
>
> Attachments: HIVE-217.log, HIVE-217.patch, HIVE-217.patch
>
>
> When running a query similar to the following:
> "insert overwrite table outputtable select a, b, cast(sum(counter) as INT) from tablea join tableb on (tablea.username=tableb.username) join tablec on (tablec.userid = tablea.userid) join tabled on (tablec.id=tabled.id) where insertdate >= 'somedate' and insertdate <= 'someotherdate' group by a, b;"
> Where one table is ~40gb or so and the others are a couple of hundred mb. The error happens in the first mapred job that processes the 40gb.
> I get the following exception (see attached file for full stack trace):
> Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: java.io.IOException: Stream closed.
> at org.apache.hadoop.hive.ql.exec.FileSinkOperator.process(FileSinkOperator.java:162)
> It happens in one reduce task and is reproducible, running the same query gives the error.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HIVE-217) Stream closed exception
Posted by "Johan Oskarsson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-217?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12662074#action_12662074 ]
Johan Oskarsson commented on HIVE-217:
--------------------------------------
Looking into this a bit further the exception is actually caused by the reduce task timing out:
"Task attempt_200901071012_0373_r_000031_0 failed to report status for 616 seconds. Killing!"
This in turn closes the FileSystem/stream and the FileSink keeps trying to write, causing the exception seen in the log.
Now the question is why the task fails to report status.
> Stream closed exception
> -----------------------
>
> Key: HIVE-217
> URL: https://issues.apache.org/jira/browse/HIVE-217
> Project: Hadoop Hive
> Issue Type: Bug
> Components: Serializers/Deserializers
> Environment: Hive from trunk, hadoop 0.18.2, ~20 machines
> Reporter: Johan Oskarsson
> Priority: Critical
> Fix For: 0.2.0
>
> Attachments: HIVE-217.log
>
>
> When running a query similar to the following:
> "insert overwrite table outputtable select a, b, cast(sum(counter) as INT) from tablea join tableb on (tablea.username=tableb.username) join tablec on (tablec.userid = tablea.userid) join tabled on (tablec.id=tabled.id) where insertdate >= 'somedate' and insertdate <= 'someotherdate' group by a, b;"
> Where one table is ~40gb or so and the others are a couple of hundred mb. The error happens in the first mapred job that processes the 40gb.
> I get the following exception (see attached file for full stack trace):
> Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: java.io.IOException: Stream closed.
> at org.apache.hadoop.hive.ql.exec.FileSinkOperator.process(FileSinkOperator.java:162)
> It happens in one reduce task and is reproducible, running the same query gives the error.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HIVE-217) Stream closed exception
Posted by "Johan Oskarsson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-217?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12662081#action_12662081 ]
Johan Oskarsson commented on HIVE-217:
--------------------------------------
The big table in the example query above is tablec, is this what you were referring to?
I'm using Hadoop 0.18.2.
> Stream closed exception
> -----------------------
>
> Key: HIVE-217
> URL: https://issues.apache.org/jira/browse/HIVE-217
> Project: Hadoop Hive
> Issue Type: Bug
> Components: Serializers/Deserializers
> Environment: Hive from trunk, hadoop 0.18.2, ~20 machines
> Reporter: Johan Oskarsson
> Priority: Critical
> Fix For: 0.2.0
>
> Attachments: HIVE-217.log
>
>
> When running a query similar to the following:
> "insert overwrite table outputtable select a, b, cast(sum(counter) as INT) from tablea join tableb on (tablea.username=tableb.username) join tablec on (tablec.userid = tablea.userid) join tabled on (tablec.id=tabled.id) where insertdate >= 'somedate' and insertdate <= 'someotherdate' group by a, b;"
> Where one table is ~40gb or so and the others are a couple of hundred mb. The error happens in the first mapred job that processes the 40gb.
> I get the following exception (see attached file for full stack trace):
> Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: java.io.IOException: Stream closed.
> at org.apache.hadoop.hive.ql.exec.FileSinkOperator.process(FileSinkOperator.java:162)
> It happens in one reduce task and is reproducible, running the same query gives the error.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HIVE-217) Stream closed exception
Posted by "Joydeep Sen Sarma (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-217?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Joydeep Sen Sarma updated HIVE-217:
-----------------------------------
Component/s: (was: Serializers/Deserializers)
Query Processor
> Stream closed exception
> -----------------------
>
> Key: HIVE-217
> URL: https://issues.apache.org/jira/browse/HIVE-217
> Project: Hadoop Hive
> Issue Type: Bug
> Components: Query Processor
> Environment: Hive from trunk, hadoop 0.18.2, ~20 machines
> Reporter: Johan Oskarsson
> Priority: Critical
> Fix For: 0.2.0
>
> Attachments: HIVE-217.log, HIVE-217.patch
>
>
> When running a query similar to the following:
> "insert overwrite table outputtable select a, b, cast(sum(counter) as INT) from tablea join tableb on (tablea.username=tableb.username) join tablec on (tablec.userid = tablea.userid) join tabled on (tablec.id=tabled.id) where insertdate >= 'somedate' and insertdate <= 'someotherdate' group by a, b;"
> Where one table is ~40gb or so and the others are a couple of hundred mb. The error happens in the first mapred job that processes the 40gb.
> I get the following exception (see attached file for full stack trace):
> Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: java.io.IOException: Stream closed.
> at org.apache.hadoop.hive.ql.exec.FileSinkOperator.process(FileSinkOperator.java:162)
> It happens in one reduce task and is reproducible, running the same query gives the error.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HIVE-217) Stream closed exception
Posted by "Johan Oskarsson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-217?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12661993#action_12661993 ]
Johan Oskarsson commented on HIVE-217:
--------------------------------------
Added some debug logging to the close and process methods in FileSinkOperator and the close method is not called before the process method fails as one could guess from the stack trace.
> Stream closed exception
> -----------------------
>
> Key: HIVE-217
> URL: https://issues.apache.org/jira/browse/HIVE-217
> Project: Hadoop Hive
> Issue Type: Bug
> Components: Serializers/Deserializers
> Environment: Hive from trunk, hadoop 0.18.2, ~20 machines
> Reporter: Johan Oskarsson
> Priority: Critical
> Fix For: 0.2.0
>
> Attachments: HIVE-217.log
>
>
> When running a query similar to the following:
> "insert overwrite table outputtable select a, b, cast(sum(counter) as INT) from tablea join tableb on (tablea.username=tableb.username) join tablec on (tablec.userid = tablea.userid) join tabled on (tablec.id=tabled.id) where insertdate >= 'somedate' and insertdate <= 'someotherdate' group by a, b;"
> Where one table is ~40gb or so and the others are a couple of hundred mb. The error happens in the first mapred job that processes the 40gb.
> I get the following exception (see attached file for full stack trace):
> Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: java.io.IOException: Stream closed.
> at org.apache.hadoop.hive.ql.exec.FileSinkOperator.process(FileSinkOperator.java:162)
> It happens in one reduce task and is reproducible, running the same query gives the error.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HIVE-217) Stream closed exception
Posted by "Johan Oskarsson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-217?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12662085#action_12662085 ]
Johan Oskarsson commented on HIVE-217:
--------------------------------------
Just had a look and it doesn't seem like the reducer that fails gets any more data to it compared to the others.
> Stream closed exception
> -----------------------
>
> Key: HIVE-217
> URL: https://issues.apache.org/jira/browse/HIVE-217
> Project: Hadoop Hive
> Issue Type: Bug
> Components: Serializers/Deserializers
> Environment: Hive from trunk, hadoop 0.18.2, ~20 machines
> Reporter: Johan Oskarsson
> Priority: Critical
> Fix For: 0.2.0
>
> Attachments: HIVE-217.log
>
>
> When running a query similar to the following:
> "insert overwrite table outputtable select a, b, cast(sum(counter) as INT) from tablea join tableb on (tablea.username=tableb.username) join tablec on (tablec.userid = tablea.userid) join tabled on (tablec.id=tabled.id) where insertdate >= 'somedate' and insertdate <= 'someotherdate' group by a, b;"
> Where one table is ~40gb or so and the others are a couple of hundred mb. The error happens in the first mapred job that processes the 40gb.
> I get the following exception (see attached file for full stack trace):
> Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: java.io.IOException: Stream closed.
> at org.apache.hadoop.hive.ql.exec.FileSinkOperator.process(FileSinkOperator.java:162)
> It happens in one reduce task and is reproducible, running the same query gives the error.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HIVE-217) Stream closed exception
Posted by "Johan Oskarsson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-217?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12662303#action_12662303 ]
Johan Oskarsson commented on HIVE-217:
--------------------------------------
That sounds like a very reasonable explanation. We have fewer machines meaning fewer disks in total then Facebook, that could explain why we're seeing this error.
Out of curiosity, how come you're not using the standard output collector?
> Stream closed exception
> -----------------------
>
> Key: HIVE-217
> URL: https://issues.apache.org/jira/browse/HIVE-217
> Project: Hadoop Hive
> Issue Type: Bug
> Components: Serializers/Deserializers
> Environment: Hive from trunk, hadoop 0.18.2, ~20 machines
> Reporter: Johan Oskarsson
> Priority: Critical
> Fix For: 0.2.0
>
> Attachments: HIVE-217.log
>
>
> When running a query similar to the following:
> "insert overwrite table outputtable select a, b, cast(sum(counter) as INT) from tablea join tableb on (tablea.username=tableb.username) join tablec on (tablec.userid = tablea.userid) join tabled on (tablec.id=tabled.id) where insertdate >= 'somedate' and insertdate <= 'someotherdate' group by a, b;"
> Where one table is ~40gb or so and the others are a couple of hundred mb. The error happens in the first mapred job that processes the 40gb.
> I get the following exception (see attached file for full stack trace):
> Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: java.io.IOException: Stream closed.
> at org.apache.hadoop.hive.ql.exec.FileSinkOperator.process(FileSinkOperator.java:162)
> It happens in one reduce task and is reproducible, running the same query gives the error.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HIVE-217) Stream closed exception
Posted by "Johan Oskarsson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-217?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Johan Oskarsson updated HIVE-217:
---------------------------------
Attachment: HIVE-217.patch
Updated with the changes mentioned.
> Stream closed exception
> -----------------------
>
> Key: HIVE-217
> URL: https://issues.apache.org/jira/browse/HIVE-217
> Project: Hadoop Hive
> Issue Type: Bug
> Components: Query Processor
> Environment: Hive from trunk, hadoop 0.18.2, ~20 machines
> Reporter: Johan Oskarsson
> Priority: Critical
> Fix For: 0.2.0
>
> Attachments: HIVE-217.log, HIVE-217.patch, HIVE-217.patch
>
>
> When running a query similar to the following:
> "insert overwrite table outputtable select a, b, cast(sum(counter) as INT) from tablea join tableb on (tablea.username=tableb.username) join tablec on (tablec.userid = tablea.userid) join tabled on (tablec.id=tabled.id) where insertdate >= 'somedate' and insertdate <= 'someotherdate' group by a, b;"
> Where one table is ~40gb or so and the others are a couple of hundred mb. The error happens in the first mapred job that processes the 40gb.
> I get the following exception (see attached file for full stack trace):
> Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: java.io.IOException: Stream closed.
> at org.apache.hadoop.hive.ql.exec.FileSinkOperator.process(FileSinkOperator.java:162)
> It happens in one reduce task and is reproducible, running the same query gives the error.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HIVE-217) Stream closed exception
Posted by "Johan Oskarsson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-217?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Johan Oskarsson updated HIVE-217:
---------------------------------
Attachment: HIVE-217.patch
To verify that this was indeed the issue I have slapped together a patch. Tried it on the same cluster and the same query as above and it worked fine this time. It's not an ideal patch, it changes the initialize method etc. I'll leave a real solution up to someone with more detailed knowledge about this area.
> Stream closed exception
> -----------------------
>
> Key: HIVE-217
> URL: https://issues.apache.org/jira/browse/HIVE-217
> Project: Hadoop Hive
> Issue Type: Bug
> Components: Serializers/Deserializers
> Environment: Hive from trunk, hadoop 0.18.2, ~20 machines
> Reporter: Johan Oskarsson
> Priority: Critical
> Fix For: 0.2.0
>
> Attachments: HIVE-217.log, HIVE-217.patch
>
>
> When running a query similar to the following:
> "insert overwrite table outputtable select a, b, cast(sum(counter) as INT) from tablea join tableb on (tablea.username=tableb.username) join tablec on (tablec.userid = tablea.userid) join tabled on (tablec.id=tabled.id) where insertdate >= 'somedate' and insertdate <= 'someotherdate' group by a, b;"
> Where one table is ~40gb or so and the others are a couple of hundred mb. The error happens in the first mapred job that processes the 40gb.
> I get the following exception (see attached file for full stack trace):
> Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: java.io.IOException: Stream closed.
> at org.apache.hadoop.hive.ql.exec.FileSinkOperator.process(FileSinkOperator.java:162)
> It happens in one reduce task and is reproducible, running the same query gives the error.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HIVE-217) Stream closed exception
Posted by "Johan Oskarsson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-217?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Johan Oskarsson updated HIVE-217:
---------------------------------
Attachment: HIVE-217.log
The full stack trace
> Stream closed exception
> -----------------------
>
> Key: HIVE-217
> URL: https://issues.apache.org/jira/browse/HIVE-217
> Project: Hadoop Hive
> Issue Type: Bug
> Components: Serializers/Deserializers
> Environment: Hive from trunk, hadoop 0.18.2, ~20 machines
> Reporter: Johan Oskarsson
> Priority: Critical
> Attachments: HIVE-217.log
>
>
> When running a query similar to the following:
> "insert overwrite table outputtable select a, b, cast(sum(counter) as INT) from tablea join tableb on (tablea.username=tableb.username) join tablec on (tablec.userid = tablea.userid) join tabled on (tablec.id=tabled.id) where insertdate >= 'somedate' and insertdate <= 'someotherdate' group by a, b;"
> Where one table is ~40gb or so and the others are a couple of hundred mb. The error happens in the first mapred job that processes the 40gb.
> I get the following exception (see attached file for full stack trace):
> Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: java.io.IOException: Stream closed.
> at org.apache.hadoop.hive.ql.exec.FileSinkOperator.process(FileSinkOperator.java:162)
> It happens in one reduce task and is reproducible, running the same query gives the error.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HIVE-217) Stream closed exception
Posted by "Joydeep Sen Sarma (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-217?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12662078#action_12662078 ]
Joydeep Sen Sarma commented on HIVE-217:
----------------------------------------
one question - which side was the big table on?
regardless though - perhaps the issue is that when we iterate through large reduce groups - hadoop is not reporting progress - maybe that's something that the application has to do.
what version of hadoop are u using?
> Stream closed exception
> -----------------------
>
> Key: HIVE-217
> URL: https://issues.apache.org/jira/browse/HIVE-217
> Project: Hadoop Hive
> Issue Type: Bug
> Components: Serializers/Deserializers
> Environment: Hive from trunk, hadoop 0.18.2, ~20 machines
> Reporter: Johan Oskarsson
> Priority: Critical
> Fix For: 0.2.0
>
> Attachments: HIVE-217.log
>
>
> When running a query similar to the following:
> "insert overwrite table outputtable select a, b, cast(sum(counter) as INT) from tablea join tableb on (tablea.username=tableb.username) join tablec on (tablec.userid = tablea.userid) join tabled on (tablec.id=tabled.id) where insertdate >= 'somedate' and insertdate <= 'someotherdate' group by a, b;"
> Where one table is ~40gb or so and the others are a couple of hundred mb. The error happens in the first mapred job that processes the 40gb.
> I get the following exception (see attached file for full stack trace):
> Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: java.io.IOException: Stream closed.
> at org.apache.hadoop.hive.ql.exec.FileSinkOperator.process(FileSinkOperator.java:162)
> It happens in one reduce task and is reproducible, running the same query gives the error.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HIVE-217) Stream closed exception
Posted by "Joydeep Sen Sarma (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-217?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12663535#action_12663535 ]
Joydeep Sen Sarma commented on HIVE-217:
----------------------------------------
For the change to get the reporter reference into the operator structure - the interface change looks good. However - why don't we just store the reporter reference in the base Operator class rather than the FileSinkOperator specifically? If we run into other cases where we have to add progress indicators - this will make it easier.
+1 otherwise.
> Stream closed exception
> -----------------------
>
> Key: HIVE-217
> URL: https://issues.apache.org/jira/browse/HIVE-217
> Project: Hadoop Hive
> Issue Type: Bug
> Components: Query Processor
> Environment: Hive from trunk, hadoop 0.18.2, ~20 machines
> Reporter: Johan Oskarsson
> Priority: Critical
> Fix For: 0.2.0
>
> Attachments: HIVE-217.log, HIVE-217.patch
>
>
> When running a query similar to the following:
> "insert overwrite table outputtable select a, b, cast(sum(counter) as INT) from tablea join tableb on (tablea.username=tableb.username) join tablec on (tablec.userid = tablea.userid) join tabled on (tablec.id=tabled.id) where insertdate >= 'somedate' and insertdate <= 'someotherdate' group by a, b;"
> Where one table is ~40gb or so and the others are a couple of hundred mb. The error happens in the first mapred job that processes the 40gb.
> I get the following exception (see attached file for full stack trace):
> Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: java.io.IOException: Stream closed.
> at org.apache.hadoop.hive.ql.exec.FileSinkOperator.process(FileSinkOperator.java:162)
> It happens in one reduce task and is reproducible, running the same query gives the error.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HIVE-217) Stream closed exception
Posted by "Johan Oskarsson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-217?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12661666#action_12661666 ]
Johan Oskarsson commented on HIVE-217:
--------------------------------------
It is also worth mentioning that the namenode was under hardly any load when this happened. I have not seen a reproducible problem like this with normal map reduce jobs on that cluster.
> Stream closed exception
> -----------------------
>
> Key: HIVE-217
> URL: https://issues.apache.org/jira/browse/HIVE-217
> Project: Hadoop Hive
> Issue Type: Bug
> Components: Serializers/Deserializers
> Environment: Hive from trunk, hadoop 0.18.2, ~20 machines
> Reporter: Johan Oskarsson
> Priority: Critical
> Attachments: HIVE-217.log
>
>
> When running a query similar to the following:
> "insert overwrite table outputtable select a, b, cast(sum(counter) as INT) from tablea join tableb on (tablea.username=tableb.username) join tablec on (tablec.userid = tablea.userid) join tabled on (tablec.id=tabled.id) where insertdate >= 'somedate' and insertdate <= 'someotherdate' group by a, b;"
> Where one table is ~40gb or so and the others are a couple of hundred mb. The error happens in the first mapred job that processes the 40gb.
> I get the following exception (see attached file for full stack trace):
> Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: java.io.IOException: Stream closed.
> at org.apache.hadoop.hive.ql.exec.FileSinkOperator.process(FileSinkOperator.java:162)
> It happens in one reduce task and is reproducible, running the same query gives the error.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HIVE-217) Stream closed exception
Posted by "Johan Oskarsson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-217?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Johan Oskarsson updated HIVE-217:
---------------------------------
Fix Version/s: 0.2.0
> Stream closed exception
> -----------------------
>
> Key: HIVE-217
> URL: https://issues.apache.org/jira/browse/HIVE-217
> Project: Hadoop Hive
> Issue Type: Bug
> Components: Serializers/Deserializers
> Environment: Hive from trunk, hadoop 0.18.2, ~20 machines
> Reporter: Johan Oskarsson
> Priority: Critical
> Fix For: 0.2.0
>
> Attachments: HIVE-217.log
>
>
> When running a query similar to the following:
> "insert overwrite table outputtable select a, b, cast(sum(counter) as INT) from tablea join tableb on (tablea.username=tableb.username) join tablec on (tablec.userid = tablea.userid) join tabled on (tablec.id=tabled.id) where insertdate >= 'somedate' and insertdate <= 'someotherdate' group by a, b;"
> Where one table is ~40gb or so and the others are a couple of hundred mb. The error happens in the first mapred job that processes the 40gb.
> I get the following exception (see attached file for full stack trace):
> Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: java.io.IOException: Stream closed.
> at org.apache.hadoop.hive.ql.exec.FileSinkOperator.process(FileSinkOperator.java:162)
> It happens in one reduce task and is reproducible, running the same query gives the error.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HIVE-217) Stream closed exception
Posted by "Carl Steinbach (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-217?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Carl Steinbach updated HIVE-217:
--------------------------------
Fix Version/s: 0.3.0
(was: 0.6.0)
> Stream closed exception
> -----------------------
>
> Key: HIVE-217
> URL: https://issues.apache.org/jira/browse/HIVE-217
> Project: Hadoop Hive
> Issue Type: Bug
> Components: Query Processor
> Environment: Hive from trunk, hadoop 0.18.2, ~20 machines
> Reporter: Johan Oskarsson
> Assignee: Johan Oskarsson
> Priority: Critical
> Fix For: 0.3.0
>
> Attachments: HIVE-217.log, HIVE-217.patch, HIVE-217.patch
>
>
> When running a query similar to the following:
> "insert overwrite table outputtable select a, b, cast(sum(counter) as INT) from tablea join tableb on (tablea.username=tableb.username) join tablec on (tablec.userid = tablea.userid) join tabled on (tablec.id=tabled.id) where insertdate >= 'somedate' and insertdate <= 'someotherdate' group by a, b;"
> Where one table is ~40gb or so and the others are a couple of hundred mb. The error happens in the first mapred job that processes the 40gb.
> I get the following exception (see attached file for full stack trace):
> Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: java.io.IOException: Stream closed.
> at org.apache.hadoop.hive.ql.exec.FileSinkOperator.process(FileSinkOperator.java:162)
> It happens in one reduce task and is reproducible, running the same query gives the error.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HIVE-217) Stream closed exception
Posted by "Joydeep Sen Sarma (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-217?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12662111#action_12662111 ]
Joydeep Sen Sarma commented on HIVE-217:
----------------------------------------
ok - got it. hadoop is ok - it's already reporting progress whenever any data is consumed or any data is emitted to output collector.
the issue is that we are not sending data to the output collector - rather we write out data to file system ourselves. the stack trace indicates that we have consumed the entire reduce group and we are writing to the filesystem. that means that hadoop gets no opportunity to report progress (it would have reported progress if we were writing to the output collector).
I am not sure why we haven't seen the problem in our environment yet - perhaps DFS IO is slow in ur environment. the fix is simple (we can report progress either in the filesinkoperator or the joinoperator). i think we should do this in the joinoperator (for the particular case where we have already gone through the entire reduce group) - since that's the only place we are vulnerable right now.
i can post a patch u can try out in a couple of hours ..
> Stream closed exception
> -----------------------
>
> Key: HIVE-217
> URL: https://issues.apache.org/jira/browse/HIVE-217
> Project: Hadoop Hive
> Issue Type: Bug
> Components: Serializers/Deserializers
> Environment: Hive from trunk, hadoop 0.18.2, ~20 machines
> Reporter: Johan Oskarsson
> Priority: Critical
> Fix For: 0.2.0
>
> Attachments: HIVE-217.log
>
>
> When running a query similar to the following:
> "insert overwrite table outputtable select a, b, cast(sum(counter) as INT) from tablea join tableb on (tablea.username=tableb.username) join tablec on (tablec.userid = tablea.userid) join tabled on (tablec.id=tabled.id) where insertdate >= 'somedate' and insertdate <= 'someotherdate' group by a, b;"
> Where one table is ~40gb or so and the others are a couple of hundred mb. The error happens in the first mapred job that processes the 40gb.
> I get the following exception (see attached file for full stack trace):
> Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: java.io.IOException: Stream closed.
> at org.apache.hadoop.hive.ql.exec.FileSinkOperator.process(FileSinkOperator.java:162)
> It happens in one reduce task and is reproducible, running the same query gives the error.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HIVE-217) Stream closed exception
Posted by "Johan Oskarsson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-217?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12665786#action_12665786 ]
Johan Oskarsson commented on HIVE-217:
--------------------------------------
If one of the committers have time to look at the patch it would be much appreciated, thanks.
> Stream closed exception
> -----------------------
>
> Key: HIVE-217
> URL: https://issues.apache.org/jira/browse/HIVE-217
> Project: Hadoop Hive
> Issue Type: Bug
> Components: Query Processor
> Environment: Hive from trunk, hadoop 0.18.2, ~20 machines
> Reporter: Johan Oskarsson
> Priority: Critical
> Fix For: 0.2.0
>
> Attachments: HIVE-217.log, HIVE-217.patch, HIVE-217.patch
>
>
> When running a query similar to the following:
> "insert overwrite table outputtable select a, b, cast(sum(counter) as INT) from tablea join tableb on (tablea.username=tableb.username) join tablec on (tablec.userid = tablea.userid) join tabled on (tablec.id=tabled.id) where insertdate >= 'somedate' and insertdate <= 'someotherdate' group by a, b;"
> Where one table is ~40gb or so and the others are a couple of hundred mb. The error happens in the first mapred job that processes the 40gb.
> I get the following exception (see attached file for full stack trace):
> Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: java.io.IOException: Stream closed.
> at org.apache.hadoop.hive.ql.exec.FileSinkOperator.process(FileSinkOperator.java:162)
> It happens in one reduce task and is reproducible, running the same query gives the error.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Resolved: (HIVE-217) Stream closed exception
Posted by "Ashish Thusoo (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-217?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Ashish Thusoo resolved HIVE-217.
--------------------------------
Resolution: Fixed
Assignee: Johan Oskarsson
Hadoop Flags: [Reviewed]
committed. Thanks Johan!!
> Stream closed exception
> -----------------------
>
> Key: HIVE-217
> URL: https://issues.apache.org/jira/browse/HIVE-217
> Project: Hadoop Hive
> Issue Type: Bug
> Components: Query Processor
> Environment: Hive from trunk, hadoop 0.18.2, ~20 machines
> Reporter: Johan Oskarsson
> Assignee: Johan Oskarsson
> Priority: Critical
> Fix For: 0.2.0
>
> Attachments: HIVE-217.log, HIVE-217.patch, HIVE-217.patch
>
>
> When running a query similar to the following:
> "insert overwrite table outputtable select a, b, cast(sum(counter) as INT) from tablea join tableb on (tablea.username=tableb.username) join tablec on (tablec.userid = tablea.userid) join tabled on (tablec.id=tabled.id) where insertdate >= 'somedate' and insertdate <= 'someotherdate' group by a, b;"
> Where one table is ~40gb or so and the others are a couple of hundred mb. The error happens in the first mapred job that processes the 40gb.
> I get the following exception (see attached file for full stack trace):
> Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: java.io.IOException: Stream closed.
> at org.apache.hadoop.hive.ql.exec.FileSinkOperator.process(FileSinkOperator.java:162)
> It happens in one reduce task and is reproducible, running the same query gives the error.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.