You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hive.apache.org by "Michael Klatt (JIRA)" <ji...@apache.org> on 2010/04/25 22:51:49 UTC
[jira] Created: (HIVE-1326) RowContainer uses hard-coded '/tmp/'
path for temporary files
RowContainer uses hard-coded '/tmp/' path for temporary files
-------------------------------------------------------------
Key: HIVE-1326
URL: https://issues.apache.org/jira/browse/HIVE-1326
Project: Hadoop Hive
Issue Type: Bug
Environment: Hadoop 0.19.2 with Hive trunk. We're using FreeBSD 7.0, but that doesn't seem relevant.
Reporter: Michael Klatt
In our production hadoop environment, the "/tmp/" is actually pretty small, and we encountered a problem when a query used the RowContainer class and filled up the /tmp/ partition. I tracked down the cause to the RowContainer class putting temporary files in the '/tmp/' path instead of using the configured Hadoop temporary path. I've attached a patch to fix this.
Here's the traceback:
2010-04-25 12:05:05,120 INFO org.apache.hadoop.hive.ql.exec.persistence.RowContainer: RowContainer created temp file /tmp/hive-rowcontainer-1244151903/RowContainer7816.tmp
2010-04-25 12:05:06,326 INFO ExecReducer: ExecReducer: processing 10000000 rows: used memory = 385520312
2010-04-25 12:05:08,513 INFO ExecReducer: ExecReducer: processing 11000000 rows: used memory = 341780472
2010-04-25 12:05:10,697 INFO ExecReducer: ExecReducer: processing 12000000 rows: used memory = 301446768
2010-04-25 12:05:12,837 INFO ExecReducer: ExecReducer: processing 13000000 rows: used memory = 399208768
2010-04-25 12:05:15,085 INFO ExecReducer: ExecReducer: processing 14000000 rows: used memory = 364507216
2010-04-25 12:05:17,260 INFO ExecReducer: ExecReducer: processing 15000000 rows: used memory = 332907280
2010-04-25 12:05:19,580 INFO ExecReducer: ExecReducer: processing 16000000 rows: used memory = 298774096
2010-04-25 12:05:21,629 INFO ExecReducer: ExecReducer: processing 17000000 rows: used memory = 396505408
2010-04-25 12:05:23,830 INFO ExecReducer: ExecReducer: processing 18000000 rows: used memory = 362477288
2010-04-25 12:05:25,914 INFO ExecReducer: ExecReducer: processing 19000000 rows: used memory = 327229744
2010-04-25 12:05:27,978 INFO ExecReducer: ExecReducer: processing 20000000 rows: used memory = 296051904
2010-04-25 12:05:28,155 FATAL ExecReducer: org.apache.hadoop.fs.FSError: java.io.IOException: No space left on device
at org.apache.hadoop.fs.RawLocalFileSystem$LocalFSFileOutputStream.write(RawLocalFileSystem.java:199)
at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:65)
at java.io.BufferedOutputStream.write(BufferedOutputStream.java:109)
at org.apache.hadoop.fs.FSDataOutputStream$PositionCache.write(FSDataOutputStream.java:49)
at java.io.DataOutputStream.write(DataOutputStream.java:90)
at org.apache.hadoop.fs.ChecksumFileSystem$ChecksumFSOutputSummer.writeChunk(ChecksumFileSystem.java:346)
at org.apache.hadoop.fs.FSOutputSummer.writeChecksumChunk(FSOutputSummer.java:150)
at org.apache.hadoop.fs.FSOutputSummer.flushBuffer(FSOutputSummer.java:132)
at org.apache.hadoop.fs.FSOutputSummer.flushBuffer(FSOutputSummer.java:121)
at org.apache.hadoop.fs.FSOutputSummer.write1(FSOutputSummer.java:112)
at org.apache.hadoop.fs.FSOutputSummer.write(FSOutputSummer.java:86)
at org.apache.hadoop.fs.FSDataOutputStream$PositionCache.write(FSDataOutputStream.java:49)
at java.io.DataOutputStream.write(DataOutputStream.java:90)
at org.apache.hadoop.io.SequenceFile$Writer.append(SequenceFile.java:1013)
at org.apache.hadoop.io.SequenceFile$Writer.append(SequenceFile.java:977)
at org.apache.hadoop.hive.ql.io.HiveSequenceFileOutputFormat$1.write(HiveSequenceFileOutputFormat.java:70)
at org.apache.hadoop.hive.ql.exec.persistence.RowContainer.spillBlock(RowContainer.java:343)
at org.apache.hadoop.hive.ql.exec.persistence.RowContainer.add(RowContainer.java:163)
at org.apache.hadoop.hive.ql.exec.JoinOperator.processOp(JoinOperator.java:118)
at org.apache.hadoop.hive.ql.exec.Operator.process(Operator.java:456)
at org.apache.hadoop.hive.ql.exec.ExecReducer.reduce(ExecReducer.java:244)
at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:436)
at org.apache.hadoop.mapred.Child.main(Child.java:158)
Caused by: java.io.IOException: No space left on device
at java.io.FileOutputStream.writeBytes(Native Method)
at java.io.FileOutputStream.write(FileOutputStream.java:260)
at org.apache.hadoop.fs.RawLocalFileSystem$LocalFSFileOutputStream.write(RawLocalFileSystem.java:197)
... 22 more
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HIVE-1326) RowContainer uses hard-coded '/tmp/'
path for temporary files
Posted by "Michael Klatt (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-1326?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Michael Klatt updated HIVE-1326:
--------------------------------
Attachment: rowcontainer_v2.patch
> RowContainer uses hard-coded '/tmp/' path for temporary files
> -------------------------------------------------------------
>
> Key: HIVE-1326
> URL: https://issues.apache.org/jira/browse/HIVE-1326
> Project: Hadoop Hive
> Issue Type: Bug
> Environment: Hadoop 0.19.2 with Hive trunk. We're using FreeBSD 7.0, but that doesn't seem relevant.
> Reporter: Michael Klatt
> Attachments: rowcontainer.patch, rowcontainer_v2.patch
>
>
> In our production hadoop environment, the "/tmp/" is actually pretty small, and we encountered a problem when a query used the RowContainer class and filled up the /tmp/ partition. I tracked down the cause to the RowContainer class putting temporary files in the '/tmp/' path instead of using the configured Hadoop temporary path. I've attached a patch to fix this.
> Here's the traceback:
> 2010-04-25 12:05:05,120 INFO org.apache.hadoop.hive.ql.exec.persistence.RowContainer: RowContainer created temp file /tmp/hive-rowcontainer-1244151903/RowContainer7816.tmp
> 2010-04-25 12:05:06,326 INFO ExecReducer: ExecReducer: processing 10000000 rows: used memory = 385520312
> 2010-04-25 12:05:08,513 INFO ExecReducer: ExecReducer: processing 11000000 rows: used memory = 341780472
> 2010-04-25 12:05:10,697 INFO ExecReducer: ExecReducer: processing 12000000 rows: used memory = 301446768
> 2010-04-25 12:05:12,837 INFO ExecReducer: ExecReducer: processing 13000000 rows: used memory = 399208768
> 2010-04-25 12:05:15,085 INFO ExecReducer: ExecReducer: processing 14000000 rows: used memory = 364507216
> 2010-04-25 12:05:17,260 INFO ExecReducer: ExecReducer: processing 15000000 rows: used memory = 332907280
> 2010-04-25 12:05:19,580 INFO ExecReducer: ExecReducer: processing 16000000 rows: used memory = 298774096
> 2010-04-25 12:05:21,629 INFO ExecReducer: ExecReducer: processing 17000000 rows: used memory = 396505408
> 2010-04-25 12:05:23,830 INFO ExecReducer: ExecReducer: processing 18000000 rows: used memory = 362477288
> 2010-04-25 12:05:25,914 INFO ExecReducer: ExecReducer: processing 19000000 rows: used memory = 327229744
> 2010-04-25 12:05:27,978 INFO ExecReducer: ExecReducer: processing 20000000 rows: used memory = 296051904
> 2010-04-25 12:05:28,155 FATAL ExecReducer: org.apache.hadoop.fs.FSError: java.io.IOException: No space left on device
> at org.apache.hadoop.fs.RawLocalFileSystem$LocalFSFileOutputStream.write(RawLocalFileSystem.java:199)
> at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:65)
> at java.io.BufferedOutputStream.write(BufferedOutputStream.java:109)
> at org.apache.hadoop.fs.FSDataOutputStream$PositionCache.write(FSDataOutputStream.java:49)
> at java.io.DataOutputStream.write(DataOutputStream.java:90)
> at org.apache.hadoop.fs.ChecksumFileSystem$ChecksumFSOutputSummer.writeChunk(ChecksumFileSystem.java:346)
> at org.apache.hadoop.fs.FSOutputSummer.writeChecksumChunk(FSOutputSummer.java:150)
> at org.apache.hadoop.fs.FSOutputSummer.flushBuffer(FSOutputSummer.java:132)
> at org.apache.hadoop.fs.FSOutputSummer.flushBuffer(FSOutputSummer.java:121)
> at org.apache.hadoop.fs.FSOutputSummer.write1(FSOutputSummer.java:112)
> at org.apache.hadoop.fs.FSOutputSummer.write(FSOutputSummer.java:86)
> at org.apache.hadoop.fs.FSDataOutputStream$PositionCache.write(FSDataOutputStream.java:49)
> at java.io.DataOutputStream.write(DataOutputStream.java:90)
> at org.apache.hadoop.io.SequenceFile$Writer.append(SequenceFile.java:1013)
> at org.apache.hadoop.io.SequenceFile$Writer.append(SequenceFile.java:977)
> at org.apache.hadoop.hive.ql.io.HiveSequenceFileOutputFormat$1.write(HiveSequenceFileOutputFormat.java:70)
> at org.apache.hadoop.hive.ql.exec.persistence.RowContainer.spillBlock(RowContainer.java:343)
> at org.apache.hadoop.hive.ql.exec.persistence.RowContainer.add(RowContainer.java:163)
> at org.apache.hadoop.hive.ql.exec.JoinOperator.processOp(JoinOperator.java:118)
> at org.apache.hadoop.hive.ql.exec.Operator.process(Operator.java:456)
> at org.apache.hadoop.hive.ql.exec.ExecReducer.reduce(ExecReducer.java:244)
> at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:436)
> at org.apache.hadoop.mapred.Child.main(Child.java:158)
> Caused by: java.io.IOException: No space left on device
> at java.io.FileOutputStream.writeBytes(Native Method)
> at java.io.FileOutputStream.write(FileOutputStream.java:260)
> at org.apache.hadoop.fs.RawLocalFileSystem$LocalFSFileOutputStream.write(RawLocalFileSystem.java:197)
> ... 22 more
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HIVE-1326) RowContainer uses hard-coded '/tmp/'
path for temporary files
Posted by "Michael Klatt (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-1326?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12861080#action_12861080 ]
Michael Klatt commented on HIVE-1326:
-------------------------------------
In my brief experiments, I found that File.createTempFile() used the work/tmp path for the individual task (in Hadoop). I suspect that the java.io.tmpdir is being set by the TaskTracker. This has the advantage of letting Hadoop do the temporary file cleanup when the task cleanup happens. I think it would be great if this was the default behavior, but admins could override that with a hive specific parameter.
I don't have any attachment to the actual implementation. I can spend more time to produce a different patch if needed, but I suspected someone would want to implement it differently.
> RowContainer uses hard-coded '/tmp/' path for temporary files
> -------------------------------------------------------------
>
> Key: HIVE-1326
> URL: https://issues.apache.org/jira/browse/HIVE-1326
> Project: Hadoop Hive
> Issue Type: Bug
> Environment: Hadoop 0.19.2 with Hive trunk. We're using FreeBSD 7.0, but that doesn't seem relevant.
> Reporter: Michael Klatt
> Attachments: rowcontainer.patch
>
>
> In our production hadoop environment, the "/tmp/" is actually pretty small, and we encountered a problem when a query used the RowContainer class and filled up the /tmp/ partition. I tracked down the cause to the RowContainer class putting temporary files in the '/tmp/' path instead of using the configured Hadoop temporary path. I've attached a patch to fix this.
> Here's the traceback:
> 2010-04-25 12:05:05,120 INFO org.apache.hadoop.hive.ql.exec.persistence.RowContainer: RowContainer created temp file /tmp/hive-rowcontainer-1244151903/RowContainer7816.tmp
> 2010-04-25 12:05:06,326 INFO ExecReducer: ExecReducer: processing 10000000 rows: used memory = 385520312
> 2010-04-25 12:05:08,513 INFO ExecReducer: ExecReducer: processing 11000000 rows: used memory = 341780472
> 2010-04-25 12:05:10,697 INFO ExecReducer: ExecReducer: processing 12000000 rows: used memory = 301446768
> 2010-04-25 12:05:12,837 INFO ExecReducer: ExecReducer: processing 13000000 rows: used memory = 399208768
> 2010-04-25 12:05:15,085 INFO ExecReducer: ExecReducer: processing 14000000 rows: used memory = 364507216
> 2010-04-25 12:05:17,260 INFO ExecReducer: ExecReducer: processing 15000000 rows: used memory = 332907280
> 2010-04-25 12:05:19,580 INFO ExecReducer: ExecReducer: processing 16000000 rows: used memory = 298774096
> 2010-04-25 12:05:21,629 INFO ExecReducer: ExecReducer: processing 17000000 rows: used memory = 396505408
> 2010-04-25 12:05:23,830 INFO ExecReducer: ExecReducer: processing 18000000 rows: used memory = 362477288
> 2010-04-25 12:05:25,914 INFO ExecReducer: ExecReducer: processing 19000000 rows: used memory = 327229744
> 2010-04-25 12:05:27,978 INFO ExecReducer: ExecReducer: processing 20000000 rows: used memory = 296051904
> 2010-04-25 12:05:28,155 FATAL ExecReducer: org.apache.hadoop.fs.FSError: java.io.IOException: No space left on device
> at org.apache.hadoop.fs.RawLocalFileSystem$LocalFSFileOutputStream.write(RawLocalFileSystem.java:199)
> at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:65)
> at java.io.BufferedOutputStream.write(BufferedOutputStream.java:109)
> at org.apache.hadoop.fs.FSDataOutputStream$PositionCache.write(FSDataOutputStream.java:49)
> at java.io.DataOutputStream.write(DataOutputStream.java:90)
> at org.apache.hadoop.fs.ChecksumFileSystem$ChecksumFSOutputSummer.writeChunk(ChecksumFileSystem.java:346)
> at org.apache.hadoop.fs.FSOutputSummer.writeChecksumChunk(FSOutputSummer.java:150)
> at org.apache.hadoop.fs.FSOutputSummer.flushBuffer(FSOutputSummer.java:132)
> at org.apache.hadoop.fs.FSOutputSummer.flushBuffer(FSOutputSummer.java:121)
> at org.apache.hadoop.fs.FSOutputSummer.write1(FSOutputSummer.java:112)
> at org.apache.hadoop.fs.FSOutputSummer.write(FSOutputSummer.java:86)
> at org.apache.hadoop.fs.FSDataOutputStream$PositionCache.write(FSDataOutputStream.java:49)
> at java.io.DataOutputStream.write(DataOutputStream.java:90)
> at org.apache.hadoop.io.SequenceFile$Writer.append(SequenceFile.java:1013)
> at org.apache.hadoop.io.SequenceFile$Writer.append(SequenceFile.java:977)
> at org.apache.hadoop.hive.ql.io.HiveSequenceFileOutputFormat$1.write(HiveSequenceFileOutputFormat.java:70)
> at org.apache.hadoop.hive.ql.exec.persistence.RowContainer.spillBlock(RowContainer.java:343)
> at org.apache.hadoop.hive.ql.exec.persistence.RowContainer.add(RowContainer.java:163)
> at org.apache.hadoop.hive.ql.exec.JoinOperator.processOp(JoinOperator.java:118)
> at org.apache.hadoop.hive.ql.exec.Operator.process(Operator.java:456)
> at org.apache.hadoop.hive.ql.exec.ExecReducer.reduce(ExecReducer.java:244)
> at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:436)
> at org.apache.hadoop.mapred.Child.main(Child.java:158)
> Caused by: java.io.IOException: No space left on device
> at java.io.FileOutputStream.writeBytes(Native Method)
> at java.io.FileOutputStream.write(FileOutputStream.java:260)
> at org.apache.hadoop.fs.RawLocalFileSystem$LocalFSFileOutputStream.write(RawLocalFileSystem.java:197)
> ... 22 more
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HIVE-1326) RowContainer uses hard-coded '/tmp/'
path for temporary files
Posted by "Ning Zhang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-1326?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12861188#action_12861188 ]
Ning Zhang commented on HIVE-1326:
----------------------------------
I guess this is due to https://issues.apache.org/jira/browse/HADOOP-2735 which set java.io.tmpdir to be dependent on mapred.tmp.dir. In that case the change of using createTempFile looks fine with me. Also can you remove the parentFile.delete() in your patch? If we set java.io.tmpdir to work/tmp then we don't have file name conflicts, But if the user choose to use shared /tmp then it may remove files created by other tasks.
Yongqiang, any further comments?
> RowContainer uses hard-coded '/tmp/' path for temporary files
> -------------------------------------------------------------
>
> Key: HIVE-1326
> URL: https://issues.apache.org/jira/browse/HIVE-1326
> Project: Hadoop Hive
> Issue Type: Bug
> Environment: Hadoop 0.19.2 with Hive trunk. We're using FreeBSD 7.0, but that doesn't seem relevant.
> Reporter: Michael Klatt
> Attachments: rowcontainer.patch
>
>
> In our production hadoop environment, the "/tmp/" is actually pretty small, and we encountered a problem when a query used the RowContainer class and filled up the /tmp/ partition. I tracked down the cause to the RowContainer class putting temporary files in the '/tmp/' path instead of using the configured Hadoop temporary path. I've attached a patch to fix this.
> Here's the traceback:
> 2010-04-25 12:05:05,120 INFO org.apache.hadoop.hive.ql.exec.persistence.RowContainer: RowContainer created temp file /tmp/hive-rowcontainer-1244151903/RowContainer7816.tmp
> 2010-04-25 12:05:06,326 INFO ExecReducer: ExecReducer: processing 10000000 rows: used memory = 385520312
> 2010-04-25 12:05:08,513 INFO ExecReducer: ExecReducer: processing 11000000 rows: used memory = 341780472
> 2010-04-25 12:05:10,697 INFO ExecReducer: ExecReducer: processing 12000000 rows: used memory = 301446768
> 2010-04-25 12:05:12,837 INFO ExecReducer: ExecReducer: processing 13000000 rows: used memory = 399208768
> 2010-04-25 12:05:15,085 INFO ExecReducer: ExecReducer: processing 14000000 rows: used memory = 364507216
> 2010-04-25 12:05:17,260 INFO ExecReducer: ExecReducer: processing 15000000 rows: used memory = 332907280
> 2010-04-25 12:05:19,580 INFO ExecReducer: ExecReducer: processing 16000000 rows: used memory = 298774096
> 2010-04-25 12:05:21,629 INFO ExecReducer: ExecReducer: processing 17000000 rows: used memory = 396505408
> 2010-04-25 12:05:23,830 INFO ExecReducer: ExecReducer: processing 18000000 rows: used memory = 362477288
> 2010-04-25 12:05:25,914 INFO ExecReducer: ExecReducer: processing 19000000 rows: used memory = 327229744
> 2010-04-25 12:05:27,978 INFO ExecReducer: ExecReducer: processing 20000000 rows: used memory = 296051904
> 2010-04-25 12:05:28,155 FATAL ExecReducer: org.apache.hadoop.fs.FSError: java.io.IOException: No space left on device
> at org.apache.hadoop.fs.RawLocalFileSystem$LocalFSFileOutputStream.write(RawLocalFileSystem.java:199)
> at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:65)
> at java.io.BufferedOutputStream.write(BufferedOutputStream.java:109)
> at org.apache.hadoop.fs.FSDataOutputStream$PositionCache.write(FSDataOutputStream.java:49)
> at java.io.DataOutputStream.write(DataOutputStream.java:90)
> at org.apache.hadoop.fs.ChecksumFileSystem$ChecksumFSOutputSummer.writeChunk(ChecksumFileSystem.java:346)
> at org.apache.hadoop.fs.FSOutputSummer.writeChecksumChunk(FSOutputSummer.java:150)
> at org.apache.hadoop.fs.FSOutputSummer.flushBuffer(FSOutputSummer.java:132)
> at org.apache.hadoop.fs.FSOutputSummer.flushBuffer(FSOutputSummer.java:121)
> at org.apache.hadoop.fs.FSOutputSummer.write1(FSOutputSummer.java:112)
> at org.apache.hadoop.fs.FSOutputSummer.write(FSOutputSummer.java:86)
> at org.apache.hadoop.fs.FSDataOutputStream$PositionCache.write(FSDataOutputStream.java:49)
> at java.io.DataOutputStream.write(DataOutputStream.java:90)
> at org.apache.hadoop.io.SequenceFile$Writer.append(SequenceFile.java:1013)
> at org.apache.hadoop.io.SequenceFile$Writer.append(SequenceFile.java:977)
> at org.apache.hadoop.hive.ql.io.HiveSequenceFileOutputFormat$1.write(HiveSequenceFileOutputFormat.java:70)
> at org.apache.hadoop.hive.ql.exec.persistence.RowContainer.spillBlock(RowContainer.java:343)
> at org.apache.hadoop.hive.ql.exec.persistence.RowContainer.add(RowContainer.java:163)
> at org.apache.hadoop.hive.ql.exec.JoinOperator.processOp(JoinOperator.java:118)
> at org.apache.hadoop.hive.ql.exec.Operator.process(Operator.java:456)
> at org.apache.hadoop.hive.ql.exec.ExecReducer.reduce(ExecReducer.java:244)
> at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:436)
> at org.apache.hadoop.mapred.Child.main(Child.java:158)
> Caused by: java.io.IOException: No space left on device
> at java.io.FileOutputStream.writeBytes(Native Method)
> at java.io.FileOutputStream.write(FileOutputStream.java:260)
> at org.apache.hadoop.fs.RawLocalFileSystem$LocalFSFileOutputStream.write(RawLocalFileSystem.java:197)
> ... 22 more
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HIVE-1326) RowContainer uses hard-coded '/tmp/'
path for temporary files
Posted by "Michael Klatt (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-1326?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12861444#action_12861444 ]
Michael Klatt commented on HIVE-1326:
-------------------------------------
The reason the parentFile.delete() call is there is because the File.createTempFile method actually creates a file. The code, as it currently is, creates a temporary directory to hold the rowcontainer file and I made the smallest change possible to continue to support this behavior.
Looking at the code, it appears that the createTempFile mechanism is used several lines down to actually create the temporary file (within the new temporary directory). I'm not sure why a temporary directory is created first, but I'll submit a new patch which doesn't try to create a temporary directory at all.
> RowContainer uses hard-coded '/tmp/' path for temporary files
> -------------------------------------------------------------
>
> Key: HIVE-1326
> URL: https://issues.apache.org/jira/browse/HIVE-1326
> Project: Hadoop Hive
> Issue Type: Bug
> Environment: Hadoop 0.19.2 with Hive trunk. We're using FreeBSD 7.0, but that doesn't seem relevant.
> Reporter: Michael Klatt
> Attachments: rowcontainer.patch
>
>
> In our production hadoop environment, the "/tmp/" is actually pretty small, and we encountered a problem when a query used the RowContainer class and filled up the /tmp/ partition. I tracked down the cause to the RowContainer class putting temporary files in the '/tmp/' path instead of using the configured Hadoop temporary path. I've attached a patch to fix this.
> Here's the traceback:
> 2010-04-25 12:05:05,120 INFO org.apache.hadoop.hive.ql.exec.persistence.RowContainer: RowContainer created temp file /tmp/hive-rowcontainer-1244151903/RowContainer7816.tmp
> 2010-04-25 12:05:06,326 INFO ExecReducer: ExecReducer: processing 10000000 rows: used memory = 385520312
> 2010-04-25 12:05:08,513 INFO ExecReducer: ExecReducer: processing 11000000 rows: used memory = 341780472
> 2010-04-25 12:05:10,697 INFO ExecReducer: ExecReducer: processing 12000000 rows: used memory = 301446768
> 2010-04-25 12:05:12,837 INFO ExecReducer: ExecReducer: processing 13000000 rows: used memory = 399208768
> 2010-04-25 12:05:15,085 INFO ExecReducer: ExecReducer: processing 14000000 rows: used memory = 364507216
> 2010-04-25 12:05:17,260 INFO ExecReducer: ExecReducer: processing 15000000 rows: used memory = 332907280
> 2010-04-25 12:05:19,580 INFO ExecReducer: ExecReducer: processing 16000000 rows: used memory = 298774096
> 2010-04-25 12:05:21,629 INFO ExecReducer: ExecReducer: processing 17000000 rows: used memory = 396505408
> 2010-04-25 12:05:23,830 INFO ExecReducer: ExecReducer: processing 18000000 rows: used memory = 362477288
> 2010-04-25 12:05:25,914 INFO ExecReducer: ExecReducer: processing 19000000 rows: used memory = 327229744
> 2010-04-25 12:05:27,978 INFO ExecReducer: ExecReducer: processing 20000000 rows: used memory = 296051904
> 2010-04-25 12:05:28,155 FATAL ExecReducer: org.apache.hadoop.fs.FSError: java.io.IOException: No space left on device
> at org.apache.hadoop.fs.RawLocalFileSystem$LocalFSFileOutputStream.write(RawLocalFileSystem.java:199)
> at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:65)
> at java.io.BufferedOutputStream.write(BufferedOutputStream.java:109)
> at org.apache.hadoop.fs.FSDataOutputStream$PositionCache.write(FSDataOutputStream.java:49)
> at java.io.DataOutputStream.write(DataOutputStream.java:90)
> at org.apache.hadoop.fs.ChecksumFileSystem$ChecksumFSOutputSummer.writeChunk(ChecksumFileSystem.java:346)
> at org.apache.hadoop.fs.FSOutputSummer.writeChecksumChunk(FSOutputSummer.java:150)
> at org.apache.hadoop.fs.FSOutputSummer.flushBuffer(FSOutputSummer.java:132)
> at org.apache.hadoop.fs.FSOutputSummer.flushBuffer(FSOutputSummer.java:121)
> at org.apache.hadoop.fs.FSOutputSummer.write1(FSOutputSummer.java:112)
> at org.apache.hadoop.fs.FSOutputSummer.write(FSOutputSummer.java:86)
> at org.apache.hadoop.fs.FSDataOutputStream$PositionCache.write(FSDataOutputStream.java:49)
> at java.io.DataOutputStream.write(DataOutputStream.java:90)
> at org.apache.hadoop.io.SequenceFile$Writer.append(SequenceFile.java:1013)
> at org.apache.hadoop.io.SequenceFile$Writer.append(SequenceFile.java:977)
> at org.apache.hadoop.hive.ql.io.HiveSequenceFileOutputFormat$1.write(HiveSequenceFileOutputFormat.java:70)
> at org.apache.hadoop.hive.ql.exec.persistence.RowContainer.spillBlock(RowContainer.java:343)
> at org.apache.hadoop.hive.ql.exec.persistence.RowContainer.add(RowContainer.java:163)
> at org.apache.hadoop.hive.ql.exec.JoinOperator.processOp(JoinOperator.java:118)
> at org.apache.hadoop.hive.ql.exec.Operator.process(Operator.java:456)
> at org.apache.hadoop.hive.ql.exec.ExecReducer.reduce(ExecReducer.java:244)
> at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:436)
> at org.apache.hadoop.mapred.Child.main(Child.java:158)
> Caused by: java.io.IOException: No space left on device
> at java.io.FileOutputStream.writeBytes(Native Method)
> at java.io.FileOutputStream.write(FileOutputStream.java:260)
> at org.apache.hadoop.fs.RawLocalFileSystem$LocalFSFileOutputStream.write(RawLocalFileSystem.java:197)
> ... 22 more
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HIVE-1326) RowContainer uses hard-coded '/tmp/'
path for temporary files
Posted by "Carl Steinbach (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-1326?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Carl Steinbach updated HIVE-1326:
---------------------------------
Component/s: Query Processor
Description:
In our production hadoop environment, the "/tmp/" is actually pretty small, and we encountered a problem when a query used the RowContainer class and filled up the /tmp/ partition. I tracked down the cause to the RowContainer class putting temporary files in the '/tmp/' path instead of using the configured Hadoop temporary path. I've attached a patch to fix this.
Here's the traceback:
2010-04-25 12:05:05,120 INFO org.apache.hadoop.hive.ql.exec.persistence.RowContainer: RowContainer created temp file /tmp/hive-rowcontainer-1244151903/RowContainer7816.tmp
2010-04-25 12:05:06,326 INFO ExecReducer: ExecReducer: processing 10000000 rows: used memory = 385520312
2010-04-25 12:05:08,513 INFO ExecReducer: ExecReducer: processing 11000000 rows: used memory = 341780472
2010-04-25 12:05:10,697 INFO ExecReducer: ExecReducer: processing 12000000 rows: used memory = 301446768
2010-04-25 12:05:12,837 INFO ExecReducer: ExecReducer: processing 13000000 rows: used memory = 399208768
2010-04-25 12:05:15,085 INFO ExecReducer: ExecReducer: processing 14000000 rows: used memory = 364507216
2010-04-25 12:05:17,260 INFO ExecReducer: ExecReducer: processing 15000000 rows: used memory = 332907280
2010-04-25 12:05:19,580 INFO ExecReducer: ExecReducer: processing 16000000 rows: used memory = 298774096
2010-04-25 12:05:21,629 INFO ExecReducer: ExecReducer: processing 17000000 rows: used memory = 396505408
2010-04-25 12:05:23,830 INFO ExecReducer: ExecReducer: processing 18000000 rows: used memory = 362477288
2010-04-25 12:05:25,914 INFO ExecReducer: ExecReducer: processing 19000000 rows: used memory = 327229744
2010-04-25 12:05:27,978 INFO ExecReducer: ExecReducer: processing 20000000 rows: used memory = 296051904
2010-04-25 12:05:28,155 FATAL ExecReducer: org.apache.hadoop.fs.FSError: java.io.IOException: No space left on device
at org.apache.hadoop.fs.RawLocalFileSystem$LocalFSFileOutputStream.write(RawLocalFileSystem.java:199)
at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:65)
at java.io.BufferedOutputStream.write(BufferedOutputStream.java:109)
at org.apache.hadoop.fs.FSDataOutputStream$PositionCache.write(FSDataOutputStream.java:49)
at java.io.DataOutputStream.write(DataOutputStream.java:90)
at org.apache.hadoop.fs.ChecksumFileSystem$ChecksumFSOutputSummer.writeChunk(ChecksumFileSystem.java:346)
at org.apache.hadoop.fs.FSOutputSummer.writeChecksumChunk(FSOutputSummer.java:150)
at org.apache.hadoop.fs.FSOutputSummer.flushBuffer(FSOutputSummer.java:132)
at org.apache.hadoop.fs.FSOutputSummer.flushBuffer(FSOutputSummer.java:121)
at org.apache.hadoop.fs.FSOutputSummer.write1(FSOutputSummer.java:112)
at org.apache.hadoop.fs.FSOutputSummer.write(FSOutputSummer.java:86)
at org.apache.hadoop.fs.FSDataOutputStream$PositionCache.write(FSDataOutputStream.java:49)
at java.io.DataOutputStream.write(DataOutputStream.java:90)
at org.apache.hadoop.io.SequenceFile$Writer.append(SequenceFile.java:1013)
at org.apache.hadoop.io.SequenceFile$Writer.append(SequenceFile.java:977)
at org.apache.hadoop.hive.ql.io.HiveSequenceFileOutputFormat$1.write(HiveSequenceFileOutputFormat.java:70)
at org.apache.hadoop.hive.ql.exec.persistence.RowContainer.spillBlock(RowContainer.java:343)
at org.apache.hadoop.hive.ql.exec.persistence.RowContainer.add(RowContainer.java:163)
at org.apache.hadoop.hive.ql.exec.JoinOperator.processOp(JoinOperator.java:118)
at org.apache.hadoop.hive.ql.exec.Operator.process(Operator.java:456)
at org.apache.hadoop.hive.ql.exec.ExecReducer.reduce(ExecReducer.java:244)
at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:436)
at org.apache.hadoop.mapred.Child.main(Child.java:158)
Caused by: java.io.IOException: No space left on device
at java.io.FileOutputStream.writeBytes(Native Method)
at java.io.FileOutputStream.write(FileOutputStream.java:260)
at org.apache.hadoop.fs.RawLocalFileSystem$LocalFSFileOutputStream.write(RawLocalFileSystem.java:197)
... 22 more
was:
In our production hadoop environment, the "/tmp/" is actually pretty small, and we encountered a problem when a query used the RowContainer class and filled up the /tmp/ partition. I tracked down the cause to the RowContainer class putting temporary files in the '/tmp/' path instead of using the configured Hadoop temporary path. I've attached a patch to fix this.
Here's the traceback:
2010-04-25 12:05:05,120 INFO org.apache.hadoop.hive.ql.exec.persistence.RowContainer: RowContainer created temp file /tmp/hive-rowcontainer-1244151903/RowContainer7816.tmp
2010-04-25 12:05:06,326 INFO ExecReducer: ExecReducer: processing 10000000 rows: used memory = 385520312
2010-04-25 12:05:08,513 INFO ExecReducer: ExecReducer: processing 11000000 rows: used memory = 341780472
2010-04-25 12:05:10,697 INFO ExecReducer: ExecReducer: processing 12000000 rows: used memory = 301446768
2010-04-25 12:05:12,837 INFO ExecReducer: ExecReducer: processing 13000000 rows: used memory = 399208768
2010-04-25 12:05:15,085 INFO ExecReducer: ExecReducer: processing 14000000 rows: used memory = 364507216
2010-04-25 12:05:17,260 INFO ExecReducer: ExecReducer: processing 15000000 rows: used memory = 332907280
2010-04-25 12:05:19,580 INFO ExecReducer: ExecReducer: processing 16000000 rows: used memory = 298774096
2010-04-25 12:05:21,629 INFO ExecReducer: ExecReducer: processing 17000000 rows: used memory = 396505408
2010-04-25 12:05:23,830 INFO ExecReducer: ExecReducer: processing 18000000 rows: used memory = 362477288
2010-04-25 12:05:25,914 INFO ExecReducer: ExecReducer: processing 19000000 rows: used memory = 327229744
2010-04-25 12:05:27,978 INFO ExecReducer: ExecReducer: processing 20000000 rows: used memory = 296051904
2010-04-25 12:05:28,155 FATAL ExecReducer: org.apache.hadoop.fs.FSError: java.io.IOException: No space left on device
at org.apache.hadoop.fs.RawLocalFileSystem$LocalFSFileOutputStream.write(RawLocalFileSystem.java:199)
at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:65)
at java.io.BufferedOutputStream.write(BufferedOutputStream.java:109)
at org.apache.hadoop.fs.FSDataOutputStream$PositionCache.write(FSDataOutputStream.java:49)
at java.io.DataOutputStream.write(DataOutputStream.java:90)
at org.apache.hadoop.fs.ChecksumFileSystem$ChecksumFSOutputSummer.writeChunk(ChecksumFileSystem.java:346)
at org.apache.hadoop.fs.FSOutputSummer.writeChecksumChunk(FSOutputSummer.java:150)
at org.apache.hadoop.fs.FSOutputSummer.flushBuffer(FSOutputSummer.java:132)
at org.apache.hadoop.fs.FSOutputSummer.flushBuffer(FSOutputSummer.java:121)
at org.apache.hadoop.fs.FSOutputSummer.write1(FSOutputSummer.java:112)
at org.apache.hadoop.fs.FSOutputSummer.write(FSOutputSummer.java:86)
at org.apache.hadoop.fs.FSDataOutputStream$PositionCache.write(FSDataOutputStream.java:49)
at java.io.DataOutputStream.write(DataOutputStream.java:90)
at org.apache.hadoop.io.SequenceFile$Writer.append(SequenceFile.java:1013)
at org.apache.hadoop.io.SequenceFile$Writer.append(SequenceFile.java:977)
at org.apache.hadoop.hive.ql.io.HiveSequenceFileOutputFormat$1.write(HiveSequenceFileOutputFormat.java:70)
at org.apache.hadoop.hive.ql.exec.persistence.RowContainer.spillBlock(RowContainer.java:343)
at org.apache.hadoop.hive.ql.exec.persistence.RowContainer.add(RowContainer.java:163)
at org.apache.hadoop.hive.ql.exec.JoinOperator.processOp(JoinOperator.java:118)
at org.apache.hadoop.hive.ql.exec.Operator.process(Operator.java:456)
at org.apache.hadoop.hive.ql.exec.ExecReducer.reduce(ExecReducer.java:244)
at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:436)
at org.apache.hadoop.mapred.Child.main(Child.java:158)
Caused by: java.io.IOException: No space left on device
at java.io.FileOutputStream.writeBytes(Native Method)
at java.io.FileOutputStream.write(FileOutputStream.java:260)
at org.apache.hadoop.fs.RawLocalFileSystem$LocalFSFileOutputStream.write(RawLocalFileSystem.java:197)
... 22 more
> RowContainer uses hard-coded '/tmp/' path for temporary files
> -------------------------------------------------------------
>
> Key: HIVE-1326
> URL: https://issues.apache.org/jira/browse/HIVE-1326
> Project: Hive
> Issue Type: Bug
> Components: Query Processor
> Environment: Hadoop 0.19.2 with Hive trunk. We're using FreeBSD 7.0, but that doesn't seem relevant.
> Reporter: Michael Klatt
> Fix For: 0.6.0
>
> Attachments: rowcontainer.patch, rowcontainer_v2.patch
>
>
> In our production hadoop environment, the "/tmp/" is actually pretty small, and we encountered a problem when a query used the RowContainer class and filled up the /tmp/ partition. I tracked down the cause to the RowContainer class putting temporary files in the '/tmp/' path instead of using the configured Hadoop temporary path. I've attached a patch to fix this.
> Here's the traceback:
> 2010-04-25 12:05:05,120 INFO org.apache.hadoop.hive.ql.exec.persistence.RowContainer: RowContainer created temp file /tmp/hive-rowcontainer-1244151903/RowContainer7816.tmp
> 2010-04-25 12:05:06,326 INFO ExecReducer: ExecReducer: processing 10000000 rows: used memory = 385520312
> 2010-04-25 12:05:08,513 INFO ExecReducer: ExecReducer: processing 11000000 rows: used memory = 341780472
> 2010-04-25 12:05:10,697 INFO ExecReducer: ExecReducer: processing 12000000 rows: used memory = 301446768
> 2010-04-25 12:05:12,837 INFO ExecReducer: ExecReducer: processing 13000000 rows: used memory = 399208768
> 2010-04-25 12:05:15,085 INFO ExecReducer: ExecReducer: processing 14000000 rows: used memory = 364507216
> 2010-04-25 12:05:17,260 INFO ExecReducer: ExecReducer: processing 15000000 rows: used memory = 332907280
> 2010-04-25 12:05:19,580 INFO ExecReducer: ExecReducer: processing 16000000 rows: used memory = 298774096
> 2010-04-25 12:05:21,629 INFO ExecReducer: ExecReducer: processing 17000000 rows: used memory = 396505408
> 2010-04-25 12:05:23,830 INFO ExecReducer: ExecReducer: processing 18000000 rows: used memory = 362477288
> 2010-04-25 12:05:25,914 INFO ExecReducer: ExecReducer: processing 19000000 rows: used memory = 327229744
> 2010-04-25 12:05:27,978 INFO ExecReducer: ExecReducer: processing 20000000 rows: used memory = 296051904
> 2010-04-25 12:05:28,155 FATAL ExecReducer: org.apache.hadoop.fs.FSError: java.io.IOException: No space left on device
> at org.apache.hadoop.fs.RawLocalFileSystem$LocalFSFileOutputStream.write(RawLocalFileSystem.java:199)
> at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:65)
> at java.io.BufferedOutputStream.write(BufferedOutputStream.java:109)
> at org.apache.hadoop.fs.FSDataOutputStream$PositionCache.write(FSDataOutputStream.java:49)
> at java.io.DataOutputStream.write(DataOutputStream.java:90)
> at org.apache.hadoop.fs.ChecksumFileSystem$ChecksumFSOutputSummer.writeChunk(ChecksumFileSystem.java:346)
> at org.apache.hadoop.fs.FSOutputSummer.writeChecksumChunk(FSOutputSummer.java:150)
> at org.apache.hadoop.fs.FSOutputSummer.flushBuffer(FSOutputSummer.java:132)
> at org.apache.hadoop.fs.FSOutputSummer.flushBuffer(FSOutputSummer.java:121)
> at org.apache.hadoop.fs.FSOutputSummer.write1(FSOutputSummer.java:112)
> at org.apache.hadoop.fs.FSOutputSummer.write(FSOutputSummer.java:86)
> at org.apache.hadoop.fs.FSDataOutputStream$PositionCache.write(FSDataOutputStream.java:49)
> at java.io.DataOutputStream.write(DataOutputStream.java:90)
> at org.apache.hadoop.io.SequenceFile$Writer.append(SequenceFile.java:1013)
> at org.apache.hadoop.io.SequenceFile$Writer.append(SequenceFile.java:977)
> at org.apache.hadoop.hive.ql.io.HiveSequenceFileOutputFormat$1.write(HiveSequenceFileOutputFormat.java:70)
> at org.apache.hadoop.hive.ql.exec.persistence.RowContainer.spillBlock(RowContainer.java:343)
> at org.apache.hadoop.hive.ql.exec.persistence.RowContainer.add(RowContainer.java:163)
> at org.apache.hadoop.hive.ql.exec.JoinOperator.processOp(JoinOperator.java:118)
> at org.apache.hadoop.hive.ql.exec.Operator.process(Operator.java:456)
> at org.apache.hadoop.hive.ql.exec.ExecReducer.reduce(ExecReducer.java:244)
> at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:436)
> at org.apache.hadoop.mapred.Child.main(Child.java:158)
> Caused by: java.io.IOException: No space left on device
> at java.io.FileOutputStream.writeBytes(Native Method)
> at java.io.FileOutputStream.write(FileOutputStream.java:260)
> at org.apache.hadoop.fs.RawLocalFileSystem$LocalFSFileOutputStream.write(RawLocalFileSystem.java:197)
> ... 22 more
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HIVE-1326) RowContainer uses hard-coded '/tmp/'
path for temporary files
Posted by "He Yongqiang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-1326?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12861370#action_12861370 ]
He Yongqiang commented on HIVE-1326:
------------------------------------
Agreed with Ning, looks fine to me.
> RowContainer uses hard-coded '/tmp/' path for temporary files
> -------------------------------------------------------------
>
> Key: HIVE-1326
> URL: https://issues.apache.org/jira/browse/HIVE-1326
> Project: Hadoop Hive
> Issue Type: Bug
> Environment: Hadoop 0.19.2 with Hive trunk. We're using FreeBSD 7.0, but that doesn't seem relevant.
> Reporter: Michael Klatt
> Attachments: rowcontainer.patch
>
>
> In our production hadoop environment, the "/tmp/" is actually pretty small, and we encountered a problem when a query used the RowContainer class and filled up the /tmp/ partition. I tracked down the cause to the RowContainer class putting temporary files in the '/tmp/' path instead of using the configured Hadoop temporary path. I've attached a patch to fix this.
> Here's the traceback:
> 2010-04-25 12:05:05,120 INFO org.apache.hadoop.hive.ql.exec.persistence.RowContainer: RowContainer created temp file /tmp/hive-rowcontainer-1244151903/RowContainer7816.tmp
> 2010-04-25 12:05:06,326 INFO ExecReducer: ExecReducer: processing 10000000 rows: used memory = 385520312
> 2010-04-25 12:05:08,513 INFO ExecReducer: ExecReducer: processing 11000000 rows: used memory = 341780472
> 2010-04-25 12:05:10,697 INFO ExecReducer: ExecReducer: processing 12000000 rows: used memory = 301446768
> 2010-04-25 12:05:12,837 INFO ExecReducer: ExecReducer: processing 13000000 rows: used memory = 399208768
> 2010-04-25 12:05:15,085 INFO ExecReducer: ExecReducer: processing 14000000 rows: used memory = 364507216
> 2010-04-25 12:05:17,260 INFO ExecReducer: ExecReducer: processing 15000000 rows: used memory = 332907280
> 2010-04-25 12:05:19,580 INFO ExecReducer: ExecReducer: processing 16000000 rows: used memory = 298774096
> 2010-04-25 12:05:21,629 INFO ExecReducer: ExecReducer: processing 17000000 rows: used memory = 396505408
> 2010-04-25 12:05:23,830 INFO ExecReducer: ExecReducer: processing 18000000 rows: used memory = 362477288
> 2010-04-25 12:05:25,914 INFO ExecReducer: ExecReducer: processing 19000000 rows: used memory = 327229744
> 2010-04-25 12:05:27,978 INFO ExecReducer: ExecReducer: processing 20000000 rows: used memory = 296051904
> 2010-04-25 12:05:28,155 FATAL ExecReducer: org.apache.hadoop.fs.FSError: java.io.IOException: No space left on device
> at org.apache.hadoop.fs.RawLocalFileSystem$LocalFSFileOutputStream.write(RawLocalFileSystem.java:199)
> at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:65)
> at java.io.BufferedOutputStream.write(BufferedOutputStream.java:109)
> at org.apache.hadoop.fs.FSDataOutputStream$PositionCache.write(FSDataOutputStream.java:49)
> at java.io.DataOutputStream.write(DataOutputStream.java:90)
> at org.apache.hadoop.fs.ChecksumFileSystem$ChecksumFSOutputSummer.writeChunk(ChecksumFileSystem.java:346)
> at org.apache.hadoop.fs.FSOutputSummer.writeChecksumChunk(FSOutputSummer.java:150)
> at org.apache.hadoop.fs.FSOutputSummer.flushBuffer(FSOutputSummer.java:132)
> at org.apache.hadoop.fs.FSOutputSummer.flushBuffer(FSOutputSummer.java:121)
> at org.apache.hadoop.fs.FSOutputSummer.write1(FSOutputSummer.java:112)
> at org.apache.hadoop.fs.FSOutputSummer.write(FSOutputSummer.java:86)
> at org.apache.hadoop.fs.FSDataOutputStream$PositionCache.write(FSDataOutputStream.java:49)
> at java.io.DataOutputStream.write(DataOutputStream.java:90)
> at org.apache.hadoop.io.SequenceFile$Writer.append(SequenceFile.java:1013)
> at org.apache.hadoop.io.SequenceFile$Writer.append(SequenceFile.java:977)
> at org.apache.hadoop.hive.ql.io.HiveSequenceFileOutputFormat$1.write(HiveSequenceFileOutputFormat.java:70)
> at org.apache.hadoop.hive.ql.exec.persistence.RowContainer.spillBlock(RowContainer.java:343)
> at org.apache.hadoop.hive.ql.exec.persistence.RowContainer.add(RowContainer.java:163)
> at org.apache.hadoop.hive.ql.exec.JoinOperator.processOp(JoinOperator.java:118)
> at org.apache.hadoop.hive.ql.exec.Operator.process(Operator.java:456)
> at org.apache.hadoop.hive.ql.exec.ExecReducer.reduce(ExecReducer.java:244)
> at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:436)
> at org.apache.hadoop.mapred.Child.main(Child.java:158)
> Caused by: java.io.IOException: No space left on device
> at java.io.FileOutputStream.writeBytes(Native Method)
> at java.io.FileOutputStream.write(FileOutputStream.java:260)
> at org.apache.hadoop.fs.RawLocalFileSystem$LocalFSFileOutputStream.write(RawLocalFileSystem.java:197)
> ... 22 more
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HIVE-1326) RowContainer uses hard-coded '/tmp/'
path for temporary files
Posted by "Ning Zhang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-1326?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12861467#action_12861467 ]
Ning Zhang commented on HIVE-1326:
----------------------------------
Michael, the temporary directory is needed for a conditional task to handle skewed joins (that's why the JobConf is cloned but with a different path). The idea here is that whenever there many too many rows for a particular job key to be held in memory, they will be first write to local disk via RowContainer. If there are still too many rows to local disk, they will be write to a DFS location and a contitional task will be triggered to handle this skewed key. So if we create a temp directory in local disk and let all mappers write their temp files in that directory, then later when we want to move the data to HDFS, we just move the whole directory instead of moving individual files.
+1 on v2. Will commit if tests pass.
> RowContainer uses hard-coded '/tmp/' path for temporary files
> -------------------------------------------------------------
>
> Key: HIVE-1326
> URL: https://issues.apache.org/jira/browse/HIVE-1326
> Project: Hadoop Hive
> Issue Type: Bug
> Environment: Hadoop 0.19.2 with Hive trunk. We're using FreeBSD 7.0, but that doesn't seem relevant.
> Reporter: Michael Klatt
> Attachments: rowcontainer.patch, rowcontainer_v2.patch
>
>
> In our production hadoop environment, the "/tmp/" is actually pretty small, and we encountered a problem when a query used the RowContainer class and filled up the /tmp/ partition. I tracked down the cause to the RowContainer class putting temporary files in the '/tmp/' path instead of using the configured Hadoop temporary path. I've attached a patch to fix this.
> Here's the traceback:
> 2010-04-25 12:05:05,120 INFO org.apache.hadoop.hive.ql.exec.persistence.RowContainer: RowContainer created temp file /tmp/hive-rowcontainer-1244151903/RowContainer7816.tmp
> 2010-04-25 12:05:06,326 INFO ExecReducer: ExecReducer: processing 10000000 rows: used memory = 385520312
> 2010-04-25 12:05:08,513 INFO ExecReducer: ExecReducer: processing 11000000 rows: used memory = 341780472
> 2010-04-25 12:05:10,697 INFO ExecReducer: ExecReducer: processing 12000000 rows: used memory = 301446768
> 2010-04-25 12:05:12,837 INFO ExecReducer: ExecReducer: processing 13000000 rows: used memory = 399208768
> 2010-04-25 12:05:15,085 INFO ExecReducer: ExecReducer: processing 14000000 rows: used memory = 364507216
> 2010-04-25 12:05:17,260 INFO ExecReducer: ExecReducer: processing 15000000 rows: used memory = 332907280
> 2010-04-25 12:05:19,580 INFO ExecReducer: ExecReducer: processing 16000000 rows: used memory = 298774096
> 2010-04-25 12:05:21,629 INFO ExecReducer: ExecReducer: processing 17000000 rows: used memory = 396505408
> 2010-04-25 12:05:23,830 INFO ExecReducer: ExecReducer: processing 18000000 rows: used memory = 362477288
> 2010-04-25 12:05:25,914 INFO ExecReducer: ExecReducer: processing 19000000 rows: used memory = 327229744
> 2010-04-25 12:05:27,978 INFO ExecReducer: ExecReducer: processing 20000000 rows: used memory = 296051904
> 2010-04-25 12:05:28,155 FATAL ExecReducer: org.apache.hadoop.fs.FSError: java.io.IOException: No space left on device
> at org.apache.hadoop.fs.RawLocalFileSystem$LocalFSFileOutputStream.write(RawLocalFileSystem.java:199)
> at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:65)
> at java.io.BufferedOutputStream.write(BufferedOutputStream.java:109)
> at org.apache.hadoop.fs.FSDataOutputStream$PositionCache.write(FSDataOutputStream.java:49)
> at java.io.DataOutputStream.write(DataOutputStream.java:90)
> at org.apache.hadoop.fs.ChecksumFileSystem$ChecksumFSOutputSummer.writeChunk(ChecksumFileSystem.java:346)
> at org.apache.hadoop.fs.FSOutputSummer.writeChecksumChunk(FSOutputSummer.java:150)
> at org.apache.hadoop.fs.FSOutputSummer.flushBuffer(FSOutputSummer.java:132)
> at org.apache.hadoop.fs.FSOutputSummer.flushBuffer(FSOutputSummer.java:121)
> at org.apache.hadoop.fs.FSOutputSummer.write1(FSOutputSummer.java:112)
> at org.apache.hadoop.fs.FSOutputSummer.write(FSOutputSummer.java:86)
> at org.apache.hadoop.fs.FSDataOutputStream$PositionCache.write(FSDataOutputStream.java:49)
> at java.io.DataOutputStream.write(DataOutputStream.java:90)
> at org.apache.hadoop.io.SequenceFile$Writer.append(SequenceFile.java:1013)
> at org.apache.hadoop.io.SequenceFile$Writer.append(SequenceFile.java:977)
> at org.apache.hadoop.hive.ql.io.HiveSequenceFileOutputFormat$1.write(HiveSequenceFileOutputFormat.java:70)
> at org.apache.hadoop.hive.ql.exec.persistence.RowContainer.spillBlock(RowContainer.java:343)
> at org.apache.hadoop.hive.ql.exec.persistence.RowContainer.add(RowContainer.java:163)
> at org.apache.hadoop.hive.ql.exec.JoinOperator.processOp(JoinOperator.java:118)
> at org.apache.hadoop.hive.ql.exec.Operator.process(Operator.java:456)
> at org.apache.hadoop.hive.ql.exec.ExecReducer.reduce(ExecReducer.java:244)
> at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:436)
> at org.apache.hadoop.mapred.Child.main(Child.java:158)
> Caused by: java.io.IOException: No space left on device
> at java.io.FileOutputStream.writeBytes(Native Method)
> at java.io.FileOutputStream.write(FileOutputStream.java:260)
> at org.apache.hadoop.fs.RawLocalFileSystem$LocalFSFileOutputStream.write(RawLocalFileSystem.java:197)
> ... 22 more
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HIVE-1326) RowContainer uses hard-coded '/tmp/'
path for temporary files
Posted by "Michael Klatt (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-1326?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Michael Klatt updated HIVE-1326:
--------------------------------
Attachment: rowcontainer.patch
> RowContainer uses hard-coded '/tmp/' path for temporary files
> -------------------------------------------------------------
>
> Key: HIVE-1326
> URL: https://issues.apache.org/jira/browse/HIVE-1326
> Project: Hadoop Hive
> Issue Type: Bug
> Environment: Hadoop 0.19.2 with Hive trunk. We're using FreeBSD 7.0, but that doesn't seem relevant.
> Reporter: Michael Klatt
> Attachments: rowcontainer.patch
>
>
> In our production hadoop environment, the "/tmp/" is actually pretty small, and we encountered a problem when a query used the RowContainer class and filled up the /tmp/ partition. I tracked down the cause to the RowContainer class putting temporary files in the '/tmp/' path instead of using the configured Hadoop temporary path. I've attached a patch to fix this.
> Here's the traceback:
> 2010-04-25 12:05:05,120 INFO org.apache.hadoop.hive.ql.exec.persistence.RowContainer: RowContainer created temp file /tmp/hive-rowcontainer-1244151903/RowContainer7816.tmp
> 2010-04-25 12:05:06,326 INFO ExecReducer: ExecReducer: processing 10000000 rows: used memory = 385520312
> 2010-04-25 12:05:08,513 INFO ExecReducer: ExecReducer: processing 11000000 rows: used memory = 341780472
> 2010-04-25 12:05:10,697 INFO ExecReducer: ExecReducer: processing 12000000 rows: used memory = 301446768
> 2010-04-25 12:05:12,837 INFO ExecReducer: ExecReducer: processing 13000000 rows: used memory = 399208768
> 2010-04-25 12:05:15,085 INFO ExecReducer: ExecReducer: processing 14000000 rows: used memory = 364507216
> 2010-04-25 12:05:17,260 INFO ExecReducer: ExecReducer: processing 15000000 rows: used memory = 332907280
> 2010-04-25 12:05:19,580 INFO ExecReducer: ExecReducer: processing 16000000 rows: used memory = 298774096
> 2010-04-25 12:05:21,629 INFO ExecReducer: ExecReducer: processing 17000000 rows: used memory = 396505408
> 2010-04-25 12:05:23,830 INFO ExecReducer: ExecReducer: processing 18000000 rows: used memory = 362477288
> 2010-04-25 12:05:25,914 INFO ExecReducer: ExecReducer: processing 19000000 rows: used memory = 327229744
> 2010-04-25 12:05:27,978 INFO ExecReducer: ExecReducer: processing 20000000 rows: used memory = 296051904
> 2010-04-25 12:05:28,155 FATAL ExecReducer: org.apache.hadoop.fs.FSError: java.io.IOException: No space left on device
> at org.apache.hadoop.fs.RawLocalFileSystem$LocalFSFileOutputStream.write(RawLocalFileSystem.java:199)
> at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:65)
> at java.io.BufferedOutputStream.write(BufferedOutputStream.java:109)
> at org.apache.hadoop.fs.FSDataOutputStream$PositionCache.write(FSDataOutputStream.java:49)
> at java.io.DataOutputStream.write(DataOutputStream.java:90)
> at org.apache.hadoop.fs.ChecksumFileSystem$ChecksumFSOutputSummer.writeChunk(ChecksumFileSystem.java:346)
> at org.apache.hadoop.fs.FSOutputSummer.writeChecksumChunk(FSOutputSummer.java:150)
> at org.apache.hadoop.fs.FSOutputSummer.flushBuffer(FSOutputSummer.java:132)
> at org.apache.hadoop.fs.FSOutputSummer.flushBuffer(FSOutputSummer.java:121)
> at org.apache.hadoop.fs.FSOutputSummer.write1(FSOutputSummer.java:112)
> at org.apache.hadoop.fs.FSOutputSummer.write(FSOutputSummer.java:86)
> at org.apache.hadoop.fs.FSDataOutputStream$PositionCache.write(FSDataOutputStream.java:49)
> at java.io.DataOutputStream.write(DataOutputStream.java:90)
> at org.apache.hadoop.io.SequenceFile$Writer.append(SequenceFile.java:1013)
> at org.apache.hadoop.io.SequenceFile$Writer.append(SequenceFile.java:977)
> at org.apache.hadoop.hive.ql.io.HiveSequenceFileOutputFormat$1.write(HiveSequenceFileOutputFormat.java:70)
> at org.apache.hadoop.hive.ql.exec.persistence.RowContainer.spillBlock(RowContainer.java:343)
> at org.apache.hadoop.hive.ql.exec.persistence.RowContainer.add(RowContainer.java:163)
> at org.apache.hadoop.hive.ql.exec.JoinOperator.processOp(JoinOperator.java:118)
> at org.apache.hadoop.hive.ql.exec.Operator.process(Operator.java:456)
> at org.apache.hadoop.hive.ql.exec.ExecReducer.reduce(ExecReducer.java:244)
> at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:436)
> at org.apache.hadoop.mapred.Child.main(Child.java:158)
> Caused by: java.io.IOException: No space left on device
> at java.io.FileOutputStream.writeBytes(Native Method)
> at java.io.FileOutputStream.write(FileOutputStream.java:260)
> at org.apache.hadoop.fs.RawLocalFileSystem$LocalFSFileOutputStream.write(RawLocalFileSystem.java:197)
> ... 22 more
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HIVE-1326) RowContainer uses hard-coded '/tmp/'
path for temporary files
Posted by "Michael Klatt (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-1326?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12861449#action_12861449 ]
Michael Klatt commented on HIVE-1326:
-------------------------------------
Hmm, on second thought, the parentFile (directory) is referenced in several places. I'm not familiar enough with Hive to know what these lines do:
HiveConf.setVar(jobCloneUsingLocalFs,
HiveConf.ConfVars.HADOOPMAPREDINPUTDIR,
org.apache.hadoop.util.StringUtils.escapeString(parentFile
.getAbsolutePath()));
If it weren't for these lines I could remove the parentFile variable all together.
> RowContainer uses hard-coded '/tmp/' path for temporary files
> -------------------------------------------------------------
>
> Key: HIVE-1326
> URL: https://issues.apache.org/jira/browse/HIVE-1326
> Project: Hadoop Hive
> Issue Type: Bug
> Environment: Hadoop 0.19.2 with Hive trunk. We're using FreeBSD 7.0, but that doesn't seem relevant.
> Reporter: Michael Klatt
> Attachments: rowcontainer.patch, rowcontainer_v2.patch
>
>
> In our production hadoop environment, the "/tmp/" is actually pretty small, and we encountered a problem when a query used the RowContainer class and filled up the /tmp/ partition. I tracked down the cause to the RowContainer class putting temporary files in the '/tmp/' path instead of using the configured Hadoop temporary path. I've attached a patch to fix this.
> Here's the traceback:
> 2010-04-25 12:05:05,120 INFO org.apache.hadoop.hive.ql.exec.persistence.RowContainer: RowContainer created temp file /tmp/hive-rowcontainer-1244151903/RowContainer7816.tmp
> 2010-04-25 12:05:06,326 INFO ExecReducer: ExecReducer: processing 10000000 rows: used memory = 385520312
> 2010-04-25 12:05:08,513 INFO ExecReducer: ExecReducer: processing 11000000 rows: used memory = 341780472
> 2010-04-25 12:05:10,697 INFO ExecReducer: ExecReducer: processing 12000000 rows: used memory = 301446768
> 2010-04-25 12:05:12,837 INFO ExecReducer: ExecReducer: processing 13000000 rows: used memory = 399208768
> 2010-04-25 12:05:15,085 INFO ExecReducer: ExecReducer: processing 14000000 rows: used memory = 364507216
> 2010-04-25 12:05:17,260 INFO ExecReducer: ExecReducer: processing 15000000 rows: used memory = 332907280
> 2010-04-25 12:05:19,580 INFO ExecReducer: ExecReducer: processing 16000000 rows: used memory = 298774096
> 2010-04-25 12:05:21,629 INFO ExecReducer: ExecReducer: processing 17000000 rows: used memory = 396505408
> 2010-04-25 12:05:23,830 INFO ExecReducer: ExecReducer: processing 18000000 rows: used memory = 362477288
> 2010-04-25 12:05:25,914 INFO ExecReducer: ExecReducer: processing 19000000 rows: used memory = 327229744
> 2010-04-25 12:05:27,978 INFO ExecReducer: ExecReducer: processing 20000000 rows: used memory = 296051904
> 2010-04-25 12:05:28,155 FATAL ExecReducer: org.apache.hadoop.fs.FSError: java.io.IOException: No space left on device
> at org.apache.hadoop.fs.RawLocalFileSystem$LocalFSFileOutputStream.write(RawLocalFileSystem.java:199)
> at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:65)
> at java.io.BufferedOutputStream.write(BufferedOutputStream.java:109)
> at org.apache.hadoop.fs.FSDataOutputStream$PositionCache.write(FSDataOutputStream.java:49)
> at java.io.DataOutputStream.write(DataOutputStream.java:90)
> at org.apache.hadoop.fs.ChecksumFileSystem$ChecksumFSOutputSummer.writeChunk(ChecksumFileSystem.java:346)
> at org.apache.hadoop.fs.FSOutputSummer.writeChecksumChunk(FSOutputSummer.java:150)
> at org.apache.hadoop.fs.FSOutputSummer.flushBuffer(FSOutputSummer.java:132)
> at org.apache.hadoop.fs.FSOutputSummer.flushBuffer(FSOutputSummer.java:121)
> at org.apache.hadoop.fs.FSOutputSummer.write1(FSOutputSummer.java:112)
> at org.apache.hadoop.fs.FSOutputSummer.write(FSOutputSummer.java:86)
> at org.apache.hadoop.fs.FSDataOutputStream$PositionCache.write(FSDataOutputStream.java:49)
> at java.io.DataOutputStream.write(DataOutputStream.java:90)
> at org.apache.hadoop.io.SequenceFile$Writer.append(SequenceFile.java:1013)
> at org.apache.hadoop.io.SequenceFile$Writer.append(SequenceFile.java:977)
> at org.apache.hadoop.hive.ql.io.HiveSequenceFileOutputFormat$1.write(HiveSequenceFileOutputFormat.java:70)
> at org.apache.hadoop.hive.ql.exec.persistence.RowContainer.spillBlock(RowContainer.java:343)
> at org.apache.hadoop.hive.ql.exec.persistence.RowContainer.add(RowContainer.java:163)
> at org.apache.hadoop.hive.ql.exec.JoinOperator.processOp(JoinOperator.java:118)
> at org.apache.hadoop.hive.ql.exec.Operator.process(Operator.java:456)
> at org.apache.hadoop.hive.ql.exec.ExecReducer.reduce(ExecReducer.java:244)
> at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:436)
> at org.apache.hadoop.mapred.Child.main(Child.java:158)
> Caused by: java.io.IOException: No space left on device
> at java.io.FileOutputStream.writeBytes(Native Method)
> at java.io.FileOutputStream.write(FileOutputStream.java:260)
> at org.apache.hadoop.fs.RawLocalFileSystem$LocalFSFileOutputStream.write(RawLocalFileSystem.java:197)
> ... 22 more
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HIVE-1326) RowContainer uses hard-coded '/tmp/'
path for temporary files
Posted by "Edward Capriolo (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-1326?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12860929#action_12860929 ]
Edward Capriolo commented on HIVE-1326:
---------------------------------------
I am +1 on the concept. Many times /tmp can be a ramdisk or mounted with some size restrictions. This type of bug an be very painful to track down when it happens.
> RowContainer uses hard-coded '/tmp/' path for temporary files
> -------------------------------------------------------------
>
> Key: HIVE-1326
> URL: https://issues.apache.org/jira/browse/HIVE-1326
> Project: Hadoop Hive
> Issue Type: Bug
> Environment: Hadoop 0.19.2 with Hive trunk. We're using FreeBSD 7.0, but that doesn't seem relevant.
> Reporter: Michael Klatt
> Attachments: rowcontainer.patch
>
>
> In our production hadoop environment, the "/tmp/" is actually pretty small, and we encountered a problem when a query used the RowContainer class and filled up the /tmp/ partition. I tracked down the cause to the RowContainer class putting temporary files in the '/tmp/' path instead of using the configured Hadoop temporary path. I've attached a patch to fix this.
> Here's the traceback:
> 2010-04-25 12:05:05,120 INFO org.apache.hadoop.hive.ql.exec.persistence.RowContainer: RowContainer created temp file /tmp/hive-rowcontainer-1244151903/RowContainer7816.tmp
> 2010-04-25 12:05:06,326 INFO ExecReducer: ExecReducer: processing 10000000 rows: used memory = 385520312
> 2010-04-25 12:05:08,513 INFO ExecReducer: ExecReducer: processing 11000000 rows: used memory = 341780472
> 2010-04-25 12:05:10,697 INFO ExecReducer: ExecReducer: processing 12000000 rows: used memory = 301446768
> 2010-04-25 12:05:12,837 INFO ExecReducer: ExecReducer: processing 13000000 rows: used memory = 399208768
> 2010-04-25 12:05:15,085 INFO ExecReducer: ExecReducer: processing 14000000 rows: used memory = 364507216
> 2010-04-25 12:05:17,260 INFO ExecReducer: ExecReducer: processing 15000000 rows: used memory = 332907280
> 2010-04-25 12:05:19,580 INFO ExecReducer: ExecReducer: processing 16000000 rows: used memory = 298774096
> 2010-04-25 12:05:21,629 INFO ExecReducer: ExecReducer: processing 17000000 rows: used memory = 396505408
> 2010-04-25 12:05:23,830 INFO ExecReducer: ExecReducer: processing 18000000 rows: used memory = 362477288
> 2010-04-25 12:05:25,914 INFO ExecReducer: ExecReducer: processing 19000000 rows: used memory = 327229744
> 2010-04-25 12:05:27,978 INFO ExecReducer: ExecReducer: processing 20000000 rows: used memory = 296051904
> 2010-04-25 12:05:28,155 FATAL ExecReducer: org.apache.hadoop.fs.FSError: java.io.IOException: No space left on device
> at org.apache.hadoop.fs.RawLocalFileSystem$LocalFSFileOutputStream.write(RawLocalFileSystem.java:199)
> at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:65)
> at java.io.BufferedOutputStream.write(BufferedOutputStream.java:109)
> at org.apache.hadoop.fs.FSDataOutputStream$PositionCache.write(FSDataOutputStream.java:49)
> at java.io.DataOutputStream.write(DataOutputStream.java:90)
> at org.apache.hadoop.fs.ChecksumFileSystem$ChecksumFSOutputSummer.writeChunk(ChecksumFileSystem.java:346)
> at org.apache.hadoop.fs.FSOutputSummer.writeChecksumChunk(FSOutputSummer.java:150)
> at org.apache.hadoop.fs.FSOutputSummer.flushBuffer(FSOutputSummer.java:132)
> at org.apache.hadoop.fs.FSOutputSummer.flushBuffer(FSOutputSummer.java:121)
> at org.apache.hadoop.fs.FSOutputSummer.write1(FSOutputSummer.java:112)
> at org.apache.hadoop.fs.FSOutputSummer.write(FSOutputSummer.java:86)
> at org.apache.hadoop.fs.FSDataOutputStream$PositionCache.write(FSDataOutputStream.java:49)
> at java.io.DataOutputStream.write(DataOutputStream.java:90)
> at org.apache.hadoop.io.SequenceFile$Writer.append(SequenceFile.java:1013)
> at org.apache.hadoop.io.SequenceFile$Writer.append(SequenceFile.java:977)
> at org.apache.hadoop.hive.ql.io.HiveSequenceFileOutputFormat$1.write(HiveSequenceFileOutputFormat.java:70)
> at org.apache.hadoop.hive.ql.exec.persistence.RowContainer.spillBlock(RowContainer.java:343)
> at org.apache.hadoop.hive.ql.exec.persistence.RowContainer.add(RowContainer.java:163)
> at org.apache.hadoop.hive.ql.exec.JoinOperator.processOp(JoinOperator.java:118)
> at org.apache.hadoop.hive.ql.exec.Operator.process(Operator.java:456)
> at org.apache.hadoop.hive.ql.exec.ExecReducer.reduce(ExecReducer.java:244)
> at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:436)
> at org.apache.hadoop.mapred.Child.main(Child.java:158)
> Caused by: java.io.IOException: No space left on device
> at java.io.FileOutputStream.writeBytes(Native Method)
> at java.io.FileOutputStream.write(FileOutputStream.java:260)
> at org.apache.hadoop.fs.RawLocalFileSystem$LocalFSFileOutputStream.write(RawLocalFileSystem.java:197)
> ... 22 more
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Resolved: (HIVE-1326) RowContainer uses hard-coded '/tmp/'
path for temporary files
Posted by "Ning Zhang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-1326?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Ning Zhang resolved HIVE-1326.
------------------------------
Fix Version/s: 0.6.0
Resolution: Fixed
Committed. Thanks Michael!
> RowContainer uses hard-coded '/tmp/' path for temporary files
> -------------------------------------------------------------
>
> Key: HIVE-1326
> URL: https://issues.apache.org/jira/browse/HIVE-1326
> Project: Hadoop Hive
> Issue Type: Bug
> Environment: Hadoop 0.19.2 with Hive trunk. We're using FreeBSD 7.0, but that doesn't seem relevant.
> Reporter: Michael Klatt
> Fix For: 0.6.0
>
> Attachments: rowcontainer.patch, rowcontainer_v2.patch
>
>
> In our production hadoop environment, the "/tmp/" is actually pretty small, and we encountered a problem when a query used the RowContainer class and filled up the /tmp/ partition. I tracked down the cause to the RowContainer class putting temporary files in the '/tmp/' path instead of using the configured Hadoop temporary path. I've attached a patch to fix this.
> Here's the traceback:
> 2010-04-25 12:05:05,120 INFO org.apache.hadoop.hive.ql.exec.persistence.RowContainer: RowContainer created temp file /tmp/hive-rowcontainer-1244151903/RowContainer7816.tmp
> 2010-04-25 12:05:06,326 INFO ExecReducer: ExecReducer: processing 10000000 rows: used memory = 385520312
> 2010-04-25 12:05:08,513 INFO ExecReducer: ExecReducer: processing 11000000 rows: used memory = 341780472
> 2010-04-25 12:05:10,697 INFO ExecReducer: ExecReducer: processing 12000000 rows: used memory = 301446768
> 2010-04-25 12:05:12,837 INFO ExecReducer: ExecReducer: processing 13000000 rows: used memory = 399208768
> 2010-04-25 12:05:15,085 INFO ExecReducer: ExecReducer: processing 14000000 rows: used memory = 364507216
> 2010-04-25 12:05:17,260 INFO ExecReducer: ExecReducer: processing 15000000 rows: used memory = 332907280
> 2010-04-25 12:05:19,580 INFO ExecReducer: ExecReducer: processing 16000000 rows: used memory = 298774096
> 2010-04-25 12:05:21,629 INFO ExecReducer: ExecReducer: processing 17000000 rows: used memory = 396505408
> 2010-04-25 12:05:23,830 INFO ExecReducer: ExecReducer: processing 18000000 rows: used memory = 362477288
> 2010-04-25 12:05:25,914 INFO ExecReducer: ExecReducer: processing 19000000 rows: used memory = 327229744
> 2010-04-25 12:05:27,978 INFO ExecReducer: ExecReducer: processing 20000000 rows: used memory = 296051904
> 2010-04-25 12:05:28,155 FATAL ExecReducer: org.apache.hadoop.fs.FSError: java.io.IOException: No space left on device
> at org.apache.hadoop.fs.RawLocalFileSystem$LocalFSFileOutputStream.write(RawLocalFileSystem.java:199)
> at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:65)
> at java.io.BufferedOutputStream.write(BufferedOutputStream.java:109)
> at org.apache.hadoop.fs.FSDataOutputStream$PositionCache.write(FSDataOutputStream.java:49)
> at java.io.DataOutputStream.write(DataOutputStream.java:90)
> at org.apache.hadoop.fs.ChecksumFileSystem$ChecksumFSOutputSummer.writeChunk(ChecksumFileSystem.java:346)
> at org.apache.hadoop.fs.FSOutputSummer.writeChecksumChunk(FSOutputSummer.java:150)
> at org.apache.hadoop.fs.FSOutputSummer.flushBuffer(FSOutputSummer.java:132)
> at org.apache.hadoop.fs.FSOutputSummer.flushBuffer(FSOutputSummer.java:121)
> at org.apache.hadoop.fs.FSOutputSummer.write1(FSOutputSummer.java:112)
> at org.apache.hadoop.fs.FSOutputSummer.write(FSOutputSummer.java:86)
> at org.apache.hadoop.fs.FSDataOutputStream$PositionCache.write(FSDataOutputStream.java:49)
> at java.io.DataOutputStream.write(DataOutputStream.java:90)
> at org.apache.hadoop.io.SequenceFile$Writer.append(SequenceFile.java:1013)
> at org.apache.hadoop.io.SequenceFile$Writer.append(SequenceFile.java:977)
> at org.apache.hadoop.hive.ql.io.HiveSequenceFileOutputFormat$1.write(HiveSequenceFileOutputFormat.java:70)
> at org.apache.hadoop.hive.ql.exec.persistence.RowContainer.spillBlock(RowContainer.java:343)
> at org.apache.hadoop.hive.ql.exec.persistence.RowContainer.add(RowContainer.java:163)
> at org.apache.hadoop.hive.ql.exec.JoinOperator.processOp(JoinOperator.java:118)
> at org.apache.hadoop.hive.ql.exec.Operator.process(Operator.java:456)
> at org.apache.hadoop.hive.ql.exec.ExecReducer.reduce(ExecReducer.java:244)
> at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:436)
> at org.apache.hadoop.mapred.Child.main(Child.java:158)
> Caused by: java.io.IOException: No space left on device
> at java.io.FileOutputStream.writeBytes(Native Method)
> at java.io.FileOutputStream.write(FileOutputStream.java:260)
> at org.apache.hadoop.fs.RawLocalFileSystem$LocalFSFileOutputStream.write(RawLocalFileSystem.java:197)
> ... 22 more
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HIVE-1326) RowContainer uses hard-coded '/tmp/'
path for temporary files
Posted by "Ning Zhang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-1326?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12861054#action_12861054 ]
Ning Zhang commented on HIVE-1326:
----------------------------------
The general idea of why we put temp files and directories under /tmp is two fold:
1) it allows easy clean up in case of any unexpected interruptions to the JVM. In the regular exits, these temp files/directories should be removed by JVM by deleteOnExit(), but if the JVM was killed/interrupted unexpected, these temp files could eventually take up all disk spaces and make the whole cluster hard to recover (imagine finding these temp files in thousands of machines). By putting it in /tmp, all these will be automatically removed whenever the machine is restarted, or cleaned up manually by just removing all files in /tmp. That's much easier in terms of Hadoop administration.
2) /tmp seems to be a universally writable mounted point on all unix-based systems. So it should be a safe place to put the temporary files. But I agree that it would a good idea to introduce a new Hive parameter to let the user choose a customizable tmp directory, which should address Edward's point.
I think we use File.createTempFile() previously and Yongqiang added another directory structure (parentFile). I think we can use File.createTempFile to create this directory as well (Yongqiang, please correct me if I'm wrong), there there are several questions about the patch:
1) File.createTempFile() will by default create temp files in java.io.tmpdir (/tmp or /var/tmp) according to JDK. I suspect most of the case JVM on Linux will use /tmp as the java.io.tmpdir, so in that caes your change won't achieve your purpose in most cases.
2) the random number generator is not needed if you can use File.createTempFile to create parentFile, because File.createTempFile guarantees its uniqueness for this JVM. The only case you need to prevent is the collision between different JVMs. This is why there is a check for parentFile.mkdir(). It actually acts like a write lock to the directory so that other JVM won't overwrite it. So parentFile.delete() should not be added because it may delete the directory created by another JVM who is still running.
> RowContainer uses hard-coded '/tmp/' path for temporary files
> -------------------------------------------------------------
>
> Key: HIVE-1326
> URL: https://issues.apache.org/jira/browse/HIVE-1326
> Project: Hadoop Hive
> Issue Type: Bug
> Environment: Hadoop 0.19.2 with Hive trunk. We're using FreeBSD 7.0, but that doesn't seem relevant.
> Reporter: Michael Klatt
> Attachments: rowcontainer.patch
>
>
> In our production hadoop environment, the "/tmp/" is actually pretty small, and we encountered a problem when a query used the RowContainer class and filled up the /tmp/ partition. I tracked down the cause to the RowContainer class putting temporary files in the '/tmp/' path instead of using the configured Hadoop temporary path. I've attached a patch to fix this.
> Here's the traceback:
> 2010-04-25 12:05:05,120 INFO org.apache.hadoop.hive.ql.exec.persistence.RowContainer: RowContainer created temp file /tmp/hive-rowcontainer-1244151903/RowContainer7816.tmp
> 2010-04-25 12:05:06,326 INFO ExecReducer: ExecReducer: processing 10000000 rows: used memory = 385520312
> 2010-04-25 12:05:08,513 INFO ExecReducer: ExecReducer: processing 11000000 rows: used memory = 341780472
> 2010-04-25 12:05:10,697 INFO ExecReducer: ExecReducer: processing 12000000 rows: used memory = 301446768
> 2010-04-25 12:05:12,837 INFO ExecReducer: ExecReducer: processing 13000000 rows: used memory = 399208768
> 2010-04-25 12:05:15,085 INFO ExecReducer: ExecReducer: processing 14000000 rows: used memory = 364507216
> 2010-04-25 12:05:17,260 INFO ExecReducer: ExecReducer: processing 15000000 rows: used memory = 332907280
> 2010-04-25 12:05:19,580 INFO ExecReducer: ExecReducer: processing 16000000 rows: used memory = 298774096
> 2010-04-25 12:05:21,629 INFO ExecReducer: ExecReducer: processing 17000000 rows: used memory = 396505408
> 2010-04-25 12:05:23,830 INFO ExecReducer: ExecReducer: processing 18000000 rows: used memory = 362477288
> 2010-04-25 12:05:25,914 INFO ExecReducer: ExecReducer: processing 19000000 rows: used memory = 327229744
> 2010-04-25 12:05:27,978 INFO ExecReducer: ExecReducer: processing 20000000 rows: used memory = 296051904
> 2010-04-25 12:05:28,155 FATAL ExecReducer: org.apache.hadoop.fs.FSError: java.io.IOException: No space left on device
> at org.apache.hadoop.fs.RawLocalFileSystem$LocalFSFileOutputStream.write(RawLocalFileSystem.java:199)
> at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:65)
> at java.io.BufferedOutputStream.write(BufferedOutputStream.java:109)
> at org.apache.hadoop.fs.FSDataOutputStream$PositionCache.write(FSDataOutputStream.java:49)
> at java.io.DataOutputStream.write(DataOutputStream.java:90)
> at org.apache.hadoop.fs.ChecksumFileSystem$ChecksumFSOutputSummer.writeChunk(ChecksumFileSystem.java:346)
> at org.apache.hadoop.fs.FSOutputSummer.writeChecksumChunk(FSOutputSummer.java:150)
> at org.apache.hadoop.fs.FSOutputSummer.flushBuffer(FSOutputSummer.java:132)
> at org.apache.hadoop.fs.FSOutputSummer.flushBuffer(FSOutputSummer.java:121)
> at org.apache.hadoop.fs.FSOutputSummer.write1(FSOutputSummer.java:112)
> at org.apache.hadoop.fs.FSOutputSummer.write(FSOutputSummer.java:86)
> at org.apache.hadoop.fs.FSDataOutputStream$PositionCache.write(FSDataOutputStream.java:49)
> at java.io.DataOutputStream.write(DataOutputStream.java:90)
> at org.apache.hadoop.io.SequenceFile$Writer.append(SequenceFile.java:1013)
> at org.apache.hadoop.io.SequenceFile$Writer.append(SequenceFile.java:977)
> at org.apache.hadoop.hive.ql.io.HiveSequenceFileOutputFormat$1.write(HiveSequenceFileOutputFormat.java:70)
> at org.apache.hadoop.hive.ql.exec.persistence.RowContainer.spillBlock(RowContainer.java:343)
> at org.apache.hadoop.hive.ql.exec.persistence.RowContainer.add(RowContainer.java:163)
> at org.apache.hadoop.hive.ql.exec.JoinOperator.processOp(JoinOperator.java:118)
> at org.apache.hadoop.hive.ql.exec.Operator.process(Operator.java:456)
> at org.apache.hadoop.hive.ql.exec.ExecReducer.reduce(ExecReducer.java:244)
> at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:436)
> at org.apache.hadoop.mapred.Child.main(Child.java:158)
> Caused by: java.io.IOException: No space left on device
> at java.io.FileOutputStream.writeBytes(Native Method)
> at java.io.FileOutputStream.write(FileOutputStream.java:260)
> at org.apache.hadoop.fs.RawLocalFileSystem$LocalFSFileOutputStream.write(RawLocalFileSystem.java:197)
> ... 22 more
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.