You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hive.apache.org by "Namit Jain (JIRA)" <ji...@apache.org> on 2008/12/05 21:36:44 UTC
[jira] Created: (HIVE-127) fetch task does not pass the job to the
serde in initialize
fetch task does not pass the job to the serde in initialize
-----------------------------------------------------------
Key: HIVE-127
URL: https://issues.apache.org/jira/browse/HIVE-127
Project: Hadoop Hive
Issue Type: Bug
Reporter: Namit Jain
Assignee: Namit Jain
Fetch task passes a null job to the serde during initialize - it should not assume that the serde does not need the job.
This assumption is valid for MetadataTypedColumnSetSerde, but need not be valid always
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HIVE-127) fetch task does not pass the job to the
serde in initialize
Posted by "Carl Steinbach (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-127?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Carl Steinbach updated HIVE-127:
--------------------------------
Fix Version/s: 0.3.0
Component/s: Query Processor
> fetch task does not pass the job to the serde in initialize
> -----------------------------------------------------------
>
> Key: HIVE-127
> URL: https://issues.apache.org/jira/browse/HIVE-127
> Project: Hadoop Hive
> Issue Type: Bug
> Components: Query Processor
> Reporter: Namit Jain
> Assignee: Namit Jain
> Fix For: 0.3.0
>
> Attachments: patch-127.txt
>
>
> Fetch task passes a null job to the serde during initialize - it should not assume that the serde does not need the job.
> This assumption is valid for MetadataTypedColumnSetSerde, but need not be valid always
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HIVE-127) fetch task does not pass the job to
the serde in initialize
Posted by "Namit Jain (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-127?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12654535#action_12654535 ]
Namit Jain commented on HIVE-127:
---------------------------------
[junit] 08/12/08 11:27:39 INFO mapred.TaskRunner: Task 'attempt_local_0001_m_000000_0' done.
[junit] 08/12/08 11:27:39 INFO thrift.TBinarySortableProtocol: Sort order is ""
[junit] 08/12/08 11:27:39 INFO thrift.TBinarySortableProtocol: Sort order is ""
[junit] 08/12/08 11:27:39 INFO mapred.Merger: Merging 1 sorted segments
[junit] 08/12/08 11:27:39 INFO mapred.Merger: Down to the last merge-pass, with 1 segments left of total size: 24 bytes
[junit] 08/12/08 11:27:39 INFO exec.GroupByOperator: Initializing Self
[junit] 08/12/08 11:27:39 INFO exec.GroupByOperator: Initializing children:
[junit] 08/12/08 11:27:39 INFO exec.SelectOperator: Initializing Self
[junit] 08/12/08 11:27:39 INFO exec.SelectOperator: Initializing children:
[junit] 08/12/08 11:27:39 INFO exec.FileSinkOperator: Initializing Self
[junit] 08/12/08 11:27:39 INFO exec.SelectOperator: Initialization Done
[junit] 08/12/08 11:27:39 INFO exec.GroupByOperator: Initialization Done
[junit] 08/12/08 11:27:40 INFO mapred.TaskRunner: Task:attempt_local_0001_r_000000_0 is done. And is in the process of commiting
[junit] 08/12/08 11:27:40 INFO mapred.LocalJobRunner:
[junit] 08/12/08 11:27:40 INFO mapred.TaskRunner: Task attempt_local_0001_r_000000_0 is allowed to commit now
[junit] 08/12/08 11:27:40 INFO mapred.FileOutputCommitter: Saved output of task 'attempt_local_0001_r_000000_0' to file:/tmp/hive-njain1339773828
[junit] 08/12/08 11:27:40 INFO mapred.LocalJobRunner: reduce > reduce
[junit] 08/12/08 11:27:40 INFO mapred.TaskRunner: Task 'attempt_local_0001_r_000000_0' done.
[junit] 08/12/08 11:27:40 INFO exec.ExecDriver: map = 100%, reduce =100%
[junit] 08/12/08 11:27:40 INFO exec.ExecDriver: Ended Job = job_local_0001
[junit] ------------- ---------------- ---------------
[junit] Testcase: testExecute took 12.928 sec
[junit] Testcase: testMetastore took 0.207 sec
[junit] Testcase: testFetch took 8.076 sec
[junit] Testcase: testDynamicSerde took 8.464 sec
BUILD SUCCESSFUL
Total time: 16 minutes 17 seconds
the tests passed successfully
> fetch task does not pass the job to the serde in initialize
> -----------------------------------------------------------
>
> Key: HIVE-127
> URL: https://issues.apache.org/jira/browse/HIVE-127
> Project: Hadoop Hive
> Issue Type: Bug
> Reporter: Namit Jain
> Assignee: Namit Jain
> Attachments: patch-127.txt
>
>
> Fetch task passes a null job to the serde during initialize - it should not assume that the serde does not need the job.
> This assumption is valid for MetadataTypedColumnSetSerde, but need not be valid always
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HIVE-127) fetch task does not pass the job to
the serde in initialize
Posted by "Zheng Shao (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-127?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12654579#action_12654579 ]
Zheng Shao commented on HIVE-127:
---------------------------------
Agree with Raghu on this.
The jobconf is a legacy thing from the first version of serde. It was not changed when migrated to serde2 mainly because we had too many other issues to deal with at that moment.
We should get rid of jobconf to make the serde interface cleaner.
> fetch task does not pass the job to the serde in initialize
> -----------------------------------------------------------
>
> Key: HIVE-127
> URL: https://issues.apache.org/jira/browse/HIVE-127
> Project: Hadoop Hive
> Issue Type: Bug
> Reporter: Namit Jain
> Assignee: Namit Jain
> Attachments: patch-127.txt
>
>
> Fetch task passes a null job to the serde during initialize - it should not assume that the serde does not need the job.
> This assumption is valid for MetadataTypedColumnSetSerde, but need not be valid always
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HIVE-127) fetch task does not pass the job to the
serde in initialize
Posted by "Namit Jain (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-127?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Namit Jain updated HIVE-127:
----------------------------
Attachment: patch-127.txt
> fetch task does not pass the job to the serde in initialize
> -----------------------------------------------------------
>
> Key: HIVE-127
> URL: https://issues.apache.org/jira/browse/HIVE-127
> Project: Hadoop Hive
> Issue Type: Bug
> Reporter: Namit Jain
> Assignee: Namit Jain
> Attachments: patch-127.txt
>
>
> Fetch task passes a null job to the serde during initialize - it should not assume that the serde does not need the job.
> This assumption is valid for MetadataTypedColumnSetSerde, but need not be valid always
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HIVE-127) fetch task does not pass the job to
the serde in initialize
Posted by "Zheng Shao (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-127?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12654610#action_12654610 ]
Zheng Shao commented on HIVE-127:
---------------------------------
Committed revision 724517.
> fetch task does not pass the job to the serde in initialize
> -----------------------------------------------------------
>
> Key: HIVE-127
> URL: https://issues.apache.org/jira/browse/HIVE-127
> Project: Hadoop Hive
> Issue Type: Bug
> Reporter: Namit Jain
> Assignee: Namit Jain
> Attachments: patch-127.txt
>
>
> Fetch task passes a null job to the serde during initialize - it should not assume that the serde does not need the job.
> This assumption is valid for MetadataTypedColumnSetSerde, but need not be valid always
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HIVE-127) fetch task does not pass the job to
the serde in initialize
Posted by "Raghotham Murthy (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-127?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12654527#action_12654527 ]
Raghotham Murthy commented on HIVE-127:
---------------------------------------
+1
Accepting for now.
In general, it seems to be a bad idea to provide a jobconf to the serde. We are already passing a Properties object which is arbitrary key-value pairs. If there are parameters in the jobconf required by the serde, they should be extracted from the jobconf and then inserted into the Properties. This will make code flow much more easy to understand. Now, we are passing an opaque object (jobconf) with arbitrary key-value pairs which can be set anywhere in the code. Can you file another jira to remove jobconf from the SerDe interface?
> fetch task does not pass the job to the serde in initialize
> -----------------------------------------------------------
>
> Key: HIVE-127
> URL: https://issues.apache.org/jira/browse/HIVE-127
> Project: Hadoop Hive
> Issue Type: Bug
> Reporter: Namit Jain
> Assignee: Namit Jain
> Attachments: patch-127.txt
>
>
> Fetch task passes a null job to the serde during initialize - it should not assume that the serde does not need the job.
> This assumption is valid for MetadataTypedColumnSetSerde, but need not be valid always
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HIVE-127) fetch task does not pass the job to the
serde in initialize
Posted by "Zheng Shao (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-127?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Zheng Shao updated HIVE-127:
----------------------------
Resolution: Fixed
Hadoop Flags: [Reviewed]
Status: Resolved (was: Patch Available)
HIVE-127. Fetch Task to pass the jobconf to SerDe. (Namit through zshao)
> fetch task does not pass the job to the serde in initialize
> -----------------------------------------------------------
>
> Key: HIVE-127
> URL: https://issues.apache.org/jira/browse/HIVE-127
> Project: Hadoop Hive
> Issue Type: Bug
> Reporter: Namit Jain
> Assignee: Namit Jain
> Attachments: patch-127.txt
>
>
> Fetch task passes a null job to the serde during initialize - it should not assume that the serde does not need the job.
> This assumption is valid for MetadataTypedColumnSetSerde, but need not be valid always
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HIVE-127) fetch task does not pass the job to the
serde in initialize
Posted by "Namit Jain (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-127?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Namit Jain updated HIVE-127:
----------------------------
Status: Patch Available (was: Open)
> fetch task does not pass the job to the serde in initialize
> -----------------------------------------------------------
>
> Key: HIVE-127
> URL: https://issues.apache.org/jira/browse/HIVE-127
> Project: Hadoop Hive
> Issue Type: Bug
> Reporter: Namit Jain
> Assignee: Namit Jain
> Attachments: patch-127.txt
>
>
> Fetch task passes a null job to the serde during initialize - it should not assume that the serde does not need the job.
> This assumption is valid for MetadataTypedColumnSetSerde, but need not be valid always
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HIVE-127) fetch task does not pass the job to
the serde in initialize
Posted by "Namit Jain (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-127?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12654486#action_12654486 ]
Namit Jain commented on HIVE-127:
---------------------------------
causes null pointer exception when the serde of the table tries to use the conf.
Mostly, we use MetadataColumnTypedSerde which does not use it - so things work fine.
But, in case of optimizations (no mapreduce job) - select * from context where partitioning column conditions only
context serde uses conf and dies
> fetch task does not pass the job to the serde in initialize
> -----------------------------------------------------------
>
> Key: HIVE-127
> URL: https://issues.apache.org/jira/browse/HIVE-127
> Project: Hadoop Hive
> Issue Type: Bug
> Reporter: Namit Jain
> Assignee: Namit Jain
>
> Fetch task passes a null job to the serde during initialize - it should not assume that the serde does not need the job.
> This assumption is valid for MetadataTypedColumnSetSerde, but need not be valid always
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.