You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hive.apache.org by "Namit Jain (JIRA)" <ji...@apache.org> on 2008/12/08 20:20:44 UTC
[jira] Created: (HIVE-134) job conf being passed to serde
(initialize)
job conf being passed to serde (initialize)
-------------------------------------------
Key: HIVE-134
URL: https://issues.apache.org/jira/browse/HIVE-134
Project: Hadoop Hive
Issue Type: Bug
Components: Serializers/Deserializers
Reporter: Namit Jain
Currently, job conf is being passed to serde - serde should not need job conf. Anyway, properties is being passed around which can contain any name/value pairs.
Once jira-127 is committed, this should be done in a follow-up
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HIVE-134) job conf being passed to serde
(initialize)
Posted by "Raghotham Murthy (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-134?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12654548#action_12654548 ]
Raghotham Murthy commented on HIVE-134:
---------------------------------------
@Joy
If there are session options that affect the serde, these options should ideally be extracted from jobconf and inserted into properties object. This documents the parameters that affect the serde atleast in the code. Right now jobconf is this opaque object which is passed around (and potentially modified) everywhere. But this is not a bug. This jira code cleanup task to make code more readable.
> job conf being passed to serde (initialize)
> -------------------------------------------
>
> Key: HIVE-134
> URL: https://issues.apache.org/jira/browse/HIVE-134
> Project: Hadoop Hive
> Issue Type: Bug
> Components: Serializers/Deserializers
> Reporter: Namit Jain
>
> Currently, job conf is being passed to serde - serde should not need job conf. Anyway, properties is being passed around which can contain any name/value pairs.
> Once jira-127 is committed, this should be done in a follow-up
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HIVE-134) job conf being passed to serde
(initialize)
Posted by "Raghotham Murthy (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-134?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12654560#action_12654560 ]
Raghotham Murthy commented on HIVE-134:
---------------------------------------
Why does adding an interface to specify required jobconf options in the serde make it harder to write serdes? Its about documenting what is required by the serde. It will be make it more clear for the serde writer (to specify) as well as the serde user (to figure out) what runtime parameters are required.
The problem right now is that there is no easy way for a user/developer to figure out what the configurable parameters are for using a serde. Of course, since we have only a handful of serdes this is not an urgent issue right now.
> job conf being passed to serde (initialize)
> -------------------------------------------
>
> Key: HIVE-134
> URL: https://issues.apache.org/jira/browse/HIVE-134
> Project: Hadoop Hive
> Issue Type: Bug
> Components: Serializers/Deserializers
> Reporter: Namit Jain
>
> Currently, job conf is being passed to serde - serde should not need job conf. Anyway, properties is being passed around which can contain any name/value pairs.
> Once jira-127 is committed, this should be done in a follow-up
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HIVE-134) job conf being passed to serde
(initialize)
Posted by "Joydeep Sen Sarma (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-134?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12654552#action_12654552 ]
Joydeep Sen Sarma commented on HIVE-134:
----------------------------------------
i disagree.
the whole idea with serde is that these are opaque to hive except for the functional interfaces exposed via the serde interface. as u can see - the interfaces don't currently have any information about 'options' that the serde may use. Hence we pass the jobconf to the serde so that it can get whatsoever it pleases.
i don't see the point of this jira. we are trying to create a more strongly coupled system. in order to do what Raghu is saying - now we will need another interface in the serde to get the list of required options, and then extract them from the jobconf and then pass those in. what good will this serve? why are we making it even harder to write serdes?
what real and existing problem are we trying to solve here?
> job conf being passed to serde (initialize)
> -------------------------------------------
>
> Key: HIVE-134
> URL: https://issues.apache.org/jira/browse/HIVE-134
> Project: Hadoop Hive
> Issue Type: Bug
> Components: Serializers/Deserializers
> Reporter: Namit Jain
>
> Currently, job conf is being passed to serde - serde should not need job conf. Anyway, properties is being passed around which can contain any name/value pairs.
> Once jira-127 is committed, this should be done in a follow-up
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HIVE-134) job conf being passed to serde
(initialize)
Posted by "Zheng Shao (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-134?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12654629#action_12654629 ]
Zheng Shao commented on HIVE-134:
---------------------------------
btw there is a difference between configuration and jobconf: jobconf is a subclass of configuration: http://hadoop.apache.org/core/docs/current/api/org/apache/hadoop/mapred/JobConf.html
We are requiring configuration instead of jobconf.
Joy and Raghu and I discussed offline and agreed this is not the current hadoop way of doing things. So we probably won't do anything unless hadoop accepts this way of configuration.
> job conf being passed to serde (initialize)
> -------------------------------------------
>
> Key: HIVE-134
> URL: https://issues.apache.org/jira/browse/HIVE-134
> Project: Hadoop Hive
> Issue Type: Bug
> Components: Serializers/Deserializers
> Reporter: Namit Jain
>
> Currently, job conf is being passed to serde - serde should not need job conf. Anyway, properties is being passed around which can contain any name/value pairs.
> Once jira-127 is committed, this should be done in a follow-up
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Resolved: (HIVE-134) job conf being passed to serde
(initialize)
Posted by "Zheng Shao (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-134?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Zheng Shao resolved HIVE-134.
-----------------------------
Resolution: Won't Fix
> job conf being passed to serde (initialize)
> -------------------------------------------
>
> Key: HIVE-134
> URL: https://issues.apache.org/jira/browse/HIVE-134
> Project: Hadoop Hive
> Issue Type: Bug
> Components: Serializers/Deserializers
> Reporter: Namit Jain
>
> Currently, job conf is being passed to serde - serde should not need job conf. Anyway, properties is being passed around which can contain any name/value pairs.
> Once jira-127 is committed, this should be done in a follow-up
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HIVE-134) job conf being passed to serde
(initialize)
Posted by "Joydeep Sen Sarma (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-134?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12654615#action_12654615 ]
Joydeep Sen Sarma commented on HIVE-134:
----------------------------------------
of course it makes it harder. it's one more interface that you are forced to implement for full functionality.
the other problem you are not considering is if the SerDe invokes some other class that requires a Conf to bootstrap (see below for why). Now the SerDe author has to know what options they might be interested in and what options whatever code they are invoking might be interested in. This is a non-starter.
here's some background: almost all hadoop classes implement Configurable - which means they get a Configuration object passed in at initialization time. If someone asked me what made it possible to implement Hive on Hadoop easily - this would be one of the single biggest reasons. Entire Hive execution system is bootstrapped by configuring all manners of parameters inside Hadoop's Conf with minimal overhead from our side.
also - the goal you have is nice (create a registry of options to control serde behavior - possibly with metadata about semantics/values). so yes - we can have additional interface. But we should not FORCE people to provide this metadata and implement this interface for correct functioning (and take away the flexibility to have parameters that are not declared via the new interface). does that make sense?
> job conf being passed to serde (initialize)
> -------------------------------------------
>
> Key: HIVE-134
> URL: https://issues.apache.org/jira/browse/HIVE-134
> Project: Hadoop Hive
> Issue Type: Bug
> Components: Serializers/Deserializers
> Reporter: Namit Jain
>
> Currently, job conf is being passed to serde - serde should not need job conf. Anyway, properties is being passed around which can contain any name/value pairs.
> Once jira-127 is committed, this should be done in a follow-up
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HIVE-134) job conf being passed to serde
(initialize)
Posted by "Joydeep Sen Sarma (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-134?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12654538#action_12654538 ]
Joydeep Sen Sarma commented on HIVE-134:
----------------------------------------
why is this a concern?
i vaguely remember discussing this long time in the past - and don't quite agree that this is a bug. table serde properties capture per-table configuration. but there are session or global configurations that can affect serde behavior. for example - suppose we add a new option to dynamicserde to do type conversions on demand (just as an example). it maybe that we want to test this strategy out - the way we would control it is by passing the configuration object in and controlling it via session options.
> job conf being passed to serde (initialize)
> -------------------------------------------
>
> Key: HIVE-134
> URL: https://issues.apache.org/jira/browse/HIVE-134
> Project: Hadoop Hive
> Issue Type: Bug
> Components: Serializers/Deserializers
> Reporter: Namit Jain
>
> Currently, job conf is being passed to serde - serde should not need job conf. Anyway, properties is being passed around which can contain any name/value pairs.
> Once jira-127 is committed, this should be done in a follow-up
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.