You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@pig.apache.org by "Greg Bowyer (JIRA)" <ji...@apache.org> on 2011/06/09 12:54:58 UTC
[jira] [Created] (PIG-2115) Pig HBaseStorage configuration and
setup issues
Pig HBaseStorage configuration and setup issues
-----------------------------------------------
Key: PIG-2115
URL: https://issues.apache.org/jira/browse/PIG-2115
Project: Pig
Issue Type: Bug
Affects Versions: 0.9.0
Reporter: Greg Bowyer
HBase storage currently configures child MapR jobs assuming that the HBase jars and cluster configuration are present on the hadoop cluster that will run the pig program. In some circumstances, and for some configurations this causes problems with both the configuration suddenly becoming defaulted during a job run as well as ClassNotFound exceptions occurring due to the child MapReduce job being incorrectly configured.
As such, the hbase built in configuration is merged with the job configuration using the inbuilt pig classes for this functionality, this prevents the jobs configuration from being overwritten by defaults.
At the same time this change has been mirrored in the relevant setup methods called when the HBaseStorage is used as a StoreFunc; this means that the StoreFunc also correctly sets up the relevant classpath preventing ClassNotFound exceptions in clusters that are not adding the HBase classes to the HADOOP_CLASSPATH.
Attached is a patch that hopefully fixes these issues for others.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (PIG-2115) Pig HBaseStorage configuration and
setup issues
Posted by "Greg Bowyer (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-2115?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13065619#comment-13065619 ]
Greg Bowyer commented on PIG-2115:
----------------------------------
Fine point, do you want me to change the patch ?
> Pig HBaseStorage configuration and setup issues
> -----------------------------------------------
>
> Key: PIG-2115
> URL: https://issues.apache.org/jira/browse/PIG-2115
> Project: Pig
> Issue Type: Bug
> Affects Versions: 0.9.0
> Reporter: Greg Bowyer
> Labels: hbase, load, storage
> Attachments: HBaseStorage-correctly-configure-child-MapR-task-v1.patch
>
>
> HBase storage currently configures child MapR jobs assuming that the HBase jars and cluster configuration are present on the hadoop cluster that will run the pig program. In some circumstances, and for some configurations this causes problems with both the configuration suddenly becoming defaulted during a job run as well as ClassNotFound exceptions occurring due to the child MapReduce job being incorrectly configured.
> As such, the hbase built in configuration is merged with the job configuration using the inbuilt pig classes for this functionality, this prevents the jobs configuration from being overwritten by defaults.
> At the same time this change has been mirrored in the relevant setup methods called when the HBaseStorage is used as a StoreFunc; this means that the StoreFunc also correctly sets up the relevant classpath preventing ClassNotFound exceptions in clusters that are not adding the HBase classes to the HADOOP_CLASSPATH.
> Attached is a patch that hopefully fixes these issues for others.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (PIG-2115) Pig HBaseStorage configuration and
setup issues
Posted by "Dmitriy V. Ryaboy (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-2115?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Dmitriy V. Ryaboy reassigned PIG-2115:
--------------------------------------
Assignee: Greg Bowyer
credit where credit's due :)
> Pig HBaseStorage configuration and setup issues
> -----------------------------------------------
>
> Key: PIG-2115
> URL: https://issues.apache.org/jira/browse/PIG-2115
> Project: Pig
> Issue Type: Bug
> Affects Versions: 0.9.0
> Reporter: Greg Bowyer
> Assignee: Greg Bowyer
> Labels: hbase, load, storage
> Fix For: 0.10.0
>
> Attachments: HBaseStorage-correctly-configure-child-MapR-task-v1.patch, HBaseStorage-correctly-configure-child-MapR-task-v2.patch, PIG-2115.3.patch
>
>
> HBase storage currently configures child MapR jobs assuming that the HBase jars and cluster configuration are present on the hadoop cluster that will run the pig program. In some circumstances, and for some configurations this causes problems with both the configuration suddenly becoming defaulted during a job run as well as ClassNotFound exceptions occurring due to the child MapReduce job being incorrectly configured.
> As such, the hbase built in configuration is merged with the job configuration using the inbuilt pig classes for this functionality, this prevents the jobs configuration from being overwritten by defaults.
> At the same time this change has been mirrored in the relevant setup methods called when the HBaseStorage is used as a StoreFunc; this means that the StoreFunc also correctly sets up the relevant classpath preventing ClassNotFound exceptions in clusters that are not adding the HBase classes to the HADOOP_CLASSPATH.
> Attached is a patch that hopefully fixes these issues for others.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (PIG-2115) Pig HBaseStorage configuration and
setup issues
Posted by "Greg Bowyer (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-2115?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Greg Bowyer updated PIG-2115:
-----------------------------
Attachment: HBaseStorage-correctly-configure-child-MapR-task-v1.patch
> Pig HBaseStorage configuration and setup issues
> -----------------------------------------------
>
> Key: PIG-2115
> URL: https://issues.apache.org/jira/browse/PIG-2115
> Project: Pig
> Issue Type: Bug
> Affects Versions: 0.9.0
> Reporter: Greg Bowyer
> Labels: hbase, load, storage
> Attachments: HBaseStorage-correctly-configure-child-MapR-task-v1.patch
>
>
> HBase storage currently configures child MapR jobs assuming that the HBase jars and cluster configuration are present on the hadoop cluster that will run the pig program. In some circumstances, and for some configurations this causes problems with both the configuration suddenly becoming defaulted during a job run as well as ClassNotFound exceptions occurring due to the child MapReduce job being incorrectly configured.
> As such, the hbase built in configuration is merged with the job configuration using the inbuilt pig classes for this functionality, this prevents the jobs configuration from being overwritten by defaults.
> At the same time this change has been mirrored in the relevant setup methods called when the HBaseStorage is used as a StoreFunc; this means that the StoreFunc also correctly sets up the relevant classpath preventing ClassNotFound exceptions in clusters that are not adding the HBase classes to the HADOOP_CLASSPATH.
> Attached is a patch that hopefully fixes these issues for others.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (PIG-2115) Pig HBaseStorage configuration and
setup issues
Posted by "Dmitriy V. Ryaboy (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-2115?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Dmitriy V. Ryaboy updated PIG-2115:
-----------------------------------
Attachment: PIG-2115.3.patch
Code drifted a bit since this was written (my fault), I had to modify the patch slightly.
Tests pass.
Greg can you take a look and let me know if this looks ok?
> Pig HBaseStorage configuration and setup issues
> -----------------------------------------------
>
> Key: PIG-2115
> URL: https://issues.apache.org/jira/browse/PIG-2115
> Project: Pig
> Issue Type: Bug
> Affects Versions: 0.9.0
> Reporter: Greg Bowyer
> Labels: hbase, load, storage
> Attachments: HBaseStorage-correctly-configure-child-MapR-task-v1.patch, HBaseStorage-correctly-configure-child-MapR-task-v2.patch, PIG-2115.3.patch
>
>
> HBase storage currently configures child MapR jobs assuming that the HBase jars and cluster configuration are present on the hadoop cluster that will run the pig program. In some circumstances, and for some configurations this causes problems with both the configuration suddenly becoming defaulted during a job run as well as ClassNotFound exceptions occurring due to the child MapReduce job being incorrectly configured.
> As such, the hbase built in configuration is merged with the job configuration using the inbuilt pig classes for this functionality, this prevents the jobs configuration from being overwritten by defaults.
> At the same time this change has been mirrored in the relevant setup methods called when the HBaseStorage is used as a StoreFunc; this means that the StoreFunc also correctly sets up the relevant classpath preventing ClassNotFound exceptions in clusters that are not adding the HBase classes to the HADOOP_CLASSPATH.
> Attached is a patch that hopefully fixes these issues for others.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (PIG-2115) Pig HBaseStorage configuration and
setup issues
Posted by "Greg Bowyer (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-2115?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13094130#comment-13094130 ]
Greg Bowyer commented on PIG-2115:
----------------------------------
did we get this looked at ? I just found this patch at the end of my todo list :S
> Pig HBaseStorage configuration and setup issues
> -----------------------------------------------
>
> Key: PIG-2115
> URL: https://issues.apache.org/jira/browse/PIG-2115
> Project: Pig
> Issue Type: Bug
> Affects Versions: 0.9.0
> Reporter: Greg Bowyer
> Labels: hbase, load, storage
> Attachments: HBaseStorage-correctly-configure-child-MapR-task-v1.patch, HBaseStorage-correctly-configure-child-MapR-task-v2.patch
>
>
> HBase storage currently configures child MapR jobs assuming that the HBase jars and cluster configuration are present on the hadoop cluster that will run the pig program. In some circumstances, and for some configurations this causes problems with both the configuration suddenly becoming defaulted during a job run as well as ClassNotFound exceptions occurring due to the child MapReduce job being incorrectly configured.
> As such, the hbase built in configuration is merged with the job configuration using the inbuilt pig classes for this functionality, this prevents the jobs configuration from being overwritten by defaults.
> At the same time this change has been mirrored in the relevant setup methods called when the HBaseStorage is used as a StoreFunc; this means that the StoreFunc also correctly sets up the relevant classpath preventing ClassNotFound exceptions in clusters that are not adding the HBase classes to the HADOOP_CLASSPATH.
> Attached is a patch that hopefully fixes these issues for others.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (PIG-2115) Pig HBaseStorage configuration and
setup issues
Posted by "Dmitriy V. Ryaboy (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-2115?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13046694#comment-13046694 ]
Dmitriy V. Ryaboy commented on PIG-2115:
----------------------------------------
Great, thanks for doing that. Looks reasonable.
I'll give it a spin and commit if it works.
> Pig HBaseStorage configuration and setup issues
> -----------------------------------------------
>
> Key: PIG-2115
> URL: https://issues.apache.org/jira/browse/PIG-2115
> Project: Pig
> Issue Type: Bug
> Affects Versions: 0.9.0
> Reporter: Greg Bowyer
> Labels: hbase, load, storage
> Attachments: HBaseStorage-correctly-configure-child-MapR-task-v1.patch
>
>
> HBase storage currently configures child MapR jobs assuming that the HBase jars and cluster configuration are present on the hadoop cluster that will run the pig program. In some circumstances, and for some configurations this causes problems with both the configuration suddenly becoming defaulted during a job run as well as ClassNotFound exceptions occurring due to the child MapReduce job being incorrectly configured.
> As such, the hbase built in configuration is merged with the job configuration using the inbuilt pig classes for this functionality, this prevents the jobs configuration from being overwritten by defaults.
> At the same time this change has been mirrored in the relevant setup methods called when the HBaseStorage is used as a StoreFunc; this means that the StoreFunc also correctly sets up the relevant classpath preventing ClassNotFound exceptions in clusters that are not adding the HBase classes to the HADOOP_CLASSPATH.
> Attached is a patch that hopefully fixes these issues for others.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (PIG-2115) Pig HBaseStorage configuration and
setup issues
Posted by "Greg Bowyer (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-2115?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Greg Bowyer updated PIG-2115:
-----------------------------
Status: Patch Available (was: Open)
> Pig HBaseStorage configuration and setup issues
> -----------------------------------------------
>
> Key: PIG-2115
> URL: https://issues.apache.org/jira/browse/PIG-2115
> Project: Pig
> Issue Type: Bug
> Affects Versions: 0.9.0
> Reporter: Greg Bowyer
> Labels: hbase, load, storage
>
> HBase storage currently configures child MapR jobs assuming that the HBase jars and cluster configuration are present on the hadoop cluster that will run the pig program. In some circumstances, and for some configurations this causes problems with both the configuration suddenly becoming defaulted during a job run as well as ClassNotFound exceptions occurring due to the child MapReduce job being incorrectly configured.
> As such, the hbase built in configuration is merged with the job configuration using the inbuilt pig classes for this functionality, this prevents the jobs configuration from being overwritten by defaults.
> At the same time this change has been mirrored in the relevant setup methods called when the HBaseStorage is used as a StoreFunc; this means that the StoreFunc also correctly sets up the relevant classpath preventing ClassNotFound exceptions in clusters that are not adding the HBase classes to the HADOOP_CLASSPATH.
> Attached is a patch that hopefully fixes these issues for others.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (PIG-2115) Pig HBaseStorage configuration and
setup issues
Posted by "Dmitriy V. Ryaboy (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-2115?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13084278#comment-13084278 ]
Dmitriy V. Ryaboy commented on PIG-2115:
----------------------------------------
It's on my list, I'll take a look this weekend.
> Pig HBaseStorage configuration and setup issues
> -----------------------------------------------
>
> Key: PIG-2115
> URL: https://issues.apache.org/jira/browse/PIG-2115
> Project: Pig
> Issue Type: Bug
> Affects Versions: 0.9.0
> Reporter: Greg Bowyer
> Labels: hbase, load, storage
> Attachments: HBaseStorage-correctly-configure-child-MapR-task-v1.patch, HBaseStorage-correctly-configure-child-MapR-task-v2.patch
>
>
> HBase storage currently configures child MapR jobs assuming that the HBase jars and cluster configuration are present on the hadoop cluster that will run the pig program. In some circumstances, and for some configurations this causes problems with both the configuration suddenly becoming defaulted during a job run as well as ClassNotFound exceptions occurring due to the child MapReduce job being incorrectly configured.
> As such, the hbase built in configuration is merged with the job configuration using the inbuilt pig classes for this functionality, this prevents the jobs configuration from being overwritten by defaults.
> At the same time this change has been mirrored in the relevant setup methods called when the HBaseStorage is used as a StoreFunc; this means that the StoreFunc also correctly sets up the relevant classpath preventing ClassNotFound exceptions in clusters that are not adding the HBase classes to the HADOOP_CLASSPATH.
> Attached is a patch that hopefully fixes these issues for others.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (PIG-2115) Pig HBaseStorage configuration and
setup issues
Posted by "Greg Bowyer (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-2115?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Greg Bowyer updated PIG-2115:
-----------------------------
Attachment: HBaseStorage-correctly-configure-child-MapR-task-v2.patch
Sorry it took so long to make these changes, I guess my life also went offline :S
Static import removed
> Pig HBaseStorage configuration and setup issues
> -----------------------------------------------
>
> Key: PIG-2115
> URL: https://issues.apache.org/jira/browse/PIG-2115
> Project: Pig
> Issue Type: Bug
> Affects Versions: 0.9.0
> Reporter: Greg Bowyer
> Labels: hbase, load, storage
> Attachments: HBaseStorage-correctly-configure-child-MapR-task-v1.patch, HBaseStorage-correctly-configure-child-MapR-task-v2.patch
>
>
> HBase storage currently configures child MapR jobs assuming that the HBase jars and cluster configuration are present on the hadoop cluster that will run the pig program. In some circumstances, and for some configurations this causes problems with both the configuration suddenly becoming defaulted during a job run as well as ClassNotFound exceptions occurring due to the child MapReduce job being incorrectly configured.
> As such, the hbase built in configuration is merged with the job configuration using the inbuilt pig classes for this functionality, this prevents the jobs configuration from being overwritten by defaults.
> At the same time this change has been mirrored in the relevant setup methods called when the HBaseStorage is used as a StoreFunc; this means that the StoreFunc also correctly sets up the relevant classpath preventing ClassNotFound exceptions in clusters that are not adding the HBase classes to the HADOOP_CLASSPATH.
> Attached is a patch that hopefully fixes these issues for others.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (PIG-2115) Pig HBaseStorage configuration and
setup issues
Posted by "Dmitriy V. Ryaboy (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-2115?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Dmitriy V. Ryaboy updated PIG-2115:
-----------------------------------
Resolution: Fixed
Fix Version/s: 0.10
Status: Resolved (was: Patch Available)
Committed to trunk.
> Pig HBaseStorage configuration and setup issues
> -----------------------------------------------
>
> Key: PIG-2115
> URL: https://issues.apache.org/jira/browse/PIG-2115
> Project: Pig
> Issue Type: Bug
> Affects Versions: 0.9.0
> Reporter: Greg Bowyer
> Labels: hbase, load, storage
> Fix For: 0.10
>
> Attachments: HBaseStorage-correctly-configure-child-MapR-task-v1.patch, HBaseStorage-correctly-configure-child-MapR-task-v2.patch, PIG-2115.3.patch
>
>
> HBase storage currently configures child MapR jobs assuming that the HBase jars and cluster configuration are present on the hadoop cluster that will run the pig program. In some circumstances, and for some configurations this causes problems with both the configuration suddenly becoming defaulted during a job run as well as ClassNotFound exceptions occurring due to the child MapReduce job being incorrectly configured.
> As such, the hbase built in configuration is merged with the job configuration using the inbuilt pig classes for this functionality, this prevents the jobs configuration from being overwritten by defaults.
> At the same time this change has been mirrored in the relevant setup methods called when the HBaseStorage is used as a StoreFunc; this means that the StoreFunc also correctly sets up the relevant classpath preventing ClassNotFound exceptions in clusters that are not adding the HBase classes to the HADOOP_CLASSPATH.
> Attached is a patch that hopefully fixes these issues for others.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (PIG-2115) Pig HBaseStorage configuration and
setup issues
Posted by "Alan Gates (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-2115?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Alan Gates updated PIG-2115:
----------------------------
Status: Patch Available (was: Open)
Marking as submit patch so this gets reviewed.
> Pig HBaseStorage configuration and setup issues
> -----------------------------------------------
>
> Key: PIG-2115
> URL: https://issues.apache.org/jira/browse/PIG-2115
> Project: Pig
> Issue Type: Bug
> Affects Versions: 0.9.0
> Reporter: Greg Bowyer
> Labels: hbase, load, storage
> Attachments: HBaseStorage-correctly-configure-child-MapR-task-v1.patch, HBaseStorage-correctly-configure-child-MapR-task-v2.patch
>
>
> HBase storage currently configures child MapR jobs assuming that the HBase jars and cluster configuration are present on the hadoop cluster that will run the pig program. In some circumstances, and for some configurations this causes problems with both the configuration suddenly becoming defaulted during a job run as well as ClassNotFound exceptions occurring due to the child MapReduce job being incorrectly configured.
> As such, the hbase built in configuration is merged with the job configuration using the inbuilt pig classes for this functionality, this prevents the jobs configuration from being overwritten by defaults.
> At the same time this change has been mirrored in the relevant setup methods called when the HBaseStorage is used as a StoreFunc; this means that the StoreFunc also correctly sets up the relevant classpath preventing ClassNotFound exceptions in clusters that are not adding the HBase classes to the HADOOP_CLASSPATH.
> Attached is a patch that hopefully fixes these issues for others.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (PIG-2115) Pig HBaseStorage configuration and
setup issues
Posted by "Dmitriy V. Ryaboy (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-2115?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Dmitriy V. Ryaboy updated PIG-2115:
-----------------------------------
Status: Open (was: Patch Available)
> Pig HBaseStorage configuration and setup issues
> -----------------------------------------------
>
> Key: PIG-2115
> URL: https://issues.apache.org/jira/browse/PIG-2115
> Project: Pig
> Issue Type: Bug
> Affects Versions: 0.9.0
> Reporter: Greg Bowyer
> Labels: hbase, load, storage
> Attachments: HBaseStorage-correctly-configure-child-MapR-task-v1.patch
>
>
> HBase storage currently configures child MapR jobs assuming that the HBase jars and cluster configuration are present on the hadoop cluster that will run the pig program. In some circumstances, and for some configurations this causes problems with both the configuration suddenly becoming defaulted during a job run as well as ClassNotFound exceptions occurring due to the child MapReduce job being incorrectly configured.
> As such, the hbase built in configuration is merged with the job configuration using the inbuilt pig classes for this functionality, this prevents the jobs configuration from being overwritten by defaults.
> At the same time this change has been mirrored in the relevant setup methods called when the HBaseStorage is used as a StoreFunc; this means that the StoreFunc also correctly sets up the relevant classpath preventing ClassNotFound exceptions in clusters that are not adding the HBase classes to the HADOOP_CLASSPATH.
> Attached is a patch that hopefully fixes these issues for others.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (PIG-2115) Pig HBaseStorage configuration and
setup issues
Posted by "Dmitriy V. Ryaboy (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-2115?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13065634#comment-13065634 ]
Dmitriy V. Ryaboy commented on PIG-2115:
----------------------------------------
ifn' you don't mind. I can do it if you are unable to, but I am about to go offline for a week, and have a bunch of other patches in flight :).
> Pig HBaseStorage configuration and setup issues
> -----------------------------------------------
>
> Key: PIG-2115
> URL: https://issues.apache.org/jira/browse/PIG-2115
> Project: Pig
> Issue Type: Bug
> Affects Versions: 0.9.0
> Reporter: Greg Bowyer
> Labels: hbase, load, storage
> Attachments: HBaseStorage-correctly-configure-child-MapR-task-v1.patch
>
>
> HBase storage currently configures child MapR jobs assuming that the HBase jars and cluster configuration are present on the hadoop cluster that will run the pig program. In some circumstances, and for some configurations this causes problems with both the configuration suddenly becoming defaulted during a job run as well as ClassNotFound exceptions occurring due to the child MapReduce job being incorrectly configured.
> As such, the hbase built in configuration is merged with the job configuration using the inbuilt pig classes for this functionality, this prevents the jobs configuration from being overwritten by defaults.
> At the same time this change has been mirrored in the relevant setup methods called when the HBaseStorage is used as a StoreFunc; this means that the StoreFunc also correctly sets up the relevant classpath preventing ClassNotFound exceptions in clusters that are not adding the HBase classes to the HADOOP_CLASSPATH.
> Attached is a patch that hopefully fixes these issues for others.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (PIG-2115) Pig HBaseStorage configuration and
setup issues
Posted by "Dmitriy V. Ryaboy (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-2115?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13065460#comment-13065460 ]
Dmitriy V. Ryaboy commented on PIG-2115:
----------------------------------------
Sorry for the long wait.
This looks fine, but stylistically, I don't see a reason for the "import static" construct being used. The use of import static is discouraged by its own documentation (http://download.oracle.com/javase/1.5.0/docs/guide/language/static-import.html -- see "when should I use import static"), because it makes it unclear where a given method is implemented.
> Pig HBaseStorage configuration and setup issues
> -----------------------------------------------
>
> Key: PIG-2115
> URL: https://issues.apache.org/jira/browse/PIG-2115
> Project: Pig
> Issue Type: Bug
> Affects Versions: 0.9.0
> Reporter: Greg Bowyer
> Labels: hbase, load, storage
> Attachments: HBaseStorage-correctly-configure-child-MapR-task-v1.patch
>
>
> HBase storage currently configures child MapR jobs assuming that the HBase jars and cluster configuration are present on the hadoop cluster that will run the pig program. In some circumstances, and for some configurations this causes problems with both the configuration suddenly becoming defaulted during a job run as well as ClassNotFound exceptions occurring due to the child MapReduce job being incorrectly configured.
> As such, the hbase built in configuration is merged with the job configuration using the inbuilt pig classes for this functionality, this prevents the jobs configuration from being overwritten by defaults.
> At the same time this change has been mirrored in the relevant setup methods called when the HBaseStorage is used as a StoreFunc; this means that the StoreFunc also correctly sets up the relevant classpath preventing ClassNotFound exceptions in clusters that are not adding the HBase classes to the HADOOP_CLASSPATH.
> Attached is a patch that hopefully fixes these issues for others.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira