You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hive.apache.org by John Sichi <js...@facebook.com> on 2010/03/05 03:51:27 UTC
storage handlers and HBase integration
Hey folks,
In case you're not following the action over at HIVE-705, we're getting close to having HBase integration committed into Hive. I've written up docs here:
http://wiki.apache.org/hadoop/Hive/HBaseIntegration
http://wiki.apache.org/hadoop/Hive/StorageHandlers
(If you happened to read the first draft of the HBaseIntegration doc a few days ago, I've made a lot of updates today to fill out the details on column mapping.)
As part of commit, we'll be doing some code reviews within Facebook next week and logging a bunch of followup tasks; if you have any comments on the approach or implementation, please pile on in JIRA.
I'll be giving this a quick mention at the Hive user's group later this month, and then a more detailed presentation at the HBase User Group meeting in April.
JVS
Re: storage handlers and HBase integration
Posted by Ray Duong <ra...@gmail.com>.
Thanks John,
That resolved the build issue. I'll let you know if I find any other issues
over the next few days as I test it against our Hbase cluster.
Hope to talk to you further at the Hive Usergroup Meeting.
Thanks,
Ray Duong@sharethis
On Fri, Mar 5, 2010 at 10:49 AM, John Sichi <js...@facebook.com> wrote:
> Oops, I left some ivy crud in the patch which shouldn't be there. I'll
> upload a new patch, but you can just delete these lines out of
> hbase-handler/ivy.xml:
>
> <dependency org="hadoop" name="hbase" rev="${hbase.version}">
> <artifact name="hbase" type="source" ext="tar.gz"/>
> </dependency>
>
> JVS
> ________________________________________
> From: Ray Duong [ray.duong@gmail.com]
> Sent: Friday, March 05, 2010 9:42 AM
> To: John Sichi
> Cc: hive-user@hadoop.apache.org
> Subject: Re: storage handlers and HBase integration
>
> Hi John,
>
> I removed the .ant directory and rebuilt the ant package but got the
> following error messages. I also made sure I downloaded the hbase and
> zookeeper jar files into the hive-trunk/lib directory.
>
> Thanks again for the help.
>
> Ray Duong@sharethis
>
> ivy-init:
>
> ivy-retrieve-hadoop-source:
> [ivy:retrieve] :: Ivy 2.1.0 - 20090925235825 :: http://ant.apache.org/ivy/::
> [ivy:retrieve] :: loading settings :: file =
> /tmp/hive-trunk/ivy/ivysettings.xml
> [ivy:retrieve] :: resolving dependencies ::
> org.apache.hadoop.hive#contrib;working@domU-12-31-38-02-21-81
> [ivy:retrieve] confs: [default]
> [ivy:retrieve] found hadoop#core;0.20.0 in hadoop-source
> [ivy:retrieve] :: resolution report :: resolve 2493ms :: artifacts dl 2ms
>
> ---------------------------------------------------------------------
> | | modules || artifacts
> |
> | conf | number| search|dwnlded|evicted||
> number|dwnlded|
>
> ---------------------------------------------------------------------
> | default | 2 | 0 | 0 | 0 || 1 | 0
> |
>
> ---------------------------------------------------------------------
> [ivy:retrieve]
> [ivy:retrieve] :: problems summary ::
> [ivy:retrieve] :::: WARNINGS
> [ivy:retrieve] module not found: hadoop#hbase;${hbase.version}
> [ivy:retrieve] ==== hadoop-source: tried
> [ivy:retrieve] -- artifact
> hadoop#hbase;${hbase.version}!hbase.tar.gz(source):
> [ivy:retrieve]
> http://mirror.facebook.net/facebook/hive-deps/hadoop/core/hbase-${hbase.version}/hbase-${hbase.version}.tar.gz<http://mirror.facebook.net/facebook/hive-deps/hadoop/core/hbase-$%7Bhbase.version%7D/hbase-$%7Bhbase.version%7D.tar.gz>
> [ivy:retrieve] ==== apache-snapshot: tried
> [ivy:retrieve]
> https://repository.apache.org/content/repositories/snapshots/hadoop/hbase/${hbase.version}/hbase-${hbase.version}.pom<https://repository.apache.org/content/repositories/snapshots/hadoop/hbase/$%7Bhbase.version%7D/hbase-$%7Bhbase.version%7D.pom>
> [ivy:retrieve] -- artifact
> hadoop#hbase;${hbase.version}!hbase.tar.gz(source):
> [ivy:retrieve]
> https://repository.apache.org/content/repositories/snapshots/hadoop/hbase/${hbase.version}/hbase-${hbase.version}.tar.gz<https://repository.apache.org/content/repositories/snapshots/hadoop/hbase/$%7Bhbase.version%7D/hbase-$%7Bhbase.version%7D.tar.gz>
> [ivy:retrieve] ==== maven2: tried
> [ivy:retrieve]
> http://repo1.maven.org/maven2/hadoop/hbase/${hbase.version}/hbase-${hbase.version}.pom<http://repo1.maven.org/maven2/hadoop/hbase/$%7Bhbase.version%7D/hbase-$%7Bhbase.version%7D.pom>
> [ivy:retrieve] -- artifact
> hadoop#hbase;${hbase.version}!hbase.tar.gz(source):
> [ivy:retrieve]
> http://repo1.maven.org/maven2/hadoop/hbase/${hbase.version}/hbase-${hbase.version}.tar.gz<http://repo1.maven.org/maven2/hadoop/hbase/$%7Bhbase.version%7D/hbase-$%7Bhbase.version%7D.tar.gz>
> [ivy:retrieve] ::::::::::::::::::::::::::::::::::::::::::::::
> [ivy:retrieve] :: UNRESOLVED DEPENDENCIES ::
> [ivy:retrieve] ::::::::::::::::::::::::::::::::::::::::::::::
> [ivy:retrieve] :: hadoop#hbase;${hbase.version}: not found
> [ivy:retrieve] ::::::::::::::::::::::::::::::::::::::::::::::
> [ivy:retrieve]
> [ivy:retrieve] :: USE VERBOSE OR DEBUG MESSAGE LEVEL FOR MORE DETAILS
>
> BUILD FAILED
> /tmp/hive-trunk/build.xml:158: The following error occurred while executing
> this line:
> /tmp/hive-trunk/build.xml:103: The following error occurred while executing
> this line:
> /tmp/hive-trunk/build-common.xml:200: The following error occurred while
> executing this line:
> /tmp/hive-trunk/build-common.xml:191: The following error occurred while
> executing this line:
> /tmp/hive-trunk/build-common.xml:177: impossible to resolve dependencies:
> resolve failed - see output for details
>
> Total time: 34 seconds
>
> On Fri, Mar 5, 2010 at 9:10 AM, John Sichi <jsichi@facebook.com<mailto:
> jsichi@facebook.com>> wrote:
>
> You need to delete your ~/.ant directory to get rid of the old ivy version
> stored there. Nothing to do with the HBase integration; it is due to a
> recent ivy upgrade on trunk.
>
> JVS
>
> On Mar 5, 2010, at 8:39 AM, "Ray Duong" <ray.duong@gmail.com<mailto:
> ray.duong@gmail.com>> wrote:
>
> Hi John,
>
> Nice work on the HBase integration adapator.
>
> I downloaded the Hive trunk and applied the HIVE-705<
> http://issues.apache.org/jira/browse/HIVE-705>.3.patch, but when I tried
> to 'ant package', I get the following error messages below. Is there any
> other settings that is required to build the hive_hbase_handler.jar file?
>
> Thanks,
> Ray Duong@sharethis
>
> >> ant package
>
> ivy-retrieve-hadoop-source:
> [ivy:retrieve] :: Ivy 2.0.0-rc2 - 20081028224207 :: <
> http://ant.apache.org/ivy/> http://ant.apache.org/ivy/ ::
> :: loading settings :: file = /tmp/hive-trunk/ivy/ivysettings.xml
>
> BUILD FAILED
> /tmp/hive-trunk/build.xml:158: The following error occurred while executing
> this line:
> /tmp/hive-trunk/build.xml:103: The following error occurred while executing
> this line:
> /tmp/hive-trunk/shims/build.xml:55: The following error occurred while
> executing this line:
> /tmp/hive-trunk/build-common.xml:177: impossible to configure ivy:settings
> with given file: /tmp/hive-trunk/ivy/ivysettings.xml :
> java.text.ParseException: failed to load settings from
> file:/tmp/hive-trunk/ivy/ivysettings.xml: impossible to set defaultTTL to
> eternal on class org.apache.ivy.core.cache.DefaultRepositoryCacheManager
>
> Total time: 0 seconds
>
> >> ant -v package
>
> Caused by: java.lang.IllegalArgumentException: impossible to set defaultTTL
> to eternal on class org.apache.ivy.core.cache.DefaultRepositoryCacheManager
> at
> org.apache.ivy.util.Configurator.setAttribute(Configurator.java:660)
> at
> org.apache.ivy.core.settings.XmlSettingsParser.inConfiguratorStarted(XmlSettingsParser.java:561)
> at
> org.apache.ivy.core.settings.XmlSettingsParser.startElement(XmlSettingsParser.java:200)
> at org.apache.xerces.parsers.AbstractSAXParser.startElement(Unknown
> Source)
> at
> org.apache.xerces.parsers.AbstractXMLDocumentParser.emptyElement(Unknown
> Source)
> at
> org.apache.xerces.impl.XMLDocumentFragmentScannerImpl.scanStartElement(Unknown
> Source)
> at
> org.apache.xerces.impl.XMLDocumentFragmentScannerImpl$FragmentContentDispatcher.dispatch(Unknown
> Source)
> at
> org.apache.xerces.impl.XMLDocumentFragmentScannerImpl.scanDocument(Unknown
> Source)
> at org.apache.xerces.parsers.XML11Configuration.parse(Unknown
> Source)
> at org.apache.xerces.parsers.XML11Configuration.parse(Unknown
> Source)
> at org.apache.xerces.parsers.XMLParser.parse(Unknown Source)
> at org.apache.xerces.parsers.AbstractSAXParser.parse(Unknown Source)
> at org.apache.xerces.jaxp.SAXParserImpl$JAXPSAXParser.parse(Unknown
> Source)
> at org.apache.xerces.jaxp.SAXParserImpl.parse(Unknown Source)
> at javax.xml.parsers.SAXParser.parse(SAXParser.java:277)
> at
> org.apache.ivy.core.settings.XmlSettingsParser.doParse(XmlSettingsParser.java:159)
> ... 65 more
> Caused by: java.lang.reflect.InvocationTargetException
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at
> org.apache.ivy.util.Configurator.setAttribute(Configurator.java:658)
> ... 80 more
> Caused by: java.lang.IllegalArgumentException: invalid duration 'eternal':
> it must match (?:(\d+)d)? ?(?:(\d+)h)? ?(?:(\d+)m)? ?(?:(\d+)s)?
> ?(?:(\d+)ms)?
> at
> org.apache.ivy.core.cache.DefaultRepositoryCacheManager.parseDuration(DefaultRepositoryCacheManager.java:275)
> at
> org.apache.ivy.core.cache.DefaultRepositoryCacheManager.setDefaultTTL(DefaultRepositoryCacheManager.java:178)
> ... 85 more
>
> Total time: 0 seconds
>
> On Thu, Mar 4, 2010 at 6:51 PM, John Sichi <<m...@facebook.com>
> jsichi@facebook.com<ma...@facebook.com>> wrote:
> Hey folks,
>
> In case you're not following the action over at HIVE-705, we're getting
> close to having HBase integration committed into Hive. I've written up docs
> here:
>
> <http://wiki.apache.org/hadoop/Hive/HBaseIntegration>
> http://wiki.apache.org/hadoop/Hive/HBaseIntegration
> <http://wiki.apache.org/hadoop/Hive/StorageHandlers>
> http://wiki.apache.org/hadoop/Hive/StorageHandlers
>
> (If you happened to read the first draft of the HBaseIntegration doc a few
> days ago, I've made a lot of updates today to fill out the details on column
> mapping.)
>
> As part of commit, we'll be doing some code reviews within Facebook next
> week and logging a bunch of followup tasks; if you have any comments on the
> approach or implementation, please pile on in JIRA.
>
> I'll be giving this a quick mention at the Hive user's group later this
> month, and then a more detailed presentation at the HBase User Group meeting
> in April.
>
> JVS
>
>
>
>
RE: storage handlers and HBase integration
Posted by John Sichi <js...@facebook.com>.
Oops, I left some ivy crud in the patch which shouldn't be there. I'll upload a new patch, but you can just delete these lines out of hbase-handler/ivy.xml:
<dependency org="hadoop" name="hbase" rev="${hbase.version}">
<artifact name="hbase" type="source" ext="tar.gz"/>
</dependency>
JVS
________________________________________
From: Ray Duong [ray.duong@gmail.com]
Sent: Friday, March 05, 2010 9:42 AM
To: John Sichi
Cc: hive-user@hadoop.apache.org
Subject: Re: storage handlers and HBase integration
Hi John,
I removed the .ant directory and rebuilt the ant package but got the following error messages. I also made sure I downloaded the hbase and zookeeper jar files into the hive-trunk/lib directory.
Thanks again for the help.
Ray Duong@sharethis
ivy-init:
ivy-retrieve-hadoop-source:
[ivy:retrieve] :: Ivy 2.1.0 - 20090925235825 :: http://ant.apache.org/ivy/ ::
[ivy:retrieve] :: loading settings :: file = /tmp/hive-trunk/ivy/ivysettings.xml
[ivy:retrieve] :: resolving dependencies :: org.apache.hadoop.hive#contrib;working@domU-12-31-38-02-21-81
[ivy:retrieve] confs: [default]
[ivy:retrieve] found hadoop#core;0.20.0 in hadoop-source
[ivy:retrieve] :: resolution report :: resolve 2493ms :: artifacts dl 2ms
---------------------------------------------------------------------
| | modules || artifacts |
| conf | number| search|dwnlded|evicted|| number|dwnlded|
---------------------------------------------------------------------
| default | 2 | 0 | 0 | 0 || 1 | 0 |
---------------------------------------------------------------------
[ivy:retrieve]
[ivy:retrieve] :: problems summary ::
[ivy:retrieve] :::: WARNINGS
[ivy:retrieve] module not found: hadoop#hbase;${hbase.version}
[ivy:retrieve] ==== hadoop-source: tried
[ivy:retrieve] -- artifact hadoop#hbase;${hbase.version}!hbase.tar.gz(source):
[ivy:retrieve] http://mirror.facebook.net/facebook/hive-deps/hadoop/core/hbase-${hbase.version}/hbase-${hbase.version}.tar.gz
[ivy:retrieve] ==== apache-snapshot: tried
[ivy:retrieve] https://repository.apache.org/content/repositories/snapshots/hadoop/hbase/${hbase.version}/hbase-${hbase.version}.pom
[ivy:retrieve] -- artifact hadoop#hbase;${hbase.version}!hbase.tar.gz(source):
[ivy:retrieve] https://repository.apache.org/content/repositories/snapshots/hadoop/hbase/${hbase.version}/hbase-${hbase.version}.tar.gz
[ivy:retrieve] ==== maven2: tried
[ivy:retrieve] http://repo1.maven.org/maven2/hadoop/hbase/${hbase.version}/hbase-${hbase.version}.pom
[ivy:retrieve] -- artifact hadoop#hbase;${hbase.version}!hbase.tar.gz(source):
[ivy:retrieve] http://repo1.maven.org/maven2/hadoop/hbase/${hbase.version}/hbase-${hbase.version}.tar.gz
[ivy:retrieve] ::::::::::::::::::::::::::::::::::::::::::::::
[ivy:retrieve] :: UNRESOLVED DEPENDENCIES ::
[ivy:retrieve] ::::::::::::::::::::::::::::::::::::::::::::::
[ivy:retrieve] :: hadoop#hbase;${hbase.version}: not found
[ivy:retrieve] ::::::::::::::::::::::::::::::::::::::::::::::
[ivy:retrieve]
[ivy:retrieve] :: USE VERBOSE OR DEBUG MESSAGE LEVEL FOR MORE DETAILS
BUILD FAILED
/tmp/hive-trunk/build.xml:158: The following error occurred while executing this line:
/tmp/hive-trunk/build.xml:103: The following error occurred while executing this line:
/tmp/hive-trunk/build-common.xml:200: The following error occurred while executing this line:
/tmp/hive-trunk/build-common.xml:191: The following error occurred while executing this line:
/tmp/hive-trunk/build-common.xml:177: impossible to resolve dependencies:
resolve failed - see output for details
Total time: 34 seconds
On Fri, Mar 5, 2010 at 9:10 AM, John Sichi <js...@facebook.com>> wrote:
You need to delete your ~/.ant directory to get rid of the old ivy version stored there. Nothing to do with the HBase integration; it is due to a recent ivy upgrade on trunk.
JVS
On Mar 5, 2010, at 8:39 AM, "Ray Duong" <ra...@gmail.com>> wrote:
Hi John,
Nice work on the HBase integration adapator.
I downloaded the Hive trunk and applied the HIVE-705<http://issues.apache.org/jira/browse/HIVE-705>.3.patch, but when I tried to 'ant package', I get the following error messages below. Is there any other settings that is required to build the hive_hbase_handler.jar file?
Thanks,
Ray Duong@sharethis
>> ant package
ivy-retrieve-hadoop-source:
[ivy:retrieve] :: Ivy 2.0.0-rc2 - 20081028224207 :: <http://ant.apache.org/ivy/> http://ant.apache.org/ivy/ ::
:: loading settings :: file = /tmp/hive-trunk/ivy/ivysettings.xml
BUILD FAILED
/tmp/hive-trunk/build.xml:158: The following error occurred while executing this line:
/tmp/hive-trunk/build.xml:103: The following error occurred while executing this line:
/tmp/hive-trunk/shims/build.xml:55: The following error occurred while executing this line:
/tmp/hive-trunk/build-common.xml:177: impossible to configure ivy:settings with given file: /tmp/hive-trunk/ivy/ivysettings.xml : java.text.ParseException: failed to load settings from file:/tmp/hive-trunk/ivy/ivysettings.xml: impossible to set defaultTTL to eternal on class org.apache.ivy.core.cache.DefaultRepositoryCacheManager
Total time: 0 seconds
>> ant -v package
Caused by: java.lang.IllegalArgumentException: impossible to set defaultTTL to eternal on class org.apache.ivy.core.cache.DefaultRepositoryCacheManager
at org.apache.ivy.util.Configurator.setAttribute(Configurator.java:660)
at org.apache.ivy.core.settings.XmlSettingsParser.inConfiguratorStarted(XmlSettingsParser.java:561)
at org.apache.ivy.core.settings.XmlSettingsParser.startElement(XmlSettingsParser.java:200)
at org.apache.xerces.parsers.AbstractSAXParser.startElement(Unknown Source)
at org.apache.xerces.parsers.AbstractXMLDocumentParser.emptyElement(Unknown Source)
at org.apache.xerces.impl.XMLDocumentFragmentScannerImpl.scanStartElement(Unknown Source)
at org.apache.xerces.impl.XMLDocumentFragmentScannerImpl$FragmentContentDispatcher.dispatch(Unknown Source)
at org.apache.xerces.impl.XMLDocumentFragmentScannerImpl.scanDocument(Unknown Source)
at org.apache.xerces.parsers.XML11Configuration.parse(Unknown Source)
at org.apache.xerces.parsers.XML11Configuration.parse(Unknown Source)
at org.apache.xerces.parsers.XMLParser.parse(Unknown Source)
at org.apache.xerces.parsers.AbstractSAXParser.parse(Unknown Source)
at org.apache.xerces.jaxp.SAXParserImpl$JAXPSAXParser.parse(Unknown Source)
at org.apache.xerces.jaxp.SAXParserImpl.parse(Unknown Source)
at javax.xml.parsers.SAXParser.parse(SAXParser.java:277)
at org.apache.ivy.core.settings.XmlSettingsParser.doParse(XmlSettingsParser.java:159)
... 65 more
Caused by: java.lang.reflect.InvocationTargetException
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.ivy.util.Configurator.setAttribute(Configurator.java:658)
... 80 more
Caused by: java.lang.IllegalArgumentException: invalid duration 'eternal': it must match (?:(\d+)d)? ?(?:(\d+)h)? ?(?:(\d+)m)? ?(?:(\d+)s)? ?(?:(\d+)ms)?
at org.apache.ivy.core.cache.DefaultRepositoryCacheManager.parseDuration(DefaultRepositoryCacheManager.java:275)
at org.apache.ivy.core.cache.DefaultRepositoryCacheManager.setDefaultTTL(DefaultRepositoryCacheManager.java:178)
... 85 more
Total time: 0 seconds
On Thu, Mar 4, 2010 at 6:51 PM, John Sichi <<m...@facebook.com>> wrote:
Hey folks,
In case you're not following the action over at HIVE-705, we're getting close to having HBase integration committed into Hive. I've written up docs here:
<http://wiki.apache.org/hadoop/Hive/HBaseIntegration>http://wiki.apache.org/hadoop/Hive/HBaseIntegration
<http://wiki.apache.org/hadoop/Hive/StorageHandlers>http://wiki.apache.org/hadoop/Hive/StorageHandlers
(If you happened to read the first draft of the HBaseIntegration doc a few days ago, I've made a lot of updates today to fill out the details on column mapping.)
As part of commit, we'll be doing some code reviews within Facebook next week and logging a bunch of followup tasks; if you have any comments on the approach or implementation, please pile on in JIRA.
I'll be giving this a quick mention at the Hive user's group later this month, and then a more detailed presentation at the HBase User Group meeting in April.
JVS
Re: storage handlers and HBase integration
Posted by Ray Duong <ra...@gmail.com>.
Hi John,
I removed the .ant directory and rebuilt the ant package but got the
following error messages. I also made sure I downloaded the hbase and
zookeeper jar files into the hive-trunk/lib directory.
Thanks again for the help.
Ray Duong@sharethis
ivy-init:
ivy-retrieve-hadoop-source:
[ivy:retrieve] :: Ivy 2.1.0 - 20090925235825 :: http://ant.apache.org/ivy/::
[ivy:retrieve] :: loading settings :: file =
/tmp/hive-trunk/ivy/ivysettings.xml
[ivy:retrieve] :: resolving dependencies ::
org.apache.hadoop.hive#contrib;working@domU-12-31-38-02-21-81
[ivy:retrieve] confs: [default]
[ivy:retrieve] found hadoop#core;0.20.0 in hadoop-source
[ivy:retrieve] :: resolution report :: resolve 2493ms :: artifacts dl 2ms
---------------------------------------------------------------------
| | modules || artifacts
|
| conf | number| search|dwnlded|evicted||
number|dwnlded|
---------------------------------------------------------------------
| default | 2 | 0 | 0 | 0 || 1 | 0
|
---------------------------------------------------------------------
[ivy:retrieve]
[ivy:retrieve] :: problems summary ::
[ivy:retrieve] :::: WARNINGS
[ivy:retrieve] module not found: hadoop#hbase;${hbase.version}
[ivy:retrieve] ==== hadoop-source: tried
[ivy:retrieve] -- artifact
hadoop#hbase;${hbase.version}!hbase.tar.gz(source):
[ivy:retrieve]
http://mirror.facebook.net/facebook/hive-deps/hadoop/core/hbase-${hbase.version}/hbase-${hbase.version}.tar.gz
[ivy:retrieve] ==== apache-snapshot: tried
[ivy:retrieve]
https://repository.apache.org/content/repositories/snapshots/hadoop/hbase/${hbase.version}/hbase-${hbase.version}.pom
[ivy:retrieve] -- artifact
hadoop#hbase;${hbase.version}!hbase.tar.gz(source):
[ivy:retrieve]
https://repository.apache.org/content/repositories/snapshots/hadoop/hbase/${hbase.version}/hbase-${hbase.version}.tar.gz
[ivy:retrieve] ==== maven2: tried
[ivy:retrieve]
http://repo1.maven.org/maven2/hadoop/hbase/${hbase.version}/hbase-${hbase.version}.pom
[ivy:retrieve] -- artifact
hadoop#hbase;${hbase.version}!hbase.tar.gz(source):
[ivy:retrieve]
http://repo1.maven.org/maven2/hadoop/hbase/${hbase.version}/hbase-${hbase.version}.tar.gz
[ivy:retrieve] ::::::::::::::::::::::::::::::::::::::::::::::
[ivy:retrieve] :: UNRESOLVED DEPENDENCIES ::
[ivy:retrieve] ::::::::::::::::::::::::::::::::::::::::::::::
[ivy:retrieve] :: hadoop#hbase;${hbase.version}: not found
[ivy:retrieve] ::::::::::::::::::::::::::::::::::::::::::::::
[ivy:retrieve]
[ivy:retrieve] :: USE VERBOSE OR DEBUG MESSAGE LEVEL FOR MORE DETAILS
BUILD FAILED
/tmp/hive-trunk/build.xml:158: The following error occurred while executing
this line:
/tmp/hive-trunk/build.xml:103: The following error occurred while executing
this line:
/tmp/hive-trunk/build-common.xml:200: The following error occurred while
executing this line:
/tmp/hive-trunk/build-common.xml:191: The following error occurred while
executing this line:
/tmp/hive-trunk/build-common.xml:177: impossible to resolve dependencies:
resolve failed - see output for details
Total time: 34 seconds
On Fri, Mar 5, 2010 at 9:10 AM, John Sichi <js...@facebook.com> wrote:
>
> You need to delete your ~/.ant directory to get rid of the old ivy version
> stored there. Nothing to do with the HBase integration; it is due to a
> recent ivy upgrade on trunk.
>
> JVS
>
> On Mar 5, 2010, at 8:39 AM, "Ray Duong" <ra...@gmail.com> wrote:
>
> Hi John,
>
> Nice work on the HBase integration adapator.
>
> I downloaded the Hive trunk and applied the HIVE-705<http://issues.apache.org/jira/browse/HIVE-705>.3.patch,
> but when I tried to 'ant package', I get the following error messages
> below. Is there any other settings that is required to build the
> hive_hbase_handler.jar file?
>
> Thanks,
> Ray Duong@sharethis
>
> >> ant package
>
> ivy-retrieve-hadoop-source:
> [ivy:retrieve] :: Ivy 2.0.0-rc2 - 20081028224207 ::
> <http://ant.apache.org/ivy/>http://ant.apache.org/ivy/ ::
> :: loading settings :: file = /tmp/hive-trunk/ivy/ivysettings.xml
>
> BUILD FAILED
> /tmp/hive-trunk/build.xml:158: The following error occurred while executing
> this line:
> /tmp/hive-trunk/build.xml:103: The following error occurred while executing
> this line:
> /tmp/hive-trunk/shims/build.xml:55: The following error occurred while
> executing this line:
> /tmp/hive-trunk/build-common.xml:177: impossible to configure ivy:settings
> with given file: /tmp/hive-trunk/ivy/ivysettings.xml :
> java.text.ParseException: failed to load settings from
> file:/tmp/hive-trunk/ivy/ivysettings.xml: impossible to set defaultTTL to
> eternal on class org.apache.ivy.core.cache.DefaultRepositoryCacheManager
>
> Total time: 0 seconds
>
> >> ant -v package
>
> Caused by: java.lang.IllegalArgumentException: impossible to set defaultTTL
> to eternal on class org.apache.ivy.core.cache.DefaultRepositoryCacheManager
> at
> org.apache.ivy.util.Configurator.setAttribute(Configurator.java:660)
> at
> org.apache.ivy.core.settings.XmlSettingsParser.inConfiguratorStarted(XmlSettingsParser.java:561)
> at
> org.apache.ivy.core.settings.XmlSettingsParser.startElement(XmlSettingsParser.java:200)
> at org.apache.xerces.parsers.AbstractSAXParser.startElement(Unknown
> Source)
> at
> org.apache.xerces.parsers.AbstractXMLDocumentParser.emptyElement(Unknown
> Source)
> at
> org.apache.xerces.impl.XMLDocumentFragmentScannerImpl.scanStartElement(Unknown
> Source)
> at
> org.apache.xerces.impl.XMLDocumentFragmentScannerImpl$FragmentContentDispatcher.dispatch(Unknown
> Source)
> at
> org.apache.xerces.impl.XMLDocumentFragmentScannerImpl.scanDocument(Unknown
> Source)
> at org.apache.xerces.parsers.XML11Configuration.parse(Unknown
> Source)
> at org.apache.xerces.parsers.XML11Configuration.parse(Unknown
> Source)
> at org.apache.xerces.parsers.XMLParser.parse(Unknown Source)
> at org.apache.xerces.parsers.AbstractSAXParser.parse(Unknown
> Source)
> at org.apache.xerces.jaxp.SAXParserImpl$JAXPSAXParser.parse(Unknown
> Source)
> at org.apache.xerces.jaxp.SAXParserImpl.parse(Unknown Source)
> at javax.xml.parsers.SAXParser.parse(SAXParser.java:277)
> at
> org.apache.ivy.core.settings.XmlSettingsParser.doParse(XmlSettingsParser.java:159)
> ... 65 more
> Caused by: java.lang.reflect.InvocationTargetException
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at
> org.apache.ivy.util.Configurator.setAttribute(Configurator.java:658)
> ... 80 more
> Caused by: java.lang.IllegalArgumentException: invalid duration 'eternal':
> it must match (?:(\d+)d)? ?(?:(\d+)h)? ?(?:(\d+)m)? ?(?:(\d+)s)?
> ?(?:(\d+)ms)?
> at
> org.apache.ivy.core.cache.DefaultRepositoryCacheManager.parseDuration(DefaultRepositoryCacheManager.java:275)
> at
> org.apache.ivy.core.cache.DefaultRepositoryCacheManager.setDefaultTTL(DefaultRepositoryCacheManager.java:178)
> ... 85 more
>
> Total time: 0 seconds
>
> On Thu, Mar 4, 2010 at 6:51 PM, John Sichi < <js...@facebook.com>
> jsichi@facebook.com> wrote:
>
>> Hey folks,
>>
>> In case you're not following the action over at HIVE-705, we're getting
>> close to having HBase integration committed into Hive. I've written up docs
>> here:
>>
>> <http://wiki.apache.org/hadoop/Hive/HBaseIntegration>
>> http://wiki.apache.org/hadoop/Hive/HBaseIntegration
>> <http://wiki.apache.org/hadoop/Hive/StorageHandlers>
>> http://wiki.apache.org/hadoop/Hive/StorageHandlers
>>
>> (If you happened to read the first draft of the HBaseIntegration doc a few
>> days ago, I've made a lot of updates today to fill out the details on column
>> mapping.)
>>
>> As part of commit, we'll be doing some code reviews within Facebook next
>> week and logging a bunch of followup tasks; if you have any comments on the
>> approach or implementation, please pile on in JIRA.
>>
>> I'll be giving this a quick mention at the Hive user's group later this
>> month, and then a more detailed presentation at the HBase User Group meeting
>> in April.
>>
>> JVS
>>
>>
>
Re: storage handlers and HBase integration
Posted by John Sichi <js...@facebook.com>.
You need to delete your ~/.ant directory to get rid of the old ivy version stored there. Nothing to do with the HBase integration; it is due to a recent ivy upgrade on trunk.
JVS
On Mar 5, 2010, at 8:39 AM, "Ray Duong" <ra...@gmail.com>> wrote:
Hi John,
Nice work on the HBase integration adapator.
I downloaded the Hive trunk and applied the HIVE-705<http://issues.apache.org/jira/browse/HIVE-705>.3.patch, but when I tried to 'ant package', I get the following error messages below. Is there any other settings that is required to build the hive_hbase_handler.jar file?
Thanks,
Ray Duong@sharethis
>> ant package
ivy-retrieve-hadoop-source:
[ivy:retrieve] :: Ivy 2.0.0-rc2 - 20081028224207 :: <http://ant.apache.org/ivy/> http://ant.apache.org/ivy/ ::
:: loading settings :: file = /tmp/hive-trunk/ivy/ivysettings.xml
BUILD FAILED
/tmp/hive-trunk/build.xml:158: The following error occurred while executing this line:
/tmp/hive-trunk/build.xml:103: The following error occurred while executing this line:
/tmp/hive-trunk/shims/build.xml:55: The following error occurred while executing this line:
/tmp/hive-trunk/build-common.xml:177: impossible to configure ivy:settings with given file: /tmp/hive-trunk/ivy/ivysettings.xml : java.text.ParseException: failed to load settings from file:/tmp/hive-trunk/ivy/ivysettings.xml: impossible to set defaultTTL to eternal on class org.apache.ivy.core.cache.DefaultRepositoryCacheManager
Total time: 0 seconds
>> ant -v package
Caused by: java.lang.IllegalArgumentException: impossible to set defaultTTL to eternal on class org.apache.ivy.core.cache.DefaultRepositoryCacheManager
at org.apache.ivy.util.Configurator.setAttribute(Configurator.java:660)
at org.apache.ivy.core.settings.XmlSettingsParser.inConfiguratorStarted(XmlSettingsParser.java:561)
at org.apache.ivy.core.settings.XmlSettingsParser.startElement(XmlSettingsParser.java:200)
at org.apache.xerces.parsers.AbstractSAXParser.startElement(Unknown Source)
at org.apache.xerces.parsers.AbstractXMLDocumentParser.emptyElement(Unknown Source)
at org.apache.xerces.impl.XMLDocumentFragmentScannerImpl.scanStartElement(Unknown Source)
at org.apache.xerces.impl.XMLDocumentFragmentScannerImpl$FragmentContentDispatcher.dispatch(Unknown Source)
at org.apache.xerces.impl.XMLDocumentFragmentScannerImpl.scanDocument(Unknown Source)
at org.apache.xerces.parsers.XML11Configuration.parse(Unknown Source)
at org.apache.xerces.parsers.XML11Configuration.parse(Unknown Source)
at org.apache.xerces.parsers.XMLParser.parse(Unknown Source)
at org.apache.xerces.parsers.AbstractSAXParser.parse(Unknown Source)
at org.apache.xerces.jaxp.SAXParserImpl$JAXPSAXParser.parse(Unknown Source)
at org.apache.xerces.jaxp.SAXParserImpl.parse(Unknown Source)
at javax.xml.parsers.SAXParser.parse(SAXParser.java:277)
at org.apache.ivy.core.settings.XmlSettingsParser.doParse(XmlSettingsParser.java:159)
... 65 more
Caused by: java.lang.reflect.InvocationTargetException
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.ivy.util.Configurator.setAttribute(Configurator.java:658)
... 80 more
Caused by: java.lang.IllegalArgumentException: invalid duration 'eternal': it must match (?:(\d+)d)? ?(?:(\d+)h)? ?(?:(\d+)m)? ?(?:(\d+)s)? ?(?:(\d+)ms)?
at org.apache.ivy.core.cache.DefaultRepositoryCacheManager.parseDuration(DefaultRepositoryCacheManager.java:275)
at org.apache.ivy.core.cache.DefaultRepositoryCacheManager.setDefaultTTL(DefaultRepositoryCacheManager.java:178)
... 85 more
Total time: 0 seconds
On Thu, Mar 4, 2010 at 6:51 PM, John Sichi <<m...@facebook.com>> wrote:
Hey folks,
In case you're not following the action over at HIVE-705, we're getting close to having HBase integration committed into Hive. I've written up docs here:
<http://wiki.apache.org/hadoop/Hive/HBaseIntegration>http://wiki.apache.org/hadoop/Hive/HBaseIntegration
<http://wiki.apache.org/hadoop/Hive/StorageHandlers>http://wiki.apache.org/hadoop/Hive/StorageHandlers
(If you happened to read the first draft of the HBaseIntegration doc a few days ago, I've made a lot of updates today to fill out the details on column mapping.)
As part of commit, we'll be doing some code reviews within Facebook next week and logging a bunch of followup tasks; if you have any comments on the approach or implementation, please pile on in JIRA.
I'll be giving this a quick mention at the Hive user's group later this month, and then a more detailed presentation at the HBase User Group meeting in April.
JVS
Re: storage handlers and HBase integration
Posted by Ray Duong <ra...@gmail.com>.
Hi John,
Nice work on the HBase integration adapator.
I downloaded the Hive trunk and applied the
HIVE-705<http://issues.apache.org/jira/browse/HIVE-705>.3.patch,
but when I tried to 'ant package', I get the following error messages
below. Is there any other settings that is required to build the
hive_hbase_handler.jar file?
Thanks,
Ray Duong@sharethis
>> ant package
ivy-retrieve-hadoop-source:
[ivy:retrieve] :: Ivy 2.0.0-rc2 - 20081028224207 ::
http://ant.apache.org/ivy/ ::
:: loading settings :: file = /tmp/hive-trunk/ivy/ivysettings.xml
BUILD FAILED
/tmp/hive-trunk/build.xml:158: The following error occurred while executing
this line:
/tmp/hive-trunk/build.xml:103: The following error occurred while executing
this line:
/tmp/hive-trunk/shims/build.xml:55: The following error occurred while
executing this line:
/tmp/hive-trunk/build-common.xml:177: impossible to configure ivy:settings
with given file: /tmp/hive-trunk/ivy/ivysettings.xml :
java.text.ParseException: failed to load settings from
file:/tmp/hive-trunk/ivy/ivysettings.xml: impossible to set defaultTTL to
eternal on class org.apache.ivy.core.cache.DefaultRepositoryCacheManager
Total time: 0 seconds
>> ant -v package
Caused by: java.lang.IllegalArgumentException: impossible to set defaultTTL
to eternal on class org.apache.ivy.core.cache.DefaultRepositoryCacheManager
at
org.apache.ivy.util.Configurator.setAttribute(Configurator.java:660)
at
org.apache.ivy.core.settings.XmlSettingsParser.inConfiguratorStarted(XmlSettingsParser.java:561)
at
org.apache.ivy.core.settings.XmlSettingsParser.startElement(XmlSettingsParser.java:200)
at org.apache.xerces.parsers.AbstractSAXParser.startElement(Unknown
Source)
at
org.apache.xerces.parsers.AbstractXMLDocumentParser.emptyElement(Unknown
Source)
at
org.apache.xerces.impl.XMLDocumentFragmentScannerImpl.scanStartElement(Unknown
Source)
at
org.apache.xerces.impl.XMLDocumentFragmentScannerImpl$FragmentContentDispatcher.dispatch(Unknown
Source)
at
org.apache.xerces.impl.XMLDocumentFragmentScannerImpl.scanDocument(Unknown
Source)
at org.apache.xerces.parsers.XML11Configuration.parse(Unknown
Source)
at org.apache.xerces.parsers.XML11Configuration.parse(Unknown
Source)
at org.apache.xerces.parsers.XMLParser.parse(Unknown Source)
at org.apache.xerces.parsers.AbstractSAXParser.parse(Unknown Source)
at org.apache.xerces.jaxp.SAXParserImpl$JAXPSAXParser.parse(Unknown
Source)
at org.apache.xerces.jaxp.SAXParserImpl.parse(Unknown Source)
at javax.xml.parsers.SAXParser.parse(SAXParser.java:277)
at
org.apache.ivy.core.settings.XmlSettingsParser.doParse(XmlSettingsParser.java:159)
... 65 more
Caused by: java.lang.reflect.InvocationTargetException
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
org.apache.ivy.util.Configurator.setAttribute(Configurator.java:658)
... 80 more
Caused by: java.lang.IllegalArgumentException: invalid duration 'eternal':
it must match (?:(\d+)d)? ?(?:(\d+)h)? ?(?:(\d+)m)? ?(?:(\d+)s)?
?(?:(\d+)ms)?
at
org.apache.ivy.core.cache.DefaultRepositoryCacheManager.parseDuration(DefaultRepositoryCacheManager.java:275)
at
org.apache.ivy.core.cache.DefaultRepositoryCacheManager.setDefaultTTL(DefaultRepositoryCacheManager.java:178)
... 85 more
Total time: 0 seconds
On Thu, Mar 4, 2010 at 6:51 PM, John Sichi <js...@facebook.com> wrote:
> Hey folks,
>
> In case you're not following the action over at HIVE-705, we're getting
> close to having HBase integration committed into Hive. I've written up docs
> here:
>
> http://wiki.apache.org/hadoop/Hive/HBaseIntegration
> http://wiki.apache.org/hadoop/Hive/StorageHandlers
>
> (If you happened to read the first draft of the HBaseIntegration doc a few
> days ago, I've made a lot of updates today to fill out the details on column
> mapping.)
>
> As part of commit, we'll be doing some code reviews within Facebook next
> week and logging a bunch of followup tasks; if you have any comments on the
> approach or implementation, please pile on in JIRA.
>
> I'll be giving this a quick mention at the Hive user's group later this
> month, and then a more detailed presentation at the HBase User Group meeting
> in April.
>
> JVS
>
>