You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hbase.apache.org by hossein ahmadzadeh <h....@gmail.com> on 2021/03/30 19:29:28 UTC
Is HBase 2.X suitable for running on Hadoop 3.X?
<https://stackoverflow.com/posts/66877227/timeline>
The HBase documentation <http://hbase.apache.org/book.html#hadoop> noted
that:
Hadoop 2.x is faster and includes features, such as short-circuit reads
(See Leveraging local data), which will help improve your HBase random read
profile. Hadoop 2.x also includes important bug fixes that will improve
your overall HBase experience. HBase does not support running with earlier
versions of Hadoop. See the table below for requirements specific to
different HBase versions. *Hadoop 3.x is still in early access releases and
has not yet been sufficiently tested by the HBase community for production
use cases.*
But right next to this point in the compatibility table says that HBase
2.3.4 is fully functional with Hadoop 3.2.2. So I got confused about
whether We can use HBase 2.3.4 in production alongside Hadoop 3.2.2 or not?
Best Regards.
Re: Is HBase 2.X suitable for running on Hadoop 3.X?
Posted by Ascot Moss <as...@gmail.com>.
I tried to build hadoop 3.2.2 with -Pnative, got error below:
Run Build Command(s):/usr/bin/gmake -f Makefile cmTC_c17b0/fast &&
/usr/bin/gmake -f CMakeFiles/cmTC_c17b0.dir/build.make
CMakeFiles/cmTC_c17b0.dir/build
gmake[1]: Entering directory
`/edh_all_sources/hadoop-3.2.2-src/hadoop-common-project/hadoop-common/target/native/CMakeFiles/CMakeTmp'
Building C object CMakeFiles/cmTC_c17b0.dir/src.c.o
/usr/bin/cc -DCMAKE_HAVE_LIBC_PTHREAD -o
CMakeFiles/cmTC_c17b0.dir/src.c.o -c
/edh_all_sources/hadoop-3.2.2-src/hadoop-common-project/hadoop-common/target/native/CMakeFiles/CMakeTmp/src.c
Linking C executable cmTC_c17b0
/usr/local/bin/cmake -E cmake_link_script
CMakeFiles/cmTC_c17b0.dir/link.txt --verbose=1
/usr/bin/cc -rdynamic CMakeFiles/cmTC_c17b0.dir/src.c.o -o cmTC_c17b0
CMakeFiles/cmTC_c17b0.dir/src.c.o: In function `main':
src.c:(.text+0x2d): undefined reference to `pthread_create'
src.c:(.text+0x39): undefined reference to `pthread_detach'
src.c:(.text+0x45): undefined reference to `pthread_cancel'
src.c:(.text+0x56): undefined reference to `pthread_join'
src.c:(.text+0x6a): undefined reference to `pthread_atfork'
collect2: error: ld returned 1 exit status
gmake[1]: *** [cmTC_c17b0] Error 1
gmake[1]: Leaving directory
`/edh_all_sources/hadoop-3.2.2-src/hadoop-common-project/hadoop-common/target/native/CMakeFiles/CMakeTmp'
gmake: *** [cmTC_c17b0/fast] Error 2
Please help
On Fri, Apr 2, 2021 at 7:01 PM Wei-Chiu Chuang <we...@cloudera.com.invalid>
wrote:
> i think it's time to remove that statement. We have lots of production
> users running HBase 2 on Hadoop 3 for several years now.
>
> On Fri, Apr 2, 2021 at 6:32 PM 张铎(Duo Zhang) <pa...@gmail.com>
> wrote:
>
> > According to the compatibility matrix, HBase 2.3.4 could work together
> with
> > Hadoop 3.2.2.
> >
> > And if you just want to connec to HDFS 3.2.2, you could just use the pre
> > built artifacts for HBase 2.3.4. If you want to use Hadoop 3.2.2 client
> in
> > HBase, you need to build the artifacts by your own.
> >
> > Thanks.
> >
> > hossein ahmadzadeh <h....@gmail.com> 于2021年3月31日周三 上午3:30写道：
> >
> > > <https://stackoverflow.com/posts/66877227/timeline>
> > >
> > > The HBase documentation <http://hbase.apache.org/book.html#hadoop>
> noted
> > > that:
> > >
> > > Hadoop 2.x is faster and includes features, such as short-circuit reads
> > > (See Leveraging local data), which will help improve your HBase random
> > read
> > > profile. Hadoop 2.x also includes important bug fixes that will improve
> > > your overall HBase experience. HBase does not support running with
> > earlier
> > > versions of Hadoop. See the table below for requirements specific to
> > > different HBase versions. *Hadoop 3.x is still in early access releases
> > and
> > > has not yet been sufficiently tested by the HBase community for
> > production
> > > use cases.*
> > >
> > > But right next to this point in the compatibility table says that HBase
> > > 2.3.4 is fully functional with Hadoop 3.2.2. So I got confused about
> > > whether We can use HBase 2.3.4 in production alongside Hadoop 3.2.2 or
> > not?
> > >
> > > Best Regards.
> > >
> >
>
Re: Is HBase 2.X suitable for running on Hadoop 3.X?
Posted by Wei-Chiu Chuang <we...@apache.org>.
Filed HBASE-25771 <https://issues.apache.org/jira/browse/HBASE-25771> to
recommend Hadoop 3.x
Feel free to chime in.
On Fri, Apr 2, 2021 at 7:00 PM Wei-Chiu Chuang <we...@cloudera.com> wrote:
> i think it's time to remove that statement. We have lots of production
> users running HBase 2 on Hadoop 3 for several years now.
>
> On Fri, Apr 2, 2021 at 6:32 PM 张铎(Duo Zhang) <pa...@gmail.com>
> wrote:
>
>> According to the compatibility matrix, HBase 2.3.4 could work together
>> with
>> Hadoop 3.2.2.
>>
>> And if you just want to connec to HDFS 3.2.2, you could just use the pre
>> built artifacts for HBase 2.3.4. If you want to use Hadoop 3.2.2 client in
>> HBase, you need to build the artifacts by your own.
>>
>> Thanks.
>>
>> hossein ahmadzadeh <h....@gmail.com> 于2021年3月31日周三 上午3:30写道：
>>
>> > <https://stackoverflow.com/posts/66877227/timeline>
>> >
>> > The HBase documentation <http://hbase.apache.org/book.html#hadoop>
>> noted
>> > that:
>> >
>> > Hadoop 2.x is faster and includes features, such as short-circuit reads
>> > (See Leveraging local data), which will help improve your HBase random
>> read
>> > profile. Hadoop 2.x also includes important bug fixes that will improve
>> > your overall HBase experience. HBase does not support running with
>> earlier
>> > versions of Hadoop. See the table below for requirements specific to
>> > different HBase versions. *Hadoop 3.x is still in early access releases
>> and
>> > has not yet been sufficiently tested by the HBase community for
>> production
>> > use cases.*
>> >
>> > But right next to this point in the compatibility table says that HBase
>> > 2.3.4 is fully functional with Hadoop 3.2.2. So I got confused about
>> > whether We can use HBase 2.3.4 in production alongside Hadoop 3.2.2 or
>> not?
>> >
>> > Best Regards.
>> >
>>
>
Re: Is HBase 2.X suitable for running on Hadoop 3.X?
Posted by Josh Elser <el...@apache.org>.
Looks like you don't have the pthread library available. Did you make
sure you installed the necessary prerequisites for your operating system?
I'd suggest you take Hadoop compilation questions to the Hadoop user
mailing list for some more prompt answers.
On 4/13/21 1:35 AM, Ascot Moss wrote:
> Hi,
>
> I tried to build Hadoop 3.2.2, but got error below, please help!
>
>
>
> Run Build Command(s):/usr/bin/gmake -f Makefile cmTC_c17b0/fast &&
> /usr/bin/gmake -f CMakeFiles/cmTC_c17b0.dir/build.make
> CMakeFiles/cmTC_c17b0.dir/build
>
> gmake[1]: Entering directory
> `/hadoop-3.2.2-src/hadoop-common-project/hadoop-common/target/native/CMakeFiles/CMakeTmp'
>
> Building C object CMakeFiles/cmTC_c17b0.dir/src.c.o
>
> /usr/bin/cc -DCMAKE_HAVE_LIBC_PTHREAD -o
> CMakeFiles/cmTC_c17b0.dir/src.c.o -c
> /hadoop-3.2.2-src/hadoop-common-project/hadoop-common/target/native/CMakeFiles/CMakeTmp/src.c
>
> Linking C executable cmTC_c17b0
>
> /usr/local/bin/cmake -E cmake_link_script
> CMakeFiles/cmTC_c17b0.dir/link.txt --verbose=1
>
> /usr/bin/cc -rdynamic CMakeFiles/cmTC_c17b0.dir/src.c.o -o cmTC_c17b0
>
> CMakeFiles/cmTC_c17b0.dir/src.c.o: In function `main':
>
> src.c:(.text+0x2d): undefined reference to `pthread_create'
>
> src.c:(.text+0x39): undefined reference to `pthread_detach'
>
> src.c:(.text+0x45): undefined reference to `pthread_cancel'
>
> src.c:(.text+0x56): undefined reference to `pthread_join'
>
> src.c:(.text+0x6a): undefined reference to `pthread_atfork'
>
> collect2: error: ld returned 1 exit status
>
> gmake[1]: *** [cmTC_c17b0] Error 1
>
> gmake[1]: Leaving directory
> `/hadoop-3.2.2-src/hadoop-common-project/hadoop-common/target/native/CMakeFiles/CMakeTmp'
>
> gmake: *** [cmTC_c17b0/fast] Error 2
>
>
>
>
> On Fri, Apr 2, 2021 at 7:01 PM Wei-Chiu Chuang <we...@cloudera.com.invalid>
> wrote:
>
>> i think it's time to remove that statement. We have lots of production
>> users running HBase 2 on Hadoop 3 for several years now.
>>
>> On Fri, Apr 2, 2021 at 6:32 PM 张铎(Duo Zhang) <pa...@gmail.com>
>> wrote:
>>
>>> According to the compatibility matrix, HBase 2.3.4 could work together
>> with
>>> Hadoop 3.2.2.
>>>
>>> And if you just want to connec to HDFS 3.2.2, you could just use the pre
>>> built artifacts for HBase 2.3.4. If you want to use Hadoop 3.2.2 client
>> in
>>> HBase, you need to build the artifacts by your own.
>>>
>>> Thanks.
>>>
>>> hossein ahmadzadeh <h....@gmail.com> 于2021年3月31日周三 上午3:30写道：
>>>
>>>> <https://stackoverflow.com/posts/66877227/timeline>
>>>>
>>>> The HBase documentation <http://hbase.apache.org/book.html#hadoop>
>> noted
>>>> that:
>>>>
>>>> Hadoop 2.x is faster and includes features, such as short-circuit reads
>>>> (See Leveraging local data), which will help improve your HBase random
>>> read
>>>> profile. Hadoop 2.x also includes important bug fixes that will improve
>>>> your overall HBase experience. HBase does not support running with
>>> earlier
>>>> versions of Hadoop. See the table below for requirements specific to
>>>> different HBase versions. *Hadoop 3.x is still in early access releases
>>> and
>>>> has not yet been sufficiently tested by the HBase community for
>>> production
>>>> use cases.*
>>>>
>>>> But right next to this point in the compatibility table says that HBase
>>>> 2.3.4 is fully functional with Hadoop 3.2.2. So I got confused about
>>>> whether We can use HBase 2.3.4 in production alongside Hadoop 3.2.2 or
>>> not?
>>>>
>>>> Best Regards.
>>>>
>>>
>>
>
Re: Is HBase 2.X suitable for running on Hadoop 3.X?
Posted by Ascot Moss <as...@gmail.com>.
Hi,
I tried to build Hadoop 3.2.2, but got error below, please help!
Run Build Command(s):/usr/bin/gmake -f Makefile cmTC_c17b0/fast &&
/usr/bin/gmake -f CMakeFiles/cmTC_c17b0.dir/build.make
CMakeFiles/cmTC_c17b0.dir/build
gmake[1]: Entering directory
`/hadoop-3.2.2-src/hadoop-common-project/hadoop-common/target/native/CMakeFiles/CMakeTmp'
Building C object CMakeFiles/cmTC_c17b0.dir/src.c.o
/usr/bin/cc -DCMAKE_HAVE_LIBC_PTHREAD -o
CMakeFiles/cmTC_c17b0.dir/src.c.o -c
/hadoop-3.2.2-src/hadoop-common-project/hadoop-common/target/native/CMakeFiles/CMakeTmp/src.c
Linking C executable cmTC_c17b0
/usr/local/bin/cmake -E cmake_link_script
CMakeFiles/cmTC_c17b0.dir/link.txt --verbose=1
/usr/bin/cc -rdynamic CMakeFiles/cmTC_c17b0.dir/src.c.o -o cmTC_c17b0
CMakeFiles/cmTC_c17b0.dir/src.c.o: In function `main':
src.c:(.text+0x2d): undefined reference to `pthread_create'
src.c:(.text+0x39): undefined reference to `pthread_detach'
src.c:(.text+0x45): undefined reference to `pthread_cancel'
src.c:(.text+0x56): undefined reference to `pthread_join'
src.c:(.text+0x6a): undefined reference to `pthread_atfork'
collect2: error: ld returned 1 exit status
gmake[1]: *** [cmTC_c17b0] Error 1
gmake[1]: Leaving directory
`/hadoop-3.2.2-src/hadoop-common-project/hadoop-common/target/native/CMakeFiles/CMakeTmp'
gmake: *** [cmTC_c17b0/fast] Error 2
On Fri, Apr 2, 2021 at 7:01 PM Wei-Chiu Chuang <we...@cloudera.com.invalid>
wrote:
> i think it's time to remove that statement. We have lots of production
> users running HBase 2 on Hadoop 3 for several years now.
>
> On Fri, Apr 2, 2021 at 6:32 PM 张铎(Duo Zhang) <pa...@gmail.com>
> wrote:
>
> > According to the compatibility matrix, HBase 2.3.4 could work together
> with
> > Hadoop 3.2.2.
> >
> > And if you just want to connec to HDFS 3.2.2, you could just use the pre
> > built artifacts for HBase 2.3.4. If you want to use Hadoop 3.2.2 client
> in
> > HBase, you need to build the artifacts by your own.
> >
> > Thanks.
> >
> > hossein ahmadzadeh <h....@gmail.com> 于2021年3月31日周三 上午3:30写道：
> >
> > > <https://stackoverflow.com/posts/66877227/timeline>
> > >
> > > The HBase documentation <http://hbase.apache.org/book.html#hadoop>
> noted
> > > that:
> > >
> > > Hadoop 2.x is faster and includes features, such as short-circuit reads
> > > (See Leveraging local data), which will help improve your HBase random
> > read
> > > profile. Hadoop 2.x also includes important bug fixes that will improve
> > > your overall HBase experience. HBase does not support running with
> > earlier
> > > versions of Hadoop. See the table below for requirements specific to
> > > different HBase versions. *Hadoop 3.x is still in early access releases
> > and
> > > has not yet been sufficiently tested by the HBase community for
> > production
> > > use cases.*
> > >
> > > But right next to this point in the compatibility table says that HBase
> > > 2.3.4 is fully functional with Hadoop 3.2.2. So I got confused about
> > > whether We can use HBase 2.3.4 in production alongside Hadoop 3.2.2 or
> > not?
> > >
> > > Best Regards.
> > >
> >
>
Re: Is HBase 2.X suitable for running on Hadoop 3.X?
Posted by Wei-Chiu Chuang <we...@cloudera.com.INVALID>.
i think it's time to remove that statement. We have lots of production
users running HBase 2 on Hadoop 3 for several years now.
On Fri, Apr 2, 2021 at 6:32 PM 张铎(Duo Zhang) <pa...@gmail.com> wrote:
> According to the compatibility matrix, HBase 2.3.4 could work together with
> Hadoop 3.2.2.
>
> And if you just want to connec to HDFS 3.2.2, you could just use the pre
> built artifacts for HBase 2.3.4. If you want to use Hadoop 3.2.2 client in
> HBase, you need to build the artifacts by your own.
>
> Thanks.
>
> hossein ahmadzadeh <h....@gmail.com> 于2021年3月31日周三 上午3:30写道：
>
> > <https://stackoverflow.com/posts/66877227/timeline>
> >
> > The HBase documentation <http://hbase.apache.org/book.html#hadoop> noted
> > that:
> >
> > Hadoop 2.x is faster and includes features, such as short-circuit reads
> > (See Leveraging local data), which will help improve your HBase random
> read
> > profile. Hadoop 2.x also includes important bug fixes that will improve
> > your overall HBase experience. HBase does not support running with
> earlier
> > versions of Hadoop. See the table below for requirements specific to
> > different HBase versions. *Hadoop 3.x is still in early access releases
> and
> > has not yet been sufficiently tested by the HBase community for
> production
> > use cases.*
> >
> > But right next to this point in the compatibility table says that HBase
> > 2.3.4 is fully functional with Hadoop 3.2.2. So I got confused about
> > whether We can use HBase 2.3.4 in production alongside Hadoop 3.2.2 or
> not?
> >
> > Best Regards.
> >
>
Re: Is HBase 2.X suitable for running on Hadoop 3.X?
Posted by "张铎 (Duo Zhang)" <pa...@gmail.com>.
According to the compatibility matrix, HBase 2.3.4 could work together with
Hadoop 3.2.2.
And if you just want to connec to HDFS 3.2.2, you could just use the pre
built artifacts for HBase 2.3.4. If you want to use Hadoop 3.2.2 client in
HBase, you need to build the artifacts by your own.
Thanks.
hossein ahmadzadeh <h....@gmail.com> 于2021年3月31日周三 上午3:30写道：
> <https://stackoverflow.com/posts/66877227/timeline>
>
> The HBase documentation <http://hbase.apache.org/book.html#hadoop> noted
> that:
>
> Hadoop 2.x is faster and includes features, such as short-circuit reads
> (See Leveraging local data), which will help improve your HBase random read
> profile. Hadoop 2.x also includes important bug fixes that will improve
> your overall HBase experience. HBase does not support running with earlier
> versions of Hadoop. See the table below for requirements specific to
> different HBase versions. *Hadoop 3.x is still in early access releases and
> has not yet been sufficiently tested by the HBase community for production
> use cases.*
>
> But right next to this point in the compatibility table says that HBase
> 2.3.4 is fully functional with Hadoop 3.2.2. So I got confused about
> whether We can use HBase 2.3.4 in production alongside Hadoop 3.2.2 or not?
>
> Best Regards.
>