You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@ignite.apache.org by ishan-jain <ja...@gmail.com> on 2017/06/21 05:39:42 UTC
Hadoop + IGFS + Hive+Tez
Hi all, I have configured hive to run over IGFS for hadoop file system. I am
trying to install tez on it but am not able to configure tez-site.xml and
bashrc file properly.
I have downloaded a binary tez bin to my system (0.8.4)
I Put the tez tar ball in hdfs dire /user/tez
I pointed my tez.lib.uri to this path
I have configured mapred-site.xml to use yarn-tez
I have hive already configured to run over igfs.
But when i execute the mapred task it says
Failed to connect to shared memory endpoint .
I am using TCP as ipc endpoint configuration and have enabled ipc endpoint
property.
--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Hadoop-IGFS-Hive-Tez-tp14012.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.
Re: Hadoop + IGFS + Hive+Tez
Posted by Taras Ledkov <tl...@gridgain.com>.
Hi,
I don't catch how you configured hadoop Job and where Ignite instances
are started (on all hadoop hosts?).
Shared memory is default and preferred type of connection to Ignite.
In this case Ignite nodes must be started on hadoop nodes.
If you started Ignite on either hosts IGFS URL must be specified with
the host/port.
Please take a look:
https://apacheignite.readme.io/v1.2/docs/file-system#section-file-system-uri
On 21.06.2017 18:28, Mimmo Celano wrote:
> Hi, thanks for reply.
>
> Ignite configuration is a default configuration in
> config/hadoop/default-configuration.xml i don't change anything.
> Hadoop configuration is a cluster o 3 node, it work properly withouth
> ignite. The problem is in a connection between the client and the 2
> server. I don't change anything in ignite file configuration, the only
> things i do is the code who i posted.
> I agree with you that is a configuration problem but i don't know how
> to configure it. The 2 server node are slave in hadoop configuration,
> but i don't think it's a problem. I don't find another ignite logs
> other that posted... I don't use tez. You think that i use another
> configuration for ignite instead the default for hadoop? I want just
> save all word of wordcount processing in a grid cache between 2 node
> server. There's no log of the task because the application crash
> before task......
>
> The node are connected via lan. Thanks
>
> 2017-06-21 16:59 GMT+02:00 Taras Ledkov <tledkov@gridgain.com
> <ma...@gridgain.com>>:
>
> Hi,
>
> Please provide:
> - ignite configuration;
> - ignite logs;
> - hadoop configurations;
> - logs of the task.
>
> I don't have tez experience but the cause of the problem looks
> like Ignite is not configured properly.
>
>
> On 21.06.2017 8:39, ishan-jain wrote:
>> Hi all, I have configured hive to run over IGFS for hadoop file system. I am
>> trying to install tez on it but am not able to configure tez-site.xml and
>> bashrc file properly.
>> I have downloaded a binary tez bin to my system (0.8.4)
>> I Put the tez tar ball in hdfs dire /user/tez
>> I pointed my tez.lib.uri to this path
>> I have configured mapred-site.xml to use yarn-tez
>> I have hive already configured to run over igfs.
>> But when i execute the mapred task it says
>> Failed to connect to shared memory endpoint .
>> I am using TCP as ipc endpoint configuration and have enabled ipc endpoint
>> property.
>>
>>
>>
>> --
>> View this message in context:http://apache-ignite-users.70518.x6.nabble.com/Hadoop-IGFS-Hive-Tez-tp14012.html
>> <http://apache-ignite-users.70518.x6.nabble.com/Hadoop-IGFS-Hive-Tez-tp14012.html>Sent
>> from the Apache Ignite Users mailing list archive at Nabble.com.
>
> --
> Taras Ledkov
> Mail-To:tledkov@gridgain.com <ma...@gridgain.com>
>
>
--
Taras Ledkov
Mail-To: tledkov@gridgain.com
Re: Hadoop + IGFS + Hive+Tez
Posted by Mimmo Celano <mi...@gmail.com>.
Hi, thanks for reply.
Ignite configuration is a default configuration in
config/hadoop/default-configuration.xml i don't change anything. Hadoop
configuration is a cluster o 3 node, it work properly withouth ignite. The
problem is in a connection between the client and the 2 server. I don't
change anything in ignite file configuration, the only things i do is the
code who i posted.
I agree with you that is a configuration problem but i don't know how to
configure it. The 2 server node are slave in hadoop configuration, but i
don't think it's a problem. I don't find another ignite logs other that
posted... I don't use tez. You think that i use another configuration for
ignite instead the default for hadoop? I want just save all word of
wordcount processing in a grid cache between 2 node server. There's no log
of the task because the application crash before task......
The node are connected via lan. Thanks
2017-06-21 16:59 GMT+02:00 Taras Ledkov <tl...@gridgain.com>:
> Hi,
>
> Please provide:
> - ignite configuration;
> - ignite logs;
> - hadoop configurations;
> - logs of the task.
>
> I don't have tez experience but the cause of the problem looks like Ignite
> is not configured properly.
>
> On 21.06.2017 8:39, ishan-jain wrote:
>
> Hi all, I have configured hive to run over IGFS for hadoop file system. I am
> trying to install tez on it but am not able to configure tez-site.xml and
> bashrc file properly.
> I have downloaded a binary tez bin to my system (0.8.4)
> I Put the tez tar ball in hdfs dire /user/tez
> I pointed my tez.lib.uri to this path
> I have configured mapred-site.xml to use yarn-tez
> I have hive already configured to run over igfs.
> But when i execute the mapred task it says
> Failed to connect to shared memory endpoint .
> I am using TCP as ipc endpoint configuration and have enabled ipc endpoint
> property.
>
>
>
> --
> View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Hadoop-IGFS-Hive-Tez-tp14012.html
> Sent from the Apache Ignite Users mailing list archive at Nabble.com.
>
>
> --
> Taras Ledkov
> Mail-To: tledkov@gridgain.com
>
>
Re: Hadoop + IGFS + Hive+Tez
Posted by Taras Ledkov <tl...@gridgain.com>.
Hi,
Please provide:
- ignite configuration;
- ignite logs;
- hadoop configurations;
- logs of the task.
I don't have tez experience but the cause of the problem looks like
Ignite is not configured properly.
On 21.06.2017 8:39, ishan-jain wrote:
> Hi all, I have configured hive to run over IGFS for hadoop file system. I am
> trying to install tez on it but am not able to configure tez-site.xml and
> bashrc file properly.
> I have downloaded a binary tez bin to my system (0.8.4)
> I Put the tez tar ball in hdfs dire /user/tez
> I pointed my tez.lib.uri to this path
> I have configured mapred-site.xml to use yarn-tez
> I have hive already configured to run over igfs.
> But when i execute the mapred task it says
> Failed to connect to shared memory endpoint .
> I am using TCP as ipc endpoint configuration and have enabled ipc endpoint
> property.
>
>
>
> --
> View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Hadoop-IGFS-Hive-Tez-tp14012.html
> Sent from the Apache Ignite Users mailing list archive at Nabble.com.
--
Taras Ledkov
Mail-To: tledkov@gridgain.com