You are viewing a plain text version of this content. The canonical link for it is here.
Posted to mapreduce-user@hadoop.apache.org by "Kartashov, Andy" <An...@mpac.ca> on 2012/10/22 16:55:26 UTC
task trackers
Gentlemen,
Can you please explain which property is responsible to tell JobTracker where TaskTruckers are running?
I get this far so far:
NN knows where SNN runs by specifying FQDN in the conf/masters of the NN
NN knows where DN runs by specifying FQDNs of the DNs in the conf/slaves of the NN
SNN knows the host:port of the NN using hdfs-core.xml
DNs know wherethehost:port of the NN using hdfs-core.xml
TTs know where the JT is using mapred-site.xml
But how does JT knows where the TTs are?
Please help.
Thanks,
Andy
NOTICE: This e-mail message and any attachments are confidential, subject to copyright and may be privileged. Any unauthorized use, copying or disclosure is prohibited. If you are not the intended recipient, please delete and contact the sender immediately. Please consider the environment before printing this e-mail. AVIS : le présent courriel et toute pièce jointe qui l'accompagne sont confidentiels, protégés par le droit d'auteur et peuvent être couverts par le secret professionnel. Toute utilisation, copie ou divulgation non autorisée est interdite. Si vous n'êtes pas le destinataire prévu de ce courriel, supprimez-le et contactez immédiatement l'expéditeur. Veuillez penser à l'environnement avant d'imprimer le présent courriel
RE: task trackers
Posted by "Kartashov, Andy" <An...@mpac.ca>.
Cheers Vinod.
From: Vinod Kumar Vavilapalli [mailto:vinodkv@hortonworks.com]
Sent: Monday, October 22, 2012 2:04 PM
To: user@hadoop.apache.org
Subject: Re: task trackers
Even JT accepts exclude/include hosts files for the slaves. If they aren't populated, JT by default accepts TTs that successfully register with it at runtime.
Thanks,
+Vinod
On Oct 22, 2012, at 7:55 AM, Kartashov, Andy wrote:
Gentlemen,
Can you please explain which property is responsible to tell JobTracker where TaskTruckers are running?
I get this far so far:
NN knows where SNN runs by specifying FQDN in the conf/masters of the NN
NN knows where DN runs by specifying FQDNs of the DNs in the conf/slaves of the NN
SNN knows the host:port of the NN using hdfs-core.xml
DNs know wherethehost:port of the NN using hdfs-core.xml
TTs know where the JT is using mapred-site.xml
But how does JT knows where the TTs are?
Please help.
Thanks,
Andy
NOTICE: This e-mail message and any attachments are confidential, subject to copyright and may be privileged. Any unauthorized use, copying or disclosure is prohibited. If you are not the intended recipient, please delete and contact the sender immediately. Please consider the environment before printing this e-mail. AVIS : le présent courriel et toute pièce jointe qui l'accompagne sont confidentiels, protégés par le droit d'auteur et peuvent être couverts par le secret professionnel. Toute utilisation, copie ou divulgation non autorisée est interdite. Si vous n'êtes pas le destinataire prévu de ce courriel, supprimez-le et contactez immédiatement l'expéditeur. Veuillez penser à l'environnement avant d'imprimer le présent courriel
NOTICE: This e-mail message and any attachments are confidential, subject to copyright and may be privileged. Any unauthorized use, copying or disclosure is prohibited. If you are not the intended recipient, please delete and contact the sender immediately. Please consider the environment before printing this e-mail. AVIS : le présent courriel et toute pièce jointe qui l'accompagne sont confidentiels, protégés par le droit d'auteur et peuvent être couverts par le secret professionnel. Toute utilisation, copie ou divulgation non autorisée est interdite. Si vous n'êtes pas le destinataire prévu de ce courriel, supprimez-le et contactez immédiatement l'expéditeur. Veuillez penser à l'environnement avant d'imprimer le présent courriel
RE: task trackers
Posted by "Kartashov, Andy" <An...@mpac.ca>.
Cheers Vinod.
From: Vinod Kumar Vavilapalli [mailto:vinodkv@hortonworks.com]
Sent: Monday, October 22, 2012 2:04 PM
To: user@hadoop.apache.org
Subject: Re: task trackers
Even JT accepts exclude/include hosts files for the slaves. If they aren't populated, JT by default accepts TTs that successfully register with it at runtime.
Thanks,
+Vinod
On Oct 22, 2012, at 7:55 AM, Kartashov, Andy wrote:
Gentlemen,
Can you please explain which property is responsible to tell JobTracker where TaskTruckers are running?
I get this far so far:
NN knows where SNN runs by specifying FQDN in the conf/masters of the NN
NN knows where DN runs by specifying FQDNs of the DNs in the conf/slaves of the NN
SNN knows the host:port of the NN using hdfs-core.xml
DNs know wherethehost:port of the NN using hdfs-core.xml
TTs know where the JT is using mapred-site.xml
But how does JT knows where the TTs are?
Please help.
Thanks,
Andy
NOTICE: This e-mail message and any attachments are confidential, subject to copyright and may be privileged. Any unauthorized use, copying or disclosure is prohibited. If you are not the intended recipient, please delete and contact the sender immediately. Please consider the environment before printing this e-mail. AVIS : le présent courriel et toute pièce jointe qui l'accompagne sont confidentiels, protégés par le droit d'auteur et peuvent être couverts par le secret professionnel. Toute utilisation, copie ou divulgation non autorisée est interdite. Si vous n'êtes pas le destinataire prévu de ce courriel, supprimez-le et contactez immédiatement l'expéditeur. Veuillez penser à l'environnement avant d'imprimer le présent courriel
NOTICE: This e-mail message and any attachments are confidential, subject to copyright and may be privileged. Any unauthorized use, copying or disclosure is prohibited. If you are not the intended recipient, please delete and contact the sender immediately. Please consider the environment before printing this e-mail. AVIS : le présent courriel et toute pièce jointe qui l'accompagne sont confidentiels, protégés par le droit d'auteur et peuvent être couverts par le secret professionnel. Toute utilisation, copie ou divulgation non autorisée est interdite. Si vous n'êtes pas le destinataire prévu de ce courriel, supprimez-le et contactez immédiatement l'expéditeur. Veuillez penser à l'environnement avant d'imprimer le présent courriel
RE: task trackers
Posted by "Kartashov, Andy" <An...@mpac.ca>.
Cheers Vinod.
From: Vinod Kumar Vavilapalli [mailto:vinodkv@hortonworks.com]
Sent: Monday, October 22, 2012 2:04 PM
To: user@hadoop.apache.org
Subject: Re: task trackers
Even JT accepts exclude/include hosts files for the slaves. If they aren't populated, JT by default accepts TTs that successfully register with it at runtime.
Thanks,
+Vinod
On Oct 22, 2012, at 7:55 AM, Kartashov, Andy wrote:
Gentlemen,
Can you please explain which property is responsible to tell JobTracker where TaskTruckers are running?
I get this far so far:
NN knows where SNN runs by specifying FQDN in the conf/masters of the NN
NN knows where DN runs by specifying FQDNs of the DNs in the conf/slaves of the NN
SNN knows the host:port of the NN using hdfs-core.xml
DNs know wherethehost:port of the NN using hdfs-core.xml
TTs know where the JT is using mapred-site.xml
But how does JT knows where the TTs are?
Please help.
Thanks,
Andy
NOTICE: This e-mail message and any attachments are confidential, subject to copyright and may be privileged. Any unauthorized use, copying or disclosure is prohibited. If you are not the intended recipient, please delete and contact the sender immediately. Please consider the environment before printing this e-mail. AVIS : le présent courriel et toute pièce jointe qui l'accompagne sont confidentiels, protégés par le droit d'auteur et peuvent être couverts par le secret professionnel. Toute utilisation, copie ou divulgation non autorisée est interdite. Si vous n'êtes pas le destinataire prévu de ce courriel, supprimez-le et contactez immédiatement l'expéditeur. Veuillez penser à l'environnement avant d'imprimer le présent courriel
NOTICE: This e-mail message and any attachments are confidential, subject to copyright and may be privileged. Any unauthorized use, copying or disclosure is prohibited. If you are not the intended recipient, please delete and contact the sender immediately. Please consider the environment before printing this e-mail. AVIS : le présent courriel et toute pièce jointe qui l'accompagne sont confidentiels, protégés par le droit d'auteur et peuvent être couverts par le secret professionnel. Toute utilisation, copie ou divulgation non autorisée est interdite. Si vous n'êtes pas le destinataire prévu de ce courriel, supprimez-le et contactez immédiatement l'expéditeur. Veuillez penser à l'environnement avant d'imprimer le présent courriel
RE: task trackers
Posted by "Kartashov, Andy" <An...@mpac.ca>.
Cheers Vinod.
From: Vinod Kumar Vavilapalli [mailto:vinodkv@hortonworks.com]
Sent: Monday, October 22, 2012 2:04 PM
To: user@hadoop.apache.org
Subject: Re: task trackers
Even JT accepts exclude/include hosts files for the slaves. If they aren't populated, JT by default accepts TTs that successfully register with it at runtime.
Thanks,
+Vinod
On Oct 22, 2012, at 7:55 AM, Kartashov, Andy wrote:
Gentlemen,
Can you please explain which property is responsible to tell JobTracker where TaskTruckers are running?
I get this far so far:
NN knows where SNN runs by specifying FQDN in the conf/masters of the NN
NN knows where DN runs by specifying FQDNs of the DNs in the conf/slaves of the NN
SNN knows the host:port of the NN using hdfs-core.xml
DNs know wherethehost:port of the NN using hdfs-core.xml
TTs know where the JT is using mapred-site.xml
But how does JT knows where the TTs are?
Please help.
Thanks,
Andy
NOTICE: This e-mail message and any attachments are confidential, subject to copyright and may be privileged. Any unauthorized use, copying or disclosure is prohibited. If you are not the intended recipient, please delete and contact the sender immediately. Please consider the environment before printing this e-mail. AVIS : le présent courriel et toute pièce jointe qui l'accompagne sont confidentiels, protégés par le droit d'auteur et peuvent être couverts par le secret professionnel. Toute utilisation, copie ou divulgation non autorisée est interdite. Si vous n'êtes pas le destinataire prévu de ce courriel, supprimez-le et contactez immédiatement l'expéditeur. Veuillez penser à l'environnement avant d'imprimer le présent courriel
NOTICE: This e-mail message and any attachments are confidential, subject to copyright and may be privileged. Any unauthorized use, copying or disclosure is prohibited. If you are not the intended recipient, please delete and contact the sender immediately. Please consider the environment before printing this e-mail. AVIS : le présent courriel et toute pièce jointe qui l'accompagne sont confidentiels, protégés par le droit d'auteur et peuvent être couverts par le secret professionnel. Toute utilisation, copie ou divulgation non autorisée est interdite. Si vous n'êtes pas le destinataire prévu de ce courriel, supprimez-le et contactez immédiatement l'expéditeur. Veuillez penser à l'environnement avant d'imprimer le présent courriel
Re: task trackers
Posted by Vinod Kumar Vavilapalli <vi...@hortonworks.com>.
Even JT accepts exclude/include hosts files for the slaves. If they aren't populated, JT by default accepts TTs that successfully register with it at runtime.
Thanks,
+Vinod
On Oct 22, 2012, at 7:55 AM, Kartashov, Andy wrote:
> Gentlemen,
>
> Can you please explain which property is responsible to tell JobTracker where TaskTruckers are running?
>
> I get this far so far:
>
> NN knows where SNN runs by specifying FQDN in the conf/masters of the NN
> NN knows where DN runs by specifying FQDNs of the DNs in the conf/slaves of the NN
> SNN knows the host:port of the NN using hdfs-core.xml
> DNs know wherethehost:port of the NN using hdfs-core.xml
>
> TTs know where the JT is using mapred-site.xml
>
> But how does JT knows where the TTs are?
>
> Please help.
>
> Thanks,
> Andy
> NOTICE: This e-mail message and any attachments are confidential, subject to copyright and may be privileged. Any unauthorized use, copying or disclosure is prohibited. If you are not the intended recipient, please delete and contact the sender immediately. Please consider the environment before printing this e-mail. AVIS : le présent courriel et toute pièce jointe qui l'accompagne sont confidentiels, protégés par le droit d'auteur et peuvent être couverts par le secret professionnel. Toute utilisation, copie ou divulgation non autorisée est interdite. Si vous n'êtes pas le destinataire prévu de ce courriel, supprimez-le et contactez immédiatement l'expéditeur. Veuillez penser à l'environnement avant d'imprimer le présent courriel
Re: task trackers
Posted by Vinod Kumar Vavilapalli <vi...@hortonworks.com>.
Even JT accepts exclude/include hosts files for the slaves. If they aren't populated, JT by default accepts TTs that successfully register with it at runtime.
Thanks,
+Vinod
On Oct 22, 2012, at 7:55 AM, Kartashov, Andy wrote:
> Gentlemen,
>
> Can you please explain which property is responsible to tell JobTracker where TaskTruckers are running?
>
> I get this far so far:
>
> NN knows where SNN runs by specifying FQDN in the conf/masters of the NN
> NN knows where DN runs by specifying FQDNs of the DNs in the conf/slaves of the NN
> SNN knows the host:port of the NN using hdfs-core.xml
> DNs know wherethehost:port of the NN using hdfs-core.xml
>
> TTs know where the JT is using mapred-site.xml
>
> But how does JT knows where the TTs are?
>
> Please help.
>
> Thanks,
> Andy
> NOTICE: This e-mail message and any attachments are confidential, subject to copyright and may be privileged. Any unauthorized use, copying or disclosure is prohibited. If you are not the intended recipient, please delete and contact the sender immediately. Please consider the environment before printing this e-mail. AVIS : le présent courriel et toute pièce jointe qui l'accompagne sont confidentiels, protégés par le droit d'auteur et peuvent être couverts par le secret professionnel. Toute utilisation, copie ou divulgation non autorisée est interdite. Si vous n'êtes pas le destinataire prévu de ce courriel, supprimez-le et contactez immédiatement l'expéditeur. Veuillez penser à l'environnement avant d'imprimer le présent courriel
Re: task trackers
Posted by Vinod Kumar Vavilapalli <vi...@hortonworks.com>.
Even JT accepts exclude/include hosts files for the slaves. If they aren't populated, JT by default accepts TTs that successfully register with it at runtime.
Thanks,
+Vinod
On Oct 22, 2012, at 7:55 AM, Kartashov, Andy wrote:
> Gentlemen,
>
> Can you please explain which property is responsible to tell JobTracker where TaskTruckers are running?
>
> I get this far so far:
>
> NN knows where SNN runs by specifying FQDN in the conf/masters of the NN
> NN knows where DN runs by specifying FQDNs of the DNs in the conf/slaves of the NN
> SNN knows the host:port of the NN using hdfs-core.xml
> DNs know wherethehost:port of the NN using hdfs-core.xml
>
> TTs know where the JT is using mapred-site.xml
>
> But how does JT knows where the TTs are?
>
> Please help.
>
> Thanks,
> Andy
> NOTICE: This e-mail message and any attachments are confidential, subject to copyright and may be privileged. Any unauthorized use, copying or disclosure is prohibited. If you are not the intended recipient, please delete and contact the sender immediately. Please consider the environment before printing this e-mail. AVIS : le présent courriel et toute pièce jointe qui l'accompagne sont confidentiels, protégés par le droit d'auteur et peuvent être couverts par le secret professionnel. Toute utilisation, copie ou divulgation non autorisée est interdite. Si vous n'êtes pas le destinataire prévu de ce courriel, supprimez-le et contactez immédiatement l'expéditeur. Veuillez penser à l'environnement avant d'imprimer le présent courriel
Re: task trackers
Posted by Vinod Kumar Vavilapalli <vi...@hortonworks.com>.
Even JT accepts exclude/include hosts files for the slaves. If they aren't populated, JT by default accepts TTs that successfully register with it at runtime.
Thanks,
+Vinod
On Oct 22, 2012, at 7:55 AM, Kartashov, Andy wrote:
> Gentlemen,
>
> Can you please explain which property is responsible to tell JobTracker where TaskTruckers are running?
>
> I get this far so far:
>
> NN knows where SNN runs by specifying FQDN in the conf/masters of the NN
> NN knows where DN runs by specifying FQDNs of the DNs in the conf/slaves of the NN
> SNN knows the host:port of the NN using hdfs-core.xml
> DNs know wherethehost:port of the NN using hdfs-core.xml
>
> TTs know where the JT is using mapred-site.xml
>
> But how does JT knows where the TTs are?
>
> Please help.
>
> Thanks,
> Andy
> NOTICE: This e-mail message and any attachments are confidential, subject to copyright and may be privileged. Any unauthorized use, copying or disclosure is prohibited. If you are not the intended recipient, please delete and contact the sender immediately. Please consider the environment before printing this e-mail. AVIS : le présent courriel et toute pièce jointe qui l'accompagne sont confidentiels, protégés par le droit d'auteur et peuvent être couverts par le secret professionnel. Toute utilisation, copie ou divulgation non autorisée est interdite. Si vous n'êtes pas le destinataire prévu de ce courriel, supprimez-le et contactez immédiatement l'expéditeur. Veuillez penser à l'environnement avant d'imprimer le présent courriel