[GitHub] [bigtop] sekikn commented on pull request #1008: BIGTOP-3814. Deploying Spark ThriftServer fails because it's launched too early.

[GitBox <gi...@apache.org>]

sekikn commented on PR #1008:
URL: https://github.com/apache/bigtop/pull/1008#issuecomment-1246519239

   @iwasakims Specifying a class directly (`Class['Hadoop::Init_hdfs']`) assumes that it's defined somewhere in the applied manifests, so it doesn't work if users try to deploy Spark without Hadoop, as follows.
   
   ```
   $ git diff
   diff --git a/bigtop-deploy/puppet/modules/spark/manifests/init.pp b/bigtop-deploy/puppet/modules/spark/manifests/init.pp
   index 06d804e5..3b9440c4 100644
   --- a/bigtop-deploy/puppet/modules/spark/manifests/init.pp
   +++ b/bigtop-deploy/puppet/modules/spark/manifests/init.pp
   @@ -63,7 +63,7 @@ class spark {
          hasrestart => true,
          hasstatus  => true,
        }
   -    
   +    Class['Hadoop::Init_hdfs'] -> Class['Spark::Spark_thriftserver']
      }
    
      class client {
   $ ./docker-hadoop.sh -d -C config_rockylinux-8.yaml -r file:///bigtop-home/output -G -k spark -c 1
   
   ...
   
   Error: Could not find resource 'Class[Hadoop::Init_hdfs]' for relationship on 'Class[Spark::Spark_thriftserver]' on node c3c4f6763873.bigtop.apache.org
   ```
   
   Such situation may be rare, but it's possible if users want to deploy only Spark onto their existing Hadoop cluster, for example.
   Puppet's resource collector (`Exec<| title == "init hdfs" |>`) can handle such case, so I think it's better here.
   
   ```
   $ git diff
   diff --git a/bigtop-deploy/puppet/modules/spark/manifests/init.pp b/bigtop-deploy/puppet/modules/spark/manifests/init.pp
   index 06d804e5..0da9f7f8 100644
   --- a/bigtop-deploy/puppet/modules/spark/manifests/init.pp
   +++ b/bigtop-deploy/puppet/modules/spark/manifests/init.pp
   @@ -63,7 +63,7 @@ class spark {
          hasrestart => true,
          hasstatus  => true,
        }
   -    
   +    Exec<| title == "init hdfs" |> -> Service["spark-thriftserver"]
      }
    
      class client {
   $ ./docker-hadoop.sh -d -C config_rockylinux-8.yaml -r file:///bigtop-home/output -G -k spark -c 1
   
   ...
   
   Notice: /Stage[main]/Spark::Datanucleus/Package[spark-datanucleus]/ensure: created
   Notice: /Stage[main]/Spark::Common/Package[spark-core]/ensure: created
   Notice: /Stage[main]/Spark::Common/File[/etc/spark/conf/spark-env.sh]/content: content changed '{md5}d5faace87add5ae723d31eb6b37e5007' to '{md5}7797eb3bd1be34a7a2d0ebc5bee1c4f4'
   Notice: /Stage[main]/Spark::Common/File[/etc/spark/conf/spark-defaults.conf]/ensure: defined content as '{md5}fc22ef68dcc50913d006da8d4466de4e'
   Notice: /Stage[main]/Spark::Common/File[/etc/spark/conf/log4j.properties]/ensure: defined content as '{md5}7fe3f0915ee2f08d6174bb80518c2b9a'
   Notice: /Stage[main]/Spark::Sparkr/Package[spark-sparkr]/ensure: created
   Notice: /Stage[main]/Spark::Client/Package[spark-python]/ensure: created
   Notice: /Stage[main]/Spark::Client/Package[spark-external]/ensure: created
   Notice: /Stage[main]/Spark::Yarn_shuffle/Package[spark-yarn-shuffle]/ensure: created
   Notice: /Stage[main]/Spark::Spark_thriftserver/Package[spark-thriftserver]/ensure: created
   Notice: /Stage[main]/Spark::Spark_thriftserver/Service[spark-thriftserver]/ensure: ensure changed 'stopped' to 'running'
   Notice: Applied catalog in 1729.48 seconds
   ```


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@bigtop.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org