You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@bigtop.apache.org by GitBox <gi...@apache.org> on 2022/07/18 05:07:23 UTC

[GitHub] [bigtop] timyuer opened a new pull request, #948: BIGTOP-3739: Add Tez support for Bigtop 3.1.0 Mpack

timyuer opened a new pull request, #948:
URL: https://github.com/apache/bigtop/pull/948

   <!--
     Thanks for sending a pull request!
       1. If this is your first time, please read our contributor guidelines: https://cwiki.apache.org/confluence/display/BIGTOP/How+to+Contribute
       2. Make sure your PR title starts with JIRA issue id, e.g., 'BIGTOP-3638: Your PR title ...'.
   -->
   
   ### Description of PR
   BIGTOP-3739: Add Tez support for Bigtop 3.1.0 Mpack
   
   Please use the following snapshot repository to run smoke test:
   https://sekikn-repo.s3.ap-southeast-1.amazonaws.com/centos-7/
   
   ### How was this patch tested?
   Tested along with #945 and #927 in centos7, Looks good to me.
   
   hadoop jar /usr/lib/tez/tez-examples-0.10.1.jar orderedwordcount /tmp/tezsmokeinput/sample-tez-test /tmp/tezsmokeoutput/
   ```
   [hdfs@ambari-server tez]$ hadoop jar /usr/lib/tez/tez-examples-0.10.1.jar orderedwordcount /tmp/tezsmokeinput/sample-tez-test /tmp/tezsmokeoutput/
   22/07/15 03:22:31 INFO shim.HadoopShimsLoader: Trying to locate HadoopShimProvider for hadoopVersion=3.2.3, majorVersion=3, minorVersion=2
   22/07/15 03:22:31 INFO shim.HadoopShimsLoader: Picked HadoopShim org.apache.tez.hadoop.shim.DefaultHadoopShim, providerName=null, overrideProviderViaConfig=null, hadoopVersion=3.2.3, majorVersion=3, minorVersion=2
   22/07/15 03:22:31 INFO counters.Limits: Counter limits initialized with parameters:  GROUP_NAME_MAX=256, MAX_GROUPS=3000, COUNTER_NAME_MAX=64, MAX_COUNTERS=10000
   22/07/15 03:22:31 INFO counters.Limits: Counter limits initialized with parameters:  GROUP_NAME_MAX=256, MAX_GROUPS=3000, COUNTER_NAME_MAX=64, MAX_COUNTERS=10000
   22/07/15 03:22:31 INFO client.TezClient: Tez Client Version: [ component=tez-api, version=0.10.1, revision=ff1ba5f1442094bca7ed53aba1faa991499dc1aa, SCM-URL=scm:git:https://gitbox.apache.org/repos/asf/tez.git, buildTime=2022-05-07T04:27:17Z, buildUser=jenkins, buildJavaVersion=1.8.0_322 ]
   22/07/15 03:22:31 INFO client.RMProxy: Connecting to ResourceManager at ambari-agent-02/172.30.0.4:8050
   22/07/15 03:22:32 INFO examples.OrderedWordCount: Running OrderedWordCount
   22/07/15 03:22:33 INFO client.TezClient: Submitting DAG application with id: application_1657843102751_0006
   22/07/15 03:22:33 INFO client.TezClientUtils: Using tez.lib.uris value from configuration: /apps/tez/tez.tar.gz
   22/07/15 03:22:33 INFO client.TezClientUtils: Using tez.lib.uris.classpath value from configuration: /etc/hadoop/conf,/usr/lib/hadoop/*,/usr/lib/hadoop/lib/*,/usr/lib/hadoop-hdfs/*,/usr/lib/hadoop-hdfs/lib/*,/usr/lib/hadoop-yarn/*,/usr/lib/hadoop-yarn/lib/*,/usr/lib/hadoop-mapreduce/*,/usr/lib/hadoop-mapreduce/lib/*,/usr/lib/tez/*,/usr/lib/tez/lib/*
   22/07/15 03:22:33 INFO client.TezClient: Stage directory /tmp/hdfs/staging doesn't exist and is created
   22/07/15 03:22:33 INFO client.TezClient: Tez system stage directory hdfs://ambari-agent-01:8020/tmp/hdfs/staging/.tez/application_1657843102751_0006 doesn't exist and is created
   22/07/15 03:22:33 INFO conf.Configuration: resource-types.xml not found
   22/07/15 03:22:33 INFO resource.ResourceUtils: Unable to find 'resource-types.xml'.
   22/07/15 03:22:34 INFO client.TezClient: Submitting DAG to YARN, applicationId=application_1657843102751_0006, dagName=OrderedWordCount, callerContext={ context=TezExamples, callerType=null, callerId=null }
   22/07/15 03:22:34 INFO impl.YarnClientImpl: Submitted application application_1657843102751_0006
   22/07/15 03:22:34 INFO client.TezClient: The url to track the Tez AM: http://ambari-agent-02:8088/proxy/application_1657843102751_0006/
   22/07/15 03:22:42 INFO client.DAGClientImpl: DAG initialized: CurrentState=Running
   22/07/15 03:22:42 INFO client.DAGClientImpl: DAG: State: RUNNING Progress: 0% TotalTasks: 3 Succeeded: 0 Running: 0 Failed: 0 Killed: 0
   22/07/15 03:22:42 INFO client.DAGClientImpl: 	VertexStatus: VertexName: Tokenizer Progress: 0% TotalTasks: 1 Succeeded: 0 Running: 0 Failed: 0 Killed: 0
   22/07/15 03:22:42 INFO client.DAGClientImpl: 	VertexStatus: VertexName: Summation Progress: 0% TotalTasks: 1 Succeeded: 0 Running: 0 Failed: 0 Killed: 0
   22/07/15 03:22:42 INFO client.DAGClientImpl: 	VertexStatus: VertexName: Sorter Progress: 0% TotalTasks: 1 Succeeded: 0 Running: 0 Failed: 0 Killed: 0
   22/07/15 03:22:46 INFO client.DAGClientImpl: DAG: State: RUNNING Progress: 33.33% TotalTasks: 3 Succeeded: 1 Running: 0 Failed: 0 Killed: 0
   22/07/15 03:22:46 INFO client.DAGClientImpl: 	VertexStatus: VertexName: Tokenizer Progress: 100% TotalTasks: 1 Succeeded: 1 Running: 0 Failed: 0 Killed: 0
   22/07/15 03:22:46 INFO client.DAGClientImpl: 	VertexStatus: VertexName: Summation Progress: 0% TotalTasks: 1 Succeeded: 0 Running: 0 Failed: 0 Killed: 0
   22/07/15 03:22:46 INFO client.DAGClientImpl: 	VertexStatus: VertexName: Sorter Progress: 0% TotalTasks: 1 Succeeded: 0 Running: 0 Failed: 0 Killed: 0
   22/07/15 03:22:49 INFO client.DAGClientImpl: DAG: State: RUNNING Progress: 66.67% TotalTasks: 3 Succeeded: 2 Running: 1 Failed: 0 Killed: 0
   22/07/15 03:22:49 INFO client.DAGClientImpl: 	VertexStatus: VertexName: Tokenizer Progress: 100% TotalTasks: 1 Succeeded: 1 Running: 0 Failed: 0 Killed: 0
   22/07/15 03:22:49 INFO client.DAGClientImpl: 	VertexStatus: VertexName: Summation Progress: 100% TotalTasks: 1 Succeeded: 1 Running: 0 Failed: 0 Killed: 0
   22/07/15 03:22:49 INFO client.DAGClientImpl: 	VertexStatus: VertexName: Sorter Progress: 0% TotalTasks: 1 Succeeded: 0 Running: 1 Failed: 0 Killed: 0
   22/07/15 03:22:49 INFO client.DAGClientImpl: DAG: State: SUCCEEDED Progress: 100% TotalTasks: 3 Succeeded: 3 Running: 0 Failed: 0 Killed: 0
   22/07/15 03:22:49 INFO client.DAGClientImpl: 	VertexStatus: VertexName: Tokenizer Progress: 100% TotalTasks: 1 Succeeded: 1 Running: 0 Failed: 0 Killed: 0
   22/07/15 03:22:49 INFO client.DAGClientImpl: 	VertexStatus: VertexName: Summation Progress: 100% TotalTasks: 1 Succeeded: 1 Running: 0 Failed: 0 Killed: 0
   22/07/15 03:22:49 INFO client.DAGClientImpl: 	VertexStatus: VertexName: Sorter Progress: 100% TotalTasks: 1 Succeeded: 1 Running: 0 Failed: 0 Killed: 0
   22/07/15 03:22:49 INFO client.DAGClientImpl: DAG completed. FinalState=SUCCEEDED
   ```
   ### For code changes:
   
   - [ ] Does the title or this PR starts with the corresponding JIRA issue id (e.g. 'BIGTOP-3638. Your PR title ...')?
   - [ ] Make sure that newly added files do not have any licensing issues. When in doubt refer to https://www.apache.org/licenses/


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@bigtop.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [bigtop] guyuqi merged pull request #948: BIGTOP-3739: Add Tez support for Bigtop 3.1.0 Mpack

Posted by GitBox <gi...@apache.org>.
guyuqi merged PR #948:
URL: https://github.com/apache/bigtop/pull/948


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@bigtop.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [bigtop] guyuqi commented on pull request #948: BIGTOP-3739: Add Tez support for Bigtop 3.1.0 Mpack

Posted by GitBox <gi...@apache.org>.
guyuqi commented on PR #948:
URL: https://github.com/apache/bigtop/pull/948#issuecomment-1187667506

   Merged. 
   thanks, @timyuer 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@bigtop.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [bigtop] guyuqi commented on pull request #948: BIGTOP-3739: Add Tez support for Bigtop 3.1.0 Mpack

Posted by GitBox <gi...@apache.org>.
guyuqi commented on PR #948:
URL: https://github.com/apache/bigtop/pull/948#issuecomment-1187658567

   It works after `role_command_order.json` defined. 
   +1, LGTM


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@bigtop.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [bigtop] timyuer commented on pull request #948: BIGTOP-3739: Add Tez support for Bigtop 3.1.0 Mpack

Posted by GitBox <gi...@apache.org>.
timyuer commented on PR #948:
URL: https://github.com/apache/bigtop/pull/948#issuecomment-1186978860

   Thanks, I didn't define `role_command_order.json`
   
   ![check_tez](https://user-images.githubusercontent.com/16263438/179483403-bcf95d12-21f4-41b0-af25-c46893d668f7.png)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@bigtop.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [bigtop] guyuqi commented on pull request #948: BIGTOP-3739: Add Tez support for Bigtop 3.1.0 Mpack

Posted by GitBox <gi...@apache.org>.
guyuqi commented on PR #948:
URL: https://github.com/apache/bigtop/pull/948#issuecomment-1186874410

   There are some issues on Checking Tez:
   
   ![check_tez](https://user-images.githubusercontent.com/20575685/179465984-238ff03e-9236-404d-9a6c-5deaa238ef98.png)
   
   
   Log:
   ```
   2022-07-18 07:29:44,615 - File['/var/lib/ambari-agent/tmp/sample-tez-test'] {'content': 'foo\nbar\nfoo\nbar\nfoo', 'mode': 0755}
   2022-07-18 07:29:44,617 - Writing File['/var/lib/ambari-agent/tmp/sample-tez-test'] because it doesn't exist
   2022-07-18 07:29:44,617 - Changing permission for /var/lib/ambari-agent/tmp/sample-tez-test from 644 to 755
   2022-07-18 07:29:44,618 - HdfsResource['/tmp/tezsmokeoutput'] {'security_enabled': False, 'hadoop_bin_dir': '/usr/bin', 'keytab': [EMPTY], 'dfs_type': 'HDFS', 'default_fs': 'hdfs://ambari-agent-01:8020', 'hdfs_resource_ignore_file': '/var/lib/ambari-agent/data/.hdfs_resource_ignore', 'hdfs_site': ..., 'kinit_path_local': '/usr/bin/kinit', 'principal_name': [EMPTY], 'user': 'hdfs', 'action': ['delete_on_execute'], 'hadoop_conf_dir': '/etc/hadoop/conf', 'type': 'directory', 'immutable_paths': [u'/mr-history/done', u'/app-logs', u'/tmp']}
   2022-07-18 07:29:44,629 - call['ambari-sudo.sh su hdfs -l -s /bin/bash -c 'curl -sS -L -w '"'"'%{http_code}'"'"' -X GET -d '"'"''"'"' -H '"'"'Content-Length: 0'"'"' '"'"'http://ambari-agent-01:50070/webhdfs/v1/tmp/tezsmokeoutput?op=GETFILESTATUS&user.name=hdfs'"'"' 1>/tmp/tmp3M81cO 2>/tmp/tmpaAtCg6''] {'logoutput': None, 'quiet': False}
   2022-07-18 07:29:44,687 - call returned (7, '')
   ```
   
   ```
   Traceback (most recent call last):
     File "/var/lib/ambari-agent/cache/stacks/BGTP/1.0/services/TEZ/package/scripts/service_check.py", line 125, in <module>
       TezServiceCheck().execute()
     File "/usr/lib/ambari-agent/lib/resource_management/libraries/script/script.py", line 352, in execute
       method(env)
     File "/var/lib/ambari-agent/cache/stacks/BGTP/1.0/services/TEZ/package/scripts/service_check.py", line 56, in service_check
       type = "directory"
     File "/usr/lib/ambari-agent/lib/resource_management/core/base.py", line 166, in __init__
       self.env.run()
     File "/usr/lib/ambari-agent/lib/resource_management/core/environment.py", line 160, in run
       self.run_action(resource, action)
     File "/usr/lib/ambari-agent/lib/resource_management/core/environment.py", line 124, in run_action
       provider_action()
     File "/usr/lib/ambari-agent/lib/resource_management/libraries/providers/hdfs_resource.py", line 680, in action_delete_on_execute
       self.action_delayed("delete")
     File "/usr/lib/ambari-agent/lib/resource_management/libraries/providers/hdfs_resource.py", line 674, in action_delayed
       self.get_hdfs_resource_executor().action_delayed(action_name, self)
     File "/usr/lib/ambari-agent/lib/resource_management/libraries/providers/hdfs_resource.py", line 373, in action_delayed
       self.action_delayed_for_nameservice(None, action_name, main_resource)
     File "/usr/lib/ambari-agent/lib/resource_management/libraries/providers/hdfs_resource.py", line 395, in action_delayed_for_nameservice
       self._assert_valid()
     File "/usr/lib/ambari-agent/lib/resource_management/libraries/providers/hdfs_resource.py", line 334, in _assert_valid
       self.target_status = self._get_file_status(target)
     File "/usr/lib/ambari-agent/lib/resource_management/libraries/providers/hdfs_resource.py", line 497, in _get_file_status
       list_status = self.util.run_command(target, 'GETFILESTATUS', method='GET', ignore_status_codes=['404'], assertable_result=False)
     File "/usr/lib/ambari-agent/lib/resource_management/libraries/providers/hdfs_resource.py", line 214, in run_command
       return self._run_command(*args, **kwargs)
     File "/usr/lib/ambari-agent/lib/resource_management/libraries/providers/hdfs_resource.py", line 282, in _run_command
       _, out, err = get_user_call_output(cmd, user=self.run_user, logoutput=self.logoutput, quiet=False)
     File "/usr/lib/ambari-agent/lib/resource_management/libraries/functions/get_user_call_output.py", line 62, in get_user_call_output
       raise ExecutionFailed(err_msg, code, files_output[0], files_output[1])
   resource_management.core.exceptions.ExecutionFailed: Execution of 'curl -sS -L -w '%{http_code}' -X GET -d '' -H 'Content-Length: 0' 'http://ambari-agent-01:50070/webhdfs/v1/tmp/tezsmokeoutput?op=GETFILESTATUS&user.name=hdfs' 1>/tmp/tmp3M81cO 2>/tmp/tmpaAtCg6' returned 7. curl: (7) Failed connect to ambari-agent-01:50070; Connection refused
   000
   ```
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@bigtop.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [bigtop] timyuer commented on pull request #948: BIGTOP-3739: Add Tez support for Bigtop 3.1.0 Mpack

Posted by GitBox <gi...@apache.org>.
timyuer commented on PR #948:
URL: https://github.com/apache/bigtop/pull/948#issuecomment-1186906426

   Are you using the Hadoop packaged by Bigtop 3.1.0? WebHDFS does not work due to ~[BIGTOP-3714](https://issues.apache.org/jira/browse/BIGTOP-3714)~ if so.
   ![20220718161924](https://user-images.githubusercontent.com/16263438/179471545-ed917ffe-d2f3-4933-b628-af6ade40db9e.png)
   
   Please use the following snapshot repository to run smoke test:
   https://sekikn-repo.s3.ap-southeast-1.amazonaws.com/centos-7/
   ![微信图片_20220718155147](https://user-images.githubusercontent.com/16263438/179467308-afc9a891-2aae-4a58-a948-f79130ce47b6.jpg)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@bigtop.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org