You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@bigtop.apache.org by GitBox <gi...@apache.org> on 2022/07/18 05:07:23 UTC
[GitHub] [bigtop] timyuer opened a new pull request, #948: BIGTOP-3739: Add Tez support for Bigtop 3.1.0 Mpack
timyuer opened a new pull request, #948:
URL: https://github.com/apache/bigtop/pull/948
<!--
Thanks for sending a pull request!
1. If this is your first time, please read our contributor guidelines: https://cwiki.apache.org/confluence/display/BIGTOP/How+to+Contribute
2. Make sure your PR title starts with JIRA issue id, e.g., 'BIGTOP-3638: Your PR title ...'.
-->
### Description of PR
BIGTOP-3739: Add Tez support for Bigtop 3.1.0 Mpack
Please use the following snapshot repository to run smoke test:
https://sekikn-repo.s3.ap-southeast-1.amazonaws.com/centos-7/
### How was this patch tested?
Tested along with #945 and #927 in centos7, Looks good to me.
hadoop jar /usr/lib/tez/tez-examples-0.10.1.jar orderedwordcount /tmp/tezsmokeinput/sample-tez-test /tmp/tezsmokeoutput/
```
[hdfs@ambari-server tez]$ hadoop jar /usr/lib/tez/tez-examples-0.10.1.jar orderedwordcount /tmp/tezsmokeinput/sample-tez-test /tmp/tezsmokeoutput/
22/07/15 03:22:31 INFO shim.HadoopShimsLoader: Trying to locate HadoopShimProvider for hadoopVersion=3.2.3, majorVersion=3, minorVersion=2
22/07/15 03:22:31 INFO shim.HadoopShimsLoader: Picked HadoopShim org.apache.tez.hadoop.shim.DefaultHadoopShim, providerName=null, overrideProviderViaConfig=null, hadoopVersion=3.2.3, majorVersion=3, minorVersion=2
22/07/15 03:22:31 INFO counters.Limits: Counter limits initialized with parameters: GROUP_NAME_MAX=256, MAX_GROUPS=3000, COUNTER_NAME_MAX=64, MAX_COUNTERS=10000
22/07/15 03:22:31 INFO counters.Limits: Counter limits initialized with parameters: GROUP_NAME_MAX=256, MAX_GROUPS=3000, COUNTER_NAME_MAX=64, MAX_COUNTERS=10000
22/07/15 03:22:31 INFO client.TezClient: Tez Client Version: [ component=tez-api, version=0.10.1, revision=ff1ba5f1442094bca7ed53aba1faa991499dc1aa, SCM-URL=scm:git:https://gitbox.apache.org/repos/asf/tez.git, buildTime=2022-05-07T04:27:17Z, buildUser=jenkins, buildJavaVersion=1.8.0_322 ]
22/07/15 03:22:31 INFO client.RMProxy: Connecting to ResourceManager at ambari-agent-02/172.30.0.4:8050
22/07/15 03:22:32 INFO examples.OrderedWordCount: Running OrderedWordCount
22/07/15 03:22:33 INFO client.TezClient: Submitting DAG application with id: application_1657843102751_0006
22/07/15 03:22:33 INFO client.TezClientUtils: Using tez.lib.uris value from configuration: /apps/tez/tez.tar.gz
22/07/15 03:22:33 INFO client.TezClientUtils: Using tez.lib.uris.classpath value from configuration: /etc/hadoop/conf,/usr/lib/hadoop/*,/usr/lib/hadoop/lib/*,/usr/lib/hadoop-hdfs/*,/usr/lib/hadoop-hdfs/lib/*,/usr/lib/hadoop-yarn/*,/usr/lib/hadoop-yarn/lib/*,/usr/lib/hadoop-mapreduce/*,/usr/lib/hadoop-mapreduce/lib/*,/usr/lib/tez/*,/usr/lib/tez/lib/*
22/07/15 03:22:33 INFO client.TezClient: Stage directory /tmp/hdfs/staging doesn't exist and is created
22/07/15 03:22:33 INFO client.TezClient: Tez system stage directory hdfs://ambari-agent-01:8020/tmp/hdfs/staging/.tez/application_1657843102751_0006 doesn't exist and is created
22/07/15 03:22:33 INFO conf.Configuration: resource-types.xml not found
22/07/15 03:22:33 INFO resource.ResourceUtils: Unable to find 'resource-types.xml'.
22/07/15 03:22:34 INFO client.TezClient: Submitting DAG to YARN, applicationId=application_1657843102751_0006, dagName=OrderedWordCount, callerContext={ context=TezExamples, callerType=null, callerId=null }
22/07/15 03:22:34 INFO impl.YarnClientImpl: Submitted application application_1657843102751_0006
22/07/15 03:22:34 INFO client.TezClient: The url to track the Tez AM: http://ambari-agent-02:8088/proxy/application_1657843102751_0006/
22/07/15 03:22:42 INFO client.DAGClientImpl: DAG initialized: CurrentState=Running
22/07/15 03:22:42 INFO client.DAGClientImpl: DAG: State: RUNNING Progress: 0% TotalTasks: 3 Succeeded: 0 Running: 0 Failed: 0 Killed: 0
22/07/15 03:22:42 INFO client.DAGClientImpl: VertexStatus: VertexName: Tokenizer Progress: 0% TotalTasks: 1 Succeeded: 0 Running: 0 Failed: 0 Killed: 0
22/07/15 03:22:42 INFO client.DAGClientImpl: VertexStatus: VertexName: Summation Progress: 0% TotalTasks: 1 Succeeded: 0 Running: 0 Failed: 0 Killed: 0
22/07/15 03:22:42 INFO client.DAGClientImpl: VertexStatus: VertexName: Sorter Progress: 0% TotalTasks: 1 Succeeded: 0 Running: 0 Failed: 0 Killed: 0
22/07/15 03:22:46 INFO client.DAGClientImpl: DAG: State: RUNNING Progress: 33.33% TotalTasks: 3 Succeeded: 1 Running: 0 Failed: 0 Killed: 0
22/07/15 03:22:46 INFO client.DAGClientImpl: VertexStatus: VertexName: Tokenizer Progress: 100% TotalTasks: 1 Succeeded: 1 Running: 0 Failed: 0 Killed: 0
22/07/15 03:22:46 INFO client.DAGClientImpl: VertexStatus: VertexName: Summation Progress: 0% TotalTasks: 1 Succeeded: 0 Running: 0 Failed: 0 Killed: 0
22/07/15 03:22:46 INFO client.DAGClientImpl: VertexStatus: VertexName: Sorter Progress: 0% TotalTasks: 1 Succeeded: 0 Running: 0 Failed: 0 Killed: 0
22/07/15 03:22:49 INFO client.DAGClientImpl: DAG: State: RUNNING Progress: 66.67% TotalTasks: 3 Succeeded: 2 Running: 1 Failed: 0 Killed: 0
22/07/15 03:22:49 INFO client.DAGClientImpl: VertexStatus: VertexName: Tokenizer Progress: 100% TotalTasks: 1 Succeeded: 1 Running: 0 Failed: 0 Killed: 0
22/07/15 03:22:49 INFO client.DAGClientImpl: VertexStatus: VertexName: Summation Progress: 100% TotalTasks: 1 Succeeded: 1 Running: 0 Failed: 0 Killed: 0
22/07/15 03:22:49 INFO client.DAGClientImpl: VertexStatus: VertexName: Sorter Progress: 0% TotalTasks: 1 Succeeded: 0 Running: 1 Failed: 0 Killed: 0
22/07/15 03:22:49 INFO client.DAGClientImpl: DAG: State: SUCCEEDED Progress: 100% TotalTasks: 3 Succeeded: 3 Running: 0 Failed: 0 Killed: 0
22/07/15 03:22:49 INFO client.DAGClientImpl: VertexStatus: VertexName: Tokenizer Progress: 100% TotalTasks: 1 Succeeded: 1 Running: 0 Failed: 0 Killed: 0
22/07/15 03:22:49 INFO client.DAGClientImpl: VertexStatus: VertexName: Summation Progress: 100% TotalTasks: 1 Succeeded: 1 Running: 0 Failed: 0 Killed: 0
22/07/15 03:22:49 INFO client.DAGClientImpl: VertexStatus: VertexName: Sorter Progress: 100% TotalTasks: 1 Succeeded: 1 Running: 0 Failed: 0 Killed: 0
22/07/15 03:22:49 INFO client.DAGClientImpl: DAG completed. FinalState=SUCCEEDED
```
### For code changes:
- [ ] Does the title or this PR starts with the corresponding JIRA issue id (e.g. 'BIGTOP-3638. Your PR title ...')?
- [ ] Make sure that newly added files do not have any licensing issues. When in doubt refer to https://www.apache.org/licenses/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@bigtop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [bigtop] guyuqi merged pull request #948: BIGTOP-3739: Add Tez support for Bigtop 3.1.0 Mpack
Posted by GitBox <gi...@apache.org>.
guyuqi merged PR #948:
URL: https://github.com/apache/bigtop/pull/948
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@bigtop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [bigtop] guyuqi commented on pull request #948: BIGTOP-3739: Add Tez support for Bigtop 3.1.0 Mpack
Posted by GitBox <gi...@apache.org>.
guyuqi commented on PR #948:
URL: https://github.com/apache/bigtop/pull/948#issuecomment-1187667506
Merged.
thanks, @timyuer
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@bigtop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [bigtop] guyuqi commented on pull request #948: BIGTOP-3739: Add Tez support for Bigtop 3.1.0 Mpack
Posted by GitBox <gi...@apache.org>.
guyuqi commented on PR #948:
URL: https://github.com/apache/bigtop/pull/948#issuecomment-1187658567
It works after `role_command_order.json` defined.
+1, LGTM
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@bigtop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [bigtop] timyuer commented on pull request #948: BIGTOP-3739: Add Tez support for Bigtop 3.1.0 Mpack
Posted by GitBox <gi...@apache.org>.
timyuer commented on PR #948:
URL: https://github.com/apache/bigtop/pull/948#issuecomment-1186978860
Thanks, I didn't define `role_command_order.json`

--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@bigtop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [bigtop] guyuqi commented on pull request #948: BIGTOP-3739: Add Tez support for Bigtop 3.1.0 Mpack
Posted by GitBox <gi...@apache.org>.
guyuqi commented on PR #948:
URL: https://github.com/apache/bigtop/pull/948#issuecomment-1186874410
There are some issues on Checking Tez:

Log:
```
2022-07-18 07:29:44,615 - File['/var/lib/ambari-agent/tmp/sample-tez-test'] {'content': 'foo\nbar\nfoo\nbar\nfoo', 'mode': 0755}
2022-07-18 07:29:44,617 - Writing File['/var/lib/ambari-agent/tmp/sample-tez-test'] because it doesn't exist
2022-07-18 07:29:44,617 - Changing permission for /var/lib/ambari-agent/tmp/sample-tez-test from 644 to 755
2022-07-18 07:29:44,618 - HdfsResource['/tmp/tezsmokeoutput'] {'security_enabled': False, 'hadoop_bin_dir': '/usr/bin', 'keytab': [EMPTY], 'dfs_type': 'HDFS', 'default_fs': 'hdfs://ambari-agent-01:8020', 'hdfs_resource_ignore_file': '/var/lib/ambari-agent/data/.hdfs_resource_ignore', 'hdfs_site': ..., 'kinit_path_local': '/usr/bin/kinit', 'principal_name': [EMPTY], 'user': 'hdfs', 'action': ['delete_on_execute'], 'hadoop_conf_dir': '/etc/hadoop/conf', 'type': 'directory', 'immutable_paths': [u'/mr-history/done', u'/app-logs', u'/tmp']}
2022-07-18 07:29:44,629 - call['ambari-sudo.sh su hdfs -l -s /bin/bash -c 'curl -sS -L -w '"'"'%{http_code}'"'"' -X GET -d '"'"''"'"' -H '"'"'Content-Length: 0'"'"' '"'"'http://ambari-agent-01:50070/webhdfs/v1/tmp/tezsmokeoutput?op=GETFILESTATUS&user.name=hdfs'"'"' 1>/tmp/tmp3M81cO 2>/tmp/tmpaAtCg6''] {'logoutput': None, 'quiet': False}
2022-07-18 07:29:44,687 - call returned (7, '')
```
```
Traceback (most recent call last):
File "/var/lib/ambari-agent/cache/stacks/BGTP/1.0/services/TEZ/package/scripts/service_check.py", line 125, in <module>
TezServiceCheck().execute()
File "/usr/lib/ambari-agent/lib/resource_management/libraries/script/script.py", line 352, in execute
method(env)
File "/var/lib/ambari-agent/cache/stacks/BGTP/1.0/services/TEZ/package/scripts/service_check.py", line 56, in service_check
type = "directory"
File "/usr/lib/ambari-agent/lib/resource_management/core/base.py", line 166, in __init__
self.env.run()
File "/usr/lib/ambari-agent/lib/resource_management/core/environment.py", line 160, in run
self.run_action(resource, action)
File "/usr/lib/ambari-agent/lib/resource_management/core/environment.py", line 124, in run_action
provider_action()
File "/usr/lib/ambari-agent/lib/resource_management/libraries/providers/hdfs_resource.py", line 680, in action_delete_on_execute
self.action_delayed("delete")
File "/usr/lib/ambari-agent/lib/resource_management/libraries/providers/hdfs_resource.py", line 674, in action_delayed
self.get_hdfs_resource_executor().action_delayed(action_name, self)
File "/usr/lib/ambari-agent/lib/resource_management/libraries/providers/hdfs_resource.py", line 373, in action_delayed
self.action_delayed_for_nameservice(None, action_name, main_resource)
File "/usr/lib/ambari-agent/lib/resource_management/libraries/providers/hdfs_resource.py", line 395, in action_delayed_for_nameservice
self._assert_valid()
File "/usr/lib/ambari-agent/lib/resource_management/libraries/providers/hdfs_resource.py", line 334, in _assert_valid
self.target_status = self._get_file_status(target)
File "/usr/lib/ambari-agent/lib/resource_management/libraries/providers/hdfs_resource.py", line 497, in _get_file_status
list_status = self.util.run_command(target, 'GETFILESTATUS', method='GET', ignore_status_codes=['404'], assertable_result=False)
File "/usr/lib/ambari-agent/lib/resource_management/libraries/providers/hdfs_resource.py", line 214, in run_command
return self._run_command(*args, **kwargs)
File "/usr/lib/ambari-agent/lib/resource_management/libraries/providers/hdfs_resource.py", line 282, in _run_command
_, out, err = get_user_call_output(cmd, user=self.run_user, logoutput=self.logoutput, quiet=False)
File "/usr/lib/ambari-agent/lib/resource_management/libraries/functions/get_user_call_output.py", line 62, in get_user_call_output
raise ExecutionFailed(err_msg, code, files_output[0], files_output[1])
resource_management.core.exceptions.ExecutionFailed: Execution of 'curl -sS -L -w '%{http_code}' -X GET -d '' -H 'Content-Length: 0' 'http://ambari-agent-01:50070/webhdfs/v1/tmp/tezsmokeoutput?op=GETFILESTATUS&user.name=hdfs' 1>/tmp/tmp3M81cO 2>/tmp/tmpaAtCg6' returned 7. curl: (7) Failed connect to ambari-agent-01:50070; Connection refused
000
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@bigtop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [bigtop] timyuer commented on pull request #948: BIGTOP-3739: Add Tez support for Bigtop 3.1.0 Mpack
Posted by GitBox <gi...@apache.org>.
timyuer commented on PR #948:
URL: https://github.com/apache/bigtop/pull/948#issuecomment-1186906426
Are you using the Hadoop packaged by Bigtop 3.1.0? WebHDFS does not work due to ~[BIGTOP-3714](https://issues.apache.org/jira/browse/BIGTOP-3714)~ if so.

Please use the following snapshot repository to run smoke test:
https://sekikn-repo.s3.ap-southeast-1.amazonaws.com/centos-7/

--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@bigtop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org