You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@metron.apache.org by 2xyo <gi...@git.apache.org> on 2016/08/21 12:25:28 UTC
[GitHub] incubator-metron pull request #220: Metron-342 Fix permissions of {{ metron_...
GitHub user 2xyo opened a pull request:
https://github.com/apache/incubator-metron/pull/220
Metron-342 Fix permissions of {{ metron_directory }}/patterns directory
This fix allow the `hdfs` user to access to grok patterns in `{{ metron_directory }}/patterns`.
This PR addresses https://issues.apache.org/jira/browse/METRON-343
Fix the following error during the installation:
```
TASK [metron_streaming : Upload Grok Patterns to hdfs:///apps/metron] **********
fatal: [node1]: FAILED! => {"changed": true, "cmd": ["hdfs", "dfs", "-put", "-f", "/usr/metron/0.2.0BETA/patterns", "/apps/metron"], "delta": "0:00:02.131558", "end": "2016-08-21 14:03:26.476524", "failed": true, "rc": 1, "start": "2016-08-21 14:03:24.344966", "stderr": "put: Error accessing file:/usr/metron/0.2.0BETA/patterns", "stdout": "", "stdout_lines": [], "warnings": []}
```
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/2xyo/incubator-metron patch-2
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/incubator-metron/pull/220.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #220
----
commit 16366dce01393bbaad59340a664a9e18a455284c
Author: 2*yo <yo...@lepage.info>
Date: 2016-08-21T12:23:59Z
Metron-342 Fix permissions of {{ metron_directory }}/patterns directory
This fix allow the `hdfs` user to upload grok pattern on HDFS directory.
This PR addresses https://issues.apache.org/jira/browse/METRON-343
Fix the following error during the installation:
```
TASK [metron_streaming : Upload Grok Patterns to hdfs:///apps/metron] **********
fatal: [node1]: FAILED! => {"changed": true, "cmd": ["hdfs", "dfs", "-put", "-f", "/usr/metron/0.2.0BETA/patterns", "/apps/metron"], "delta": "0:00:02.131558", "end": "2016-08-21 14:03:26.476524", "failed": true, "rc": 1, "start": "2016-08-21 14:03:24.344966", "stderr": "put: Error accessing file:/usr/metron/0.2.0BETA/patterns", "stdout": "", "stdout_lines": [], "warnings": []}
```
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron issue #220: Metron-342 Fix permissions of {{ metron_directo...
Posted by gspeter <gi...@git.apache.org>.
Github user gspeter commented on the issue:
https://github.com/apache/incubator-metron/pull/220
hi @simonellistonball
I installed nodejs on Ubuntu Server.
```
curl -sL https://deb.nodesource.com/setup_6.x | sudo -E bash -
sudo apt-get install -y nodejs
```
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron issue #220: Metron-342 Fix permissions of {{ metron_directo...
Posted by gspeter <gi...@git.apache.org>.
Github user gspeter commented on the issue:
https://github.com/apache/incubator-metron/pull/220
ok. thank you.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron issue #220: Metron-342 Fix permissions of {{ metron_directo...
Posted by simonellistonball <gi...@git.apache.org>.
Github user simonellistonball commented on the issue:
https://github.com/apache/incubator-metron/pull/220
@gspeter would you mind opening this separately as an issue, or a jira (to avoid confusion around this pull request) and posting the output of the -X switch maven suggests.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron issue #220: Metron-342 Fix permissions of {{ metron_directo...
Posted by gspeter <gi...@git.apache.org>.
Github user gspeter commented on the issue:
https://github.com/apache/incubator-metron/pull/220
Hi @ottobackwards
I run "mvn clean package -DskipTests | tee build.log". I send my log file. Can you help me.
[build.log.tar.gz](https://github.com/apache/incubator-metron/files/972696/build.log.tar.gz)
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron issue #220: Metron-342 Fix permissions of {{ metron_directo...
Posted by ottobackwards <gi...@git.apache.org>.
Github user ottobackwards commented on the issue:
https://github.com/apache/incubator-metron/pull/220
Also - above what you pasted is the real error in the log. that is what would be needed. NPM install is run in a project directory to install any specific node modules that the project uses locally. So node is installed, but it is trying to install that stuff.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron issue #220: Metron-342 Fix permissions of {{ metron_directo...
Posted by dlyle65535 <gi...@git.apache.org>.
Github user dlyle65535 commented on the issue:
https://github.com/apache/incubator-metron/pull/220
I don't see that error when I deploy. Can you tell me a little bit about your environment?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron issue #220: Metron-342 Fix permissions of {{ metron_directo...
Posted by mmiklavc <gi...@git.apache.org>.
Github user mmiklavc commented on the issue:
https://github.com/apache/incubator-metron/pull/220
@2xyo This PR has been here a while - are you still seeing this on current master?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron issue #220: Metron-342 Fix permissions of {{ metron_directo...
Posted by gspeter <gi...@git.apache.org>.
Github user gspeter commented on the issue:
https://github.com/apache/incubator-metron/pull/220
@simonellistonball ok. Thank you.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron issue #220: Metron-342 Fix permissions of {{ metron_directo...
Posted by ottobackwards <gi...@git.apache.org>.
Github user ottobackwards commented on the issue:
https://github.com/apache/incubator-metron/pull/220
Can we close this then?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron issue #220: Metron-342 Fix permissions of {{ metron_directo...
Posted by ottobackwards <gi...@git.apache.org>.
Github user ottobackwards commented on the issue:
https://github.com/apache/incubator-metron/pull/220
It would be better if we understood why it was happening in this setup a little better
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron issue #220: Metron-342 Fix permissions of {{ metron_directo...
Posted by 2xyo <gi...@git.apache.org>.
Github user 2xyo commented on the issue:
https://github.com/apache/incubator-metron/pull/220
I still see the issue on current master with a Centos 6 VM.
Steps to reproduce the error on the ansible client:
```
$ sudo -H pip uninstall -y ansible && sudo -H pip install ansible==2.0.0.2
$ ssh-keygen -t rsa -b 4096
$ echo "192.168.1.11 node1" | sudo tee -a /etc/hosts >/dev/null
$ ssh-copy-id root@node1
$ sudo add-apt-repository ppa:webupd8team/java
$ sudo apt update
$ sudo apt install oracle-java8-installer git oracle-java8-set-default maven
$ . /etc/profile.d/jdk.sh
$ env |grep JAVA
$ cd && git clone https://github.com/apache/incubator-metron.git
$ cd incubator-metron && git rev-parse HEAD
23ff817ebdd3b63be69ad9b6c493b41bb0be4880
$ rm -rf ~/.m2/
$ sed -i "s#<id></id>#<id>archive</id>#g" metron-deployment/packaging/ambari/metron-mpack/src/main/assemblies/metron-mpack.xml #METRON-500
$ mvn clean package -DskipTests
$ cd ~/incubator-metron/metron-deployment
$ sed -i '/\[defaults\]/a remote_user = root' vagrant/full-dev-platform/ansible.cfg
$ sed -i 's/eth1/eth0/g' inventory/full-dev-platform/group_vars/all
$ cd ~/incubator-metron/metron-deployment/vagrant/full-dev-platform
$ ansible-playbook -i ../../inventory/full-dev-platform/hosts ../../playbooks/metron_full_install.yml --skip-tags="solr,yaf"
PLAY ***************************************************************************
skipping: no hosts matched
PLAY ***************************************************************************
skipping: no hosts matched
PLAY ***************************************************************************
TASK [setup] *******************************************************************
ok: [node1]
..............
TASK [metron_streaming : Assign permissions of HDFS /apps/metron/patterns directory] ***
changed: [node1]
TASK [metron_streaming : Upload Grok Patterns to hdfs:///apps/metron] **********
fatal: [node1]: FAILED! => {"changed": true, "cmd": ["hdfs", "dfs", "-put", "-f", "/usr/metron/0.2.1BETA/patterns", "/apps/metron"], "delta": "0:00:01.976636", "end": "2016-10-17 16:00:46.418570", "failed": true, "rc": 1, "start": "2016-10-17 16:00:44.441934", "stderr": "put: Error accessing file:/usr/metron/0.2.1BETA/patterns", "stdout": "", "stdout_lines": [], "warnings": []}
PLAY RECAP *********************************************************************
node1 : ok=145 changed=41 unreachable=0 failed=1
```
On the node1:
````
[root@node1 ~]# tail -n 3 /var/log/messages
Oct 17 16:00:40 node1 ansible-command: Invoked with warn=True executable=None chdir=None _raw_params=hdfs dfs -chown -R hdfs:hadoop /apps/metron/patterns removes=None creates=None _uses_shell=False
Oct 17 16:00:42 node1 ansible-command: Invoked with warn=True executable=None chdir=None _raw_params=hdfs dfs -chmod -R 775 /apps/metron/patterns removes=None creates=None _uses_shell=False
Oct 17 16:00:44 node1 ansible-command: Invoked with warn=True executable=None chdir=None _raw_params=hdfs dfs -put -f /usr/metron/0.2.1BETA/patterns /apps/metron removes=None creates=None _uses_shell=False
```
Let's try with the command as root:
```
[root@node1 ~]# hdfs dfs -put -f /usr/metron/0.2.1BETA/patterns /apps/metron
16/10/17 16:19:12 WARN retry.RetryInvocationHandler: Exception while invoking class org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.create over null. Not retrying because try once and fail.
org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.AccessControlException): Permission denied: user=root, access=WRITE, inode="/apps/metron/patterns/asa._COPYING_":hdfs:hadoop:drwxrwxr-x
at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.check(FSPermissionChecker.java:319)
at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.check(FSPermissionChecker.java:292)
at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkPermission(FSPermissionChecker.java:213)
.....
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:90)
at org.apache.hadoop.fs.FsShell.main(FsShell.java:350)
put: Permission denied: user=root, access=WRITE, inode="/apps/metron/patterns/asa._COPYING_":hdfs:hadoop:drwxrwxr-x
```
hdfs user can't access to /usr/metron/0.2.1BETA/patterns:
```
[root@node1 ~]# su hdfs
[hdfs@node1 ~]$ ls -lR /usr/metron/0.2.1BETA/patterns
ls: cannot open directory /usr/metron/0.2.1BETA/patterns: Permission denied
```
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron issue #220: Metron-342 Fix permissions of {{ metron_directo...
Posted by 2xyo <gi...@git.apache.org>.
Github user 2xyo commented on the issue:
https://github.com/apache/incubator-metron/pull/220
@mmiklavc ok, let me check that this week.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron issue #220: Metron-342 Fix permissions of {{ metron_directo...
Posted by dlyle65535 <gi...@git.apache.org>.
Github user dlyle65535 commented on the issue:
https://github.com/apache/incubator-metron/pull/220
Sorry it's been so long getting back to you.
I don't have any technical objection to this PR, but I'm a bit concerned about changing permissions of a share on the ansible host as a side-effect of installation. Does anyone else have any thoughts on this? I'm kind of -0 right now, but I may be being over-cautious.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron issue #220: Metron-342 Fix permissions of {{ metron_directo...
Posted by gspeter <gi...@git.apache.org>.
Github user gspeter commented on the issue:
https://github.com/apache/incubator-metron/pull/220
Hi @ottobackwards. What are you idea for problem?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron issue #220: Metron-342 Fix permissions of {{ metron_directo...
Posted by simonellistonball <gi...@git.apache.org>.
Github user simonellistonball commented on the issue:
https://github.com/apache/incubator-metron/pull/220
@gspeter This looks like an entirely unrelated error. The most likely issue here is that you are missing nodejs from your platform.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron issue #220: Metron-342 Fix permissions of {{ metron_directo...
Posted by 2xyo <gi...@git.apache.org>.
Github user 2xyo commented on the issue:
https://github.com/apache/incubator-metron/pull/220
Hi, sorry for not getting back to you. I am no longer able to try this setup (new job, no more server at work for this setup and not enough ram at home :/)
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron pull request #220: Metron-342 Fix permissions of {{ metron_...
Posted by 2xyo <gi...@git.apache.org>.
Github user 2xyo closed the pull request at:
https://github.com/apache/incubator-metron/pull/220
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron issue #220: Metron-342 Fix permissions of {{ metron_directo...
Posted by gspeter <gi...@git.apache.org>.
Github user gspeter commented on the issue:
https://github.com/apache/incubator-metron/pull/220
Hi everyone.,
I have configured according to your instructions. But I got the error below.
```
[INFO] metron-config ...................................... FAILURE [04:14 min]
[INFO] metron-rest-client ................................. SKIPPED
[INFO] metron-rest ........................................ SKIPPED
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 58:45 min
[INFO] Finished at: 2017-04-24T14:16:03+07:00
[INFO] Final Memory: 142M/577M
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal com.github.eirslett:frontend-maven-plugin:1.3:npm (npm install) on project metron-config: Failed to run task: 'npm install' failed. (error code 1) -> [Help 1]
[ERROR]
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR]
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoFailureException
[ERROR]
[ERROR] After correcting the problems, you can resume the build with the command
[ERROR] mvn <goals> -rf :metron-config
```
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron issue #220: Metron-342 Fix permissions of {{ metron_directo...
Posted by ottobackwards <gi...@git.apache.org>.
Github user ottobackwards commented on the issue:
https://github.com/apache/incubator-metron/pull/220
please open the new jira and attach this log to that. As @simonellistonball has requested, then mention me in the jira
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron issue #220: Metron-342 Fix permissions of {{ metron_directo...
Posted by 2xyo <gi...@git.apache.org>.
Github user 2xyo commented on the issue:
https://github.com/apache/incubator-metron/pull/220
here is my setup:
# Metron full dev platform (Windows - 21/08/2016)
Two Virtualbox (5.1.41) VM
- Client :
- Ubuntu (from ubuntu-16.04.1-desktop-amd64.iso)
- Disk : 40 GB
- RAM : 4 GB
- vCPU : 2
- Hostname : yoyo-vm
- Metron :
- Centos (from CentOS-6.7-x86_64-minimal.iso)
- Disk : 200 GB
- RAM : 16 GB
- vCPU : 4
- Hostname : node1
## Metron
Update/upgrade
```
# yum update && reboot
# yum install lsof vim
# ip addr # grab the ip
```
## Client
Update/upgrade
```
$ sudo apt update && sudo apt upgrade && sudo apt install vim && sudo reboot
```
Install basic dependencies for ansible
```
$ wget https://bootstrap.pypa.io/get-pip.py && sudo -H python get-pip.py
$ sudo apt install python-dev libssl-dev
$ sudo -H pip install --upgrade ansible
```
Setup connection:
```
$ ssh-keygen -t rsa -b 4096
$ echo "192.168.1.13 node1" | sudo tee -a /etc/hosts >/dev/null
```
Check connection:
```
$ ping node1
$ ssh root@node1
```
Upload ssh key and check that password is not needed anymore
```
$ ssh-copy-id root@node1
$ ssh root@node1
```
Setup dev environment:
```
$ sudo add-apt-repository ppa:webupd8team/java
$ sudo apt update
$ sudo apt install oracle-java8-installer git oracle-java8-set-default maven
$ . /etc/profile.d/jdk.sh
$ env |grep JAVA
$ cd && git clone https://github.com/apache/incubator-metron.git
$ cd incubator-metron && git rev-parse HEAD
225a490cf4a475e728bf9129ea3e662b543f5cc4
```
Build metron
```
$ mvn clean package -DskipTests
```
Setup Ansible connection:
```
$ cd ~/incubator-metron/metron-deployment
$ sed -i '/\[defaults\]/a remote_user = root' vagrant/full-dev-platform/ansible.cfg
$ sed -i 's/eth1/eth0/g' inventory/full-dev-platform/group_vars/all
$ cd ~/incubator-metron/metron-deployment/vagrant/full-dev-platform
$ ansible -i inventory/full-dev-platform/hosts all -m ping
```
Fix some stuff :
- Support of ansible 2.1 : https://github.com/apache/incubator-metron/pull/219
- Permissions https://github.com/apache/incubator-metron/pull/220
- Elasticsearch https://github.com/apache/incubator-metron/pull/221
Install everything:
```
$ cd ~/incubator-metron/metron-deployment/vagrant/full-dev-platform
$ ansible-playbook -i ../../inventory/full-dev-platform/hosts ../../playbooks/metron_full_install.yml
...
TASK [deployment-report : debug] ***********************************************
ok: [node1] => {
"Success": [
"Apache Metron deployed successfully",
" Metron @ http://node1:5000",
" Ambari @ http://node1:8080",
" Sensor Status @ http://node1:2812",
" Topology Status @ http://node1:2812",
" Zookeeper @ node1:2181",
" Kafka @ node1:6667",
"For additional information, see https://metron.incubator.apache.org/'"
]
}
PLAY RECAP *********************************************************************
node1 : ok=307 changed=147 unreachable=0 failed=0
```
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron issue #220: Metron-342 Fix permissions of {{ metron_directo...
Posted by ottobackwards <gi...@git.apache.org>.
Github user ottobackwards commented on the issue:
https://github.com/apache/incubator-metron/pull/220
When you open the separate issue, attach the entire log so that the error is there.
if you have the 'tee' command installed
you can do:
>mvn clean package -DskipTests | tee build.log
and attach build.log to that lira.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---