You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@metron.apache.org by nickwallen <gi...@git.apache.org> on 2017/11/15 23:00:09 UTC

[GitHub] metron pull request #841: METRON-1316 Fastcapa Fails to Compile in Test Envi...

GitHub user nickwallen opened a pull request:

    https://github.com/apache/metron/pull/841

    METRON-1316 Fastcapa Fails to Compile in Test Environment

    
    DPDK 16.11.1 will no longer compile against newer Linux kernels due to an interface change in the kernel.  
    
    This can be replicated in the Fastcapa test environment.  We use the `bentos/centos-7.1` image for the test environment.  In this image, the underlying Linux kernel has been updated which makes it no longer work with DPDK 16.11.1.
    
    This change upgrades DPDK to 17.08 in the test environment and also updates Fastcapa to accomodate a function signature change in the newer version of DPDK.
    
    I also added the ability to run the same Fastcapa test code under multiple operating systems.  So now we can validate that Fastcapa runs in CentOS 7.1 and also CentOS 7.4, for example.  Additional platforms could easily be added.
    
    I also removed an old Ansible role that is no longer being used.
    
    ## Testing
    
    1. Run the following.
        ```
        cd metron-sensors/fastcapa/centos-7.1
        vagrant up
        ```
    
    2.  The deployment should complete with a message like so.
        ```
        Successfully received a Kafka message from fastcapa!
        ````
    
    
    ## Pull Request Checklist
    
    - [ ] Is there a JIRA ticket associated with this PR? If not one needs to be created at [Metron Jira](https://issues.apache.org/jira/browse/METRON/?selectedTab=com.atlassian.jira.jira-projects-plugin:summary-panel). 
    - [ ] Does your PR title start with METRON-XXXX where XXXX is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character.
    - [ ] Has your PR been rebased against the latest commit within the target branch (typically master)?
    - [ ] Have you included steps to reproduce the behavior or problem that is being changed or addressed?
    - [ ] Have you included steps or a guide to how the change may be verified and tested manually?
    - [ ] Have you ensured that the full suite of tests and checks have been executed in the root metron folder via:
    - [ ] Have you written or updated unit tests and or integration tests to verify your changes?
    - [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? 
    - [ ] Have you verified the basic functionality of the build by building and running locally with Vagrant full-dev environment or the equivalent?


You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/nickwallen/metron METRON-1316

Alternatively you can review and apply these changes as the patch at:

    https://github.com/apache/metron/pull/841.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #841
    
----
commit a00166187794a996b37c910e525ea7469b0241fb
Author: Nick Allen <ni...@nickallen.org>
Date:   2017-11-15T22:42:38Z

    METRON-1316

----


---

[GitHub] metron issue #841: METRON-1316 Fastcapa Fails to Compile in Test Environment

Posted by anandsubbu <gi...@git.apache.org>.
Github user anandsubbu commented on the issue:

    https://github.com/apache/metron/pull/841
  
    Hi @nickwallen , tested on the vagrant fastcapa deployment. Deployed fine, and I was able to see this message as well:
    ```
    TASK [debug] *******************************************************************
    ok: [source] => {
        "msg": "Successfully received packets sent from pcap-replay!"
    }
    ```
    During the deployment, I noticed the following message. I was not sure if this can be ignored or needs to be looked at.
    
    ```
    TASK [fastcapa : Restart for modified kernel params] ***************************
    fatal: [sink]: FAILED! => {"changed": false, "failed": true, "module_stderr": "", "module_stdout": "", "msg": "MODULE FAILURE", "parsed": false}
    ...ignoring
    ```



---

[GitHub] metron issue #841: METRON-1316 Fastcapa Fails to Compile in Test Environment

Posted by anandsubbu <gi...@git.apache.org>.
Github user anandsubbu commented on the issue:

    https://github.com/apache/metron/pull/841
  
    As always, very crisp @nickwallen ! My +1 (non-binding) still holds.


---

[GitHub] metron issue #841: METRON-1316 Fastcapa Fails to Compile in Test Environment

Posted by nickwallen <gi...@git.apache.org>.
Github user nickwallen commented on the issue:

    https://github.com/apache/metron/pull/841
  
    That's weird @anandsubbu .  What version of Vagrant and VirtualBox are you running?
    
    Here are the versions that I am running...
      * Virtualbox Version 5.1.22 r115126
      * Vagrant 1.9.4


---

[GitHub] metron issue #841: METRON-1316 Fastcapa Fails to Compile in Test Environment

Posted by anandsubbu <gi...@git.apache.org>.
Github user anandsubbu commented on the issue:

    https://github.com/apache/metron/pull/841
  
    +1 (non-binding), btw


---

[GitHub] metron issue #841: METRON-1316 Fastcapa Fails to Compile in Test Environment

Posted by nickwallen <gi...@git.apache.org>.
Github user nickwallen commented on the issue:

    https://github.com/apache/metron/pull/841
  
    @anandsubbu Yes, I think that is a good point.  I can add that to a README in the centos-7.4/ directory.  I'm sure someone will run across this problem.  
    
    Can you share what exact versions you were running when it did not work?  Also, the versions that you ran when it did work?  



---

[GitHub] metron issue #841: METRON-1316 Fastcapa Fails to Compile in Test Environment

Posted by anandsubbu <gi...@git.apache.org>.
Github user anandsubbu commented on the issue:

    https://github.com/apache/metron/pull/841
  
    @nickwallen - Yup, it makes sense to update the README for just the centos 7.4. I was able to replicate the timeout issue for centos 7.4 with VirtualBox ver 5.0.16 r105871. When I upgraded to VirtualBox 5.1.22 r115126, I was able to get it working. 
    



---

[GitHub] metron issue #841: METRON-1316 Fastcapa Fails to Compile in Test Environment

Posted by ottobackwards <gi...@git.apache.org>.
Github user ottobackwards commented on the issue:

    https://github.com/apache/metron/pull/841
  
    +1, ship it


---

[GitHub] metron issue #841: METRON-1316 Fastcapa Fails to Compile in Test Environment

Posted by anandsubbu <gi...@git.apache.org>.
Github user anandsubbu commented on the issue:

    https://github.com/apache/metron/pull/841
  
    Hm... you're right @nickwallen . It was indeed due to older vagrant and VirtualBox versions. I confirmed that after upgrading to the versions you noted, I was able to spin up both CentOS 7.1 and 7.4 fastcapa platforms without issues.
    
    As a side note, I was running the recommended versions of VirtualBox (5.0+) and vagrant (1.8+) which we note in our pre-requisites [here](https://github.com/nickwallen/metron/tree/master/metron-deployment/vagrant/full-dev-platform#prerequisites). Do you think it would be good idea to bump up the versions (as VirtualBox 5.1+ and vagrant 1.9+)  in our pre-requisites? Let me know and I can submit a pull request with this doc modification.


---

[GitHub] metron pull request #841: METRON-1316 Fastcapa Fails to Compile in Test Envi...

Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:

    https://github.com/apache/metron/pull/841


---

[GitHub] metron issue #841: METRON-1316 Fastcapa Fails to Compile in Test Environment

Posted by anandsubbu <gi...@git.apache.org>.
Github user anandsubbu commented on the issue:

    https://github.com/apache/metron/pull/841
  
    Hi @nickwallen 
    
    > I am not sure exactly what the problem is, but the same condition occurs in master. I would call this a pre-existing condition that we can handle with a separate PR.
    > 
    > Of course that is just my opinion and you or others may feel differently. Let me know what you think.
    > 
    
    Sure, I am fine by it, since I did not see any anomalies due to this error.
    
    Also, I was trying to test the same with CentOS 7.4 image and ran into some challenges. I am constantly facing a 'Timed out' issue when attempting 'vagrant up' on my local system. 
    
    ```
    ➜  centos-7.4 git:(master) ✗ vagrant up
    Bringing machine 'source' up with 'virtualbox' provider...
    Bringing machine 'sink' up with 'virtualbox' provider...
    ==> source: Box 'bento/centos-7.4' could not be found. Attempting to find and install...
        source: Box Provider: virtualbox
        source: Box Version: >= 0
    ==> source: Loading metadata for box 'bento/centos-7.4'
        source: URL: https://atlas.hashicorp.com/bento/centos-7.4
    ==> source: Adding box 'bento/centos-7.4' (v201710.25.0) for provider: virtualbox
        source: Downloading: https://vagrantcloud.com/bento/boxes/centos-7.4/versions/201710.25.0/providers/virtualbox.box
    ==> source: Box download is resuming from prior download progress
    ==> source: Successfully added box 'bento/centos-7.4' (v201710.25.0) for 'virtualbox'!
    ==> source: Importing base box 'bento/centos-7.4'...
    ==> source: Matching MAC address for NAT networking...
    ==> source: Checking if box 'bento/centos-7.4' is up to date...
    ==> source: Setting the name of the VM: centos-74_source_1511196258710_55009
    ==> source: Clearing any previously set network interfaces...
    ==> source: Preparing network interfaces based on configuration...
        source: Adapter 1: nat
        source: Adapter 2: hostonly
    ==> source: Forwarding ports...
        source: 22 (guest) => 2222 (host) (adapter 1)
    ==> source: Running 'pre-boot' VM customizations...
    ==> source: Booting VM...
    ==> source: Waiting for machine to boot. This may take a few minutes...
        source: SSH address: 127.0.0.1:2222
        source: SSH username: vagrant
        source: SSH auth method: private key
    Timed out while waiting for the machine to boot. This means that
    Vagrant was unable to communicate with the guest machine within
    the configured ("config.vm.boot_timeout" value) time period.
    If you look above, you should be able to see the error(s) that
    Vagrant had when attempting to connect to the machine. These errors
    are usually good hints as to what may be wrong.
    If you're using a custom box, make sure that networking is properly
    working and you're able to connect to the machine. It is a common
    problem that networking isn't setup properly in these boxes.
    Verify that authentication configurations are also setup properly,
    as well.
    If the box appears to be booting properly, you may want to increase
    the timeout ("config.vm.boot_timeout") value.
    ➜  centos-7.4 git:(master) ✗ vagrant status
    Current machine states:
    source                    running (virtualbox)
    sink                      not created (virtualbox)
    
    ```
    
    I did a fresh import of the 'bento/centos-7.4' box image multiple times after cleaning up (`vagrant box remove bento/centos-7.4` followed by fresh import), but I ran into the same timeout issue. Also tried bumping up `config.vm.boot_timeout` in `Vagrantfile` to 600, but to no avail.
    
    Further I am also not able to vagrant ssh into the 'source' VM despite it showing as running. 
    
    Does it look like I missed something in my configuration? 


---

[GitHub] metron issue #841: METRON-1316 Fastcapa Fails to Compile in Test Environment

Posted by nickwallen <gi...@git.apache.org>.
Github user nickwallen commented on the issue:

    https://github.com/apache/metron/pull/841
  
    > During the deployment, I noticed the following message. I was not sure if this can be ignored or needs to be looked at.
    
    Hi Anand - Thanks for testing!  I saw that a few times also.  During deployment it has to reboot the VM to pickup new kernel options.  Sometimes Ansible fails to wait long enough for the reboot.  If this error does happen, running `vagrant provision` it will continue the progress.  That is what I did during my testing.  
    
    I am not sure exactly what the problem is, but the same condition occurs in master.  I would call this a pre-existing condition that we can handle with a separate PR.  
    
    Of course that is just my opinion and you or others may feel differently.  Let me know what you think.


---