[jira] [Commented] (KNOX-1091) Knox Audit Logging - duplicate correlation ids

["Kevin Risden (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/KNOX-1091?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16364271#comment-16364271 ] 

Kevin Risden commented on KNOX-1091:
------------------------------------

I finally got some time to try to track this down further. Reproducing with standard Knox release without LDAP was possible for me. This is with Knox 0.9.1 but will try with latest Knox release as well to see if this still occurs.

*Setup*
 * Java version

{code:java}
java -version
java version "1.8.0_151"
Java(TM) SE Runtime Environment (build 1.8.0_151-b12)
Java HotSpot(TM) 64-Bit Server VM (build 25.151-b12, mixed mode){code}
 * OS information

{code:java}
cat /etc/os-release
NAME="CentOS Linux"
VERSION="7 (Core)"
ID="centos"
ID_LIKE="rhel fedora"
VERSION_ID="7"
PRETTY_NAME="CentOS Linux 7 (Core)"
ANSI_COLOR="0;31"
CPE_NAME="cpe:/o:centos:centos:7"
HOME_URL="https://www.centos.org/"
BUG_REPORT_URL="https://bugs.centos.org/"

CENTOS_MANTISBT_PROJECT="CentOS-7"
CENTOS_MANTISBT_PROJECT_VERSION="7"
REDHAT_SUPPORT_PRODUCT="centos"
REDHAT_SUPPORT_PRODUCT_VERSION="7"
{code}
 * Download Knox 0.9.1 (this is the closest to HDP Knox version we are on)
 * Extract Knox (unzip knox-0.9.1.zip)
 * Cd to Knox directory (cd knox-0.9.1)
 * Create master secret (./bin/knoxcli.sh create-master)
 * Create conf/topologies/health.xml with following content (simple health only topology with no LDAP)

{code:xml}
<topology>
    <gateway>
        <provider>
            <role>authentication</role>
            <name>Anonymous</name>
            <enabled>true</enabled>
        </provider>
        <provider>
            <role>identity-assertion</role>
            <name>Default</name>
            <enabled>false</enabled>
        </provider>
    </gateway>
    <service>
        <role>KNOX</role>
    </service>
</topology>
{code}
 * Disable SSL for Knox (add the following to conf/gateway-site.xml)

{code:xml}
<property>
   <name>ssl.enabled</name>
   <value>false</value>
</property>
{code}
 * Start Knox (./bin/gateway.sh start)
 * Wait for Knox to come up ([http://localhost:8443/gateway/health/api/v1/version])
 * Use Apache Bench to test with repeated load

*Test 1 concurrent connection has proper audit*
 * Truncate gateway-audit.log (>logs/gateway-audit.log)
 * Run test: ab -n 1000 -c 1 [http://localhost:8443/gateway/health/api/v1/version]
 * Check unique UUID: grep -F access logs/gateway-audit.log | cut -d'|' -f3 | sort | uniq -c | sort -n | wc -l
 * Result should be 1000
 * grep -F access logs/gateway-audit.log | cut -d'|' -f3 | sort | uniq -c | sort -n | tail -n5
 * Count for each UUID should be 2 (not sure why 2 but that is consistent)

*Test 100 concurrent connections*
 * Truncate gateway-audit.log (>logs/gateway-audit.log)
 * Run test: ab -n 1000 -c 100 [http://localhost:8443/gateway/health/api/v1/version]
 * Check unique UUID: grep -F access logs/gateway-audit.log | cut -d'|' -f3 | sort | uniq -c | sort -n | wc -l
 * Result *will not* be 1000. It will be <1000.
 * grep -F access logs/gateway-audit.log | cut -d'|' -f3 | sort | uniq -c | sort -n | tail -n5
 * Count for each UUID *will not* be 2

> Knox Audit Logging - duplicate correlation ids
> ----------------------------------------------
>
>                 Key: KNOX-1091
>                 URL: https://issues.apache.org/jira/browse/KNOX-1091
>             Project: Apache Knox
>          Issue Type: Bug
>          Components: Server
>            Reporter: Kevin Risden
>            Priority: Major
>
> From the Knox User list thread: "Multiple topology audit logging", it came to my attention that Knox seems to be logging duplicate correlation ids. Separating out the topic specifically here to dig a bit deeper.
> While looking at our Knox audit logs (Knox 0.9 on HDP 2.5) the "correlation id" doesn't seem to be unique across requests. Is this to be expected? Here is a snippet (anonymized):
> grep 7557c91b-2a48-4e09-aefc-44e9892372da /var/knox/gateway-audit.log
>  {code}
> 17/10/10 12:50:09 ||7557c91b-2a48-4e09-aefc-44e9892372da|audit|WEBHBASE||||access|uri|/gateway/HADOOPTEST/hbase/hbase/NAMESPACE1:TABLE1/ID1//|unavailable|Request method: GET
> 17/10/10 12:50:09 ||7557c91b-2a48-4e09-aefc-44e9892372da|audit|WEBHBASE|USER1|||authentication|uri|/gateway/HADOOPPROD/hbase/NAMESPACE2:TABLE2/multiget?row=ID2%2fd%3araw&|success|
> 17/10/10 12:50:09 ||7557c91b-2a48-4e09-aefc-44e9892372da|audit|WEBHBASE|USER1|||authentication|uri|/gateway/HADOOPPROD/hbase/NAMESPACE2:TABLE2/multiget?row=ID2%2fd%3araw&|success|Groups: []
> 17/10/10 12:50:09 ||7557c91b-2a48-4e09-aefc-44e9892372da|audit|WEBHBASE|USER1|||dispatch|uri|http://WEBHBASE.example.com:8084/NAMESPACE2:TABLE2/multiget?doAs=USER1&row=ID2%2Fd%3Araw|unavailable|Request method: GET
> 17/10/10 12:50:09 ||7557c91b-2a48-4e09-aefc-44e9892372da|audit|WEBHBASE|USER1|||dispatch|uri|http://WEBHBASE.example.com:8084/NAMESPACE2:TABLE2/multiget?doAs=USER1&row=ID2%2Fd%3Araw|success|Response status: 200
> 17/10/10 12:50:09 ||7557c91b-2a48-4e09-aefc-44e9892372da|audit|WEBHBASE|USER1|||access|uri|/gateway/HADOOPPROD/hbase/NAMESPACE2:TABLE2/multiget?row=ID2%2fd%3araw&|success|Response status: 200
> 17/10/10 12:50:09 ||7557c91b-2a48-4e09-aefc-44e9892372da|audit|WEBHBASE||||authentication|principal|USER2|failure|LDAP authentication failed.
> 17/10/10 12:50:09 ||7557c91b-2a48-4e09-aefc-44e9892372da|audit|WEBHBASE||||access|uri|/gateway/HADOOPTEST/hbase/hbase/NAMESPACE1:TABLE2/ID1//|success|Response status: 401
> {code}
> The things to highlight here for the same correlation id:
> * different topologies are being used
> * different uris are being used
> * different users are being used
> Some of the things that we have configured that could impact results:
> * authentication caching
> * multiple Knox servers
> * load balancer in front of Knox



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)