[jira] [Created] (RANGER-4792) Fix issue with creating index and import data in ElasticSearch as Audit database

["ognjenit (Jira)" <ji...@apache.org>]

ognjenit created RANGER-4792:
--------------------------------

             Summary: Fix issue with creating index and import data in ElasticSearch as Audit database
                 Key: RANGER-4792
                 URL: https://issues.apache.org/jira/browse/RANGER-4792
             Project: Ranger
          Issue Type: Bug
          Components: admin, audit
    Affects Versions: 2.4.0
         Environment: Container:
- Linux: Debian buster
- Java: openjdk- 11
- Tested on kubernetes and openshift on AWS/Azure and on-prem
            Reporter: ognjenit


Hi all,

I apologize in advance if I haven't adjusted this issue properly.

Short description:

I have deployed Trino with ranger-trino-plugin and I wanted to use ElasticSearch (7.10.2) as a place where I want to store the audit. When I configured ranger-admin to use elasticsearch (audit_store=elasticsearch and all other parameters audit_elasticsearch_*) I started getting errors in the catalina.out: java.lang.NoSuchFieldError: LUCENE_8_5_1. As I increased the version of Lucena, it was written in the logs that an even higher version was needed. So in the end, I moved it to 8.11.3 and 8.4.0 for lucene-spatial since it is the latest. 

After it was fixed, I tried to use https for elasticsearch protocol (audit_elasticsearch_protocol) however, it always showed that ranger-admin use http instead of https. I show in code that audit_elasticsearch_protocol is not configured well.

As soon as it done, ranger admin successfully created ES index. However, I need to move from MiscUtil.toDate to MiscUtil.toLocalDate for evtTime "column" since I was getting error: Error converting value to date. Value = 2024-05-13T13:08:47.905Z

As soon as I fixed it, I found an error in Trino that the plugin couldn't insert data into elasticsearch. After I upgraded httpcomponents bug-fix version, it's started inserting data.

I opened PR with the fix 2.4.0 version, do I need to do the same on the master branch?

PR: https://github.com/apache/ranger/pull/314/files
h4. 1. Lucene version - fixed problem with writing data to ElasticSearch

{*}Error{*}: java.lang.NoSuchFieldError: LUCENE_8_5_1

I tried to change minor version one by one, but only latest version fit for me.

Changes:
 * agents-audit/pom.xml: 311
 * pom.xml: 241

h4. 2. Elastic search protocol - fixed problem with changing protocol

Even though I changed ranger.audit.elasticsearch.protocol from http to https, audit plugin still using http protocol.

Changes:
 * security-admin/scripts/ranger-admin-site-template.xml: 167-170
 * security-admin/scripts/setup.sh: 79, 794-797
 * security-admin/scripts/upgrade_admin.py: 116
 * security-admin/src/main/resources/conf.dist/ranger-admin-site.xml: 53-57
 * security-admin/src/test/java/org/apache/ranger/elasticsearch/ElasticSearchAccessAuditsServiceTest.java: 56

h4. 3. Audit plugin - cannot write audit to ES

{*}Error{*}: bootstrap method initialization exception

After changing the version of httpcomponents I started seeing audit

Changes:
 * pom.xml: 137, 138, 140

h4. 4. Ranger admin console - Audit show 1-1-1970

{*}Erro{*}: Error converting value to date. Value = 2024-05-13T13:08:47.905Z

Even though evtTime was ok in ElasticSearch, ranger couldn't show it on GUI.

Changes:
 * security-admin/src/main/java/org/apache/ranger/elasticsearch/ElasticSearchAccessAuditsService.java: 260
 * security-admin/src/main/java/org/apache/ranger/solr/SolrAccessAuditsService.java: 239



--
This message was sent by Atlassian Jira
(v8.20.10#820010)