[jira] [Created] (METRON-1535) IndexDao patch function is not atomic

["Ryan Merriman (JIRA)" <ji...@apache.org>]

Ryan Merriman created METRON-1535:
-------------------------------------

             Summary: IndexDao patch function is not atomic
                 Key: METRON-1535
                 URL: https://issues.apache.org/jira/browse/METRON-1535
             Project: Metron
          Issue Type: Bug
            Reporter: Ryan Merriman


Our Index and Elasticsearch DAO patch operations are not atomic.  Currently a [patch|https://github.com/apache/metron/blob/master/metron-platform/metron-indexing/src/main/java/org/apache/metron/indexing/dao/IndexDao.java#L125] involves looking up the current version of a document, patching it, then reindexing the whole document in separate steps.  This could lead to a race condition that causes incorrect or corrupt data.

This pattern also exists in a couple different places in the ElasticsearchMetaAlertDao including [here|https://github.com/apache/metron/blob/master/metron-platform/metron-indexing/src/main/java/org/apache/metron/indexing/dao/IndexDao.java#L125].

Elasticsearch offers an [Update API|https://www.elastic.co/guide/en/elasticsearch/reference/5.6/docs-update.html#docs-update] that may be more appropriate in these cases.  I don't know if it works on nested documents so that would need to be tested.

Looking up the whole doc as part of an update is also causing this [bug|https://github.com/apache/metron/pull/995] in the SolrMetaAlertDao feature because Solr returns extra fields that can't be directly reindexed.  There is also an [Atomic Update|https://lucene.apache.org/solr/guide/6_6/updating-parts-of-documents.html#UpdatingPartsofDocuments-AtomicUpdates] in Solr that is similar to the Elasticsearch Update API.  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)