[jira] [Created] (ATLAS-3639) Enable to select BigQuery as a GraphStorageBackend instead of HBase

["Masaru Uchida (Jira)" <ji...@apache.org>]

Masaru Uchida created ATLAS-3639:
------------------------------------

             Summary: Enable to select BigQuery as a GraphStorageBackend instead of HBase
                 Key: ATLAS-3639
                 URL: https://issues.apache.org/jira/browse/ATLAS-3639
             Project: Atlas
          Issue Type: Wish
            Reporter: Masaru Uchida


Hi, I'm developing an ETL infrastructure and researching Atlas as part of a metadata management infrastructure replacement.

h2. Overview

I want to be able to use Bigquery for graph storage backend.

h2. Why
* Building hbase on your own is very painful.
* Introducing managed services lowers installation costs.

h2. What did and did not work

By replacing Hbase of {{HbaseStoreManager}} (atlas-janusgraph-hbase2) with {{BigtableConfiguration}} , operation such as startup and registration of Entity could be confirmed.

{code:java}
-        this.hconf = HBaseConfiguration.create();
+        String PROJECT_ID = "my-gcp-project-id";
+        String INSTANCE_ID = "my-bigtable-instance-id";
+        String APP_PROFILE_ID = "default";
+        org.apache.hadoop.conf.Configuration conf = BigtableConfiguration.configure(PROJECT_ID, INSTANCE_ID);
+        Credentials credentials = CredentialFactory.getApplicationDefaultCredential();
+
+        this.hconf = BigtableConfiguration.withCredentials(conf, credentials);
+        hconf.set(BigtableOptionsFactory.APP_PROFILE_ID_KEY, APP_PROFILE_ID);
{code}

https://github.com/usadamasa/atlas/blob/e2c2420aebe9dd6ebbd8ef7a41667a76ed621657/graphdb/janus-hbase2/src/main/java/org/janusgraph/diskstorage/hbase2/HBaseStoreManager.java#L295-L303

https://github.com/usadamasa/atlas/blob/e2c2420aebe9dd6ebbd8ef7a41667a76ed621657/graphdb/janus-hbase2/src/main/java/org/janusgraph/diskstorage/hbase2/HBaseCompat2_0.java#L48
(this branch from {{origin/branch-2.0}} )

However, {{Scan().SetReversed(true)}} calling in {{HBaseBasedAuditRepository}} fails in audit log search due to API not supported by Bigtable.

bq. Reverse scans are not supported. You cannot call the method org.apache.hadoop.hbase.client.Scan#setReversed(boolean reversed).

https://cloud.google.com/bigtable/docs/hbase-differences#gets_and_scans

h2. Proposal

* Enable to change the implementation of {{HBaseStoreManager}} if bigtable is specified in {{atlas.graph.storage.backend}} of atlas-application.properties.
* It would be nice if Solr or elasticsearch could be specified in {{atlas.EntityAuditRepository.impl}} .

Sorry in poor english. thank you.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)