[jira] [Updated] (ATLAS-3639) Enable to select BigQuery as a GraphStorageBackend instead of HBase

["Masaru Uchida (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/ATLAS-3639?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Masaru Uchida updated ATLAS-3639:
---------------------------------
    Issue Type: New Feature  (was: Wish)

> Enable to select BigQuery as a GraphStorageBackend instead of HBase
> -------------------------------------------------------------------
>
>                 Key: ATLAS-3639
>                 URL: https://issues.apache.org/jira/browse/ATLAS-3639
>             Project: Atlas
>          Issue Type: New Feature
>            Reporter: Masaru Uchida
>            Priority: Major
>              Labels: GCP
>
> Hi, I'm developing an ETL infrastructure and researching Atlas as part of a metadata management infrastructure replacement.
> h2. Overview
> I want to be able to use Bigquery for graph storage backend.
> h2. Why
> * Building hbase on your own is very painful.
> * Introducing managed services lowers installation costs.
> h2. What did and did not work
> By replacing Hbase of {{HbaseStoreManager}} (atlas-janusgraph-hbase2) with {{BigtableConfiguration}} , operation such as startup and registration of Entity could be confirmed.
> {code:java}
> -        this.hconf = HBaseConfiguration.create();
> +        String PROJECT_ID = "my-gcp-project-id";
> +        String INSTANCE_ID = "my-bigtable-instance-id";
> +        String APP_PROFILE_ID = "default";
> +        org.apache.hadoop.conf.Configuration conf = BigtableConfiguration.configure(PROJECT_ID, INSTANCE_ID);
> +        Credentials credentials = CredentialFactory.getApplicationDefaultCredential();
> +
> +        this.hconf = BigtableConfiguration.withCredentials(conf, credentials);
> +        hconf.set(BigtableOptionsFactory.APP_PROFILE_ID_KEY, APP_PROFILE_ID);
> {code}
> https://github.com/usadamasa/atlas/blob/e2c2420aebe9dd6ebbd8ef7a41667a76ed621657/graphdb/janus-hbase2/src/main/java/org/janusgraph/diskstorage/hbase2/HBaseStoreManager.java#L295-L303
> https://github.com/usadamasa/atlas/blob/e2c2420aebe9dd6ebbd8ef7a41667a76ed621657/graphdb/janus-hbase2/src/main/java/org/janusgraph/diskstorage/hbase2/HBaseCompat2_0.java#L48
> (this branch from {{origin/branch-2.0}} )
> However, {{Scan().SetReversed(true)}} calling in {{HBaseBasedAuditRepository}} fails in audit log search due to API not supported by Bigtable.
> bq. Reverse scans are not supported. You cannot call the method org.apache.hadoop.hbase.client.Scan#setReversed(boolean reversed).
> https://cloud.google.com/bigtable/docs/hbase-differences#gets_and_scans
> h2. Proposal
> * Enable to change the implementation of {{HBaseStoreManager}} if bigtable is specified in {{atlas.graph.storage.backend}} of atlas-application.properties.
> * It would be nice if Solr or elasticsearch could be specified in {{atlas.EntityAuditRepository.impl}} .
> Sorry in poor english. thank you.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)