You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hive.apache.org by Denys Kuzmenko <dk...@apache.org> on 2023/02/01 09:30:57 UTC
Re: Odp: Re: hive 4.0.0-alpha2 external table locks
If you are using external tables only, you could try switching to DummyTxnManager. It supports explicit lock requests.
Re: Odp: Re: hive 4.0.0-alpha2 external table locks
Posted by Denys Kuzmenko <dk...@apache.org>.
DbTxnManager doesn't have support for explicit locking.
You could use HMSClient to issue an explicit lock request or create a lock record manually in the backend DB `HIVE_LOCKS` table.
Maybe you could temporarily restrict access to those tables via Ranger or drop them from the HMS while performing maintenance?
Re: Odp: Re: hive 4.0.0-alpha2 external table locks
Posted by Wojtek Meler <wm...@wp.pl>.
Any other options with DbTxnManager to prevent the access from hive queries to files on hdfs while performing maintenance of external tables ?
Re: Odp: Re: hive 4.0.0-alpha2 external table locks
Posted by Denys Kuzmenko <dk...@apache.org>.
Unfortunately no. DummyTxnManager doesn't support DB locks, only Zookeeper and in-memory. So you won't be able to transparently lock the external table from other non-DummyTxtManager's.
Re: Odp: Re: hive 4.0.0-alpha2 external table locks
Posted by Wojtek Meler <wm...@wp.pl>.
Would it force non-DummyTxtManager instances to not access external tables during exclusive locks? Dnia 1 lutego 2023 15:55 Denys Kuzmenko <dkuzmenko@apache.org> napisał(a): DbTxnManager comes with DbLockManager implementation that doesn't support explicit lock/unlock functionality. That's something we plan to address in a future. For now you can configure a dedicated HS2 instance with DummyTxnManager to have explicit locking functionality for ext tables.
Re: Odp: Re: hive 4.0.0-alpha2 external table locks
Posted by Denys Kuzmenko <dk...@apache.org>.
DbTxnManager comes with DbLockManager implementation that doesn't support explicit lock/unlock functionality.
It was designed to be used only for ACID tables, however, due to code refactor challenges is reused for external tables as well. This comes with few drawbacks like suboptimal behavior (opens a txn for an ext table) and reduced functionality.
That's something we plan to address in a future. For now you can configure a dedicated HS2 instance with DummyTxnManager to have explicit locking functionality for ext tables.
Re: Odp: Re: hive 4.0.0-alpha2 external table locks
Posted by Wojtek Meler <wm...@wp.pl>.
We are using external tables on early stages of ETL. Next we switch to managed tables as we don't need to worry about partition management. With DummyTxnManager we had problems dealing with managed tables. Regards, Wojtek Dnia 1 lutego 2023 10:31 Denys Kuzmenko <dkuzmenko@apache.org> napisał(a): If you are using external tables only, you could try switching to DummyTxnManager. It supports explicit lock requests.