[incubator-hudi] branch asf-site updated: Add NOTE about Hive Incremental pull queries executed by Fetch operation (#742)

[vi...@apache.org]

This is an automated email from the ASF dual-hosted git repository.

vinoth pushed a commit to branch asf-site
in repository https://gitbox.apache.org/repos/asf/incubator-hudi.git


The following commit(s) were added to refs/heads/asf-site by this push:
     new 35ae4dc  Add NOTE about Hive Incremental pull queries executed by Fetch operation (#742)
35ae4dc is described below

commit 35ae4dc4e252f3e7babdf1e83a123d72287f89db
Author: Bhavani Sudha Saktheeswaran <bh...@uber.com>
AuthorDate: Tue Jun 25 12:57:00 2019 -0700

    Add NOTE about Hive Incremental pull queries executed by Fetch operation (#742)
---
 docs/querying_data.md | 7 +++++++
 1 file changed, 7 insertions(+)

diff --git a/docs/querying_data.md b/docs/querying_data.md
index 81186e3..f96b328 100644
--- a/docs/querying_data.md
+++ b/docs/querying_data.md
@@ -71,6 +71,13 @@ then the utility can determine if the target dataset has no commits or is behind
 it will automatically use the backfill configuration, since applying the last 24 hours incrementally could take more time than doing a backfill. The current limitation of the tool
 is the lack of support for self-joining the same table in mixed mode (normal and incremental modes).
 
+**NOTE on Hive queries that are executed using Fetch task:**
+Since Fetch tasks invoke InputFormat.listStatus() per partition, Hoodie metadata can be listed in
+every such listStatus() call. In order to avoid this, it might be useful to disable fetch tasks
+using the hive session property for incremental queries: `set hive.fetch.task.conversion=none;` This
+would ensure Map Reduce execution is chosen for a Hive query, which combines partitions (comma
+separated) and calls InputFormat.listStatus() only once with all those partitions.
+
 ## Spark
 
 Spark provides much easier deployment & management of Hudi jars and bundles into jobs/notebooks. At a high level, there are two ways to access Hudi datasets in Spark.