You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@impala.apache.org by mi...@apache.org on 2023/04/17 22:06:10 UTC

[impala] branch master updated (f0289f3cb -> df2e03ce0)

This is an automated email from the ASF dual-hosted git repository.

michaelsmith pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/impala.git


    from f0289f3cb IMPALA-11273: Remove APIs deprecated in Java 11
     new b8915c9a4 IMPALA-11878: Document that AVRO file format is supported in Iceberg tables
     new bb237660a IMPALA-12047: [DOCS] Fix a URL mistake when using "git clone" in docs/README.md
     new df2e03ce0 IMPALA-11974: Fix xrange, split in collect_diagnostics.py

The 3 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 bin/diagnostics/collect_diagnostics.py |  7 ++++---
 docs/README.md                         | 23 ++++++++++++++++++-----
 docs/topics/impala_iceberg.xml         |  9 ++++++---
 3 files changed, 28 insertions(+), 11 deletions(-)


[impala] 02/03: IMPALA-12047: [DOCS] Fix a URL mistake when using "git clone" in docs/README.md

Posted by mi...@apache.org.
This is an automated email from the ASF dual-hosted git repository.

michaelsmith pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/impala.git

commit bb237660a0cca7a2315ac419058cd8b7cc84073c
Author: whsyourdaddy <li...@foxmail.com>
AuthorDate: Mon Mar 20 22:34:39 2023 +0800

    IMPALA-12047: [DOCS] Fix a URL mistake when using "git clone" in docs/README.md
    
    There's a URL mistake when using "git clone" in docs/README.md:
      git clone https://gitbox.apache.org/repos/asf/impala.git/docs
    
    This doesn't work and it will prompt "repository not found".
    
    This change corrects the description, providing two ways to download
    the docs - either by downloading the whole repository and going to the
    docs/ directory or by downloading only the docs using git sparse-checkout.
    
    Change-Id: Ib00c37e28e67cca5b3630742b4c366dea4e967b7
    Reviewed-on: http://gerrit.cloudera.org:8080/19634
    Tested-by: Impala Public Jenkins <im...@cloudera.com>
    Reviewed-by: Daniel Becker <da...@cloudera.com>
    Reviewed-by: Yingchun Lai <la...@apache.org>
---
 docs/README.md | 23 ++++++++++++++++++-----
 1 file changed, 18 insertions(+), 5 deletions(-)

diff --git a/docs/README.md b/docs/README.md
index 02d1f5dc2..4365dd330 100644
--- a/docs/README.md
+++ b/docs/README.md
@@ -9,16 +9,29 @@ instructions.
 
 ## Download Docs Source
 
-* Open a terminal window and run the following commands to get the Impala
-  documentation source files from Git:
+* There are two ways to obtain docs sources.
+  * Clone the whole repository. Open a terminal window and run the following
+    commands to get the whole Impala repository from Git and go to the docs folder:
     ```
-    git clone https://gitbox.apache.org/repos/asf/impala.git/docs
-    cd <local_directory>
-    git checkout master
+      git clone https://gitbox.apache.org/repos/asf/impala.git
+      cd <local_directory>
+      git checkout master
+      cd docs/
     ```
 
     Where `master` is the branch where Impala documentation source files
     are uploaded.
+  * Clone only the docs directory. Open a terminal window and run the following
+    commands to get only the Impala documentation source files from Git:
+    ```
+      git init impala_docs
+      cd impala_docs
+      git remote add origin https://gitbox.apache.org/repos/asf/impala.git
+      git sparse-checkout set docs/
+      git pull origin master
+      cd docs/
+    ```
+    You'll see only the 'docs/' sub-directory is downloaded.
 
 ## Download DITA Open Toolkit
 


[impala] 01/03: IMPALA-11878: Document that AVRO file format is supported in Iceberg tables

Posted by mi...@apache.org.
This is an automated email from the ASF dual-hosted git repository.

michaelsmith pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/impala.git

commit b8915c9a4d77484b32d2fcc0166a78ddde22a59e
Author: Noemi Pap-Takacs <np...@cloudera.com>
AuthorDate: Wed Mar 1 13:48:34 2023 +0100

    IMPALA-11878: Document that AVRO file format is supported in Iceberg tables
    
    Change-Id: I0e3fdade3a12570e488a3a58819697134dbb4c15
    Reviewed-on: http://gerrit.cloudera.org:8080/19680
    Tested-by: Impala Public Jenkins <im...@cloudera.com>
    Reviewed-by: Tamas Mate <tm...@apache.org>
---
 docs/topics/impala_iceberg.xml | 9 ++++++---
 1 file changed, 6 insertions(+), 3 deletions(-)

diff --git a/docs/topics/impala_iceberg.xml b/docs/topics/impala_iceberg.xml
index df66d5a0b..871b1ba56 100644
--- a/docs/topics/impala_iceberg.xml
+++ b/docs/topics/impala_iceberg.xml
@@ -196,8 +196,8 @@ CREATE TABLE ice_t (i INT) STORED AS ICEBERG;
         </codeblock>
       </p>
       <p>
-        By default Impala assumes that the Iceberg table uses Parquet data files. ORC is also supported,
-        but we need to tell Impala via setting the table property 'write.format.default' to 'ORC'.
+        By default Impala assumes that the Iceberg table uses Parquet data files. ORC and AVRO are also supported,
+        but we need to tell Impala via setting the table property 'write.format.default' to e.g. 'ORC'.
       </p>
       <p>
         You can also use <codeph>CREATE TABLE AS SELECT</codeph> to create new Iceberg tables, e.g.:
@@ -354,6 +354,9 @@ CREATE TABLE ice_ctas_part_spec PARTITIONED BY SPEC (truncate(3, s)) STORED AS I
           <li>decimal(P, S) to decimal(P', S) if P' > P – widen the precision of decimal types.</li>
         </ul>
       </p>
+      <p>
+        Impala currently does not support schema evolution for tables with AVRO file format.
+      </p>
       <p>
         See
         <xref href="https://iceberg.apache.org/docs/latest/evolution/#schema-evolution" scope="external" format="html">
@@ -614,7 +617,7 @@ ALTER TABLE ice_tbl EXECUTE expire_snapshots(now() - interval 5 days);
           </li>
           <li><codeph>iceberg.catalog_location</codeph>: Iceberg table catalog location when <codeph>iceberg.catalog</codeph> is <codeph>'hadoop.catalog'</codeph></li>
           <li><codeph>iceberg.table_identifier</codeph>: Iceberg table identifier. We use &lt;database&gt;.&lt;table&gt; instead if this property is not set</li>
-          <li><codeph>write.format.default</codeph>: data file format of the table. Impala can read ORC and PARQUET data files in Iceberg tables, and can write PARQUET data files only.</li>
+          <li><codeph>write.format.default</codeph>: data file format of the table. Impala can read AVRO, ORC and PARQUET data files in Iceberg tables, and can write PARQUET data files only.</li>
           <li><codeph>write.parquet.compression-codec</codeph>:
             Parquet compression codec. Supported values are: NONE, GZIP, SNAPPY
             (default value), LZ4, ZSTD. The table property will be ignored if


[impala] 03/03: IMPALA-11974: Fix xrange, split in collect_diagnostics.py

Posted by mi...@apache.org.
This is an automated email from the ASF dual-hosted git repository.

michaelsmith pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/impala.git

commit df2e03ce0bc54cfd142c39a4e48f8b555248c969
Author: Michael Smith <mi...@cloudera.com>
AuthorDate: Fri Apr 14 09:49:33 2023 -0700

    IMPALA-11974: Fix xrange, split in collect_diagnostics.py
    
    Python3 deprecates xrange operator, this commit replaces it with the
    range operator similar to earlier replacements in IMPALA-11974.
    
    Adds universal_newlines to Popen so that we return text in Python 3.
    
    Testing:
    - Ran '$IMPALA_HOME/bin/diagnostics/collect_diagnostics.py --pid <pid>
      --minidumps 2 1 --minidumps_dir $IMPALA_HOME/logs/cluster/minidumps
      --stacks 2 1' with Python 2/3 and inspected the results.
    
    Change-Id: I52f075825d47613293b106a7c50d4499c19cd3f4
    Reviewed-on: http://gerrit.cloudera.org:8080/19746
    Reviewed-by: Impala Public Jenkins <im...@cloudera.com>
    Tested-by: Impala Public Jenkins <im...@cloudera.com>
---
 bin/diagnostics/collect_diagnostics.py | 7 ++++---
 1 file changed, 4 insertions(+), 3 deletions(-)

diff --git a/bin/diagnostics/collect_diagnostics.py b/bin/diagnostics/collect_diagnostics.py
index f4b3298c4..a305a0d0a 100755
--- a/bin/diagnostics/collect_diagnostics.py
+++ b/bin/diagnostics/collect_diagnostics.py
@@ -87,7 +87,8 @@ class Command(object):
     cmd_string = " ".join(self.cmd)
     logging.info("Starting command %s with a timeout of %s"
         % (cmd_string, str(self.timeout)))
-    self.child = subprocess.Popen(self.cmd, stdin=cmd_stdin, stdout=cmd_stdout)
+    self.child = subprocess.Popen(self.cmd, stdin=cmd_stdin, stdout=cmd_stdout,
+        universal_newlines=True)
     timer = Timer(self.timeout, self.kill_child)
     try:
       timer.start()
@@ -312,7 +313,7 @@ class ImpalaDiagnosticsHandler(object):
       cmds_to_run.append((Command(cmd_args, self.args.timeout), "pstack", False))
 
     collection_start_ts = time.time()
-    for i in xrange(stacks_count):
+    for i in range(stacks_count):
       for cmd, file_prefix, is_minidump in cmds_to_run:
         if file_prefix:
           stdout_file = os.path.join(stacks_dir, file_prefix + "-" + str(i) + ".txt")
@@ -341,7 +342,7 @@ class ImpalaDiagnosticsHandler(object):
     cmd_args = ["kill", "-SIGUSR1", str(self.args.pid)]
     cmd = Command(cmd_args, self.args.timeout)
     collection_start_ts = time.time()
-    for i in xrange(minidump_count):
+    for i in range(minidump_count):
       cmd.run()
       self.wait_for_minidump()
       time.sleep(minidump_interval_secs)