You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@drill.apache.org by dz...@apache.org on 2022/12/25 04:33:06 UTC
[drill] branch 1.20 updated (af2f48ab32 -> 85aa70c2b3)
This is an automated email from the ASF dual-hosted git repository.
dzamo pushed a change to branch 1.20
in repository https://gitbox.apache.org/repos/asf/drill.git
from af2f48ab32 [MINOR-UPDATE] Bump commons-net from 3.6 to 3.9.0 (#2720)
new 85c353c542 DRILL-8326: Bump SnakeYaml to 1.33 (#2666)
new 5851f86579 DRILL-8329: Close HTTP Caching Resources (#2669)
new 809a7880e5 DRILL-8332: upgrade to jackson 2.13.4.20221012 (#2674)
new 0e3b144d72 [MINOR UPDATE]: Upgrade org.apache.kafka:kafka_2.13 to 2.8.2 (#2675)
new 2d7b49f66d [MINOR UPDATE]: Upgrade org.freemarker:freemarker to 2.3.30 (#2676)
new 8e2eacc873 DRILL-8332: upgrade jackson due to issue in gradle module of previous release (#2677)
new 2b88eafc67 [MINOR UPDATE]: Update com.drewnoakes:metadata-extractor 2.17.0 to 2.18.0 (#2682)
new 85aa70c2b3 DRILL-8333: Resource leak when JsonLoader is built from a stream (#2678)
The 8 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
Summary of changes:
contrib/format-image/pom.xml | 2 +-
.../drill/exec/store/http/HttpBatchReader.java | 3 ++
.../drill/exec/store/http/HttpCSVBatchReader.java | 2 +
.../drill/exec/store/http/HttpXMLBatchReader.java | 3 ++
.../drill/exec/store/http/util/SimpleHttp.java | 15 +++++++-
contrib/storage-kafka/pom.xml | 2 +-
.../store/kafka/decoders/JsonMessageReader.java | 25 +-----------
.../easy/json/loader/ClosingStreamIterator.java} | 44 +++++++++++++++-------
.../store/easy/json/loader/JsonLoaderImpl.java | 10 +++++
pom.xml | 6 +--
10 files changed, 70 insertions(+), 42 deletions(-)
copy exec/{vector/src/main/java/org/apache/drill/exec/vector/complex/StateTool.java => java-exec/src/main/java/org/apache/drill/exec/store/easy/json/loader/ClosingStreamIterator.java} (52%)
[drill] 03/08: DRILL-8332: upgrade to jackson 2.13.4.20221012 (#2674)
Posted by dz...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
dzamo pushed a commit to branch 1.20
in repository https://gitbox.apache.org/repos/asf/drill.git
commit 809a7880e5ff1744292e34ef9f4dc004fdea9591
Author: PJ Fanning <pj...@users.noreply.github.com>
AuthorDate: Thu Oct 13 15:01:01 2022 +0100
DRILL-8332: upgrade to jackson 2.13.4.20221012 (#2674)
---
pom.xml | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/pom.xml b/pom.xml
index cebbf0d671..45dac77528 100644
--- a/pom.xml
+++ b/pom.xml
@@ -64,7 +64,7 @@
<avatica.version>1.17.0</avatica.version>
<janino.version>3.0.11</janino.version>
<sqlline.version>1.12.0</sqlline.version>
- <jackson.version>2.13.2.20220328</jackson.version>
+ <jackson.version>2.13.4.20221012</jackson.version>
<zookeeper.version>3.5.7</zookeeper.version>
<mapr.release.version>6.1.0-mapr</mapr.release.version>
<ojai.version>3.0-mapr-1808</ojai.version>
[drill] 02/08: DRILL-8329: Close HTTP Caching Resources (#2669)
Posted by dz...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
dzamo pushed a commit to branch 1.20
in repository https://gitbox.apache.org/repos/asf/drill.git
commit 5851f86579a2f34464b9562cae43abeea5abc5e9
Author: Charles S. Givre <cg...@apache.org>
AuthorDate: Tue Oct 4 20:50:51 2022 +0800
DRILL-8329: Close HTTP Caching Resources (#2669)
---
.../org/apache/drill/exec/store/http/HttpBatchReader.java | 3 +++
.../apache/drill/exec/store/http/HttpCSVBatchReader.java | 2 ++
.../apache/drill/exec/store/http/HttpXMLBatchReader.java | 3 +++
.../org/apache/drill/exec/store/http/util/SimpleHttp.java | 15 ++++++++++++++-
4 files changed, 22 insertions(+), 1 deletion(-)
diff --git a/contrib/storage-http/src/main/java/org/apache/drill/exec/store/http/HttpBatchReader.java b/contrib/storage-http/src/main/java/org/apache/drill/exec/store/http/HttpBatchReader.java
index 697b569f10..a2ce3ad8ae 100644
--- a/contrib/storage-http/src/main/java/org/apache/drill/exec/store/http/HttpBatchReader.java
+++ b/contrib/storage-http/src/main/java/org/apache/drill/exec/store/http/HttpBatchReader.java
@@ -146,9 +146,12 @@ public class HttpBatchReader implements ManagedReader<SchemaNegotiator> {
// Paranoia: ensure stream is closed if anything goes wrong.
// After this, the JSON loader will close the stream.
AutoCloseables.closeSilently(inStream);
+ AutoCloseables.closeSilently(http);
throw t;
}
+ // Close the http client
+ http.close();
return true;
}
diff --git a/contrib/storage-http/src/main/java/org/apache/drill/exec/store/http/HttpCSVBatchReader.java b/contrib/storage-http/src/main/java/org/apache/drill/exec/store/http/HttpCSVBatchReader.java
index 2341a876a4..df55433678 100644
--- a/contrib/storage-http/src/main/java/org/apache/drill/exec/store/http/HttpCSVBatchReader.java
+++ b/contrib/storage-http/src/main/java/org/apache/drill/exec/store/http/HttpCSVBatchReader.java
@@ -126,6 +126,8 @@ public class HttpCSVBatchReader extends HttpBatchReader {
rowWriter = resultLoader.writer();
populateWriterArray();
+ // Close cache resources
+ http.close();
return true;
}
diff --git a/contrib/storage-http/src/main/java/org/apache/drill/exec/store/http/HttpXMLBatchReader.java b/contrib/storage-http/src/main/java/org/apache/drill/exec/store/http/HttpXMLBatchReader.java
index c7a2857b74..ca04a534e1 100644
--- a/contrib/storage-http/src/main/java/org/apache/drill/exec/store/http/HttpXMLBatchReader.java
+++ b/contrib/storage-http/src/main/java/org/apache/drill/exec/store/http/HttpXMLBatchReader.java
@@ -115,6 +115,9 @@ public class HttpXMLBatchReader extends HttpBatchReader {
.addContext(errorContext)
.build(logger);
}
+
+ // Close cache resources
+ http.close();
return true;
}
diff --git a/contrib/storage-http/src/main/java/org/apache/drill/exec/store/http/util/SimpleHttp.java b/contrib/storage-http/src/main/java/org/apache/drill/exec/store/http/util/SimpleHttp.java
index e5940de184..c1a18d889c 100644
--- a/contrib/storage-http/src/main/java/org/apache/drill/exec/store/http/util/SimpleHttp.java
+++ b/contrib/storage-http/src/main/java/org/apache/drill/exec/store/http/util/SimpleHttp.java
@@ -76,7 +76,7 @@ import java.util.regex.Pattern;
* method is the getInputStream() method which accepts a url and opens an
* InputStream with that URL's contents.
*/
-public class SimpleHttp {
+public class SimpleHttp implements AutoCloseable {
private static final Logger logger = LoggerFactory.getLogger(SimpleHttp.class);
private static final Pattern URL_PARAM_REGEX = Pattern.compile("\\{(\\w+)(?:=(\\w*))?\\}");
@@ -568,6 +568,19 @@ public class SimpleHttp {
return tempUrl;
}
+ @Override
+ public void close() {
+ Cache cache;
+ try {
+ cache = client.cache();
+ if (cache != null) {
+ cache.close();
+ }
+ } catch (IOException e) {
+ logger.warn("Error closing cache. {}", e.getMessage());
+ }
+ }
+
/**
* Intercepts requests and adds authentication headers to the request
*/
[drill] 01/08: DRILL-8326: Bump SnakeYaml to 1.33 (#2666)
Posted by dz...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
dzamo pushed a commit to branch 1.20
in repository https://gitbox.apache.org/repos/asf/drill.git
commit 85c353c54291a6a8c2a41c74392dfc61920ca223
Author: PJ Fanning <pj...@users.noreply.github.com>
AuthorDate: Sun Oct 2 21:51:12 2022 +0800
DRILL-8326: Bump SnakeYaml to 1.33 (#2666)
---
pom.xml | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/pom.xml b/pom.xml
index d80d18881a..cebbf0d671 100644
--- a/pom.xml
+++ b/pom.xml
@@ -124,7 +124,7 @@
<libthrift.version>0.14.0</libthrift.version>
<derby.version>10.14.2.0</derby.version>
<commons.cli.version>1.4</commons.cli.version>
- <snakeyaml.version>1.31</snakeyaml.version>
+ <snakeyaml.version>1.33</snakeyaml.version>
<commons.lang3.version>3.10</commons.lang3.version>
<testcontainers.version>1.16.3</testcontainers.version>
<typesafe.config.version>1.0.0</typesafe.config.version>
[drill] 05/08: [MINOR UPDATE]: Upgrade org.freemarker:freemarker to 2.3.30 (#2676)
Posted by dz...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
dzamo pushed a commit to branch 1.20
in repository https://gitbox.apache.org/repos/asf/drill.git
commit 2d7b49f66dc5654398cc36647ade13d0e0d79d2c
Author: taorong <14...@users.noreply.github.com>
AuthorDate: Fri Oct 14 23:02:37 2022 +0800
[MINOR UPDATE]: Upgrade org.freemarker:freemarker to 2.3.30 (#2676)
---
pom.xml | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/pom.xml b/pom.xml
index 45dac77528..0bbbcb8132 100644
--- a/pom.xml
+++ b/pom.xml
@@ -88,7 +88,7 @@
<hadoop.version>3.2.4</hadoop.version>
<hbase.version>2.4.9</hbase.version>
<fmpp.version>1.0</fmpp.version>
- <freemarker.version>2.3.28</freemarker.version>
+ <freemarker.version>2.3.30</freemarker.version>
<javassist.version>3.28.0-GA</javassist.version>
<msgpack.version>0.6.6</msgpack.version>
<reflections.version>0.9.10</reflections.version>
[drill] 08/08: DRILL-8333: Resource leak when JsonLoader is built from a stream (#2678)
Posted by dz...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
dzamo pushed a commit to branch 1.20
in repository https://gitbox.apache.org/repos/asf/drill.git
commit 85aa70c2b3629291e0e80a3eae70df03c00a1192
Author: James Turton <91...@users.noreply.github.com>
AuthorDate: Sat Oct 15 05:02:49 2022 +0800
DRILL-8333: Resource leak when JsonLoader is built from a stream (#2678)
---
.../store/kafka/decoders/JsonMessageReader.java | 25 +---------
.../easy/json/loader/ClosingStreamIterator.java | 54 ++++++++++++++++++++++
.../store/easy/json/loader/JsonLoaderImpl.java | 10 ++++
3 files changed, 66 insertions(+), 23 deletions(-)
diff --git a/contrib/storage-kafka/src/main/java/org/apache/drill/exec/store/kafka/decoders/JsonMessageReader.java b/contrib/storage-kafka/src/main/java/org/apache/drill/exec/store/kafka/decoders/JsonMessageReader.java
index a9aee5a990..de9ce644a6 100644
--- a/contrib/storage-kafka/src/main/java/org/apache/drill/exec/store/kafka/decoders/JsonMessageReader.java
+++ b/contrib/storage-kafka/src/main/java/org/apache/drill/exec/store/kafka/decoders/JsonMessageReader.java
@@ -26,6 +26,7 @@ import org.apache.drill.exec.physical.resultSet.RowSetLoader;
import org.apache.drill.exec.record.metadata.ColumnMetadata;
import org.apache.drill.exec.record.metadata.MetadataUtils;
import org.apache.drill.exec.store.easy.json.loader.JsonLoaderOptions;
+import org.apache.drill.exec.store.easy.json.loader.ClosingStreamIterator;
import org.apache.drill.exec.store.easy.json.parser.TokenIterator;
import org.apache.drill.exec.store.kafka.KafkaStoragePlugin;
import org.apache.drill.exec.store.kafka.MetaDataField;
@@ -37,8 +38,6 @@ import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import java.io.ByteArrayInputStream;
-import java.io.InputStream;
-import java.util.Iterator;
import java.util.Properties;
import java.util.StringJoiner;
@@ -50,7 +49,7 @@ public class JsonMessageReader implements MessageReader {
private static final Logger logger = LoggerFactory.getLogger(JsonMessageReader.class);
- private final SingleElementIterator<InputStream> stream = new SingleElementIterator<>();
+ private final ClosingStreamIterator stream = new ClosingStreamIterator();
private KafkaJsonLoader kafkaJsonLoader;
private ResultSetLoader resultSetLoader;
@@ -156,24 +155,4 @@ public class JsonMessageReader implements MessageReader {
.add("resultSetLoader=" + resultSetLoader)
.toString();
}
-
- public static class SingleElementIterator<T> implements Iterator<T> {
- private T value;
-
- @Override
- public boolean hasNext() {
- return value != null;
- }
-
- @Override
- public T next() {
- T value = this.value;
- this.value = null;
- return value;
- }
-
- public void setValue(T value) {
- this.value = value;
- }
- }
}
diff --git a/exec/java-exec/src/main/java/org/apache/drill/exec/store/easy/json/loader/ClosingStreamIterator.java b/exec/java-exec/src/main/java/org/apache/drill/exec/store/easy/json/loader/ClosingStreamIterator.java
new file mode 100644
index 0000000000..0b67049e57
--- /dev/null
+++ b/exec/java-exec/src/main/java/org/apache/drill/exec/store/easy/json/loader/ClosingStreamIterator.java
@@ -0,0 +1,54 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.store.easy.json.loader;
+
+import java.io.InputStream;
+
+import org.apache.drill.common.AutoCloseables;
+
+import java.util.Iterator;
+
+/**
+ * It allows setting the current value in the iterator and can be used once after {@link #next} call
+ *
+ * @param <T> type of the value
+ */
+public class ClosingStreamIterator implements Iterator<InputStream> {
+ private InputStream value, last;
+
+ @Override
+ public boolean hasNext() {
+ if (value == null) {
+ AutoCloseables.closeSilently(last);
+ return false;
+ }
+ return true;
+ }
+
+ @Override
+ public InputStream next() {
+ this.last = this.value;
+ this.value = null;
+ return this.last;
+ }
+
+ public void setValue(InputStream value) {
+ AutoCloseables.closeSilently(this.value);
+ this.value = value;
+ }
+ }
diff --git a/exec/java-exec/src/main/java/org/apache/drill/exec/store/easy/json/loader/JsonLoaderImpl.java b/exec/java-exec/src/main/java/org/apache/drill/exec/store/easy/json/loader/JsonLoaderImpl.java
index edc687f4aa..2886815db8 100644
--- a/exec/java-exec/src/main/java/org/apache/drill/exec/store/easy/json/loader/JsonLoaderImpl.java
+++ b/exec/java-exec/src/main/java/org/apache/drill/exec/store/easy/json/loader/JsonLoaderImpl.java
@@ -24,6 +24,7 @@ import java.util.ArrayList;
import java.util.Arrays;
import java.util.List;
+import org.apache.drill.common.AutoCloseables;
import org.apache.drill.common.exceptions.CustomErrorContext;
import org.apache.drill.common.exceptions.EmptyErrorContext;
import org.apache.drill.common.exceptions.UserException;
@@ -232,6 +233,7 @@ public class JsonLoaderImpl implements JsonLoader, ErrorFactory {
private final JsonStructureParser parser;
private final FieldFactory fieldFactory;
private final ImplicitColumns implicitFields;
+ private final Iterable<InputStream> streams;
private final int maxRows;
private boolean eof;
@@ -254,6 +256,7 @@ public class JsonLoaderImpl implements JsonLoader, ErrorFactory {
this.implicitFields = builder.implicitFields;
this.maxRows = builder.maxRows;
this.fieldFactory = buildFieldFactory(builder);
+ this.streams = builder.streams;
this.parser = buildParser(builder);
}
@@ -354,6 +357,13 @@ public class JsonLoaderImpl implements JsonLoader, ErrorFactory {
@Override // JsonLoader
public void close() {
parser.close();
+ for (InputStream stream: streams) {
+ try {
+ AutoCloseables.close(stream);
+ } catch (Exception ex) {
+ logger.warn("Failed to close an input stream, a system resource leak may ensue.", ex);
+ }
+ }
}
@Override // ErrorFactory
[drill] 07/08: [MINOR UPDATE]: Update com.drewnoakes:metadata-extractor 2.17.0 to 2.18.0 (#2682)
Posted by dz...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
dzamo pushed a commit to branch 1.20
in repository https://gitbox.apache.org/repos/asf/drill.git
commit 2b88eafc67885151eb0bc899576c459b138cbec9
Author: claire9910 <11...@users.noreply.github.com>
AuthorDate: Sun Oct 16 12:05:42 2022 +0800
[MINOR UPDATE]: Update com.drewnoakes:metadata-extractor 2.17.0 to 2.18.0 (#2682)
---
contrib/format-image/pom.xml | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/contrib/format-image/pom.xml b/contrib/format-image/pom.xml
index a96d9890a2..3ab13b9dd9 100644
--- a/contrib/format-image/pom.xml
+++ b/contrib/format-image/pom.xml
@@ -39,7 +39,7 @@
<dependency>
<groupId>com.drewnoakes</groupId>
<artifactId>metadata-extractor</artifactId>
- <version>2.17.0</version>
+ <version>2.18.0</version>
</dependency>
<!-- Test dependencies -->
[drill] 06/08: DRILL-8332: upgrade jackson due to issue in gradle module of previous release (#2677)
Posted by dz...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
dzamo pushed a commit to branch 1.20
in repository https://gitbox.apache.org/repos/asf/drill.git
commit 8e2eacc8733ad0a21f8cb6f4b705ab6bf9604d12
Author: PJ Fanning <pj...@users.noreply.github.com>
AuthorDate: Fri Oct 14 16:02:04 2022 +0100
DRILL-8332: upgrade jackson due to issue in gradle module of previous release (#2677)
---
pom.xml | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/pom.xml b/pom.xml
index 0bbbcb8132..7a70b4566d 100644
--- a/pom.xml
+++ b/pom.xml
@@ -64,7 +64,7 @@
<avatica.version>1.17.0</avatica.version>
<janino.version>3.0.11</janino.version>
<sqlline.version>1.12.0</sqlline.version>
- <jackson.version>2.13.4.20221012</jackson.version>
+ <jackson.version>2.13.4.20221013</jackson.version>
<zookeeper.version>3.5.7</zookeeper.version>
<mapr.release.version>6.1.0-mapr</mapr.release.version>
<ojai.version>3.0-mapr-1808</ojai.version>
[drill] 04/08: [MINOR UPDATE]: Upgrade org.apache.kafka:kafka_2.13 to 2.8.2 (#2675)
Posted by dz...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
dzamo pushed a commit to branch 1.20
in repository https://gitbox.apache.org/repos/asf/drill.git
commit 0e3b144d722b913b003b8814b47e72b314c4a1b5
Author: denglunfuren <98...@users.noreply.github.com>
AuthorDate: Fri Oct 14 23:03:23 2022 +0800
[MINOR UPDATE]: Upgrade org.apache.kafka:kafka_2.13 to 2.8.2 (#2675)
---
contrib/storage-kafka/pom.xml | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/contrib/storage-kafka/pom.xml b/contrib/storage-kafka/pom.xml
index 4e12a11dd1..eaef5b735d 100644
--- a/contrib/storage-kafka/pom.xml
+++ b/contrib/storage-kafka/pom.xml
@@ -31,7 +31,7 @@
<name>Drill : Contrib : Storage : Kafka</name>
<properties>
- <kafka.version>2.8.0</kafka.version>
+ <kafka.version>2.8.2</kafka.version>
<kafka.TestSuite>**/TestKafkaSuit.class</kafka.TestSuite>
</properties>