You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by GitBox <gi...@apache.org> on 2020/04/24 09:44:41 UTC
[GitHub] [arrow] zhztheplayer opened a new pull request #7030: ARROW-7808: [Java][Dataset] Implement Datasets Java API
zhztheplayer opened a new pull request #7030:
URL: https://github.com/apache/arrow/pull/7030
Add following Datasets APIs to Java:
- DatasetFactory
- Dataset
- Scanner
- ScanTask
Add a native dataset path to bridge c++ Datasets components to Java:
- NativeDatasetFactory
- NativeDataset
- NativeScanner
- NativeScanTask
Add a simple reference `NativeDatasetFactory` implementation, which is already being useful in some scenarios:
- SingleFileDatasetFactory
- SingleFileDataset
Unit tests are based on `SingleFileDataset`/`NativeDatasetFactory`. I may add more test cases under this PR in a time.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] zhztheplayer commented on a change in pull request #7030: ARROW-7808: [Java][Dataset] Implement Datasets Java API by JNI to C++
Posted by GitBox <gi...@apache.org>.
zhztheplayer commented on a change in pull request #7030:
URL: https://github.com/apache/arrow/pull/7030#discussion_r426472567
##########
File path: java/dataset/src/test/resources/avroschema/user.avsc
##########
@@ -0,0 +1,26 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+{
Review comment:
I think I understand your point here... But my feeling is that the JSON representation of Avro schema is more human-readable than code. e.g. The same schema we create in Java will end up like
```java
public static final Schema AVRO_SCHEMA_USER = Schema.createRecord(
Arrays.asList(
new Schema.Field("id",
Schema.createUnion(Schema.create(Schema.Type.INT), Schema.create(Schema.Type.NULL)),
null, (Object) null),
new Schema.Field("name",
Schema.createUnion(Schema.create(Schema.Type.STRING), Schema.create(Schema.Type.NULL)),
null, (Object) null)));
```
I think it's a little bit messy but if it's considered better I can change to use this way. I am ok to each.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] zhztheplayer commented on a change in pull request #7030: ARROW-7808: [Java][Dataset] Implement Datasets Java API by JNI to C++
Posted by GitBox <gi...@apache.org>.
zhztheplayer commented on a change in pull request #7030:
URL: https://github.com/apache/arrow/pull/7030#discussion_r429753826
##########
File path: java/dataset/src/main/java/org/apache/arrow/dataset/jni/JniLoader.java
##########
@@ -0,0 +1,71 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.arrow.dataset.jni;
+
+import java.io.File;
+import java.io.FileNotFoundException;
+import java.io.IOException;
+import java.io.InputStream;
+import java.nio.file.Files;
+import java.nio.file.StandardCopyOption;
+import java.util.Collections;
+import java.util.List;
+import java.util.concurrent.atomic.AtomicBoolean;
+
+/**
+ * The JniLoader for Datasets API's native implementation.
+ */
+public final class JniLoader {
+ private static final JniLoader INSTANCE = new JniLoader();
+ private static final List<String> LIBRARY_NAMES = Collections.singletonList("arrow_dataset_jni");
+
+ private AtomicBoolean loaded = new AtomicBoolean(false);
+
+ public static JniLoader get() {
+ return INSTANCE;
+ }
+
+ private JniLoader() {
+ }
+
+ /**
+ * If required JNI libraries are not loaded, then load them.
+ */
+ public void ensureLoaded() {
+ if (loaded.compareAndSet(false, true)) {
Review comment:
Thanks for the catching. I think the key to synchronize such stuffs is to protect against loading the same lib multiple times.
I've pushed a fix which is similar to your suggestion. Just handled an extra case of loading multiple libraries.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] zhztheplayer commented on pull request #7030: ARROW-7808: [Java][Dataset] Implement Datasets Java API by JNI to C++
Posted by GitBox <gi...@apache.org>.
zhztheplayer commented on pull request #7030:
URL: https://github.com/apache/arrow/pull/7030#issuecomment-644494295
@fsaintjacques Rebased now.
As we have a unresolved [discussion](https://github.com/apache/arrow/pull/7030#discussion_r440075959) about memory management of native buffers. Should we clarify a bit more before writing any fixes?
May first question is what's the major problem now? I list some that I can see in comments:
1. We are package-hacking `org.apache.arrow.memory` in module `arrow-dataset`.
Yes `Ownerships.java` uses some hacks, so I've removed the class in latest commits (the whole class is not necessary for current use case). Now only `NativeUnderlyingMemory.java` is still in the package `org.apache.arrow.memory`.
2. We lose java native heap caps.
Can we clarify this? I think the explanation can be either we lose caps from JVM direct memory, or we lose caps from BufferAllocator. For the latter one I think buffers are already registered to allocator.
3. OOM-check is proceed after buffer is created.
Yes. But is this really a big problem? see [comment](https://github.com/apache/arrow/pull/7030#discussion_r440075959).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] emkornfield commented on a change in pull request #7030: ARROW-7808: [Java][Dataset] Implement Dataset Java API by JNI to C++
Posted by GitBox <gi...@apache.org>.
emkornfield commented on a change in pull request #7030:
URL: https://github.com/apache/arrow/pull/7030#discussion_r551609281
##########
File path: cpp/src/arrow/memory_pool.h
##########
@@ -149,6 +149,43 @@ class ARROW_EXPORT ProxyMemoryPool : public MemoryPool {
std::unique_ptr<ProxyMemoryPoolImpl> impl_;
};
+class ARROW_EXPORT ReservationListener {
+ public:
+ virtual ~ReservationListener();
Review comment:
```suggestion
virtual ~ReservationListener() = default;
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] zhztheplayer commented on a change in pull request #7030: ARROW-7808: [Java][Dataset] Implement Datasets Java API by JNI to C++
Posted by GitBox <gi...@apache.org>.
zhztheplayer commented on a change in pull request #7030:
URL: https://github.com/apache/arrow/pull/7030#discussion_r417789282
##########
File path: java/dataset/src/main/java/org/apache/arrow/dataset/jni/JniWrapper.java
##########
@@ -0,0 +1,61 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.arrow.dataset.jni;
+
+/**
+ * JNI wrapper for Datasets API's native implementation.
+ */
+public class JniWrapper {
+
+ private static final JniWrapper INSTANCE = new JniWrapper();
+
+ public static JniWrapper get() {
+ return INSTANCE;
+ }
+
+ private JniWrapper() {
+ JniLoader.get().ensureLoaded();
+ }
+
+ public native void closeDatasetFactory(long datasetFactoryId);
Review comment:
Javadoc added
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] fsaintjacques commented on a change in pull request #7030: ARROW-7808: [Java][Dataset] Implement Datasets Java API by JNI to C++
Posted by GitBox <gi...@apache.org>.
fsaintjacques commented on a change in pull request #7030:
URL: https://github.com/apache/arrow/pull/7030#discussion_r426869642
##########
File path: java/dataset/src/main/java/org/apache/arrow/dataset/jni/JniLoader.java
##########
@@ -0,0 +1,71 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.arrow.dataset.jni;
+
+import java.io.File;
+import java.io.FileNotFoundException;
+import java.io.IOException;
+import java.io.InputStream;
+import java.nio.file.Files;
+import java.nio.file.StandardCopyOption;
+import java.util.Collections;
+import java.util.List;
+import java.util.concurrent.atomic.AtomicBoolean;
+
+/**
+ * The JniLoader for Datasets API's native implementation.
+ */
+public final class JniLoader {
+ private static final JniLoader INSTANCE = new JniLoader();
+ private static final List<String> LIBRARY_NAMES = Collections.singletonList("arrow_dataset_jni");
+
+ private AtomicBoolean loaded = new AtomicBoolean(false);
+
+ public static JniLoader get() {
+ return INSTANCE;
+ }
+
+ private JniLoader() {
+ }
+
+ /**
+ * If required JNI libraries are not loaded, then load them.
+ */
+ public void ensureLoaded() {
+ if (loaded.compareAndSet(false, true)) {
Review comment:
It seems the logic would be to set `loaded` _after_ iterating and loading said libraries. I don't know the concurrency requirements of `System.load` so I can't comment. You could fix this by adding a single method:
```java
public void ensureLoaded() {
if (!loaded) {
loadAll();
}
}
public void synchronized loadAll() {
// The method is protected by a mutex via synchronized, if more than
// one thread race to call loadAll, only one will do the actual loading and the
// others will be a noop once they acquire the mutex.
if (loaded) return;
LIBRARY_NAMES.forEach(this::load);
loaded.set(true);
}
...
##########
File path: cpp/src/jni/dataset/jni_wrapper.cpp
##########
@@ -0,0 +1,517 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include <arrow/dataset/api.h>
+#include <arrow/dataset/file_base.h>
+#include <arrow/filesystem/hdfs.h>
+#include <arrow/filesystem/localfs.h>
+#include <arrow/io/api.h>
+#include <arrow/ipc/api.h>
+#include <arrow/util/iterator.h>
+
+#include "org_apache_arrow_dataset_file_JniWrapper.h"
+#include "org_apache_arrow_dataset_jni_JniWrapper.h"
+
+static jclass illegal_access_exception_class;
+static jclass illegal_argument_exception_class;
+static jclass runtime_exception_class;
+
+static jclass record_batch_handle_class;
+static jclass record_batch_handle_field_class;
+static jclass record_batch_handle_buffer_class;
+
+static jmethodID record_batch_handle_constructor;
+static jmethodID record_batch_handle_field_constructor;
+static jmethodID record_batch_handle_buffer_constructor;
+
+static jint JNI_VERSION = JNI_VERSION_1_6;
+
+class JniPendingException : public std::runtime_error {
+ public:
+ explicit JniPendingException(const std::string& arg) : runtime_error(arg) {}
+};
+
+void ThrowPendingException(const std::string& message) {
+ throw JniPendingException(message);
+}
+
+template <typename T>
+T JniGetOrThrow(arrow::Result<T> result) {
+ if (!result.status().ok()) {
+ ThrowPendingException(result.status().message());
+ }
+ return std::move(result).ValueOrDie();
+}
+
+void JniAssertOkOrThrow(arrow::Status status) {
+ if (!status.ok()) {
+ ThrowPendingException(status.message());
+ }
+}
+
+void JniThrow(std::string message) { ThrowPendingException(message); }
+
+#define JNI_METHOD_START try {
+// macro ended
+
+#define JNI_METHOD_END(fallback_expr) \
+ } \
+ catch (JniPendingException & e) { \
+ env->ThrowNew(runtime_exception_class, e.what()); \
+ return fallback_expr; \
+ }
+// macro ended
+
+jclass CreateGlobalClassReference(JNIEnv* env, const char* class_name) {
+ jclass local_class = env->FindClass(class_name);
+ jclass global_class = (jclass)env->NewGlobalRef(local_class);
+ env->DeleteLocalRef(local_class);
+ return global_class;
+}
+
+arrow::Result<jmethodID> GetMethodID(JNIEnv* env, jclass this_class, const char* name,
+ const char* sig) {
+ jmethodID ret = env->GetMethodID(this_class, name, sig);
+ if (ret == nullptr) {
+ std::string error_message = "Unable to find method " + std::string(name) +
+ " within signature" + std::string(sig);
+ return arrow::Status::Invalid(error_message);
+ }
+ return ret;
+}
+
+jint JNI_OnLoad(JavaVM* vm, void* reserved) {
+ JNIEnv* env;
+ if (vm->GetEnv(reinterpret_cast<void**>(&env), JNI_VERSION) != JNI_OK) {
+ return JNI_ERR;
+ }
+ JNI_METHOD_START
+ illegal_access_exception_class =
+ CreateGlobalClassReference(env, "Ljava/lang/IllegalAccessException;");
+ illegal_argument_exception_class =
+ CreateGlobalClassReference(env, "Ljava/lang/IllegalArgumentException;");
+ runtime_exception_class =
+ CreateGlobalClassReference(env, "Ljava/lang/RuntimeException;");
+
+ record_batch_handle_class =
+ CreateGlobalClassReference(env,
+ "Lorg/apache/arrow/"
+ "dataset/jni/NativeRecordBatchHandle;");
+ record_batch_handle_field_class =
+ CreateGlobalClassReference(env,
+ "Lorg/apache/arrow/"
+ "dataset/jni/NativeRecordBatchHandle$Field;");
+ record_batch_handle_buffer_class =
+ CreateGlobalClassReference(env,
+ "Lorg/apache/arrow/"
+ "dataset/jni/NativeRecordBatchHandle$Buffer;");
+
+ record_batch_handle_constructor =
+ JniGetOrThrow(GetMethodID(env, record_batch_handle_class, "<init>",
+ "(J[Lorg/apache/arrow/dataset/"
+ "jni/NativeRecordBatchHandle$Field;"
+ "[Lorg/apache/arrow/dataset/"
+ "jni/NativeRecordBatchHandle$Buffer;)V"));
+ record_batch_handle_field_constructor =
+ JniGetOrThrow(GetMethodID(env, record_batch_handle_field_class, "<init>", "(JJ)V"));
+ record_batch_handle_buffer_constructor = JniGetOrThrow(
+ GetMethodID(env, record_batch_handle_buffer_class, "<init>", "(JJJJ)V"));
+
+ return JNI_VERSION;
+ JNI_METHOD_END(JNI_ERR)
+}
+
+void JNI_OnUnload(JavaVM* vm, void* reserved) {
+ JNIEnv* env;
+ vm->GetEnv(reinterpret_cast<void**>(&env), JNI_VERSION);
+ env->DeleteGlobalRef(illegal_access_exception_class);
+ env->DeleteGlobalRef(illegal_argument_exception_class);
+ env->DeleteGlobalRef(runtime_exception_class);
+ env->DeleteGlobalRef(record_batch_handle_class);
+ env->DeleteGlobalRef(record_batch_handle_field_class);
+ env->DeleteGlobalRef(record_batch_handle_buffer_class);
+}
+
+std::shared_ptr<arrow::Schema> SchemaFromColumnNames(
+ const std::shared_ptr<arrow::Schema>& input,
+ const std::vector<std::string>& column_names) {
+ std::vector<std::shared_ptr<arrow::Field>> columns;
+ for (const auto& name : column_names) {
+ columns.push_back(input->GetFieldByName(name));
+ }
+ return std::make_shared<arrow::Schema>(columns);
+}
+
+arrow::Result<std::shared_ptr<arrow::dataset::FileFormat>> GetFileFormat(jint id) {
+ switch (id) {
+ case 0:
+ return std::make_shared<arrow::dataset::ParquetFileFormat>();
+ default:
+ std::string error_message = "illegal file format id: " + std::to_string(id);
+ return arrow::Status::Invalid(error_message);
+ }
+}
+
+arrow::Result<std::shared_ptr<arrow::fs::FileSystem>> GetFileSystem(
+ jint id, std::string path, std::string* out_path) {
+ switch (id) {
+ case 0:
+ *out_path = path;
+ return std::make_shared<arrow::fs::LocalFileSystem>();
+ case 1: {
+ ARROW_ASSIGN_OR_RAISE(std::shared_ptr<arrow::fs::FileSystem> ret,
+ arrow::fs::FileSystemFromUri(path, out_path))
+ return ret;
+ }
+ default:
+ std::string error_message = "illegal file system id: " + std::to_string(id);
+ return arrow::Status::Invalid(error_message);
+ }
+}
+
+std::string JStringToCString(JNIEnv* env, jstring string) {
+ if (string == nullptr) {
+ return std::string();
+ }
+ jboolean copied;
+ int32_t length = env->GetStringUTFLength(string);
+ const char* chars = env->GetStringUTFChars(string, &copied);
+ std::string str = std::string(chars, length);
+ // fixme calling ReleaseStringUTFChars if memory leak faced
+ return str;
+}
+
+std::vector<std::string> ToStringVector(JNIEnv* env, jobjectArray& str_array) {
+ int length = env->GetArrayLength(str_array);
+ std::vector<std::string> vector;
+ for (int i = 0; i < length; i++) {
+ auto string = (jstring)(env->GetObjectArrayElement(str_array, i));
+ vector.push_back(JStringToCString(env, string));
+ }
+ return vector;
+}
+
+template <typename T>
+jlong CreateNativeRef(std::shared_ptr<T> t) {
+ std::shared_ptr<T>* retained_ptr = new std::shared_ptr<T>(t);
+ return reinterpret_cast<jlong>(retained_ptr);
+}
+
+template <typename T>
+std::shared_ptr<T> RetrieveNativeInstance(jlong ref) {
+ std::shared_ptr<T>* retrieved_ptr = reinterpret_cast<std::shared_ptr<T>*>(ref);
+ return *retrieved_ptr;
+}
+
+template <typename T>
+void ReleaseNativeRef(jlong ref) {
+ std::shared_ptr<T>* retrieved_ptr = reinterpret_cast<std::shared_ptr<T>*>(ref);
+ delete retrieved_ptr;
+}
+
+arrow::Result<jbyteArray> ToSchemaByteArray(JNIEnv* env,
+ std::shared_ptr<arrow::Schema> schema) {
+ ARROW_ASSIGN_OR_RAISE(
+ std::shared_ptr<arrow::Buffer> buffer,
+ arrow::ipc::SerializeSchema(*schema, nullptr, arrow::default_memory_pool()))
+
+ jbyteArray out = env->NewByteArray(buffer->size());
+ auto src = reinterpret_cast<const jbyte*>(buffer->data());
+ env->SetByteArrayRegion(out, 0, buffer->size(), src);
+ return out;
+}
+
+arrow::Result<std::shared_ptr<arrow::Schema>> FromSchemaByteArray(
+ JNIEnv* env, jbyteArray schemaBytes) {
+ arrow::ipc::DictionaryMemo in_memo;
+ int schemaBytes_len = env->GetArrayLength(schemaBytes);
+ jbyte* schemaBytes_data = env->GetByteArrayElements(schemaBytes, nullptr);
+ auto serialized_schema = std::make_shared<arrow::Buffer>(
+ reinterpret_cast<uint8_t*>(schemaBytes_data), schemaBytes_len);
+ arrow::io::BufferReader buf_reader(serialized_schema);
+ ARROW_ASSIGN_OR_RAISE(std::shared_ptr<arrow::Schema> schema,
+ arrow::ipc::ReadSchema(&buf_reader, &in_memo))
+ env->ReleaseByteArrayElements(schemaBytes, schemaBytes_data, JNI_ABORT);
+ return schema;
+}
+
+/*
+ * Class: org_apache_arrow_dataset_jni_JniWrapper
+ * Method: closeDatasetFactory
+ * Signature: (J)V
+ */
+JNIEXPORT void JNICALL Java_org_apache_arrow_dataset_jni_JniWrapper_closeDatasetFactory(
+ JNIEnv* env, jobject, jlong id) {
+ JNI_METHOD_START
+ ReleaseNativeRef<arrow::dataset::DatasetFactory>(id);
+ JNI_METHOD_END()
+}
+
+/*
+ * Class: org_apache_arrow_dataset_jni_JniWrapper
+ * Method: inspectSchema
+ * Signature: (J)[B
+ */
+JNIEXPORT jbyteArray JNICALL Java_org_apache_arrow_dataset_jni_JniWrapper_inspectSchema(
+ JNIEnv* env, jobject, jlong dataset_factor_id) {
+ JNI_METHOD_START
+ std::shared_ptr<arrow::dataset::DatasetFactory> d =
+ RetrieveNativeInstance<arrow::dataset::DatasetFactory>(dataset_factor_id);
+ std::shared_ptr<arrow::Schema> schema = JniGetOrThrow(d->Inspect());
+ return JniGetOrThrow(ToSchemaByteArray(env, schema));
+ JNI_METHOD_END(nullptr)
+}
+
+/*
+ * Class: org_apache_arrow_dataset_jni_JniWrapper
+ * Method: createDataset
+ * Signature: (J[B)J
+ */
+JNIEXPORT jlong JNICALL Java_org_apache_arrow_dataset_jni_JniWrapper_createDataset(
+ JNIEnv* env, jobject, jlong dataset_factory_id, jbyteArray schema_bytes) {
+ JNI_METHOD_START
+ std::shared_ptr<arrow::dataset::DatasetFactory> d =
+ RetrieveNativeInstance<arrow::dataset::DatasetFactory>(dataset_factory_id);
+ std::shared_ptr<arrow::Schema> schema;
+ schema = JniGetOrThrow(FromSchemaByteArray(env, schema_bytes));
+ std::shared_ptr<arrow::dataset::Dataset> dataset = JniGetOrThrow(d->Finish(schema));
+ return CreateNativeRef(dataset);
+ JNI_METHOD_END(-1L)
+}
+
+/*
+ * Class: org_apache_arrow_dataset_jni_JniWrapper
+ * Method: closeDataset
+ * Signature: (J)V
+ */
+JNIEXPORT void JNICALL Java_org_apache_arrow_dataset_jni_JniWrapper_closeDataset(
+ JNIEnv* env, jobject, jlong id) {
+ JNI_METHOD_START
+ ReleaseNativeRef<arrow::dataset::Dataset>(id);
+ JNI_METHOD_END()
+}
+
+/// \class DisposableScannerAdaptor
+/// \brief An adaptor that iterates over a Scanner instance then returns RecordBatches
+/// directly.
+///
+/// This lessens the complexity of the JNI bridge to make sure it to be easier to
+/// maintain. On Java-side, NativeScanner can only produces a single NativeScanTask
+/// instance during its whole lifecycle. Each task stands for a DisposableScannerAdaptor
+/// instance through JNI bridge.
+///
+class DisposableScannerAdaptor {
Review comment:
This class limits parallelism to one thread. How do you plan to cope with this in the future?
##########
File path: cpp/.gitignore
##########
@@ -24,6 +24,8 @@ cmake_install.cmake
build/
*-build/
Testing/
+cmake-build-debug/
Review comment:
This shouldn't be here, just name your directories `build-cmake-*`.
##########
File path: cpp/src/arrow/dataset/discovery.h
##########
@@ -238,5 +238,27 @@ class ARROW_DS_EXPORT FileSystemDatasetFactory : public DatasetFactory {
util::optional<util::string_view> RemovePartitionBaseDir(util::string_view path);
};
+class ARROW_DS_EXPORT SingleFileDatasetFactory : public DatasetFactory {
Review comment:
You can use `FileSystemDataFactory::Make({path}, fs, format, opts)`, I don't think this class is warranted (see next comment with SingleFileDataset).
##########
File path: cpp/src/arrow/dataset/file_base.h
##########
@@ -290,5 +290,35 @@ class ARROW_DS_EXPORT WritePlan {
std::vector<std::string> paths;
};
+class ARROW_DS_EXPORT SingleFileDataset : public Dataset {
Review comment:
With ARROW-8318 (introduced in #7073), this class shouldn't be needed. Is there an explicit reason to do so?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] zhztheplayer commented on a change in pull request #7030: ARROW-7808: [Java][Dataset] Implement Datasets Java API by JNI to C++
Posted by GitBox <gi...@apache.org>.
zhztheplayer commented on a change in pull request #7030:
URL: https://github.com/apache/arrow/pull/7030#discussion_r415661628
##########
File path: cpp/src/jni/dataset/jni_wrapper.cpp
##########
@@ -0,0 +1,577 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include <arrow/dataset/file_base.h>
+#include <arrow/dataset/api.h>
+#include <arrow/filesystem/localfs.h>
+#include <arrow/ipc/api.h>
+#include <arrow/util/iterator.h>
+#include <arrow/filesystem/hdfs.h>
+#include <arrow/io/api.h>
+#include <google/protobuf/io/coded_stream.h>
+#include <google/protobuf/message.h>
+#include "concurrent_map.h"
+#include "arrow/compute/kernel.h"
+#include "arrow/compute/kernels/cast.h"
+#include "arrow/compute/kernels/compare.h"
+#include "jni/dataset/Types.pb.h"
+
+#include "org_apache_arrow_dataset_jni_JniWrapper.h"
+#include "org_apache_arrow_dataset_file_JniWrapper.h"
+
+static jclass illegal_access_exception_class;
+static jclass illegal_argument_exception_class;
+static jclass runtime_exception_class;
+
+static jclass record_batch_handle_class;
+static jclass record_batch_handle_field_class;
+static jclass record_batch_handle_buffer_class;
+
+static jmethodID record_batch_handle_constructor;
+static jmethodID record_batch_handle_field_constructor;
+static jmethodID record_batch_handle_buffer_constructor;
+
+static jint JNI_VERSION = JNI_VERSION_1_6;
+
+using arrow::jni::ConcurrentMap;
+
+static ConcurrentMap<std::shared_ptr<arrow::dataset::DatasetFactory>> dataset_factory_holder_;
Review comment:
done
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] zhztheplayer commented on a change in pull request #7030: ARROW-7808: [Java][Dataset] Implement Dataset Java API by JNI to C++
Posted by GitBox <gi...@apache.org>.
zhztheplayer commented on a change in pull request #7030:
URL: https://github.com/apache/arrow/pull/7030#discussion_r567767970
##########
File path: cpp/src/jni/dataset/jni_wrapper.cpp
##########
@@ -0,0 +1,859 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include <mutex>
+
+#include <arrow/array.h>
+#include <arrow/dataset/api.h>
+#include <arrow/dataset/file_base.h>
+#include <arrow/filesystem/localfs.h>
+#include <arrow/io/api.h>
+#include <arrow/ipc/api.h>
+#include <arrow/util/iterator.h>
+#include <arrow/util/logging.h>
+
+#include "org_apache_arrow_dataset_file_JniWrapper.h"
+#include "org_apache_arrow_dataset_jni_JniWrapper.h"
+#include "org_apache_arrow_dataset_jni_NativeMemoryPool.h"
+
+namespace arrow {
+namespace dataset {
+namespace jni {
+
+static jclass illegal_access_exception_class;
+static jclass illegal_argument_exception_class;
+static jclass runtime_exception_class;
+
+static jclass record_batch_handle_class;
+static jclass record_batch_handle_field_class;
+static jclass record_batch_handle_buffer_class;
+static jclass java_reservation_listener_class;
+
+static jmethodID record_batch_handle_constructor;
+static jmethodID record_batch_handle_field_constructor;
+static jmethodID record_batch_handle_buffer_constructor;
+static jmethodID reserve_memory_method;
+static jmethodID unreserve_memory_method;
+
+static jlong default_memory_pool_id = -1L;
+
+static jint JNI_VERSION = JNI_VERSION_1_6;
+
+class JniPendingException : public std::runtime_error {
+ public:
+ explicit JniPendingException(const std::string& arg) : runtime_error(arg) {}
+};
+
+void ThrowPendingException(const std::string& message) {
+ throw JniPendingException(message);
+}
+
+template <typename T>
+T JniGetOrThrow(arrow::Result<T> result) {
+ if (!result.status().ok()) {
+ ThrowPendingException(result.status().message());
+ }
+ return std::move(result).ValueOrDie();
+}
+
+void JniAssertOkOrThrow(arrow::Status status) {
+ if (!status.ok()) {
+ ThrowPendingException(status.message());
+ }
+}
+
+void JniThrow(std::string message) { ThrowPendingException(message); }
+
+jclass CreateGlobalClassReference(JNIEnv* env, const char* class_name) {
+ jclass local_class = env->FindClass(class_name);
+ jclass global_class = (jclass)env->NewGlobalRef(local_class);
+ env->DeleteLocalRef(local_class);
+ return global_class;
+}
+
+arrow::Result<jmethodID> GetMethodID(JNIEnv* env, jclass this_class, const char* name,
+ const char* sig) {
+ jmethodID ret = env->GetMethodID(this_class, name, sig);
+ if (ret == nullptr) {
+ std::string error_message = "Unable to find method " + std::string(name) +
+ " within signature" + std::string(sig);
+ return arrow::Status::Invalid(error_message);
+ }
+ return ret;
+}
+
+arrow::Result<jmethodID> GetStaticMethodID(JNIEnv* env, jclass this_class,
+ const char* name, const char* sig) {
+ jmethodID ret = env->GetStaticMethodID(this_class, name, sig);
+ if (ret == nullptr) {
+ std::string error_message = "Unable to find static method " + std::string(name) +
+ " within signature" + std::string(sig);
+ return arrow::Status::Invalid(error_message);
+ }
+ return ret;
+}
+
+std::shared_ptr<arrow::Schema> SchemaFromColumnNames(
+ const std::shared_ptr<arrow::Schema>& input,
+ const std::vector<std::string>& column_names) {
+ std::vector<std::shared_ptr<arrow::Field>> columns;
+ for (const auto& name : column_names) {
+ columns.push_back(input->GetFieldByName(name));
+ }
+ return std::make_shared<arrow::Schema>(columns);
+}
+
+arrow::Result<std::shared_ptr<arrow::dataset::FileFormat>> GetFileFormat(jint id) {
+ switch (id) {
+ case 0:
+ return std::make_shared<arrow::dataset::ParquetFileFormat>();
+ default:
+ std::string error_message = "illegal file format id: " + std::to_string(id);
+ return arrow::Status::Invalid(error_message);
+ }
+}
+
+std::string JStringToCString(JNIEnv* env, jstring string) {
+ if (string == nullptr) {
+ return std::string();
+ }
+ jboolean copied;
+ const char* chars = env->GetStringUTFChars(string, &copied);
+ std::string ret = strdup(chars);
+ env->ReleaseStringUTFChars(string, chars);
+ return ret;
+}
+
+std::vector<std::string> ToStringVector(JNIEnv* env, jobjectArray& str_array) {
+ int length = env->GetArrayLength(str_array);
+ std::vector<std::string> vector;
+ for (int i = 0; i < length; i++) {
+ auto string = (jstring)(env->GetObjectArrayElement(str_array, i));
+ vector.push_back(JStringToCString(env, string));
+ }
+ return vector;
+}
+
+template <typename T>
+jlong CreateNativeRef(std::shared_ptr<T> t) {
+ std::shared_ptr<T>* retained_ptr = new std::shared_ptr<T>(t);
+ return reinterpret_cast<jlong>(retained_ptr);
+}
+
+template <typename T>
+std::shared_ptr<T> RetrieveNativeInstance(jlong ref) {
+ std::shared_ptr<T>* retrieved_ptr = reinterpret_cast<std::shared_ptr<T>*>(ref);
+ return *retrieved_ptr;
+}
+
+template <typename T>
+void ReleaseNativeRef(jlong ref) {
+ std::shared_ptr<T>* retrieved_ptr = reinterpret_cast<std::shared_ptr<T>*>(ref);
+ delete retrieved_ptr;
+}
+
+arrow::Result<jbyteArray> ToSchemaByteArray(JNIEnv* env,
+ std::shared_ptr<arrow::Schema> schema) {
+ ARROW_ASSIGN_OR_RAISE(
+ std::shared_ptr<arrow::Buffer> buffer,
+ arrow::ipc::SerializeSchema(*schema, arrow::default_memory_pool()))
+
+ jbyteArray out = env->NewByteArray(buffer->size());
+ auto src = reinterpret_cast<const jbyte*>(buffer->data());
+ env->SetByteArrayRegion(out, 0, buffer->size(), src);
+ return out;
+}
+
+arrow::Result<std::shared_ptr<arrow::Schema>> FromSchemaByteArray(
+ JNIEnv* env, jbyteArray schemaBytes) {
+ arrow::ipc::DictionaryMemo in_memo;
+ int schemaBytes_len = env->GetArrayLength(schemaBytes);
+ jbyte* schemaBytes_data = env->GetByteArrayElements(schemaBytes, nullptr);
+ auto serialized_schema = std::make_shared<arrow::Buffer>(
+ reinterpret_cast<uint8_t*>(schemaBytes_data), schemaBytes_len);
+ arrow::io::BufferReader buf_reader(serialized_schema);
+ ARROW_ASSIGN_OR_RAISE(std::shared_ptr<arrow::Schema> schema,
+ arrow::ipc::ReadSchema(&buf_reader, &in_memo))
+ env->ReleaseByteArrayElements(schemaBytes, schemaBytes_data, JNI_ABORT);
+ return schema;
+}
+
+/// Listener to act on reservations/unreservations from ReservationListenableMemoryPool.
+///
+/// Note the memory pool will call this listener only on block-level memory
+/// reservation/unreservation is granted. So the invocation parameter "size" is always
+/// multiple of block size (by default, 512k) specified in memory pool.
+class ReservationListener {
+ public:
+ virtual ~ReservationListener() = default;
+
+ virtual arrow::Status OnReservation(int64_t size) = 0;
+ virtual arrow::Status OnRelease(int64_t size) = 0;
+
+ protected:
+ ReservationListener() = default;
+};
+
+class ReservationListenableMemoryPoolImpl {
Review comment:
See above. Do we still have to do this after the refactors
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] Sebastiaan-Alvarez-Rodriguez edited a comment on pull request #7030: ARROW-7808: [Java][Dataset] Implement Datasets Java API by JNI to C++
Posted by GitBox <gi...@apache.org>.
Sebastiaan-Alvarez-Rodriguez edited a comment on pull request #7030:
URL: https://github.com/apache/arrow/pull/7030#issuecomment-694925466
hi there,
Could you please add a few compilation/installation instructions to your branch?
I just completed a long debug session to find out why the JNI runtime did not work.
### CPP
Compiled `<project-root>/cpp/` project using:
```bash
cd <project-root>/cpp
cmake -DARROW_DATASET=ON -DARROW_JNI=ON -DARROW_PARQUET=ON -DARROW_IPC=ON .
sudo make install
```
Confirmed shared libs are indeed visible using `whereis libarrow_dataset_jni`.
### Java
Compiled `<project-root>/java/dataset/` using
```bash
cd <project-root>/java/dataset
mvn clean install -Dmaven.test.skip=true # skipping tests to let build succeed
```
To test whether it works, successfully ran a sample program:
```java
public class test {
public static void main(String[] args) {
System.loadLibrary("arrow_dataset_jni");
System.out.println("System loading works fine");
}
}
```
So, my sample program's classloader can access `libarrow_dataset_jni.so` just fine.
However, the following does not work:
```java
import org.apache.arrow.memory.RootAllocator;
import org.apache.arrow.dataset.file.FileFormat;
import org.apache.arrow.dataset.file.FileSystemDatasetFactory;
public class test {
private static FileSystemDatasetFactory getDatasetFactory() {
RootAllocator allocator = new RootAllocator(Long.MAX_VALUE);
return new FileSystemDatasetFactory(allocator, FileFormat.PARQUET, "file:///path/to/pq.parquet"); //crash
}
public static void main(String[] args) {
System.loadLibrary("arrow_dataset_jni");
System.out.println("Own classLoader works fine");
FileSystemDatasetFactory test = getDatasetFactory();
System.out.println("Not so good! I crash before I can print this");
}
}
```
The relevant part of the crash is:
```
Caused by: java.lang.IllegalStateException: error loading native libraries: java.io.FileNotFoundException: libarrow_dataset_jni.so
at org.apache.arrow.dataset.jni.JniLoader.load(JniLoader.java:91)
at org.apache.arrow.dataset.jni.JniLoader.loadRemaining(JniLoader.java:73)
at org.apache.arrow.dataset.jni.JniLoader.ensureLoaded(JniLoader.java:60)
at org.apache.arrow.dataset.file.JniWrapper.<init>(JniWrapper.java:34)
```
The source code of the Dataset Java library causing the error is found [here](https://github.com/zhztheplayer/arrow-1/blob/fd163e199a10a7225765b0c30fbf60d8df8d20db/java/dataset/src/main/java/org/apache/arrow/dataset/jni/JniLoader.java#L78-L94).
The problem occurs because `JniWrapper.class.getClassLoader().getResourceAsStream(libraryToLoad)` always returns `null`.
Only when adding a symlink (`ln -s /usr/local/lib/libarrow_dataset_jni.so libarrow_dataset_jni.so`) to the package root directory, it is able to find the correct file.
Why do you create a temporary file, store the contents of the shared library in it, and then call `System.load(tmpfile)` and not just directly call it using `System.loadLibrary(libraryToLoad);`?
Maybe it is a good idea to pick one of the following:
1. Change the `'java.io.FileNotFoundException: libarrow_dataset_jni.so'` to display the full path (so people know where your library looks to find the shared object)
2. Make a small `JNI_dataset_dev_install_note.md` somewhere on getting this library to work.
3. Use `System.loadLibrary(libraryToLoad);`
4. Use some other way to find the location of a shared library and copy it to a tmpfile
I believe it would help many people who are not familiar with `getClassLoader().getResourceAsStream(...)` (like myself)
Have a nice day,
Sebastiaan
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] zhztheplayer commented on a change in pull request #7030: ARROW-7808: [Java][Dataset] Implement Datasets Java API by JNI to C++
Posted by GitBox <gi...@apache.org>.
zhztheplayer commented on a change in pull request #7030:
URL: https://github.com/apache/arrow/pull/7030#discussion_r449956827
##########
File path: java/dataset/src/main/java/org/apache/arrow/memory/NativeUnderlingMemory.java
##########
@@ -0,0 +1,65 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.arrow.memory;
+
+import org.apache.arrow.dataset.jni.JniWrapper;
+
+/**
+ * AllocationManager implementation for Native allocated memory.
+ */
+public class NativeUnderlingMemory extends AllocationManager {
Review comment:
> In reality, this is not at all the case. There are two ways that you run out of memory in Java: you run out of available allocation space OR you hit the limit at the JVM level. The limit at the JVM level can be influenced by things like fragmentation so in nearly all cases you hit that limit before the allocator managed limit. In your patch, you only are respecting the allocator set limit, not the jvm limit. This means a user could configure java to allow X of memory and the process actually will use 2X of memory, leading to issues in many systems.
Making sense overall. So we may have to use a C++-to-Java/Netty allocation path in the first version right? I'll try it out.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] emkornfield commented on a change in pull request #7030: ARROW-7808: [Java][Dataset] Implement Datasets Java API by JNI to C++
Posted by GitBox <gi...@apache.org>.
emkornfield commented on a change in pull request #7030:
URL: https://github.com/apache/arrow/pull/7030#discussion_r425564595
##########
File path: java/dataset/src/main/java/org/apache/arrow/dataset/jni/NativeDataset.java
##########
@@ -0,0 +1,48 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.arrow.dataset.jni;
+
+import org.apache.arrow.dataset.scanner.ScanOptions;
+import org.apache.arrow.dataset.scanner.Scanner;
+import org.apache.arrow.dataset.source.Dataset;
+
+/**
+ * Native implementation of {@link Dataset}.
+ */
+public class NativeDataset implements Dataset, AutoCloseable {
+
+ private final NativeContext context;
+ private final long datasetId;
+
+ public NativeDataset(NativeContext context, long datasetId) {
+ this.context = context;
+ this.datasetId = datasetId;
+ }
+
+ @Override
+ public Scanner newScan(ScanOptions options) {
+ long scannerId = JniWrapper.get().createScanner(datasetId, options.getColumns(),
+ options.getFilter().toByteArray(), options.getBatchSize());
+ return new NativeScanner(context, scannerId);
+ }
+
+ @Override
+ public void close() throws Exception {
Review comment:
be specific in the exception it throws or make it a runtime exception.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] wesm commented on pull request #7030: ARROW-7808: [Java][Dataset] Implement Datasets Java API by JNI to C++
Posted by GitBox <gi...@apache.org>.
wesm commented on pull request #7030:
URL: https://github.com/apache/arrow/pull/7030#issuecomment-649079161
There's a small rebase conflict here. Need help from Java folks here, @rymurr can you help?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] emkornfield commented on a change in pull request #7030: ARROW-7808: [Java][Dataset] Implement Datasets Java API
Posted by GitBox <gi...@apache.org>.
emkornfield commented on a change in pull request #7030:
URL: https://github.com/apache/arrow/pull/7030#discussion_r414990082
##########
File path: cpp/src/jni/dataset/jni_wrapper.cpp
##########
@@ -0,0 +1,577 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include <arrow/dataset/file_base.h>
+#include <arrow/dataset/api.h>
+#include <arrow/filesystem/localfs.h>
+#include <arrow/ipc/api.h>
+#include <arrow/util/iterator.h>
+#include <arrow/filesystem/hdfs.h>
+#include <arrow/io/api.h>
+#include <google/protobuf/io/coded_stream.h>
+#include <google/protobuf/message.h>
+#include "concurrent_map.h"
+#include "arrow/compute/kernel.h"
+#include "arrow/compute/kernels/cast.h"
+#include "arrow/compute/kernels/compare.h"
+#include "jni/dataset/Types.pb.h"
+
+#include "org_apache_arrow_dataset_jni_JniWrapper.h"
+#include "org_apache_arrow_dataset_file_JniWrapper.h"
+
+static jclass illegal_access_exception_class;
+static jclass illegal_argument_exception_class;
+static jclass runtime_exception_class;
+
+static jclass record_batch_handle_class;
+static jclass record_batch_handle_field_class;
+static jclass record_batch_handle_buffer_class;
+
+static jmethodID record_batch_handle_constructor;
+static jmethodID record_batch_handle_field_constructor;
+static jmethodID record_batch_handle_buffer_constructor;
+
+static jint JNI_VERSION = JNI_VERSION_1_6;
+
+using arrow::jni::ConcurrentMap;
+
+static ConcurrentMap<std::shared_ptr<arrow::dataset::DatasetFactory>> dataset_factory_holder_;
+static ConcurrentMap<std::shared_ptr<arrow::dataset::Dataset>> dataset_holder_;
+static ConcurrentMap<std::shared_ptr<arrow::dataset::ScanTask>> scan_task_holder_;
+static ConcurrentMap<std::shared_ptr<arrow::dataset::Scanner>> scanner_holder_;
+static ConcurrentMap<std::shared_ptr<arrow::RecordBatchIterator>> iterator_holder_;
+static ConcurrentMap<std::shared_ptr<arrow::Buffer>> buffer_holder_;
+
+#define JNI_ASSIGN_OR_THROW_NAME(x, y) ARROW_CONCAT(x, y)
+
+#define JNI_ASSIGN_OR_THROW_IMPL(t, lhs, rexpr) \
+ auto t = (rexpr); \
+ if (!t.status().ok()) { \
+ env->ThrowNew(runtime_exception_class, t.status().message().c_str()); \
+ } \
+ lhs = std::move(t).ValueOrDie();
+
+#define JNI_ASSIGN_OR_THROW(lhs, rexpr) \
+ JNI_ASSIGN_OR_THROW_IMPL(JNI_ASSIGN_OR_THROW_NAME(_tmp_var, __COUNTER__), lhs, rexpr)
+
+#define JNI_ASSERT_OK_OR_THROW(expr) \
+ do { \
+ auto _res = (expr); \
+ arrow::Status _st = ::arrow::internal::GenericToStatus(_res); \
+ if (!_st.ok()) { \
+ env->ThrowNew(runtime_exception_class, _st.message().c_str()); \
+ } \
+ } while (false);
+
+jclass CreateGlobalClassReference(JNIEnv* env, const char* class_name) {
+ jclass local_class = env->FindClass(class_name);
+ jclass global_class = (jclass)env->NewGlobalRef(local_class);
+ env->DeleteLocalRef(local_class);
+ return global_class;
+}
+
+jmethodID GetMethodID(JNIEnv* env, jclass this_class, const char* name, const char* sig) {
+ jmethodID ret = env->GetMethodID(this_class, name, sig);
+ if (ret == nullptr) {
+ std::string error_message = "Unable to find method " + std::string(name) +
+ " within signature" + std::string(sig);
+ env->ThrowNew(illegal_access_exception_class, error_message.c_str());
+ }
+ return ret;
+}
+
+jint JNI_OnLoad(JavaVM* vm, void* reserved) {
+ JNIEnv* env;
+ if (vm->GetEnv(reinterpret_cast<void**>(&env), JNI_VERSION) != JNI_OK) {
+ return JNI_ERR;
+ }
+
+ illegal_access_exception_class =
+ CreateGlobalClassReference(env, "Ljava/lang/IllegalAccessException;");
+ illegal_argument_exception_class =
+ CreateGlobalClassReference(env, "Ljava/lang/IllegalArgumentException;");
+ runtime_exception_class =
+ CreateGlobalClassReference(env, "Ljava/lang/RuntimeException;");
+
+ record_batch_handle_class =
+ CreateGlobalClassReference(env, "Lorg/apache/arrow/dataset/jni/NativeRecordBatchHandle;");
+ record_batch_handle_field_class =
+ CreateGlobalClassReference(env, "Lorg/apache/arrow/dataset/jni/NativeRecordBatchHandle$Field;");
+ record_batch_handle_buffer_class =
+ CreateGlobalClassReference(env, "Lorg/apache/arrow/dataset/jni/NativeRecordBatchHandle$Buffer;");
+
+ record_batch_handle_constructor = GetMethodID(env, record_batch_handle_class, "<init>",
+ "(J[Lorg/apache/arrow/dataset/jni/NativeRecordBatchHandle$Field;"
+ "[Lorg/apache/arrow/dataset/jni/NativeRecordBatchHandle$Buffer;)V");
+ record_batch_handle_field_constructor = GetMethodID(env, record_batch_handle_field_class, "<init>",
+ "(JJ)V");
+ record_batch_handle_buffer_constructor = GetMethodID(env, record_batch_handle_buffer_class, "<init>",
+ "(JJJJ)V");
+
+ env->ExceptionDescribe();
+
+ return JNI_VERSION;
+}
+
+
+void JNI_OnUnload(JavaVM* vm, void* reserved) {
+ JNIEnv* env;
+ vm->GetEnv(reinterpret_cast<void**>(&env), JNI_VERSION);
+ env->DeleteGlobalRef(illegal_access_exception_class);
+ env->DeleteGlobalRef(illegal_argument_exception_class);
+ env->DeleteGlobalRef(runtime_exception_class);
+ env->DeleteGlobalRef(record_batch_handle_class);
+ env->DeleteGlobalRef(record_batch_handle_field_class);
+ env->DeleteGlobalRef(record_batch_handle_buffer_class);
+
+ dataset_factory_holder_.Clear();
+ dataset_holder_.Clear();
+ scan_task_holder_.Clear();
+ scanner_holder_.Clear();
+ iterator_holder_.Clear();
+ buffer_holder_.Clear();
+}
+
+std::shared_ptr<arrow::Schema> SchemaFromColumnNames(
+ const std::shared_ptr<arrow::Schema>& input, const std::vector<std::string>& column_names) {
+ std::vector<std::shared_ptr<arrow::Field>> columns;
+ for (const auto& name : column_names) {
+ columns.push_back(input->GetFieldByName(name));
+ }
+ return std::make_shared<arrow::Schema>(columns);
+}
+
+std::shared_ptr<arrow::dataset::FileFormat> GetFileFormat(JNIEnv *env, jint id) {
+ switch (id) {
+ case 0:
+ return std::make_shared<arrow::dataset::ParquetFileFormat>();
+ default:
+ std::string error_message = "illegal file format id: " + std::to_string(id);
+ env->ThrowNew(illegal_argument_exception_class, error_message.c_str());
+ return nullptr; // unreachable
+ }
+}
+
+std::shared_ptr<arrow::fs::FileSystem> GetFileSystem(JNIEnv *env, jint id, std::string path,
+ std::string* out_path) {
+ switch (id) {
+ case 0:
+ *out_path = path;
+ return std::make_shared<arrow::fs::LocalFileSystem>();
+ case 1: {
+ JNI_ASSIGN_OR_THROW(std::shared_ptr<arrow::fs::FileSystem> ret, arrow::fs::FileSystemFromUri(path, out_path))
+ return ret;
+ }
+ default:std::string error_message = "illegal filesystem id: " + std::to_string(id);
+ env->ThrowNew(illegal_argument_exception_class, error_message.c_str());
+ return nullptr; // unreachable
+ }
+}
+
+std::string JStringToCString(JNIEnv* env, jstring string) {
+ jboolean copied;
+ int32_t length = env->GetStringUTFLength(string);
+ const char *chars = env->GetStringUTFChars(string, &copied);
+ std::string str = std::string(chars, length);
+ // fixme calling ReleaseStringUTFChars if memory leak faced
+ return str;
+}
+
+std::vector<std::string> ToStringVector(JNIEnv* env, jobjectArray& str_array) {
+ int length = env->GetArrayLength(str_array);
+ std::vector<std::string> vector;
+ for (int i = 0; i < length; i++) {
+ auto string = (jstring) (env->GetObjectArrayElement(str_array, i));
+ vector.push_back(JStringToCString(env, string));
+ }
+ return vector;
+}
+
+template <typename T>
+std::vector<T> collect(JNIEnv* env, arrow::Iterator<T> itr) {
+ std::vector<T> vector;
+ while(true) {
+ JNI_ASSIGN_OR_THROW(T t, itr.Next())
+ if (!t) {
+ break;
+ }
+ vector.push_back(t);
+ }
+ return vector;
+}
+
+// FIXME: COPIED FROM intel/master on which this branch is not rebased yet
+// FIXME: https://github.com/Intel-bigdata/arrow/blob/02502a4eb59834c2471dd629e77dbeed19559f68/cpp/src/jni/jni_common.h#L239-L254
+jbyteArray ToSchemaByteArray(JNIEnv* env, std::shared_ptr<arrow::Schema> schema) {
+ JNI_ASSIGN_OR_THROW(std::shared_ptr<arrow::Buffer> buffer,
+ arrow::ipc::SerializeSchema(*schema.get(), nullptr, arrow::default_memory_pool()))
+
+ jbyteArray out = env->NewByteArray(buffer->size());
+ auto src = reinterpret_cast<const jbyte*>(buffer->data());
+ env->SetByteArrayRegion(out, 0, buffer->size(), src);
+ return out;
+}
+
+// FIXME: COPIED FROM intel/master on which this branch is not rebased yet
+// FIXME: https://github.com/Intel-bigdata/arrow/blob/02502a4eb59834c2471dd629e77dbeed19559f68/cpp/src/jni/jni_common.h#L256-L272
+std::shared_ptr<arrow::Schema> FromSchemaByteArray(JNIEnv* env, jbyteArray schemaBytes) {
+ arrow::ipc::DictionaryMemo in_memo;
+ int schemaBytes_len = env->GetArrayLength(schemaBytes);
+ jbyte* schemaBytes_data = env->GetByteArrayElements(schemaBytes, 0);
+ auto serialized_schema =
+ std::make_shared<arrow::Buffer>((uint8_t*)schemaBytes_data, schemaBytes_len);
+ arrow::io::BufferReader buf_reader(serialized_schema);
+ JNI_ASSIGN_OR_THROW(std::shared_ptr<arrow::Schema> schema, arrow::ipc::ReadSchema(&buf_reader, &in_memo))
+ env->ReleaseByteArrayElements(schemaBytes, schemaBytes_data, JNI_ABORT);
+ return schema;
+}
+
+bool ParseProtobuf(uint8_t* buf, int bufLen, google::protobuf::Message* msg) {
+ google::protobuf::io::CodedInputStream cis(buf, bufLen);
+ cis.SetRecursionLimit(1000);
+ return msg->ParseFromCodedStream(&cis);
+}
+
+void releaseFilterInput(jbyteArray condition_arr, jbyte* condition_bytes, JNIEnv* env) {
+ env->ReleaseByteArrayElements(condition_arr, condition_bytes, JNI_ABORT);
+}
+
+// fixme in development. Not all node types considered.
+std::shared_ptr<arrow::dataset::Expression> translateNode(types::TreeNode node, JNIEnv* env) {
+ if (node.has_fieldnode()) {
+ const types::FieldNode& f_node = node.fieldnode();
+ const std::string& name = f_node.name();
+ return std::make_shared<arrow::dataset::FieldExpression>(name);
+ }
+ if (node.has_intnode()) {
+ const types::IntNode& int_node = node.intnode();
+ int32_t val = int_node.value();
+ return std::make_shared<arrow::dataset::ScalarExpression>(std::make_shared<arrow::Int32Scalar>(val));
+ }
+ if (node.has_longnode()) {
+ const types::LongNode& long_node = node.longnode();
+ int64_t val = long_node.value();
+ return std::make_shared<arrow::dataset::ScalarExpression>(std::make_shared<arrow::Int64Scalar>(val));
+ }
+ if (node.has_floatnode()) {
+ const types::FloatNode& float_node = node.floatnode();
+ float_t val = float_node.value();
+ return std::make_shared<arrow::dataset::ScalarExpression>(std::make_shared<arrow::FloatScalar>(val));
+ }
+ if (node.has_doublenode()) {
+ const types::DoubleNode& double_node = node.doublenode();
+ double_t val = double_node.value();
+ return std::make_shared<arrow::dataset::ScalarExpression>(std::make_shared<arrow::DoubleScalar>(val));
+ }
+ if (node.has_booleannode()) {
+ const types::BooleanNode& boolean_node = node.booleannode();
+ bool val = boolean_node.value();
+ return std::make_shared<arrow::dataset::ScalarExpression>(std::make_shared<arrow::BooleanScalar>(val));
+ }
+ if (node.has_andnode()) {
+ const types::AndNode& and_node = node.andnode();
+ const types::TreeNode& left_arg = and_node.leftarg();
+ const types::TreeNode& right_arg = and_node.rightarg();
+ return std::make_shared<arrow::dataset::AndExpression>(translateNode(left_arg, env), translateNode(right_arg, env));
+ }
+ if (node.has_ornode()) {
+ const types::OrNode& or_node = node.ornode();
+ const types::TreeNode& left_arg = or_node.leftarg();
+ const types::TreeNode& right_arg = or_node.rightarg();
+ return std::make_shared<arrow::dataset::OrExpression>(translateNode(left_arg, env), translateNode(right_arg, env));
+ }
+ if (node.has_cpnode()) {
+ const types::ComparisonNode& cp_node = node.cpnode();
+ const std::string& op_name = cp_node.opname();
+ arrow::compute::CompareOperator op;
+ if (op_name == "equal") {
+ op = arrow::compute::CompareOperator::EQUAL;
+ } else if (op_name == "greaterThan") {
+ op = arrow::compute::CompareOperator::GREATER;
+ } else if (op_name == "greaterThanOrEqual") {
+ op = arrow::compute::CompareOperator::GREATER_EQUAL;
+ } else if (op_name == "lessThan") {
+ op = arrow::compute::CompareOperator::LESS;
+ } else if (op_name == "lessThanOrEqual") {
+ op = arrow::compute::CompareOperator::LESS_EQUAL;
+ } else {
+ std::string error_message = "Unknown operation name in comparison node";
+ env->ThrowNew(illegal_argument_exception_class, error_message.c_str());
+ return nullptr; // unreachable
+ }
+ const types::TreeNode& left_arg = cp_node.leftarg();
+ const types::TreeNode& right_arg = cp_node.rightarg();
+ return std::make_shared<arrow::dataset::ComparisonExpression>(op,
+ translateNode(left_arg, env), translateNode(right_arg, env));
+ }
+ if (node.has_notnode()) {
+ const types::NotNode& not_node = node.notnode();
+ const ::types::TreeNode& child = not_node.args();
+ std::shared_ptr<arrow::dataset::Expression> translatedChild = translateNode(child, env);
+ return std::make_shared<arrow::dataset::NotExpression>(translatedChild);
+ }
+ if (node.has_isvalidnode()) {
+ const types::IsValidNode& is_valid_node = node.isvalidnode();
+ const ::types::TreeNode& child = is_valid_node.args();
+ std::shared_ptr<arrow::dataset::Expression> translatedChild = translateNode(child, env);
+ return std::make_shared<arrow::dataset::IsValidExpression>(translatedChild);
+ }
+ std::string error_message = "Unknown node type";
+ env->ThrowNew(illegal_argument_exception_class, error_message.c_str());
+ return nullptr; // unreachable
+}
+
+std::shared_ptr<arrow::dataset::Expression> translateFilter(types::Condition condition, JNIEnv* env) {
Review comment:
style nit: I think Pure C++ methods should be TranslateFilter
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] emkornfield commented on pull request #7030: ARROW-7808: [Java][Dataset] Implement Datasets Java API by JNI to C++
Posted by GitBox <gi...@apache.org>.
emkornfield commented on pull request #7030:
URL: https://github.com/apache/arrow/pull/7030#issuecomment-643058410
@fsaintjacques do you have time to finish the review otherwise I can take a hopefully final look tomorrow.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] emkornfield commented on pull request #7030: ARROW-7808: [Java][Dataset] Implement Datasets Java API by JNI to C++
Posted by GitBox <gi...@apache.org>.
emkornfield commented on pull request #7030:
URL: https://github.com/apache/arrow/pull/7030#issuecomment-679355309
> Also I suppose if we want to consider block level memory reservation then we should dive into jemalloc and use some thing like a hook API right? That will appear to take me sometime as I am new to jemalloc :)
I think this might be overcomplicating things. I think a new MemoryPool implementation that takes a std::function of "reserve" that confirms that memory can be allocated before hand. For JVM purposes, the reserve method would have a preconfigured block size that it requests from Bits.java and it can then request bytes when it consumes its initial reservation.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] zhztheplayer commented on a change in pull request #7030: ARROW-7808: [Java][Dataset] Implement Datasets Java API by JNI to C++
Posted by GitBox <gi...@apache.org>.
zhztheplayer commented on a change in pull request #7030:
URL: https://github.com/apache/arrow/pull/7030#discussion_r415483479
##########
File path: cpp/src/jni/dataset/jni_wrapper.cpp
##########
@@ -0,0 +1,577 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include <arrow/dataset/file_base.h>
+#include <arrow/dataset/api.h>
+#include <arrow/filesystem/localfs.h>
+#include <arrow/ipc/api.h>
+#include <arrow/util/iterator.h>
+#include <arrow/filesystem/hdfs.h>
+#include <arrow/io/api.h>
+#include <google/protobuf/io/coded_stream.h>
+#include <google/protobuf/message.h>
+#include "concurrent_map.h"
+#include "arrow/compute/kernel.h"
+#include "arrow/compute/kernels/cast.h"
+#include "arrow/compute/kernels/compare.h"
+#include "jni/dataset/Types.pb.h"
+
+#include "org_apache_arrow_dataset_jni_JniWrapper.h"
+#include "org_apache_arrow_dataset_file_JniWrapper.h"
+
+static jclass illegal_access_exception_class;
+static jclass illegal_argument_exception_class;
+static jclass runtime_exception_class;
+
+static jclass record_batch_handle_class;
+static jclass record_batch_handle_field_class;
+static jclass record_batch_handle_buffer_class;
+
+static jmethodID record_batch_handle_constructor;
+static jmethodID record_batch_handle_field_constructor;
+static jmethodID record_batch_handle_buffer_constructor;
+
+static jint JNI_VERSION = JNI_VERSION_1_6;
+
+using arrow::jni::ConcurrentMap;
+
+static ConcurrentMap<std::shared_ptr<arrow::dataset::DatasetFactory>> dataset_factory_holder_;
Review comment:
Will do. But I kind of like the current pattern - it benefits more or less when debugging/tracing (self-growing id, instance cache, etc.), right? Is there any other consideration making it bad?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] zhztheplayer commented on pull request #7030: ARROW-7808: [Java][Dataset] Implement Dataset Java API by JNI to C++
Posted by GitBox <gi...@apache.org>.
zhztheplayer commented on pull request #7030:
URL: https://github.com/apache/arrow/pull/7030#issuecomment-727213403
Hi @emkornfield, It seems like CI failure was due to [ARROW-10499](https://issues.apache.org/jira/browse/ARROW-10499). I've rebased the PR onto newest master then the previous failure was gone, but there is still new ones which don't seem to be related to this PR (an error from flight_test.cc).
So, if assuming CI passed is it a good time for a re-review? I think it would be good to see that the feature can be shown in the next major release so that if I can do anything to push this forward please kindly let me know. Thanks!
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] emkornfield commented on a change in pull request #7030: ARROW-7808: [Java][Dataset] Implement Dataset Java API by JNI to C++
Posted by GitBox <gi...@apache.org>.
emkornfield commented on a change in pull request #7030:
URL: https://github.com/apache/arrow/pull/7030#discussion_r551610971
##########
File path: cpp/src/arrow/memory_pool.h
##########
@@ -149,6 +149,43 @@ class ARROW_EXPORT ProxyMemoryPool : public MemoryPool {
std::unique_ptr<ProxyMemoryPoolImpl> impl_;
};
+class ARROW_EXPORT ReservationListener {
+ public:
+ virtual ~ReservationListener();
+
+ virtual Status OnReservation(int64_t size) = 0;
+ virtual Status OnRelease(int64_t size) = 0;
+
+ protected:
+ ReservationListener();
+};
+
+class ARROW_EXPORT ReservationListenableMemoryPool : public MemoryPool {
+ public:
+ explicit ReservationListenableMemoryPool(MemoryPool* pool,
+ std::shared_ptr<ReservationListener> listener,
Review comment:
why shared ptr?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] zhztheplayer commented on a change in pull request #7030: ARROW-7808: [Java][Dataset] Implement Datasets Java API by JNI to C++
Posted by GitBox <gi...@apache.org>.
zhztheplayer commented on a change in pull request #7030:
URL: https://github.com/apache/arrow/pull/7030#discussion_r429552386
##########
File path: cpp/src/arrow/dataset/file_base.h
##########
@@ -290,5 +290,35 @@ class ARROW_DS_EXPORT WritePlan {
std::vector<std::string> paths;
};
+class ARROW_DS_EXPORT SingleFileDataset : public Dataset {
Review comment:
Commented above.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] zhztheplayer edited a comment on pull request #7030: ARROW-7808: [Java][Dataset] Implement Datasets Java API by JNI to C++
Posted by GitBox <gi...@apache.org>.
zhztheplayer edited a comment on pull request #7030:
URL: https://github.com/apache/arrow/pull/7030#issuecomment-678839259
@emkornfield Sorry for the late reply. And yes I was planing to try adding a `Bits.java` based implementation to this PR, but I may not be able to work on it instantly as I am occupied by some other projects these days. Also I suppose if we want to consider block level memory reservation then we should dive into jemalloc and use some thing like a hook API right? That will appear to take me sometime as I am new to jemalloc :)
If we'd like to see the feature in a release quickly maybe we can possibly resolve this one in advance and I could continue working on the above in a separated JIRA ticket. Either way is fine to me.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] wesm commented on a change in pull request #7030: ARROW-7808: [Java][Dataset] Implement Datasets Java API by JNI to C++
Posted by GitBox <gi...@apache.org>.
wesm commented on a change in pull request #7030:
URL: https://github.com/apache/arrow/pull/7030#discussion_r416638207
##########
File path: cpp/src/jni/dataset/proto/Types.proto
##########
@@ -0,0 +1,149 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+syntax = "proto2";
+package types;
+
+option java_package = "org.apache.arrow.dataset";
Review comment:
> As for filter, If I understand correctly it's OK to keep Java API but the JNI mapping for filters is considered fragile, right? I can remove the mapping anyway but when users read parquet files from Java they'll not be able to filter row groups to reduce I/O. Which is extremely important when low-selectivity filter is specified.
I'm advising you to define some kind of abstraction layer so that the _details_ of the C++ API are not exposed in the Java API. So if there are changes in the C++ API, then refactoring can take place only at the JNI C++ code and no changes will be required in Java.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] zhztheplayer commented on a change in pull request #7030: ARROW-7808: [Java][Dataset] Implement Datasets Java API
Posted by GitBox <gi...@apache.org>.
zhztheplayer commented on a change in pull request #7030:
URL: https://github.com/apache/arrow/pull/7030#discussion_r415256413
##########
File path: cpp/src/jni/dataset/concurrent_map.h
##########
@@ -0,0 +1,81 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ */
+
+#ifndef JNI_ID_TO_MODULE_MAP_H
+#define JNI_ID_TO_MODULE_MAP_H
+
+#include <memory>
+#include <mutex>
+#include <unordered_map>
+#include <utility>
+
+#include "jni.h"
+#include "arrow/util/macros.h"
+
+namespace arrow {
Review comment:
Maybe move concurrent_map.h out into `src/jni/`? By the way, should we somehow migrate `src/gandiva/jni` to `src/jni/gandiva`, or migrate `src/jni` to `src/arrow/jni`? It seems that currently they are organized inconsistently.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] zhztheplayer commented on a change in pull request #7030: ARROW-7808: [Java][Dataset] Implement Datasets Java API by JNI to C++
Posted by GitBox <gi...@apache.org>.
zhztheplayer commented on a change in pull request #7030:
URL: https://github.com/apache/arrow/pull/7030#discussion_r417789771
##########
File path: java/dataset/src/main/java/org/apache/arrow/dataset/source/DatasetFactory.java
##########
@@ -0,0 +1,34 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.arrow.dataset.source;
+
+import org.apache.arrow.vector.types.pojo.Schema;
+
+/**
+ * DataSourceDiscovery provides a way to inspect a DataSource potential
+ * schema before materializing it. Thus, the user can peek the schema for
+ * data sources and decide on a unified schema.
+ */
+public interface DatasetFactory {
+
+ Schema inspect();
+
Review comment:
added
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] emkornfield commented on a change in pull request #7030: ARROW-7808: [Java][Dataset] Implement Datasets Java API
Posted by GitBox <gi...@apache.org>.
emkornfield commented on a change in pull request #7030:
URL: https://github.com/apache/arrow/pull/7030#discussion_r414988755
##########
File path: java/dataset/src/test/java/org/apache/arrow/util/SchemaUtilsTest.java
##########
@@ -0,0 +1,50 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.arrow.util;
+
+import static java.util.Arrays.asList;
+
+import java.io.IOException;
+
+import org.apache.arrow.memory.RootAllocator;
+import org.apache.arrow.vector.types.pojo.ArrowType;
+import org.apache.arrow.vector.types.pojo.Field;
+import org.apache.arrow.vector.types.pojo.FieldType;
+import org.apache.arrow.vector.types.pojo.Schema;
+import org.junit.Assert;
+import org.junit.Test;
+
+public class SchemaUtilsTest {
+
+ private static Field field(String name, boolean nullable, ArrowType type, Field... children) {
+ return new Field(name, new FieldType(nullable, type, null, null), asList(children));
+ }
+
+ @Test
+ public void testSerializationAndDeserialization() throws IOException {
+ Schema schema = new Schema(asList(
+ field("a", false, new ArrowType.Null()),
+ field("b", true, new ArrowType.Utf8()),
+ field("c", true, new ArrowType.Binary()))
+ );
+
+ byte[] serialized = SchemaUtils.get().serialize(schema);
+ Schema deserialized = SchemaUtils.get().deserialize(serialized, new RootAllocator(Long.MAX_VALUE));
+ Assert.assertEquals(schema, deserialized);
Review comment:
generally, I think we static import assertEquals.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] zhztheplayer commented on a change in pull request #7030: ARROW-7808: [Java][Dataset] Implement Datasets Java API by JNI to C++
Posted by GitBox <gi...@apache.org>.
zhztheplayer commented on a change in pull request #7030:
URL: https://github.com/apache/arrow/pull/7030#discussion_r426348356
##########
File path: cpp/src/jni/dataset/proto/Types.proto
##########
@@ -0,0 +1,149 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+syntax = "proto2";
+package types;
+
+option java_package = "org.apache.arrow.dataset";
Review comment:
Thanks for your suggestion, but they don't seem like the same functionality to me for dataset expression and Gandiva expression... In C++ code they are designed and placed differently and we'll face problems merging the JNI abstraction together. e.g. arrow::dataset::FieldExpression doesn't retain data type information (it is only represented by field name) where gandiva::FieldNode is type sensitive... I don't think it's a good idea to use any workaround to make them suitable for each other. What do you think?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] zhztheplayer commented on a change in pull request #7030: ARROW-7808: [Java][Dataset] Implement Datasets Java API by JNI to C++
Posted by GitBox <gi...@apache.org>.
zhztheplayer commented on a change in pull request #7030:
URL: https://github.com/apache/arrow/pull/7030#discussion_r429717566
##########
File path: cpp/.gitignore
##########
@@ -24,6 +24,8 @@ cmake_install.cmake
build/
*-build/
Testing/
+cmake-build-debug/
Review comment:
done
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] zhztheplayer commented on pull request #7030: ARROW-7808: [Java][Dataset] Implement Datasets Java API by JNI to C++
Posted by GitBox <gi...@apache.org>.
zhztheplayer commented on pull request #7030:
URL: https://github.com/apache/arrow/pull/7030#issuecomment-651422684
Thanks for the comments! I've got some stuffs to deal with these days. Will address as soon as possible.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] zhztheplayer commented on pull request #7030: ARROW-7808: [Java][Dataset] Implement Datasets Java API by JNI to C++
Posted by GitBox <gi...@apache.org>.
zhztheplayer commented on pull request #7030:
URL: https://github.com/apache/arrow/pull/7030#issuecomment-653997609
@rymurr You are right, schemas are not frequently transferred over JNI so it's more straightforward to use byte arrays in development. I think your suggestion makes sense hence we can try optimize a bit on this behaviour (maybe in following tickets? not sure).
I'll remove the schema files and put schemas in code.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] zhztheplayer commented on a change in pull request #7030: ARROW-7808: [Java][Dataset] Implement Datasets Java API by JNI to C++
Posted by GitBox <gi...@apache.org>.
zhztheplayer commented on a change in pull request #7030:
URL: https://github.com/apache/arrow/pull/7030#discussion_r422738964
##########
File path: cpp/src/jni/dataset/jni_wrapper.cpp
##########
@@ -0,0 +1,577 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include <arrow/dataset/file_base.h>
+#include <arrow/dataset/api.h>
+#include <arrow/filesystem/localfs.h>
+#include <arrow/ipc/api.h>
+#include <arrow/util/iterator.h>
+#include <arrow/filesystem/hdfs.h>
+#include <arrow/io/api.h>
+#include <google/protobuf/io/coded_stream.h>
+#include <google/protobuf/message.h>
+#include "concurrent_map.h"
+#include "arrow/compute/kernel.h"
+#include "arrow/compute/kernels/cast.h"
+#include "arrow/compute/kernels/compare.h"
+#include "jni/dataset/Types.pb.h"
+
+#include "org_apache_arrow_dataset_jni_JniWrapper.h"
+#include "org_apache_arrow_dataset_file_JniWrapper.h"
+
+static jclass illegal_access_exception_class;
+static jclass illegal_argument_exception_class;
+static jclass runtime_exception_class;
+
+static jclass record_batch_handle_class;
+static jclass record_batch_handle_field_class;
+static jclass record_batch_handle_buffer_class;
+
+static jmethodID record_batch_handle_constructor;
+static jmethodID record_batch_handle_field_constructor;
+static jmethodID record_batch_handle_buffer_constructor;
+
+static jint JNI_VERSION = JNI_VERSION_1_6;
+
+using arrow::jni::ConcurrentMap;
+
+static ConcurrentMap<std::shared_ptr<arrow::dataset::DatasetFactory>> dataset_factory_holder_;
+static ConcurrentMap<std::shared_ptr<arrow::dataset::Dataset>> dataset_holder_;
+static ConcurrentMap<std::shared_ptr<arrow::dataset::ScanTask>> scan_task_holder_;
+static ConcurrentMap<std::shared_ptr<arrow::dataset::Scanner>> scanner_holder_;
+static ConcurrentMap<std::shared_ptr<arrow::RecordBatchIterator>> iterator_holder_;
+static ConcurrentMap<std::shared_ptr<arrow::Buffer>> buffer_holder_;
+
+#define JNI_ASSIGN_OR_THROW_NAME(x, y) ARROW_CONCAT(x, y)
+
+#define JNI_ASSIGN_OR_THROW_IMPL(t, lhs, rexpr) \
+ auto t = (rexpr); \
+ if (!t.status().ok()) { \
+ env->ThrowNew(runtime_exception_class, t.status().message().c_str()); \
+ } \
+ lhs = std::move(t).ValueOrDie();
+
+#define JNI_ASSIGN_OR_THROW(lhs, rexpr) \
+ JNI_ASSIGN_OR_THROW_IMPL(JNI_ASSIGN_OR_THROW_NAME(_tmp_var, __COUNTER__), lhs, rexpr)
+
+#define JNI_ASSERT_OK_OR_THROW(expr) \
+ do { \
+ auto _res = (expr); \
+ arrow::Status _st = ::arrow::internal::GenericToStatus(_res); \
+ if (!_st.ok()) { \
+ env->ThrowNew(runtime_exception_class, _st.message().c_str()); \
+ } \
+ } while (false);
+
+jclass CreateGlobalClassReference(JNIEnv* env, const char* class_name) {
+ jclass local_class = env->FindClass(class_name);
+ jclass global_class = (jclass)env->NewGlobalRef(local_class);
+ env->DeleteLocalRef(local_class);
+ return global_class;
+}
+
+jmethodID GetMethodID(JNIEnv* env, jclass this_class, const char* name, const char* sig) {
+ jmethodID ret = env->GetMethodID(this_class, name, sig);
+ if (ret == nullptr) {
+ std::string error_message = "Unable to find method " + std::string(name) +
+ " within signature" + std::string(sig);
+ env->ThrowNew(illegal_access_exception_class, error_message.c_str());
+ }
+ return ret;
+}
+
+jint JNI_OnLoad(JavaVM* vm, void* reserved) {
+ JNIEnv* env;
+ if (vm->GetEnv(reinterpret_cast<void**>(&env), JNI_VERSION) != JNI_OK) {
+ return JNI_ERR;
+ }
+
+ illegal_access_exception_class =
+ CreateGlobalClassReference(env, "Ljava/lang/IllegalAccessException;");
+ illegal_argument_exception_class =
+ CreateGlobalClassReference(env, "Ljava/lang/IllegalArgumentException;");
+ runtime_exception_class =
+ CreateGlobalClassReference(env, "Ljava/lang/RuntimeException;");
+
+ record_batch_handle_class =
+ CreateGlobalClassReference(env, "Lorg/apache/arrow/dataset/jni/NativeRecordBatchHandle;");
+ record_batch_handle_field_class =
+ CreateGlobalClassReference(env, "Lorg/apache/arrow/dataset/jni/NativeRecordBatchHandle$Field;");
+ record_batch_handle_buffer_class =
+ CreateGlobalClassReference(env, "Lorg/apache/arrow/dataset/jni/NativeRecordBatchHandle$Buffer;");
+
+ record_batch_handle_constructor = GetMethodID(env, record_batch_handle_class, "<init>",
+ "(J[Lorg/apache/arrow/dataset/jni/NativeRecordBatchHandle$Field;"
+ "[Lorg/apache/arrow/dataset/jni/NativeRecordBatchHandle$Buffer;)V");
+ record_batch_handle_field_constructor = GetMethodID(env, record_batch_handle_field_class, "<init>",
+ "(JJ)V");
+ record_batch_handle_buffer_constructor = GetMethodID(env, record_batch_handle_buffer_class, "<init>",
+ "(JJJJ)V");
+
+ env->ExceptionDescribe();
+
+ return JNI_VERSION;
+}
+
+
+void JNI_OnUnload(JavaVM* vm, void* reserved) {
+ JNIEnv* env;
+ vm->GetEnv(reinterpret_cast<void**>(&env), JNI_VERSION);
+ env->DeleteGlobalRef(illegal_access_exception_class);
+ env->DeleteGlobalRef(illegal_argument_exception_class);
+ env->DeleteGlobalRef(runtime_exception_class);
+ env->DeleteGlobalRef(record_batch_handle_class);
+ env->DeleteGlobalRef(record_batch_handle_field_class);
+ env->DeleteGlobalRef(record_batch_handle_buffer_class);
+
+ dataset_factory_holder_.Clear();
+ dataset_holder_.Clear();
+ scan_task_holder_.Clear();
+ scanner_holder_.Clear();
+ iterator_holder_.Clear();
+ buffer_holder_.Clear();
+}
+
+std::shared_ptr<arrow::Schema> SchemaFromColumnNames(
+ const std::shared_ptr<arrow::Schema>& input, const std::vector<std::string>& column_names) {
+ std::vector<std::shared_ptr<arrow::Field>> columns;
+ for (const auto& name : column_names) {
+ columns.push_back(input->GetFieldByName(name));
+ }
+ return std::make_shared<arrow::Schema>(columns);
+}
+
+std::shared_ptr<arrow::dataset::FileFormat> GetFileFormat(JNIEnv *env, jint id) {
+ switch (id) {
+ case 0:
+ return std::make_shared<arrow::dataset::ParquetFileFormat>();
+ default:
+ std::string error_message = "illegal file format id: " + std::to_string(id);
+ env->ThrowNew(illegal_argument_exception_class, error_message.c_str());
+ return nullptr; // unreachable
+ }
+}
+
+std::shared_ptr<arrow::fs::FileSystem> GetFileSystem(JNIEnv *env, jint id, std::string path,
+ std::string* out_path) {
+ switch (id) {
+ case 0:
+ *out_path = path;
+ return std::make_shared<arrow::fs::LocalFileSystem>();
+ case 1: {
+ JNI_ASSIGN_OR_THROW(std::shared_ptr<arrow::fs::FileSystem> ret, arrow::fs::FileSystemFromUri(path, out_path))
+ return ret;
+ }
+ default:std::string error_message = "illegal filesystem id: " + std::to_string(id);
+ env->ThrowNew(illegal_argument_exception_class, error_message.c_str());
+ return nullptr; // unreachable
+ }
+}
+
+std::string JStringToCString(JNIEnv* env, jstring string) {
+ jboolean copied;
+ int32_t length = env->GetStringUTFLength(string);
+ const char *chars = env->GetStringUTFChars(string, &copied);
+ std::string str = std::string(chars, length);
+ // fixme calling ReleaseStringUTFChars if memory leak faced
+ return str;
+}
+
+std::vector<std::string> ToStringVector(JNIEnv* env, jobjectArray& str_array) {
+ int length = env->GetArrayLength(str_array);
+ std::vector<std::string> vector;
+ for (int i = 0; i < length; i++) {
+ auto string = (jstring) (env->GetObjectArrayElement(str_array, i));
+ vector.push_back(JStringToCString(env, string));
+ }
+ return vector;
+}
+
+template <typename T>
+std::vector<T> collect(JNIEnv* env, arrow::Iterator<T> itr) {
+ std::vector<T> vector;
+ while(true) {
+ JNI_ASSIGN_OR_THROW(T t, itr.Next())
+ if (!t) {
+ break;
+ }
+ vector.push_back(t);
+ }
+ return vector;
+}
+
+// FIXME: COPIED FROM intel/master on which this branch is not rebased yet
+// FIXME: https://github.com/Intel-bigdata/arrow/blob/02502a4eb59834c2471dd629e77dbeed19559f68/cpp/src/jni/jni_common.h#L239-L254
+jbyteArray ToSchemaByteArray(JNIEnv* env, std::shared_ptr<arrow::Schema> schema) {
+ JNI_ASSIGN_OR_THROW(std::shared_ptr<arrow::Buffer> buffer,
+ arrow::ipc::SerializeSchema(*schema.get(), nullptr, arrow::default_memory_pool()))
+
+ jbyteArray out = env->NewByteArray(buffer->size());
+ auto src = reinterpret_cast<const jbyte*>(buffer->data());
+ env->SetByteArrayRegion(out, 0, buffer->size(), src);
+ return out;
+}
+
+// FIXME: COPIED FROM intel/master on which this branch is not rebased yet
Review comment:
ok. fixmes like this one is by my mistake.
##########
File path: cpp/src/jni/dataset/jni_wrapper.cpp
##########
@@ -0,0 +1,577 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include <arrow/dataset/file_base.h>
+#include <arrow/dataset/api.h>
+#include <arrow/filesystem/localfs.h>
+#include <arrow/ipc/api.h>
+#include <arrow/util/iterator.h>
+#include <arrow/filesystem/hdfs.h>
+#include <arrow/io/api.h>
+#include <google/protobuf/io/coded_stream.h>
+#include <google/protobuf/message.h>
+#include "concurrent_map.h"
+#include "arrow/compute/kernel.h"
+#include "arrow/compute/kernels/cast.h"
+#include "arrow/compute/kernels/compare.h"
+#include "jni/dataset/Types.pb.h"
+
+#include "org_apache_arrow_dataset_jni_JniWrapper.h"
+#include "org_apache_arrow_dataset_file_JniWrapper.h"
+
+static jclass illegal_access_exception_class;
+static jclass illegal_argument_exception_class;
+static jclass runtime_exception_class;
+
+static jclass record_batch_handle_class;
+static jclass record_batch_handle_field_class;
+static jclass record_batch_handle_buffer_class;
+
+static jmethodID record_batch_handle_constructor;
+static jmethodID record_batch_handle_field_constructor;
+static jmethodID record_batch_handle_buffer_constructor;
+
+static jint JNI_VERSION = JNI_VERSION_1_6;
+
+using arrow::jni::ConcurrentMap;
+
+static ConcurrentMap<std::shared_ptr<arrow::dataset::DatasetFactory>> dataset_factory_holder_;
+static ConcurrentMap<std::shared_ptr<arrow::dataset::Dataset>> dataset_holder_;
+static ConcurrentMap<std::shared_ptr<arrow::dataset::ScanTask>> scan_task_holder_;
+static ConcurrentMap<std::shared_ptr<arrow::dataset::Scanner>> scanner_holder_;
+static ConcurrentMap<std::shared_ptr<arrow::RecordBatchIterator>> iterator_holder_;
+static ConcurrentMap<std::shared_ptr<arrow::Buffer>> buffer_holder_;
+
+#define JNI_ASSIGN_OR_THROW_NAME(x, y) ARROW_CONCAT(x, y)
+
+#define JNI_ASSIGN_OR_THROW_IMPL(t, lhs, rexpr) \
+ auto t = (rexpr); \
+ if (!t.status().ok()) { \
+ env->ThrowNew(runtime_exception_class, t.status().message().c_str()); \
+ } \
+ lhs = std::move(t).ValueOrDie();
+
+#define JNI_ASSIGN_OR_THROW(lhs, rexpr) \
+ JNI_ASSIGN_OR_THROW_IMPL(JNI_ASSIGN_OR_THROW_NAME(_tmp_var, __COUNTER__), lhs, rexpr)
+
+#define JNI_ASSERT_OK_OR_THROW(expr) \
+ do { \
+ auto _res = (expr); \
+ arrow::Status _st = ::arrow::internal::GenericToStatus(_res); \
+ if (!_st.ok()) { \
+ env->ThrowNew(runtime_exception_class, _st.message().c_str()); \
+ } \
+ } while (false);
+
+jclass CreateGlobalClassReference(JNIEnv* env, const char* class_name) {
+ jclass local_class = env->FindClass(class_name);
+ jclass global_class = (jclass)env->NewGlobalRef(local_class);
+ env->DeleteLocalRef(local_class);
+ return global_class;
+}
+
+jmethodID GetMethodID(JNIEnv* env, jclass this_class, const char* name, const char* sig) {
+ jmethodID ret = env->GetMethodID(this_class, name, sig);
+ if (ret == nullptr) {
+ std::string error_message = "Unable to find method " + std::string(name) +
+ " within signature" + std::string(sig);
+ env->ThrowNew(illegal_access_exception_class, error_message.c_str());
+ }
+ return ret;
+}
+
+jint JNI_OnLoad(JavaVM* vm, void* reserved) {
+ JNIEnv* env;
+ if (vm->GetEnv(reinterpret_cast<void**>(&env), JNI_VERSION) != JNI_OK) {
+ return JNI_ERR;
+ }
+
+ illegal_access_exception_class =
+ CreateGlobalClassReference(env, "Ljava/lang/IllegalAccessException;");
+ illegal_argument_exception_class =
+ CreateGlobalClassReference(env, "Ljava/lang/IllegalArgumentException;");
+ runtime_exception_class =
+ CreateGlobalClassReference(env, "Ljava/lang/RuntimeException;");
+
+ record_batch_handle_class =
+ CreateGlobalClassReference(env, "Lorg/apache/arrow/dataset/jni/NativeRecordBatchHandle;");
+ record_batch_handle_field_class =
+ CreateGlobalClassReference(env, "Lorg/apache/arrow/dataset/jni/NativeRecordBatchHandle$Field;");
+ record_batch_handle_buffer_class =
+ CreateGlobalClassReference(env, "Lorg/apache/arrow/dataset/jni/NativeRecordBatchHandle$Buffer;");
+
+ record_batch_handle_constructor = GetMethodID(env, record_batch_handle_class, "<init>",
+ "(J[Lorg/apache/arrow/dataset/jni/NativeRecordBatchHandle$Field;"
+ "[Lorg/apache/arrow/dataset/jni/NativeRecordBatchHandle$Buffer;)V");
+ record_batch_handle_field_constructor = GetMethodID(env, record_batch_handle_field_class, "<init>",
+ "(JJ)V");
+ record_batch_handle_buffer_constructor = GetMethodID(env, record_batch_handle_buffer_class, "<init>",
+ "(JJJJ)V");
+
+ env->ExceptionDescribe();
+
+ return JNI_VERSION;
+}
+
+
+void JNI_OnUnload(JavaVM* vm, void* reserved) {
+ JNIEnv* env;
+ vm->GetEnv(reinterpret_cast<void**>(&env), JNI_VERSION);
+ env->DeleteGlobalRef(illegal_access_exception_class);
+ env->DeleteGlobalRef(illegal_argument_exception_class);
+ env->DeleteGlobalRef(runtime_exception_class);
+ env->DeleteGlobalRef(record_batch_handle_class);
+ env->DeleteGlobalRef(record_batch_handle_field_class);
+ env->DeleteGlobalRef(record_batch_handle_buffer_class);
+
+ dataset_factory_holder_.Clear();
+ dataset_holder_.Clear();
+ scan_task_holder_.Clear();
+ scanner_holder_.Clear();
+ iterator_holder_.Clear();
+ buffer_holder_.Clear();
+}
+
+std::shared_ptr<arrow::Schema> SchemaFromColumnNames(
+ const std::shared_ptr<arrow::Schema>& input, const std::vector<std::string>& column_names) {
+ std::vector<std::shared_ptr<arrow::Field>> columns;
+ for (const auto& name : column_names) {
+ columns.push_back(input->GetFieldByName(name));
+ }
+ return std::make_shared<arrow::Schema>(columns);
+}
+
+std::shared_ptr<arrow::dataset::FileFormat> GetFileFormat(JNIEnv *env, jint id) {
+ switch (id) {
+ case 0:
+ return std::make_shared<arrow::dataset::ParquetFileFormat>();
+ default:
+ std::string error_message = "illegal file format id: " + std::to_string(id);
+ env->ThrowNew(illegal_argument_exception_class, error_message.c_str());
+ return nullptr; // unreachable
+ }
+}
+
+std::shared_ptr<arrow::fs::FileSystem> GetFileSystem(JNIEnv *env, jint id, std::string path,
+ std::string* out_path) {
+ switch (id) {
+ case 0:
+ *out_path = path;
+ return std::make_shared<arrow::fs::LocalFileSystem>();
+ case 1: {
+ JNI_ASSIGN_OR_THROW(std::shared_ptr<arrow::fs::FileSystem> ret, arrow::fs::FileSystemFromUri(path, out_path))
+ return ret;
+ }
+ default:std::string error_message = "illegal filesystem id: " + std::to_string(id);
+ env->ThrowNew(illegal_argument_exception_class, error_message.c_str());
+ return nullptr; // unreachable
+ }
+}
+
+std::string JStringToCString(JNIEnv* env, jstring string) {
+ jboolean copied;
+ int32_t length = env->GetStringUTFLength(string);
+ const char *chars = env->GetStringUTFChars(string, &copied);
+ std::string str = std::string(chars, length);
+ // fixme calling ReleaseStringUTFChars if memory leak faced
+ return str;
+}
+
+std::vector<std::string> ToStringVector(JNIEnv* env, jobjectArray& str_array) {
+ int length = env->GetArrayLength(str_array);
+ std::vector<std::string> vector;
+ for (int i = 0; i < length; i++) {
+ auto string = (jstring) (env->GetObjectArrayElement(str_array, i));
+ vector.push_back(JStringToCString(env, string));
+ }
+ return vector;
+}
+
+template <typename T>
+std::vector<T> collect(JNIEnv* env, arrow::Iterator<T> itr) {
+ std::vector<T> vector;
+ while(true) {
+ JNI_ASSIGN_OR_THROW(T t, itr.Next())
+ if (!t) {
+ break;
+ }
+ vector.push_back(t);
+ }
+ return vector;
+}
+
+// FIXME: COPIED FROM intel/master on which this branch is not rebased yet
+// FIXME: https://github.com/Intel-bigdata/arrow/blob/02502a4eb59834c2471dd629e77dbeed19559f68/cpp/src/jni/jni_common.h#L239-L254
+jbyteArray ToSchemaByteArray(JNIEnv* env, std::shared_ptr<arrow::Schema> schema) {
+ JNI_ASSIGN_OR_THROW(std::shared_ptr<arrow::Buffer> buffer,
+ arrow::ipc::SerializeSchema(*schema.get(), nullptr, arrow::default_memory_pool()))
+
+ jbyteArray out = env->NewByteArray(buffer->size());
+ auto src = reinterpret_cast<const jbyte*>(buffer->data());
+ env->SetByteArrayRegion(out, 0, buffer->size(), src);
+ return out;
+}
+
+// FIXME: COPIED FROM intel/master on which this branch is not rebased yet
Review comment:
ok. fixmes like this one are by my mistake.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] Sebastiaan-Alvarez-Rodriguez edited a comment on pull request #7030: ARROW-7808: [Java][Dataset] Implement Datasets Java API by JNI to C++
Posted by GitBox <gi...@apache.org>.
Sebastiaan-Alvarez-Rodriguez edited a comment on pull request #7030:
URL: https://github.com/apache/arrow/pull/7030#issuecomment-694925466
hi there,
Could you please add a few compilation/installation instructions to your branch?
I just completed a long debug session to find out why the JNI runtime did not work.
### CPP
Compiled `<project-root>/cpp/` project using:
```bash
cd <project-root>/cpp
cmake -DARROW_DATASET=ON -DARROW_JNI=ON -DARROW_PARQUET=ON -DARROW_IPC=ON .
sudo make install
```
Confirmed shared libs are indeed visible using `whereis libarrow_dataset_jni`.
### Java
Compiled `<project-root>/java/dataset/` using
```bash
cd <project-root>/java/dataset
mvn clean install -Dmaven.test.skip=true # skipping tests to let build succeed
```
To test whether it works, successfully ran a sample program:
```java
public class test {
public static void main(String[] args) {
System.loadLibrary("arrow_dataset_jni");
System.out.println("System loading works fine");
}
}
```
So, my sample program's classloader can access `libarrow_dataset_jni.so` just fine.
However, the following does not work:
```java
import org.apache.arrow.memory.RootAllocator;
import org.apache.arrow.dataset.file.FileFormat;
import org.apache.arrow.dataset.file.FileSystemDatasetFactory;
public class test {
private static FileSystemDatasetFactory getDatasetFactory() {
RootAllocator allocator = new RootAllocator(Long.MAX_VALUE);
return new FileSystemDatasetFactory(allocator, FileFormat.PARQUET, "/path/to/pq.parquet"); //crash
}
public static void main(String[] args) {
System.loadLibrary("arrow_dataset_jni");
System.out.println("Own classLoader works fine");
FileSystemDatasetFactory test = getDatasetFactory();
System.out.println("Not so good! I crash before I can print this");
}
}
```
The relevant part of the crash is:
```
Caused by: java.lang.IllegalStateException: error loading native libraries: java.io.FileNotFoundException: libarrow_dataset_jni.so
at org.apache.arrow.dataset.jni.JniLoader.load(JniLoader.java:91)
at org.apache.arrow.dataset.jni.JniLoader.loadRemaining(JniLoader.java:73)
at org.apache.arrow.dataset.jni.JniLoader.ensureLoaded(JniLoader.java:60)
at org.apache.arrow.dataset.file.JniWrapper.<init>(JniWrapper.java:34)
```
The source code of the Dataset Java library causing the error is found [here](https://github.com/zhztheplayer/arrow-1/blob/fd163e199a10a7225765b0c30fbf60d8df8d20db/java/dataset/src/main/java/org/apache/arrow/dataset/jni/JniLoader.java#L78-L94).
The problem occurs because `JniWrapper.class.getClassLoader().getResourceAsStream(libraryToLoad)` always returns `null`.
Only when adding a symlink (`ln -s /usr/local/lib/libarrow_dataset_jni.so libarrow_dataset_jni.so`) to the package root directory, it is able to find the correct file.
Why do you create a temporary file, store the contents of the shared library in it, and then call `System.load(tmpfile)` and not just directly call it using `System.loadLibrary(libraryToLoad);`?
Maybe it is a good idea to pick one of the following:
1. Change the `'java.io.FileNotFoundException: libarrow_dataset_jni.so'` to display the full path (so people know where your library looks to find the shared object)
2. Make a small `JNI_dataset_dev_install_note.md` somewhere on getting this library to work.
3. Use `System.loadLibrary(libraryToLoad);`
4. Use some other way to find the location of a shared library and copy it to a tmpfile
I believe it would help many people who are not familiar with `getClassLoader().getResourceAsStream(...)` (like myself)
Have a nice day,
Sebastiaan
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] zhztheplayer commented on pull request #7030: ARROW-7808: [Java][Dataset] Implement Dataset Java API by JNI to C++
Posted by GitBox <gi...@apache.org>.
zhztheplayer commented on pull request #7030:
URL: https://github.com/apache/arrow/pull/7030#issuecomment-755902629
@emkornfield And would you mind to take another look on latest code? Thanks a lot.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] wesm commented on a change in pull request #7030: ARROW-7808: [Java][Dataset] Implement Datasets Java API by JNI to C++
Posted by GitBox <gi...@apache.org>.
wesm commented on a change in pull request #7030:
URL: https://github.com/apache/arrow/pull/7030#discussion_r416073397
##########
File path: cpp/src/jni/dataset/proto/Types.proto
##########
@@ -0,0 +1,149 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+syntax = "proto2";
+package types;
+
+option java_package = "org.apache.arrow.dataset";
Review comment:
I think it's fine if you have a filter API in Java, but I think you should try to limit the exposure of the particular details of the current C++ API in Java. If the Java API has a 1-1 correspondence with the C++ API that's what I'm advising you against
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] zhztheplayer commented on a change in pull request #7030: ARROW-7808: [Java][Dataset] Implement Datasets Java API by JNI to C++
Posted by GitBox <gi...@apache.org>.
zhztheplayer commented on a change in pull request #7030:
URL: https://github.com/apache/arrow/pull/7030#discussion_r440075959
##########
File path: java/dataset/src/main/java/org/apache/arrow/memory/NativeUnderlingMemory.java
##########
@@ -0,0 +1,65 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.arrow.memory;
+
+import org.apache.arrow.dataset.jni.JniWrapper;
+
+/**
+ * AllocationManager implementation for Native allocated memory.
+ */
+public class NativeUnderlingMemory extends AllocationManager {
Review comment:
Hi @jacques-n,
Thanks for your suggestion but I think the "weaknesses" of current implementation might be somehow overrated. Specifically for orc adaptor it's of course urgent to improve its memory allocation process because all of the allocated memory by orc is escaping from Java allocator. However we don't have the same problem here in Datasets as allocator is going to manage all the native memory. The only issue we may have is that OOM error might be thrown after a buffer is allocated rather than before we create it. But because OOM will be finally thrown and over-allocated memory will be finally released, the impact will be very very small.
And I knew we can possibly change to allocate Java memory directly from C++ via JNI, but codes will become messy and performance will turn to worse. That is why I didn't propose such a solution in this PR.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] zhztheplayer edited a comment on pull request #7030: ARROW-7808: [Java][Dataset] Implement Dataset Java API by JNI to C++
Posted by GitBox <gi...@apache.org>.
zhztheplayer edited a comment on pull request #7030:
URL: https://github.com/apache/arrow/pull/7030#issuecomment-766616108
Hi @emkornfield @pitrou
I think I have addressed existing comments so would you like to take another look? Thanks a lot.
Based on some recent comments I think we may don't have to make 100% consensus on the current design of `ReservationListenableMemoryPool` or relevant things instantly, if this part is in a way controversial I could split out the part from this PR then create a new one for it if we can possibly get this merged.
And if you have other suggestions on the whole design or architecture of the basic functionality please be free to add comments and I am happy to make further refactors accordingly.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] emkornfield commented on a change in pull request #7030: ARROW-7808: [Java][Dataset] Implement Datasets Java API
Posted by GitBox <gi...@apache.org>.
emkornfield commented on a change in pull request #7030:
URL: https://github.com/apache/arrow/pull/7030#discussion_r414985230
##########
File path: cpp/src/jni/dataset/jni_wrapper.cpp
##########
@@ -0,0 +1,577 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include <arrow/dataset/file_base.h>
+#include <arrow/dataset/api.h>
+#include <arrow/filesystem/localfs.h>
+#include <arrow/ipc/api.h>
+#include <arrow/util/iterator.h>
+#include <arrow/filesystem/hdfs.h>
+#include <arrow/io/api.h>
+#include <google/protobuf/io/coded_stream.h>
+#include <google/protobuf/message.h>
+#include "concurrent_map.h"
+#include "arrow/compute/kernel.h"
+#include "arrow/compute/kernels/cast.h"
+#include "arrow/compute/kernels/compare.h"
+#include "jni/dataset/Types.pb.h"
+
+#include "org_apache_arrow_dataset_jni_JniWrapper.h"
+#include "org_apache_arrow_dataset_file_JniWrapper.h"
+
+static jclass illegal_access_exception_class;
+static jclass illegal_argument_exception_class;
+static jclass runtime_exception_class;
+
+static jclass record_batch_handle_class;
+static jclass record_batch_handle_field_class;
+static jclass record_batch_handle_buffer_class;
+
+static jmethodID record_batch_handle_constructor;
+static jmethodID record_batch_handle_field_constructor;
+static jmethodID record_batch_handle_buffer_constructor;
+
+static jint JNI_VERSION = JNI_VERSION_1_6;
+
+using arrow::jni::ConcurrentMap;
+
+static ConcurrentMap<std::shared_ptr<arrow::dataset::DatasetFactory>> dataset_factory_holder_;
Review comment:
this still strikes me as a bad pattern.
https://stackoverflow.com/questions/26375215/c-shared-ptr-and-java-native-object-ownership/26377515 seems better.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] zhztheplayer commented on a change in pull request #7030: ARROW-7808: [Java][Dataset] Implement Datasets Java API by JNI to C++
Posted by GitBox <gi...@apache.org>.
zhztheplayer commented on a change in pull request #7030:
URL: https://github.com/apache/arrow/pull/7030#discussion_r416295541
##########
File path: cpp/src/jni/dataset/proto/Types.proto
##########
@@ -0,0 +1,149 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+syntax = "proto2";
+package types;
+
+option java_package = "org.apache.arrow.dataset";
Review comment:
Thanks, as I knew some of the base design purposes of datasets API, dataframe API, and query engine system, so I think I would understand any of the concerns against a too detailed mapping. I'll try remove more JNI stuffs.
As for filter, If I understand correctly it's OK to keep Java API but the JNI mapping for filters is considered fragile, right? I can remove the mapping anyway but when users read parquet files from Java they'll not be able to filter row groups to reduce I/O. Which is extremely important when low-selectivity filters is specified.
##########
File path: cpp/src/jni/dataset/proto/Types.proto
##########
@@ -0,0 +1,149 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+syntax = "proto2";
+package types;
+
+option java_package = "org.apache.arrow.dataset";
Review comment:
Thanks, as I knew some of the base design purposes of datasets API, dataframe API, and query engine system, so I think I would understand any of the concerns against a too detailed mapping. I'll try remove more JNI stuffs.
As for filter, If I understand correctly it's OK to keep Java API but the JNI mapping for filters is considered fragile, right? I can remove the mapping anyway but when users read parquet files from Java they'll not be able to filter row groups to reduce I/O. Which is extremely important when low-selectivity filter is specified.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] wesm commented on pull request #7030: ARROW-7808: [Java][Dataset] Implement Datasets Java API by JNI to C++
Posted by GitBox <gi...@apache.org>.
wesm commented on pull request #7030:
URL: https://github.com/apache/arrow/pull/7030#issuecomment-643020617
@fsaintjacques @emkornfield ping on this, could we get this into the next release
@zhztheplayer the CI failed, could you investigate?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] fsaintjacques commented on a change in pull request #7030: ARROW-7808: [Java][Dataset] Implement Datasets Java API by JNI to C++
Posted by GitBox <gi...@apache.org>.
fsaintjacques commented on a change in pull request #7030:
URL: https://github.com/apache/arrow/pull/7030#discussion_r439413765
##########
File path: cpp/src/arrow/dataset/discovery.h
##########
@@ -216,6 +216,16 @@ class ARROW_DS_EXPORT FileSystemDatasetFactory : public DatasetFactory {
std::shared_ptr<fs::FileSystem> filesystem, fs::FileSelector selector,
std::shared_ptr<FileFormat> format, FileSystemFactoryOptions options);
+ /// \brief Build a FileSystemDatasetFactory from an uri including filesystem
+ /// information.
+ ///
+ /// \param[in] uri passed to FileSystemDataset
+ /// \param[in] format passed to FileSystemDataset
+ /// \param[in] options see FileSystemFactoryOptions for more information.
+ static Result<std::shared_ptr<DatasetFactory>> Make(std::string uri,
Review comment:
:+1:
##########
File path: java/dataset/src/main/java/org/apache/arrow/dataset/jni/NativeScanner.java
##########
@@ -0,0 +1,143 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.arrow.dataset.jni;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.NoSuchElementException;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.stream.Collectors;
+
+import org.apache.arrow.dataset.scanner.ScanTask;
+import org.apache.arrow.dataset.scanner.Scanner;
+import org.apache.arrow.memory.ArrowBuf;
+import org.apache.arrow.memory.BufferAllocator;
+import org.apache.arrow.memory.BufferLedger;
+import org.apache.arrow.memory.NativeUnderlingMemory;
+import org.apache.arrow.memory.Ownerships;
+import org.apache.arrow.vector.ipc.message.ArrowFieldNode;
+import org.apache.arrow.vector.ipc.message.ArrowRecordBatch;
+import org.apache.arrow.vector.types.pojo.Schema;
+import org.apache.arrow.vector.util.SchemaUtility;
+
+/**
+ * Native implementation of {@link Scanner}. Note that it currently emits only a single scan task of type
+ * {@link NativeScanTask}, which is internally a combination of all scan task instances returned by the
+ * native scanner.
+ */
+public class NativeScanner implements Scanner {
+
+ private final AtomicBoolean closed = new AtomicBoolean(false);
+ private final AtomicBoolean executed = new AtomicBoolean(false);
+ private final NativeContext context;
+ private final long scannerId;
+
+ public NativeScanner(NativeContext context, long scannerId) {
+ this.context = context;
+ this.scannerId = scannerId;
+ }
+
+ ScanTask.BatchIterator execute() {
+ if (closed.get()) {
+ throw new NativeInstanceClosedException();
+ }
+ if (!executed.compareAndSet(false, true)) {
+ throw new UnsupportedOperationException("NativeScanner cannot be executed more than once. Consider creating " +
+ "new scanner instead");
+ }
+ return new ScanTask.BatchIterator() {
+ private ArrowRecordBatch peek = null;
+
+ @Override
+ public void close() {
+ NativeScanner.this.close();
+ }
+
+ @Override
+ public boolean hasNext() {
+ if (closed.get()) {
Review comment:
What's the point of checking close before the method if any concurrent thread can come and close said object while this method is running? In other words, this doesn't protect against "use-after-close". Since `close()` will delete the `shared_ptr` underneath, you're very open to user-after-free in C++.
I think you need to protect with close state with a read-write-mutex, or document that the Scanner is single threaded only, which it is in C++. The important part is that ScanTasks can be dispatched concurrently.
##########
File path: java/dataset/src/main/java/org/apache/arrow/dataset/jni/NativeScanner.java
##########
@@ -0,0 +1,143 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.arrow.dataset.jni;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.NoSuchElementException;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.stream.Collectors;
+
+import org.apache.arrow.dataset.scanner.ScanTask;
+import org.apache.arrow.dataset.scanner.Scanner;
+import org.apache.arrow.memory.ArrowBuf;
+import org.apache.arrow.memory.BufferAllocator;
+import org.apache.arrow.memory.BufferLedger;
+import org.apache.arrow.memory.NativeUnderlingMemory;
+import org.apache.arrow.memory.Ownerships;
+import org.apache.arrow.vector.ipc.message.ArrowFieldNode;
+import org.apache.arrow.vector.ipc.message.ArrowRecordBatch;
+import org.apache.arrow.vector.types.pojo.Schema;
+import org.apache.arrow.vector.util.SchemaUtility;
+
+/**
+ * Native implementation of {@link Scanner}. Note that it currently emits only a single scan task of type
+ * {@link NativeScanTask}, which is internally a combination of all scan task instances returned by the
+ * native scanner.
+ */
+public class NativeScanner implements Scanner {
+
+ private final AtomicBoolean closed = new AtomicBoolean(false);
+ private final AtomicBoolean executed = new AtomicBoolean(false);
+ private final NativeContext context;
+ private final long scannerId;
+
+ public NativeScanner(NativeContext context, long scannerId) {
+ this.context = context;
+ this.scannerId = scannerId;
+ }
+
+ ScanTask.BatchIterator execute() {
+ if (closed.get()) {
+ throw new NativeInstanceClosedException();
+ }
+ if (!executed.compareAndSet(false, true)) {
+ throw new UnsupportedOperationException("NativeScanner cannot be executed more than once. Consider creating " +
+ "new scanner instead");
+ }
+ return new ScanTask.BatchIterator() {
+ private ArrowRecordBatch peek = null;
+
+ @Override
+ public void close() {
+ NativeScanner.this.close();
+ }
+
+ @Override
+ public boolean hasNext() {
+ if (closed.get()) {
+ throw new NativeInstanceClosedException();
+ }
+ if (peek != null) {
+ return true;
+ }
+ NativeRecordBatchHandle handle = JniWrapper.get().nextRecordBatch(scannerId);
+ if (handle == null) {
+ return false;
+ }
+ final ArrayList<ArrowBuf> buffers = new ArrayList<>();
+ for (NativeRecordBatchHandle.Buffer buffer : handle.getBuffers()) {
+ final BufferAllocator allocator = context.getAllocator();
+ final NativeUnderlingMemory am = NativeUnderlingMemory.create(allocator,
+ (int) buffer.size, buffer.nativeInstanceId, buffer.memoryAddress);
Review comment:
You should minimally check if any of the size properties are bigger than INT32_MAX and raise accordingly. This will avoid nasty segfault/corrupted data down the road.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] zhztheplayer commented on pull request #7030: ARROW-7808: [Java][Dataset] Implement Datasets Java API by JNI to C++
Posted by GitBox <gi...@apache.org>.
zhztheplayer commented on pull request #7030:
URL: https://github.com/apache/arrow/pull/7030#issuecomment-630624723
Hi @emkornfield @wesm, comments addressed and filter implementation is removed from this PR (I suppose we expect a separate topic about it). Any further changes needed please kindly let me know.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] emkornfield commented on a change in pull request #7030: ARROW-7808: [Java][Dataset] Implement Datasets Java API
Posted by GitBox <gi...@apache.org>.
emkornfield commented on a change in pull request #7030:
URL: https://github.com/apache/arrow/pull/7030#discussion_r414984900
##########
File path: cpp/src/jni/dataset/jni_wrapper.cpp
##########
@@ -0,0 +1,577 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include <arrow/dataset/file_base.h>
+#include <arrow/dataset/api.h>
+#include <arrow/filesystem/localfs.h>
+#include <arrow/ipc/api.h>
+#include <arrow/util/iterator.h>
+#include <arrow/filesystem/hdfs.h>
+#include <arrow/io/api.h>
+#include <google/protobuf/io/coded_stream.h>
+#include <google/protobuf/message.h>
+#include "concurrent_map.h"
+#include "arrow/compute/kernel.h"
+#include "arrow/compute/kernels/cast.h"
+#include "arrow/compute/kernels/compare.h"
+#include "jni/dataset/Types.pb.h"
+
+#include "org_apache_arrow_dataset_jni_JniWrapper.h"
+#include "org_apache_arrow_dataset_file_JniWrapper.h"
+
+static jclass illegal_access_exception_class;
+static jclass illegal_argument_exception_class;
+static jclass runtime_exception_class;
+
+static jclass record_batch_handle_class;
+static jclass record_batch_handle_field_class;
+static jclass record_batch_handle_buffer_class;
+
+static jmethodID record_batch_handle_constructor;
+static jmethodID record_batch_handle_field_constructor;
+static jmethodID record_batch_handle_buffer_constructor;
+
+static jint JNI_VERSION = JNI_VERSION_1_6;
+
+using arrow::jni::ConcurrentMap;
+
+static ConcurrentMap<std::shared_ptr<arrow::dataset::DatasetFactory>> dataset_factory_holder_;
+static ConcurrentMap<std::shared_ptr<arrow::dataset::Dataset>> dataset_holder_;
+static ConcurrentMap<std::shared_ptr<arrow::dataset::ScanTask>> scan_task_holder_;
+static ConcurrentMap<std::shared_ptr<arrow::dataset::Scanner>> scanner_holder_;
+static ConcurrentMap<std::shared_ptr<arrow::RecordBatchIterator>> iterator_holder_;
+static ConcurrentMap<std::shared_ptr<arrow::Buffer>> buffer_holder_;
+
+#define JNI_ASSIGN_OR_THROW_NAME(x, y) ARROW_CONCAT(x, y)
+
+#define JNI_ASSIGN_OR_THROW_IMPL(t, lhs, rexpr) \
+ auto t = (rexpr); \
+ if (!t.status().ok()) { \
+ env->ThrowNew(runtime_exception_class, t.status().message().c_str()); \
+ } \
+ lhs = std::move(t).ValueOrDie();
+
+#define JNI_ASSIGN_OR_THROW(lhs, rexpr) \
+ JNI_ASSIGN_OR_THROW_IMPL(JNI_ASSIGN_OR_THROW_NAME(_tmp_var, __COUNTER__), lhs, rexpr)
+
+#define JNI_ASSERT_OK_OR_THROW(expr) \
+ do { \
+ auto _res = (expr); \
+ arrow::Status _st = ::arrow::internal::GenericToStatus(_res); \
+ if (!_st.ok()) { \
+ env->ThrowNew(runtime_exception_class, _st.message().c_str()); \
+ } \
+ } while (false);
+
+jclass CreateGlobalClassReference(JNIEnv* env, const char* class_name) {
+ jclass local_class = env->FindClass(class_name);
+ jclass global_class = (jclass)env->NewGlobalRef(local_class);
+ env->DeleteLocalRef(local_class);
+ return global_class;
+}
+
+jmethodID GetMethodID(JNIEnv* env, jclass this_class, const char* name, const char* sig) {
+ jmethodID ret = env->GetMethodID(this_class, name, sig);
+ if (ret == nullptr) {
+ std::string error_message = "Unable to find method " + std::string(name) +
+ " within signature" + std::string(sig);
+ env->ThrowNew(illegal_access_exception_class, error_message.c_str());
+ }
+ return ret;
+}
+
+jint JNI_OnLoad(JavaVM* vm, void* reserved) {
+ JNIEnv* env;
+ if (vm->GetEnv(reinterpret_cast<void**>(&env), JNI_VERSION) != JNI_OK) {
+ return JNI_ERR;
+ }
+
+ illegal_access_exception_class =
+ CreateGlobalClassReference(env, "Ljava/lang/IllegalAccessException;");
+ illegal_argument_exception_class =
+ CreateGlobalClassReference(env, "Ljava/lang/IllegalArgumentException;");
+ runtime_exception_class =
+ CreateGlobalClassReference(env, "Ljava/lang/RuntimeException;");
+
+ record_batch_handle_class =
+ CreateGlobalClassReference(env, "Lorg/apache/arrow/dataset/jni/NativeRecordBatchHandle;");
+ record_batch_handle_field_class =
+ CreateGlobalClassReference(env, "Lorg/apache/arrow/dataset/jni/NativeRecordBatchHandle$Field;");
+ record_batch_handle_buffer_class =
+ CreateGlobalClassReference(env, "Lorg/apache/arrow/dataset/jni/NativeRecordBatchHandle$Buffer;");
+
+ record_batch_handle_constructor = GetMethodID(env, record_batch_handle_class, "<init>",
+ "(J[Lorg/apache/arrow/dataset/jni/NativeRecordBatchHandle$Field;"
+ "[Lorg/apache/arrow/dataset/jni/NativeRecordBatchHandle$Buffer;)V");
+ record_batch_handle_field_constructor = GetMethodID(env, record_batch_handle_field_class, "<init>",
+ "(JJ)V");
+ record_batch_handle_buffer_constructor = GetMethodID(env, record_batch_handle_buffer_class, "<init>",
+ "(JJJJ)V");
+
+ env->ExceptionDescribe();
+
+ return JNI_VERSION;
+}
+
+
+void JNI_OnUnload(JavaVM* vm, void* reserved) {
+ JNIEnv* env;
+ vm->GetEnv(reinterpret_cast<void**>(&env), JNI_VERSION);
+ env->DeleteGlobalRef(illegal_access_exception_class);
+ env->DeleteGlobalRef(illegal_argument_exception_class);
+ env->DeleteGlobalRef(runtime_exception_class);
+ env->DeleteGlobalRef(record_batch_handle_class);
+ env->DeleteGlobalRef(record_batch_handle_field_class);
+ env->DeleteGlobalRef(record_batch_handle_buffer_class);
+
+ dataset_factory_holder_.Clear();
+ dataset_holder_.Clear();
+ scan_task_holder_.Clear();
+ scanner_holder_.Clear();
+ iterator_holder_.Clear();
+ buffer_holder_.Clear();
+}
+
+std::shared_ptr<arrow::Schema> SchemaFromColumnNames(
+ const std::shared_ptr<arrow::Schema>& input, const std::vector<std::string>& column_names) {
+ std::vector<std::shared_ptr<arrow::Field>> columns;
+ for (const auto& name : column_names) {
+ columns.push_back(input->GetFieldByName(name));
+ }
+ return std::make_shared<arrow::Schema>(columns);
+}
+
+std::shared_ptr<arrow::dataset::FileFormat> GetFileFormat(JNIEnv *env, jint id) {
+ switch (id) {
+ case 0:
+ return std::make_shared<arrow::dataset::ParquetFileFormat>();
+ default:
+ std::string error_message = "illegal file format id: " + std::to_string(id);
+ env->ThrowNew(illegal_argument_exception_class, error_message.c_str());
+ return nullptr; // unreachable
+ }
+}
+
+std::shared_ptr<arrow::fs::FileSystem> GetFileSystem(JNIEnv *env, jint id, std::string path,
+ std::string* out_path) {
+ switch (id) {
+ case 0:
+ *out_path = path;
+ return std::make_shared<arrow::fs::LocalFileSystem>();
+ case 1: {
+ JNI_ASSIGN_OR_THROW(std::shared_ptr<arrow::fs::FileSystem> ret, arrow::fs::FileSystemFromUri(path, out_path))
+ return ret;
+ }
+ default:std::string error_message = "illegal filesystem id: " + std::to_string(id);
+ env->ThrowNew(illegal_argument_exception_class, error_message.c_str());
+ return nullptr; // unreachable
+ }
+}
+
+std::string JStringToCString(JNIEnv* env, jstring string) {
+ jboolean copied;
+ int32_t length = env->GetStringUTFLength(string);
+ const char *chars = env->GetStringUTFChars(string, &copied);
+ std::string str = std::string(chars, length);
+ // fixme calling ReleaseStringUTFChars if memory leak faced
+ return str;
+}
+
+std::vector<std::string> ToStringVector(JNIEnv* env, jobjectArray& str_array) {
+ int length = env->GetArrayLength(str_array);
+ std::vector<std::string> vector;
+ for (int i = 0; i < length; i++) {
+ auto string = (jstring) (env->GetObjectArrayElement(str_array, i));
+ vector.push_back(JStringToCString(env, string));
+ }
+ return vector;
+}
+
+template <typename T>
+std::vector<T> collect(JNIEnv* env, arrow::Iterator<T> itr) {
+ std::vector<T> vector;
+ while(true) {
+ JNI_ASSIGN_OR_THROW(T t, itr.Next())
+ if (!t) {
+ break;
+ }
+ vector.push_back(t);
+ }
+ return vector;
+}
+
+// FIXME: COPIED FROM intel/master on which this branch is not rebased yet
+// FIXME: https://github.com/Intel-bigdata/arrow/blob/02502a4eb59834c2471dd629e77dbeed19559f68/cpp/src/jni/jni_common.h#L239-L254
+jbyteArray ToSchemaByteArray(JNIEnv* env, std::shared_ptr<arrow::Schema> schema) {
+ JNI_ASSIGN_OR_THROW(std::shared_ptr<arrow::Buffer> buffer,
+ arrow::ipc::SerializeSchema(*schema.get(), nullptr, arrow::default_memory_pool()))
+
+ jbyteArray out = env->NewByteArray(buffer->size());
+ auto src = reinterpret_cast<const jbyte*>(buffer->data());
+ env->SetByteArrayRegion(out, 0, buffer->size(), src);
+ return out;
+}
+
+// FIXME: COPIED FROM intel/master on which this branch is not rebased yet
Review comment:
it looks like there are a lot of FIXMEs can you address those?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] emkornfield commented on a change in pull request #7030: ARROW-7808: [Java][Dataset] Implement Datasets Java API
Posted by GitBox <gi...@apache.org>.
emkornfield commented on a change in pull request #7030:
URL: https://github.com/apache/arrow/pull/7030#discussion_r414987860
##########
File path: java/dataset/src/main/java/org/apache/arrow/dataset/source/DatasetFactory.java
##########
@@ -0,0 +1,34 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.arrow.dataset.source;
+
+import org.apache.arrow.vector.types.pojo.Schema;
+
+/**
+ * DataSourceDiscovery provides a way to inspect a DataSource potential
+ * schema before materializing it. Thus, the user can peek the schema for
+ * data sources and decide on a unified schema.
+ */
+public interface DatasetFactory {
+
+ Schema inspect();
+
Review comment:
javadoc.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] emkornfield commented on a change in pull request #7030: ARROW-7808: [Java][Dataset] Implement Datasets Java API
Posted by GitBox <gi...@apache.org>.
emkornfield commented on a change in pull request #7030:
URL: https://github.com/apache/arrow/pull/7030#discussion_r414987799
##########
File path: java/dataset/src/main/java/org/apache/arrow/dataset/scanner/ScanTask.java
##########
@@ -0,0 +1,42 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.arrow.dataset.scanner;
+
+import java.util.Iterator;
+
+import org.apache.arrow.vector.VectorSchemaRoot;
+
+/**
+ * Read record batches from a range of a single data fragment. A
+ * ScanTask is meant to be a unit of work to be dispatched. The implementation
+ * must be thread and concurrent safe.
+ */
+public interface ScanTask {
+
+ /**
+ * Creates and returns a {@link Itr} instance.
+ */
+ Itr scan();
+
+ /**
+ * The iterator implementation for {@link VectorSchemaRoot}s.
+ */
+ interface Itr extends Iterator<VectorSchemaRoot>, AutoCloseable {
+ // FIXME VectorSchemaRoot is not actually something ITERABLE. Using a reader convention instead.
Review comment:
Fix this?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] zhztheplayer commented on a change in pull request #7030: ARROW-7808: [Java][Dataset] Implement Datasets Java API by JNI to C++
Posted by GitBox <gi...@apache.org>.
zhztheplayer commented on a change in pull request #7030:
URL: https://github.com/apache/arrow/pull/7030#discussion_r417804784
##########
File path: java/dataset/src/test/java/org/apache/arrow/util/SchemaUtilsTest.java
##########
@@ -0,0 +1,50 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.arrow.util;
+
+import static java.util.Arrays.asList;
+
+import java.io.IOException;
+
+import org.apache.arrow.memory.RootAllocator;
+import org.apache.arrow.vector.types.pojo.ArrowType;
+import org.apache.arrow.vector.types.pojo.Field;
+import org.apache.arrow.vector.types.pojo.FieldType;
+import org.apache.arrow.vector.types.pojo.Schema;
+import org.junit.Assert;
+import org.junit.Test;
+
+public class SchemaUtilsTest {
+
+ private static Field field(String name, boolean nullable, ArrowType type, Field... children) {
+ return new Field(name, new FieldType(nullable, type, null, null), asList(children));
+ }
+
+ @Test
+ public void testSerializationAndDeserialization() throws IOException {
+ Schema schema = new Schema(asList(
+ field("a", false, new ArrowType.Null()),
+ field("b", true, new ArrowType.Utf8()),
+ field("c", true, new ArrowType.Binary()))
+ );
+
+ byte[] serialized = SchemaUtils.get().serialize(schema);
+ Schema deserialized = SchemaUtils.get().deserialize(serialized, new RootAllocator(Long.MAX_VALUE));
+ Assert.assertEquals(schema, deserialized);
Review comment:
done
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] zhztheplayer commented on a change in pull request #7030: ARROW-7808: [Java][Dataset] Implement Datasets Java API by JNI to C++
Posted by GitBox <gi...@apache.org>.
zhztheplayer commented on a change in pull request #7030:
URL: https://github.com/apache/arrow/pull/7030#discussion_r417725284
##########
File path: cpp/src/jni/dataset/proto/Types.proto
##########
@@ -0,0 +1,149 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+syntax = "proto2";
+package types;
+
+option java_package = "org.apache.arrow.dataset";
Review comment:
I see. At this time I'll try keeping them detached as far as possible.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] zhztheplayer commented on a change in pull request #7030: ARROW-7808: [Java][Dataset] Implement Datasets Java API by JNI to C++
Posted by GitBox <gi...@apache.org>.
zhztheplayer commented on a change in pull request #7030:
URL: https://github.com/apache/arrow/pull/7030#discussion_r426395161
##########
File path: java/dataset/src/test/java/org/apache/arrow/dataset/file/TestSingleFileDataset.java
##########
@@ -0,0 +1,237 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.arrow.dataset.file;
+
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertTrue;
+
+import java.util.Collections;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.Objects;
+import java.util.stream.Collectors;
+
+import org.apache.arrow.dataset.DatasetTypes;
+import org.apache.arrow.dataset.filter.Filter;
+import org.apache.arrow.dataset.jni.TestNativeDataset;
+import org.apache.arrow.dataset.scanner.ScanOptions;
+import org.apache.arrow.util.AutoCloseables;
+import org.apache.arrow.vector.FieldVector;
+import org.apache.arrow.vector.VectorLoader;
+import org.apache.arrow.vector.VectorSchemaRoot;
+import org.apache.arrow.vector.ipc.message.ArrowRecordBatch;
+import org.apache.arrow.vector.types.Types;
+import org.apache.arrow.vector.types.pojo.Schema;
+import org.apache.avro.generic.GenericData;
+import org.apache.avro.generic.GenericRecord;
+import org.apache.avro.generic.GenericRecordBuilder;
+import org.junit.ClassRule;
+import org.junit.Test;
+import org.junit.rules.TemporaryFolder;
+
+import com.google.common.collect.ImmutableList;
+
+public class TestSingleFileDataset extends TestNativeDataset {
+
+ @ClassRule
+ public static final TemporaryFolder TMP = new TemporaryFolder();
+
+ public static final String AVRO_SCHEMA_USER = "user.avsc";
+
+ @Test
+ public void testParquetRead() throws Exception {
+ ParquetWriteSupport writeSupport = new ParquetWriteSupport(AVRO_SCHEMA_USER, TMP.newFolder());
+ GenericRecord record = new GenericData.Record(writeSupport.getAvroSchema());
+ record.put("id", 1);
+ record.put("name", "a");
+ writeSupport.writeRecord(record);
+ writeSupport.close();
+
+ SingleFileDatasetFactory factory = new SingleFileDatasetFactory(rootAllocator(),
+ FileFormat.PARQUET, FileSystem.LOCAL, writeSupport.getOutputFile());
+ ScanOptions options = new ScanOptions(new String[0], Filter.EMPTY, 100);
+
+ Schema schema = inferResultSchemaFromFactory(factory, options);
+ assertEquals(2, schema.getFields().size());
+ assertEquals("id", schema.getFields().get(0).getName());
+ assertEquals("name", schema.getFields().get(1).getName());
+ assertEquals(Types.MinorType.INT.getType(), schema.getFields().get(0).getType());
+ assertEquals(Types.MinorType.VARCHAR.getType(), schema.getFields().get(1).getType());
+
+ assertSingleTaskProduced(factory, options);
+
+ List<ArrowRecordBatch> datum = collectResultFromFactory(factory, options);
+ assertEquals(1, datum.size());
+ checkParquetReadResult(schema, Collections.singletonList(record), datum);
+ AutoCloseables.close(datum);
+ }
+
+ @Test
+ public void testParquetProjector() throws Exception {
+ ParquetWriteSupport writeSupport = new ParquetWriteSupport(AVRO_SCHEMA_USER, TMP.newFolder());
Review comment:
done. thanks for the suggestion.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] zhztheplayer commented on pull request #7030: ARROW-7808: [Java][Dataset] Implement Dataset Java API by JNI to C++
Posted by GitBox <gi...@apache.org>.
zhztheplayer commented on pull request #7030:
URL: https://github.com/apache/arrow/pull/7030#issuecomment-755898896
@pitrou I've moved the pool class out to jni_wrapper.cpp. I saw your comments in https://issues.apache.org/jira/browse/ARROW-11143, thanks for that.
Regarding the static variables, I was just following current patterns in modules like gandiva-jni. Is there a better practice for this? Should we factor them out to specific namespaces (maybe in other tickets)?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] emkornfield commented on a change in pull request #7030: ARROW-7808: [Java][Dataset] Implement Datasets Java API
Posted by GitBox <gi...@apache.org>.
emkornfield commented on a change in pull request #7030:
URL: https://github.com/apache/arrow/pull/7030#discussion_r414986958
##########
File path: java/dataset/src/main/java/org/apache/arrow/dataset/filter/Filter.java
##########
@@ -0,0 +1,37 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.arrow.dataset.filter;
+
+// todo filter tree implementation
Review comment:
leave this out if it isn't implemented.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] zhztheplayer commented on a change in pull request #7030: ARROW-7808: [Java][Dataset] Implement Datasets Java API by JNI to C++
Posted by GitBox <gi...@apache.org>.
zhztheplayer commented on a change in pull request #7030:
URL: https://github.com/apache/arrow/pull/7030#discussion_r426995832
##########
File path: java/dataset/src/test/resources/avroschema/user.avsc
##########
@@ -0,0 +1,26 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+{
Review comment:
Also there will be a name collision between org.apache.arrow.vector.types.pojo.Schema and org.apache.avro.Schema if we use them in the same file. To avoid writing complicated code due to this we may tend to use a separated Java file to create avro schemas e.g. AvroSchemas.java, which finally makes user look into two places too.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] zhztheplayer commented on a change in pull request #7030: ARROW-7808: [Java][Dataset] Implement Datasets Java API by JNI to C++
Posted by GitBox <gi...@apache.org>.
zhztheplayer commented on a change in pull request #7030:
URL: https://github.com/apache/arrow/pull/7030#discussion_r426376401
##########
File path: java/dataset/src/main/java/org/apache/arrow/dataset/jni/NativeDataset.java
##########
@@ -0,0 +1,48 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.arrow.dataset.jni;
+
+import org.apache.arrow.dataset.scanner.ScanOptions;
+import org.apache.arrow.dataset.scanner.Scanner;
+import org.apache.arrow.dataset.source.Dataset;
+
+/**
+ * Native implementation of {@link Dataset}.
+ */
+public class NativeDataset implements Dataset, AutoCloseable {
+
+ private final NativeContext context;
+ private final long datasetId;
+
+ public NativeDataset(NativeContext context, long datasetId) {
+ this.context = context;
+ this.datasetId = datasetId;
+ }
+
+ @Override
+ public Scanner newScan(ScanOptions options) {
+ long scannerId = JniWrapper.get().createScanner(datasetId, options.getColumns(),
+ options.getFilter().toByteArray(), options.getBatchSize());
+ return new NativeScanner(context, scannerId);
+ }
+
+ @Override
+ public void close() throws Exception {
Review comment:
done
##########
File path: java/dataset/src/main/java/org/apache/arrow/dataset/jni/NativeScanTask.java
##########
@@ -0,0 +1,110 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.arrow.dataset.jni;
+
+import java.util.ArrayList;
+import java.util.NoSuchElementException;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.stream.Collectors;
+
+import org.apache.arrow.dataset.scanner.ScanTask;
+import org.apache.arrow.memory.ArrowBuf;
+import org.apache.arrow.memory.BaseAllocator;
+import org.apache.arrow.memory.BufferLedger;
+import org.apache.arrow.memory.NativeUnderlingMemory;
+import org.apache.arrow.memory.Ownerships;
+import org.apache.arrow.vector.ipc.message.ArrowFieldNode;
+import org.apache.arrow.vector.ipc.message.ArrowRecordBatch;
+
+/**
+ * Native implementation of {@link ScanTask}. Currently RecordBatches are iterated directly by the scanner
+ * id via {@link JniWrapper}, thus we allow only one-time execution of method {@link #scan()}. If a re-scan
+ * operation is expected, call {@link NativeDataset#newScan} to create a new scanner instance.
+ */
+public class NativeScanTask implements ScanTask, AutoCloseable {
+ private final NativeScanner scanner;
+ private final AtomicBoolean executed = new AtomicBoolean(false);
+
+ /**
+ * Constructor.
+ */
+ public NativeScanTask(NativeScanner scanner) {
+ this.scanner = scanner;
+ }
+
+ @Override
+ public BatchIterator scan() {
+ if (!executed.compareAndSet(false, true)) {
+ throw new UnsupportedOperationException("Method scan() cannot be executed more than once if it is a " +
+ "native scan task");
+ }
+ return new BatchIterator() {
+ private ArrowRecordBatch peek = null;
+
+ @Override
+ public void close() throws Exception {
+ scanner.close();
+ }
+
+ @Override
+ public boolean hasNext() {
+ if (peek != null) {
+ return true;
+ }
+ NativeRecordBatchHandle handle = JniWrapper.get().nextRecordBatch(scanner.getId());
+ if (handle == null) {
+ return false;
+ }
+ final ArrayList<ArrowBuf> buffers = new ArrayList<>();
+ for (NativeRecordBatchHandle.Buffer buffer : handle.getBuffers()) {
+ final BaseAllocator allocator = scanner.getContext().getAllocator();
+ final NativeUnderlingMemory am = new NativeUnderlingMemory(allocator,
+ (int) buffer.size, buffer.nativeInstanceId, buffer.memoryAddress);
+ final BufferLedger ledger = Ownerships.get().takeOwnership(allocator, am);
+ ArrowBuf buf = new ArrowBuf(ledger, null, (int) buffer.size, buffer.memoryAddress, false);
+ buffers.add(buf);
+ }
+ try {
+ peek = new ArrowRecordBatch((int) handle.getNumRows(), handle.getFields().stream()
+ .map(field -> new ArrowFieldNode((int) field.length, (int) field.nullCount))
+ .collect(Collectors.toList()), buffers);
+ return true;
+ } finally {
+ buffers.forEach(buffer -> buffer.getReferenceManager().release());
+ }
+ }
+
+ @Override
+ public ArrowRecordBatch next() {
+ if (!hasNext()) {
+ throw new NoSuchElementException();
+ }
+ try {
Review comment:
done, thanks
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] zhztheplayer commented on a change in pull request #7030: ARROW-7808: [Java][Dataset] Implement Datasets Java API by JNI to C++
Posted by GitBox <gi...@apache.org>.
zhztheplayer commented on a change in pull request #7030:
URL: https://github.com/apache/arrow/pull/7030#discussion_r426995832
##########
File path: java/dataset/src/test/resources/avroschema/user.avsc
##########
@@ -0,0 +1,26 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+{
Review comment:
Also there will be a name collision between org.apache.arrow.vector.types.pojo.Schema and org.apache.avro.Schema if we use them in the same file. To avoid writing complicated code we may tend to use a separated Java file to create avro schemas e.g. AvroSchemas.java, which finally makes user look into two places too.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] zhztheplayer commented on pull request #7030: ARROW-7808: [Java][Dataset] Implement Datasets Java API by JNI to C++
Posted by GitBox <gi...@apache.org>.
zhztheplayer commented on pull request #7030:
URL: https://github.com/apache/arrow/pull/7030#issuecomment-660927861
Sorry I am reconsidering whether to use JVM direct memory as a limit of native buffer allocation. By Oracle's explanation[1] about direct memory, direct memory seems only to limit direct byte buffer allocations. As the discussion has lasted for days I'll send a mail for this topic. Hopefully we can make a conclusion :)
[1] https://docs.oracle.com/cd/E15289_01/JRCLR/optionxx.htm#BABGCFFB
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] zhztheplayer edited a comment on pull request #7030: ARROW-7808: [Java][Dataset] Implement Datasets Java API by JNI to C++
Posted by GitBox <gi...@apache.org>.
zhztheplayer edited a comment on pull request #7030:
URL: https://github.com/apache/arrow/pull/7030#issuecomment-660927861
Sorry I am reconsidering whether to use JVM direct memory as a limit of native buffer allocation. Technically by Oracle's explanation[1] about direct memory, direct memory seems only to limit direct byte buffer allocations. As the discussion has lasted for days I'll send a mail for this topic. Hopefully we can make a conclusion :)
[1] https://docs.oracle.com/cd/E15289_01/JRCLR/optionxx.htm#BABGCFFB
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] zhztheplayer commented on a change in pull request #7030: ARROW-7808: [Java][Dataset] Implement Dataset Java API by JNI to C++
Posted by GitBox <gi...@apache.org>.
zhztheplayer commented on a change in pull request #7030:
URL: https://github.com/apache/arrow/pull/7030#discussion_r563299997
##########
File path: cpp/src/arrow/memory_pool.h
##########
@@ -149,6 +149,43 @@ class ARROW_EXPORT ProxyMemoryPool : public MemoryPool {
std::unique_ptr<ProxyMemoryPoolImpl> impl_;
};
+class ARROW_EXPORT ReservationListener {
+ public:
+ virtual ~ReservationListener();
+
+ virtual Status OnReservation(int64_t size) = 0;
+ virtual Status OnRelease(int64_t size) = 0;
+
+ protected:
+ ReservationListener();
+};
+
+class ARROW_EXPORT ReservationListenableMemoryPool : public MemoryPool {
Review comment:
@pitrou Here is a previous discussion about "reserve" bytes from Bits.java before the actual allocation https://github.com/apache/arrow/pull/7030#issuecomment-679355309. "pre-reserve" does not actually allocate memory but call a customized listener to reserve from other resource managers such as JVM native memory, for other examples, Spark or Yarn or something.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] emkornfield commented on a change in pull request #7030: ARROW-7808: [Java][Dataset] Implement Datasets Java API
Posted by GitBox <gi...@apache.org>.
emkornfield commented on a change in pull request #7030:
URL: https://github.com/apache/arrow/pull/7030#discussion_r414988086
##########
File path: java/dataset/src/main/java/org/apache/arrow/memory/NativeUnderlingMemory.java
##########
@@ -0,0 +1,65 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.arrow.memory;
+
+import org.apache.arrow.dataset.jni.JniWrapper;
+
+/**
+ * AllocationManager implementation for Native allocated memory.
+ */
+public class NativeUnderlingMemory extends AllocationManager {
Review comment:
this should probably be in a different common package if we want to wrap the C++ allocator. It isn't clear to me that we should do this because we lose java native heap caps IIUC.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] emkornfield commented on a change in pull request #7030: ARROW-7808: [Java][Dataset] Implement Datasets Java API by JNI to C++
Posted by GitBox <gi...@apache.org>.
emkornfield commented on a change in pull request #7030:
URL: https://github.com/apache/arrow/pull/7030#discussion_r417074805
##########
File path: java/dataset/src/main/java/org/apache/arrow/dataset/scanner/ScanTask.java
##########
@@ -0,0 +1,42 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.arrow.dataset.scanner;
+
+import java.util.Iterator;
+
+import org.apache.arrow.vector.VectorSchemaRoot;
+
+/**
+ * Read record batches from a range of a single data fragment. A
+ * ScanTask is meant to be a unit of work to be dispatched. The implementation
+ * must be thread and concurrent safe.
+ */
+public interface ScanTask {
+
+ /**
+ * Creates and returns a {@link Itr} instance.
+ */
+ Itr scan();
+
+ /**
+ * The iterator implementation for {@link VectorSchemaRoot}s.
+ */
+ interface Itr extends Iterator<VectorSchemaRoot>, AutoCloseable {
Review comment:
I would expect ScanTask.Iterator would be used consistently to avoid collision. BatchIterator is also code with me.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] zhztheplayer commented on a change in pull request #7030: ARROW-7808: [Java][Dataset] Implement Datasets Java API by JNI to C++
Posted by GitBox <gi...@apache.org>.
zhztheplayer commented on a change in pull request #7030:
URL: https://github.com/apache/arrow/pull/7030#discussion_r416412538
##########
File path: java/dataset/src/main/java/org/apache/arrow/dataset/scanner/ScanTask.java
##########
@@ -0,0 +1,42 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.arrow.dataset.scanner;
+
+import java.util.Iterator;
+
+import org.apache.arrow.vector.VectorSchemaRoot;
+
+/**
+ * Read record batches from a range of a single data fragment. A
+ * ScanTask is meant to be a unit of work to be dispatched. The implementation
+ * must be thread and concurrent safe.
+ */
+public interface ScanTask {
+
+ /**
+ * Creates and returns a {@link Itr} instance.
+ */
+ Itr scan();
+
+ /**
+ * The iterator implementation for {@link VectorSchemaRoot}s.
+ */
+ interface Itr extends Iterator<VectorSchemaRoot>, AutoCloseable {
Review comment:
Using Iterator is OK but it may collide with java.util.Iterator in scenarios. Should it be "BatchIterator" or something?
##########
File path: java/dataset/src/main/java/org/apache/arrow/dataset/scanner/ScanTask.java
##########
@@ -0,0 +1,42 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.arrow.dataset.scanner;
+
+import java.util.Iterator;
+
+import org.apache.arrow.vector.VectorSchemaRoot;
+
+/**
+ * Read record batches from a range of a single data fragment. A
+ * ScanTask is meant to be a unit of work to be dispatched. The implementation
+ * must be thread and concurrent safe.
+ */
+public interface ScanTask {
+
+ /**
+ * Creates and returns a {@link Itr} instance.
+ */
+ Itr scan();
+
+ /**
+ * The iterator implementation for {@link VectorSchemaRoot}s.
+ */
+ interface Itr extends Iterator<VectorSchemaRoot>, AutoCloseable {
Review comment:
renamed
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] zhztheplayer commented on a change in pull request #7030: ARROW-7808: [Java][Dataset] Implement Datasets Java API by JNI to C++
Posted by GitBox <gi...@apache.org>.
zhztheplayer commented on a change in pull request #7030:
URL: https://github.com/apache/arrow/pull/7030#discussion_r429553399
##########
File path: cpp/src/jni/dataset/jni_wrapper.cpp
##########
@@ -0,0 +1,517 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include <arrow/dataset/api.h>
+#include <arrow/dataset/file_base.h>
+#include <arrow/filesystem/hdfs.h>
+#include <arrow/filesystem/localfs.h>
+#include <arrow/io/api.h>
+#include <arrow/ipc/api.h>
+#include <arrow/util/iterator.h>
+
+#include "org_apache_arrow_dataset_file_JniWrapper.h"
+#include "org_apache_arrow_dataset_jni_JniWrapper.h"
+
+static jclass illegal_access_exception_class;
+static jclass illegal_argument_exception_class;
+static jclass runtime_exception_class;
+
+static jclass record_batch_handle_class;
+static jclass record_batch_handle_field_class;
+static jclass record_batch_handle_buffer_class;
+
+static jmethodID record_batch_handle_constructor;
+static jmethodID record_batch_handle_field_constructor;
+static jmethodID record_batch_handle_buffer_constructor;
+
+static jint JNI_VERSION = JNI_VERSION_1_6;
+
+class JniPendingException : public std::runtime_error {
+ public:
+ explicit JniPendingException(const std::string& arg) : runtime_error(arg) {}
+};
+
+void ThrowPendingException(const std::string& message) {
+ throw JniPendingException(message);
+}
+
+template <typename T>
+T JniGetOrThrow(arrow::Result<T> result) {
+ if (!result.status().ok()) {
+ ThrowPendingException(result.status().message());
+ }
+ return std::move(result).ValueOrDie();
+}
+
+void JniAssertOkOrThrow(arrow::Status status) {
+ if (!status.ok()) {
+ ThrowPendingException(status.message());
+ }
+}
+
+void JniThrow(std::string message) { ThrowPendingException(message); }
+
+#define JNI_METHOD_START try {
+// macro ended
+
+#define JNI_METHOD_END(fallback_expr) \
+ } \
+ catch (JniPendingException & e) { \
+ env->ThrowNew(runtime_exception_class, e.what()); \
+ return fallback_expr; \
+ }
+// macro ended
+
+jclass CreateGlobalClassReference(JNIEnv* env, const char* class_name) {
+ jclass local_class = env->FindClass(class_name);
+ jclass global_class = (jclass)env->NewGlobalRef(local_class);
+ env->DeleteLocalRef(local_class);
+ return global_class;
+}
+
+arrow::Result<jmethodID> GetMethodID(JNIEnv* env, jclass this_class, const char* name,
+ const char* sig) {
+ jmethodID ret = env->GetMethodID(this_class, name, sig);
+ if (ret == nullptr) {
+ std::string error_message = "Unable to find method " + std::string(name) +
+ " within signature" + std::string(sig);
+ return arrow::Status::Invalid(error_message);
+ }
+ return ret;
+}
+
+jint JNI_OnLoad(JavaVM* vm, void* reserved) {
+ JNIEnv* env;
+ if (vm->GetEnv(reinterpret_cast<void**>(&env), JNI_VERSION) != JNI_OK) {
+ return JNI_ERR;
+ }
+ JNI_METHOD_START
+ illegal_access_exception_class =
+ CreateGlobalClassReference(env, "Ljava/lang/IllegalAccessException;");
+ illegal_argument_exception_class =
+ CreateGlobalClassReference(env, "Ljava/lang/IllegalArgumentException;");
+ runtime_exception_class =
+ CreateGlobalClassReference(env, "Ljava/lang/RuntimeException;");
+
+ record_batch_handle_class =
+ CreateGlobalClassReference(env,
+ "Lorg/apache/arrow/"
+ "dataset/jni/NativeRecordBatchHandle;");
+ record_batch_handle_field_class =
+ CreateGlobalClassReference(env,
+ "Lorg/apache/arrow/"
+ "dataset/jni/NativeRecordBatchHandle$Field;");
+ record_batch_handle_buffer_class =
+ CreateGlobalClassReference(env,
+ "Lorg/apache/arrow/"
+ "dataset/jni/NativeRecordBatchHandle$Buffer;");
+
+ record_batch_handle_constructor =
+ JniGetOrThrow(GetMethodID(env, record_batch_handle_class, "<init>",
+ "(J[Lorg/apache/arrow/dataset/"
+ "jni/NativeRecordBatchHandle$Field;"
+ "[Lorg/apache/arrow/dataset/"
+ "jni/NativeRecordBatchHandle$Buffer;)V"));
+ record_batch_handle_field_constructor =
+ JniGetOrThrow(GetMethodID(env, record_batch_handle_field_class, "<init>", "(JJ)V"));
+ record_batch_handle_buffer_constructor = JniGetOrThrow(
+ GetMethodID(env, record_batch_handle_buffer_class, "<init>", "(JJJJ)V"));
+
+ return JNI_VERSION;
+ JNI_METHOD_END(JNI_ERR)
+}
+
+void JNI_OnUnload(JavaVM* vm, void* reserved) {
+ JNIEnv* env;
+ vm->GetEnv(reinterpret_cast<void**>(&env), JNI_VERSION);
+ env->DeleteGlobalRef(illegal_access_exception_class);
+ env->DeleteGlobalRef(illegal_argument_exception_class);
+ env->DeleteGlobalRef(runtime_exception_class);
+ env->DeleteGlobalRef(record_batch_handle_class);
+ env->DeleteGlobalRef(record_batch_handle_field_class);
+ env->DeleteGlobalRef(record_batch_handle_buffer_class);
+}
+
+std::shared_ptr<arrow::Schema> SchemaFromColumnNames(
+ const std::shared_ptr<arrow::Schema>& input,
+ const std::vector<std::string>& column_names) {
+ std::vector<std::shared_ptr<arrow::Field>> columns;
+ for (const auto& name : column_names) {
+ columns.push_back(input->GetFieldByName(name));
+ }
+ return std::make_shared<arrow::Schema>(columns);
+}
+
+arrow::Result<std::shared_ptr<arrow::dataset::FileFormat>> GetFileFormat(jint id) {
+ switch (id) {
+ case 0:
+ return std::make_shared<arrow::dataset::ParquetFileFormat>();
+ default:
+ std::string error_message = "illegal file format id: " + std::to_string(id);
+ return arrow::Status::Invalid(error_message);
+ }
+}
+
+arrow::Result<std::shared_ptr<arrow::fs::FileSystem>> GetFileSystem(
+ jint id, std::string path, std::string* out_path) {
+ switch (id) {
+ case 0:
+ *out_path = path;
+ return std::make_shared<arrow::fs::LocalFileSystem>();
+ case 1: {
+ ARROW_ASSIGN_OR_RAISE(std::shared_ptr<arrow::fs::FileSystem> ret,
+ arrow::fs::FileSystemFromUri(path, out_path))
+ return ret;
+ }
+ default:
+ std::string error_message = "illegal file system id: " + std::to_string(id);
+ return arrow::Status::Invalid(error_message);
+ }
+}
+
+std::string JStringToCString(JNIEnv* env, jstring string) {
+ if (string == nullptr) {
+ return std::string();
+ }
+ jboolean copied;
+ int32_t length = env->GetStringUTFLength(string);
+ const char* chars = env->GetStringUTFChars(string, &copied);
+ std::string str = std::string(chars, length);
+ // fixme calling ReleaseStringUTFChars if memory leak faced
+ return str;
+}
+
+std::vector<std::string> ToStringVector(JNIEnv* env, jobjectArray& str_array) {
+ int length = env->GetArrayLength(str_array);
+ std::vector<std::string> vector;
+ for (int i = 0; i < length; i++) {
+ auto string = (jstring)(env->GetObjectArrayElement(str_array, i));
+ vector.push_back(JStringToCString(env, string));
+ }
+ return vector;
+}
+
+template <typename T>
+jlong CreateNativeRef(std::shared_ptr<T> t) {
+ std::shared_ptr<T>* retained_ptr = new std::shared_ptr<T>(t);
+ return reinterpret_cast<jlong>(retained_ptr);
+}
+
+template <typename T>
+std::shared_ptr<T> RetrieveNativeInstance(jlong ref) {
+ std::shared_ptr<T>* retrieved_ptr = reinterpret_cast<std::shared_ptr<T>*>(ref);
+ return *retrieved_ptr;
+}
+
+template <typename T>
+void ReleaseNativeRef(jlong ref) {
+ std::shared_ptr<T>* retrieved_ptr = reinterpret_cast<std::shared_ptr<T>*>(ref);
+ delete retrieved_ptr;
+}
+
+arrow::Result<jbyteArray> ToSchemaByteArray(JNIEnv* env,
+ std::shared_ptr<arrow::Schema> schema) {
+ ARROW_ASSIGN_OR_RAISE(
+ std::shared_ptr<arrow::Buffer> buffer,
+ arrow::ipc::SerializeSchema(*schema, nullptr, arrow::default_memory_pool()))
+
+ jbyteArray out = env->NewByteArray(buffer->size());
+ auto src = reinterpret_cast<const jbyte*>(buffer->data());
+ env->SetByteArrayRegion(out, 0, buffer->size(), src);
+ return out;
+}
+
+arrow::Result<std::shared_ptr<arrow::Schema>> FromSchemaByteArray(
+ JNIEnv* env, jbyteArray schemaBytes) {
+ arrow::ipc::DictionaryMemo in_memo;
+ int schemaBytes_len = env->GetArrayLength(schemaBytes);
+ jbyte* schemaBytes_data = env->GetByteArrayElements(schemaBytes, nullptr);
+ auto serialized_schema = std::make_shared<arrow::Buffer>(
+ reinterpret_cast<uint8_t*>(schemaBytes_data), schemaBytes_len);
+ arrow::io::BufferReader buf_reader(serialized_schema);
+ ARROW_ASSIGN_OR_RAISE(std::shared_ptr<arrow::Schema> schema,
+ arrow::ipc::ReadSchema(&buf_reader, &in_memo))
+ env->ReleaseByteArrayElements(schemaBytes, schemaBytes_data, JNI_ABORT);
+ return schema;
+}
+
+/*
+ * Class: org_apache_arrow_dataset_jni_JniWrapper
+ * Method: closeDatasetFactory
+ * Signature: (J)V
+ */
+JNIEXPORT void JNICALL Java_org_apache_arrow_dataset_jni_JniWrapper_closeDatasetFactory(
+ JNIEnv* env, jobject, jlong id) {
+ JNI_METHOD_START
+ ReleaseNativeRef<arrow::dataset::DatasetFactory>(id);
+ JNI_METHOD_END()
+}
+
+/*
+ * Class: org_apache_arrow_dataset_jni_JniWrapper
+ * Method: inspectSchema
+ * Signature: (J)[B
+ */
+JNIEXPORT jbyteArray JNICALL Java_org_apache_arrow_dataset_jni_JniWrapper_inspectSchema(
+ JNIEnv* env, jobject, jlong dataset_factor_id) {
+ JNI_METHOD_START
+ std::shared_ptr<arrow::dataset::DatasetFactory> d =
+ RetrieveNativeInstance<arrow::dataset::DatasetFactory>(dataset_factor_id);
+ std::shared_ptr<arrow::Schema> schema = JniGetOrThrow(d->Inspect());
+ return JniGetOrThrow(ToSchemaByteArray(env, schema));
+ JNI_METHOD_END(nullptr)
+}
+
+/*
+ * Class: org_apache_arrow_dataset_jni_JniWrapper
+ * Method: createDataset
+ * Signature: (J[B)J
+ */
+JNIEXPORT jlong JNICALL Java_org_apache_arrow_dataset_jni_JniWrapper_createDataset(
+ JNIEnv* env, jobject, jlong dataset_factory_id, jbyteArray schema_bytes) {
+ JNI_METHOD_START
+ std::shared_ptr<arrow::dataset::DatasetFactory> d =
+ RetrieveNativeInstance<arrow::dataset::DatasetFactory>(dataset_factory_id);
+ std::shared_ptr<arrow::Schema> schema;
+ schema = JniGetOrThrow(FromSchemaByteArray(env, schema_bytes));
+ std::shared_ptr<arrow::dataset::Dataset> dataset = JniGetOrThrow(d->Finish(schema));
+ return CreateNativeRef(dataset);
+ JNI_METHOD_END(-1L)
+}
+
+/*
+ * Class: org_apache_arrow_dataset_jni_JniWrapper
+ * Method: closeDataset
+ * Signature: (J)V
+ */
+JNIEXPORT void JNICALL Java_org_apache_arrow_dataset_jni_JniWrapper_closeDataset(
+ JNIEnv* env, jobject, jlong id) {
+ JNI_METHOD_START
+ ReleaseNativeRef<arrow::dataset::Dataset>(id);
+ JNI_METHOD_END()
+}
+
+/// \class DisposableScannerAdaptor
+/// \brief An adaptor that iterates over a Scanner instance then returns RecordBatches
+/// directly.
+///
+/// This lessens the complexity of the JNI bridge to make sure it to be easier to
+/// maintain. On Java-side, NativeScanner can only produces a single NativeScanTask
+/// instance during its whole lifecycle. Each task stands for a DisposableScannerAdaptor
+/// instance through JNI bridge.
+///
+class DisposableScannerAdaptor {
Review comment:
So let's just remove this and add a method `Iterator<ScanTask> Scanner.scan()` to JNI bridge? I have same concern with you and for me this code is temporary for several versions. I think we wanted to keep JNI bridge simple enough at the first implementation so that if C++ API get factored again in future we may save time from changing JNI code to make Java side work.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] emkornfield commented on a change in pull request #7030: ARROW-7808: [Java][Dataset] Implement Datasets Java API by JNI to C++
Posted by GitBox <gi...@apache.org>.
emkornfield commented on a change in pull request #7030:
URL: https://github.com/apache/arrow/pull/7030#discussion_r425565612
##########
File path: java/dataset/src/main/java/org/apache/arrow/dataset/source/DatasetFactory.java
##########
@@ -0,0 +1,51 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.arrow.dataset.source;
+
+import org.apache.arrow.vector.types.pojo.Schema;
+
+/**
+ * DataSourceDiscovery provides a way to inspect a DataSource potential
Review comment:
this name doesn't match the class.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] emkornfield commented on a change in pull request #7030: ARROW-7808: [Java][Dataset] Implement Datasets Java API by JNI to C++
Posted by GitBox <gi...@apache.org>.
emkornfield commented on a change in pull request #7030:
URL: https://github.com/apache/arrow/pull/7030#discussion_r426355718
##########
File path: cpp/src/jni/dataset/proto/Types.proto
##########
@@ -0,0 +1,149 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+syntax = "proto2";
+package types;
+
+option java_package = "org.apache.arrow.dataset";
Review comment:
I do not have a solution off the top of my head. But creating a new java->C++ protobuf abstraction seems like the the wrong way to go.
As far as i can tell gandiva [protobuf declare column type optional](https://github.com/apache/arrow/blob/116672fed5cd8b7d80e5e8bd93615704aa10eca9/cpp/src/gandiva/proto/Types.proto#L93). This needs more design which is why a discussion/proposal on the ML is warranted.
It might make sense to split out the filtering components of this PR so it doesn't get held up by the conversation.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] emkornfield commented on a change in pull request #7030: ARROW-7808: [Java][Dataset] Implement Datasets Java API
Posted by GitBox <gi...@apache.org>.
emkornfield commented on a change in pull request #7030:
URL: https://github.com/apache/arrow/pull/7030#discussion_r414989258
##########
File path: java/dataset/src/test/java/org/apache/arrow/dataset/jni/NativeDatasetTest.java
##########
@@ -0,0 +1,209 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.arrow.dataset.jni;
+
+import java.io.File;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Spliterator;
+import java.util.Spliterators;
+import java.util.stream.Collectors;
+import java.util.stream.StreamSupport;
+
+import org.apache.arrow.dataset.DatasetTypes;
+import org.apache.arrow.dataset.file.FileFormat;
+import org.apache.arrow.dataset.file.FileSystem;
+import org.apache.arrow.dataset.file.SingleFileDatasetFactory;
+import org.apache.arrow.dataset.filter.Filter;
+import org.apache.arrow.dataset.filter.FilterImpl;
+import org.apache.arrow.dataset.scanner.ScanOptions;
+import org.apache.arrow.dataset.scanner.ScanTask;
+import org.apache.arrow.dataset.scanner.Scanner;
+import org.apache.arrow.dataset.source.Dataset;
+import org.apache.arrow.dataset.source.DatasetFactory;
+import org.apache.arrow.memory.RootAllocator;
+import org.apache.arrow.vector.VectorSchemaRoot;
+import org.apache.arrow.vector.types.pojo.Schema;
+import org.junit.Assert;
+import org.junit.Ignore;
+import org.junit.Test;
+
+public class NativeDatasetTest {
+
+ private String sampleParquetPath() {
+ return NativeDatasetTest.class.getResource(File.separator + "userdata.parquet").getPath();
Review comment:
please don't check the binary file into this repo. It would be better to generate a file on the fly, so readers of the test can understand what data exists there. If a standalone parquet file is really needed [arrow-testing](https://github.com/apache/arrow-testing) is the place to put it
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] zhztheplayer commented on pull request #7030: ARROW-7808: [Java][Dataset] Implement Datasets Java API by JNI to C++
Posted by GitBox <gi...@apache.org>.
zhztheplayer commented on pull request #7030:
URL: https://github.com/apache/arrow/pull/7030#issuecomment-620528808
Hi @wesm @emkornfield , based on the discussion in the [latest commit](https://github.com/apache/arrow/pull/7030/commits/7b16a41d0f9f3265f00c92ce50304b68ee03315f) I removed some of the JNI mappings of c++ components. Now only Dataset(Factory) and Scanner are there to be bridged. Would you think of this a feasible direction?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] emkornfield commented on a change in pull request #7030: ARROW-7808: [Java][Dataset] Implement Dataset Java API by JNI to C++
Posted by GitBox <gi...@apache.org>.
emkornfield commented on a change in pull request #7030:
URL: https://github.com/apache/arrow/pull/7030#discussion_r567201270
##########
File path: cpp/src/jni/dataset/jni_wrapper.cpp
##########
@@ -0,0 +1,859 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include <mutex>
+
+#include <arrow/array.h>
+#include <arrow/dataset/api.h>
+#include <arrow/dataset/file_base.h>
+#include <arrow/filesystem/localfs.h>
+#include <arrow/io/api.h>
+#include <arrow/ipc/api.h>
+#include <arrow/util/iterator.h>
+#include <arrow/util/logging.h>
+
+#include "org_apache_arrow_dataset_file_JniWrapper.h"
+#include "org_apache_arrow_dataset_jni_JniWrapper.h"
+#include "org_apache_arrow_dataset_jni_NativeMemoryPool.h"
+
+namespace arrow {
+namespace dataset {
+namespace jni {
+
+static jclass illegal_access_exception_class;
+static jclass illegal_argument_exception_class;
+static jclass runtime_exception_class;
+
+static jclass record_batch_handle_class;
+static jclass record_batch_handle_field_class;
+static jclass record_batch_handle_buffer_class;
+static jclass java_reservation_listener_class;
+
+static jmethodID record_batch_handle_constructor;
+static jmethodID record_batch_handle_field_constructor;
+static jmethodID record_batch_handle_buffer_constructor;
+static jmethodID reserve_memory_method;
+static jmethodID unreserve_memory_method;
+
+static jlong default_memory_pool_id = -1L;
+
+static jint JNI_VERSION = JNI_VERSION_1_6;
+
+class JniPendingException : public std::runtime_error {
+ public:
+ explicit JniPendingException(const std::string& arg) : runtime_error(arg) {}
+};
+
+void ThrowPendingException(const std::string& message) {
+ throw JniPendingException(message);
+}
+
+template <typename T>
+T JniGetOrThrow(arrow::Result<T> result) {
+ if (!result.status().ok()) {
+ ThrowPendingException(result.status().message());
+ }
+ return std::move(result).ValueOrDie();
+}
+
+void JniAssertOkOrThrow(arrow::Status status) {
+ if (!status.ok()) {
+ ThrowPendingException(status.message());
+ }
+}
+
+void JniThrow(std::string message) { ThrowPendingException(message); }
+
+jclass CreateGlobalClassReference(JNIEnv* env, const char* class_name) {
+ jclass local_class = env->FindClass(class_name);
+ jclass global_class = (jclass)env->NewGlobalRef(local_class);
+ env->DeleteLocalRef(local_class);
+ return global_class;
+}
+
+arrow::Result<jmethodID> GetMethodID(JNIEnv* env, jclass this_class, const char* name,
+ const char* sig) {
+ jmethodID ret = env->GetMethodID(this_class, name, sig);
+ if (ret == nullptr) {
+ std::string error_message = "Unable to find method " + std::string(name) +
+ " within signature" + std::string(sig);
+ return arrow::Status::Invalid(error_message);
+ }
+ return ret;
+}
+
+arrow::Result<jmethodID> GetStaticMethodID(JNIEnv* env, jclass this_class,
+ const char* name, const char* sig) {
+ jmethodID ret = env->GetStaticMethodID(this_class, name, sig);
+ if (ret == nullptr) {
+ std::string error_message = "Unable to find static method " + std::string(name) +
+ " within signature" + std::string(sig);
+ return arrow::Status::Invalid(error_message);
+ }
+ return ret;
+}
+
+std::shared_ptr<arrow::Schema> SchemaFromColumnNames(
+ const std::shared_ptr<arrow::Schema>& input,
+ const std::vector<std::string>& column_names) {
+ std::vector<std::shared_ptr<arrow::Field>> columns;
+ for (const auto& name : column_names) {
+ columns.push_back(input->GetFieldByName(name));
+ }
+ return std::make_shared<arrow::Schema>(columns);
+}
+
+arrow::Result<std::shared_ptr<arrow::dataset::FileFormat>> GetFileFormat(jint id) {
+ switch (id) {
+ case 0:
+ return std::make_shared<arrow::dataset::ParquetFileFormat>();
+ default:
+ std::string error_message = "illegal file format id: " + std::to_string(id);
+ return arrow::Status::Invalid(error_message);
+ }
+}
+
+std::string JStringToCString(JNIEnv* env, jstring string) {
+ if (string == nullptr) {
+ return std::string();
+ }
+ jboolean copied;
+ const char* chars = env->GetStringUTFChars(string, &copied);
+ std::string ret = strdup(chars);
Review comment:
this causes an extra copy just construct directly with chars.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] zhztheplayer edited a comment on pull request #7030: ARROW-7808: [Java][Dataset] Implement Dataset Java API by JNI to C++
Posted by GitBox <gi...@apache.org>.
zhztheplayer edited a comment on pull request #7030:
URL: https://github.com/apache/arrow/pull/7030#issuecomment-772537663
Hi @emkornfield. I've addressed the new comments within the latest commit, please review. Previous commits have been merged into a single one.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] emkornfield commented on a change in pull request #7030: ARROW-7808: [Java][Dataset] Implement Dataset Java API by JNI to C++
Posted by GitBox <gi...@apache.org>.
emkornfield commented on a change in pull request #7030:
URL: https://github.com/apache/arrow/pull/7030#discussion_r551610223
##########
File path: cpp/src/arrow/memory_pool.cc
##########
@@ -534,4 +535,139 @@ int64_t ProxyMemoryPool::max_memory() const { return impl_->max_memory(); }
std::string ProxyMemoryPool::backend_name() const { return impl_->backend_name(); }
+ReservationListener::~ReservationListener() {}
Review comment:
this and the constructor aren't needed if you use = default as suggested in the header.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] Sebastiaan-Alvarez-Rodriguez edited a comment on pull request #7030: ARROW-7808: [Java][Dataset] Implement Datasets Java API by JNI to C++
Posted by GitBox <gi...@apache.org>.
Sebastiaan-Alvarez-Rodriguez edited a comment on pull request #7030:
URL: https://github.com/apache/arrow/pull/7030#issuecomment-694925466
hi there,
Could you please add a few compilation/installation instructions to your branch?
I just completed a long debug session to find out why the JNI runtime did not work.
### CPP
Compiled `<project-root>/cpp/` project using:
```bash
cd <project-root>/cpp
cmake -DARROW_DATASET=ON -DARROW_JNI=ON -DARROW_PARQUET=ON -DARROW_IPC=ON .
sudo make install
```
Confirmed shared libs are indeed visible using `whereis libarrow_dataset_jni`.
### Java
Compiled `<project-root>/java/dataset/` using
```bash
cd <project-root>/java/dataset
mvn clean install -Dmaven.test.skip=true # skipping tests to let build succeed
```
To test whether it works, successfully ran a sample program:
```java
public class test {
public static void main(String[] args) {
System.loadLibrary("arrow_dataset_jni");
System.out.println("System loading works fine");
}
}
```
So, my sample program's classloader can access `libarrow_dataset_jni.so` just fine.
However, the following does not work:
```java
import org.apache.arrow.memory.RootAllocator;
import org.apache.arrow.dataset.file.FileFormat;
import org.apache.arrow.dataset.file.FileSystemDatasetFactory;
public class test {
private static FileSystemDatasetFactory getDatasetFactory() {
RootAllocator allocator = new RootAllocator(Long.MAX_VALUE);
return new FileSystemDatasetFactory(allocator, FileFormat.PARQUET, "/path/to/pq.parquet"); //crash
}
public static void main(String[] args) {
System.loadLibrary("arrow_dataset_jni");
System.out.println("Own classLoader works fine");
FileSystemDatasetFactory test = getDatasetFactory();
System.out.println("Not so good! I crash before I can print this");
}
}
```
The relevant part of the crash is:
```
Caused by: java.lang.IllegalStateException: error loading native libraries: java.io.FileNotFoundException: libarrow_dataset_jni.so
at org.apache.arrow.dataset.jni.JniLoader.load(JniLoader.java:91)
at org.apache.arrow.dataset.jni.JniLoader.loadRemaining(JniLoader.java:73)
at org.apache.arrow.dataset.jni.JniLoader.ensureLoaded(JniLoader.java:60)
at org.apache.arrow.dataset.file.JniWrapper.<init>(JniWrapper.java:34)
```
The source code of the Dataset Java library causing the error is found [here](https://github.com/zhztheplayer/arrow-1/blob/fd163e199a10a7225765b0c30fbf60d8df8d20db/java/dataset/src/main/java/org/apache/arrow/dataset/jni/JniLoader.java#L78-L94).
The problem occurs because `JniWrapper.class.getClassLoader().getResourceAsStream(libraryToLoad)` always returns `null`.
Only when adding a symlink (`ln -s /usr/local/lib/libarrow_dataset_jni.so libarrow_dataset_jni.so`) to the package directory, it is able to find the correct file.
Why do you create a temporary file, store the contents of the shared library in it, and then call `System.load(tmpfile)` and not just directly call it using `System.loadLibrary(libraryToLoad);`?
Maybe it is a good idea to pick one of the following:
1. Change the `'java.io.FileNotFoundException: libarrow_dataset_jni.so'` to display the full path (so people know where your library looks to find the shared object)
2. Make a small `JNI_dataset_dev_install_note.md` somewhere on getting this library to work.
3. Use `System.loadLibrary(libraryToLoad);`
4. Use some other way to find the location of a shared library and copy it to a tmpfile
I believe it would help many people who are not familiar with `getClassLoader().getResourceAsStream(...)` (like myself)
Have a nice day,
Sebastiaan
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] zhztheplayer commented on a change in pull request #7030: ARROW-7808: [Java][Dataset] Implement Dataset Java API by JNI to C++
Posted by GitBox <gi...@apache.org>.
zhztheplayer commented on a change in pull request #7030:
URL: https://github.com/apache/arrow/pull/7030#discussion_r567762467
##########
File path: cpp/src/jni/dataset/jni_wrapper.cpp
##########
@@ -0,0 +1,859 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include <mutex>
+
+#include <arrow/array.h>
+#include <arrow/dataset/api.h>
+#include <arrow/dataset/file_base.h>
+#include <arrow/filesystem/localfs.h>
+#include <arrow/io/api.h>
+#include <arrow/ipc/api.h>
+#include <arrow/util/iterator.h>
+#include <arrow/util/logging.h>
+
+#include "org_apache_arrow_dataset_file_JniWrapper.h"
+#include "org_apache_arrow_dataset_jni_JniWrapper.h"
+#include "org_apache_arrow_dataset_jni_NativeMemoryPool.h"
+
+namespace arrow {
+namespace dataset {
+namespace jni {
+
+static jclass illegal_access_exception_class;
+static jclass illegal_argument_exception_class;
+static jclass runtime_exception_class;
+
+static jclass record_batch_handle_class;
+static jclass record_batch_handle_field_class;
+static jclass record_batch_handle_buffer_class;
+static jclass java_reservation_listener_class;
+
+static jmethodID record_batch_handle_constructor;
+static jmethodID record_batch_handle_field_constructor;
+static jmethodID record_batch_handle_buffer_constructor;
+static jmethodID reserve_memory_method;
+static jmethodID unreserve_memory_method;
+
+static jlong default_memory_pool_id = -1L;
+
+static jint JNI_VERSION = JNI_VERSION_1_6;
+
+class JniPendingException : public std::runtime_error {
+ public:
+ explicit JniPendingException(const std::string& arg) : runtime_error(arg) {}
+};
+
+void ThrowPendingException(const std::string& message) {
+ throw JniPendingException(message);
+}
+
+template <typename T>
+T JniGetOrThrow(arrow::Result<T> result) {
+ if (!result.status().ok()) {
+ ThrowPendingException(result.status().message());
+ }
+ return std::move(result).ValueOrDie();
+}
+
+void JniAssertOkOrThrow(arrow::Status status) {
+ if (!status.ok()) {
+ ThrowPendingException(status.message());
+ }
+}
+
+void JniThrow(std::string message) { ThrowPendingException(message); }
+
+jclass CreateGlobalClassReference(JNIEnv* env, const char* class_name) {
+ jclass local_class = env->FindClass(class_name);
+ jclass global_class = (jclass)env->NewGlobalRef(local_class);
+ env->DeleteLocalRef(local_class);
+ return global_class;
+}
+
+arrow::Result<jmethodID> GetMethodID(JNIEnv* env, jclass this_class, const char* name,
+ const char* sig) {
+ jmethodID ret = env->GetMethodID(this_class, name, sig);
+ if (ret == nullptr) {
+ std::string error_message = "Unable to find method " + std::string(name) +
+ " within signature" + std::string(sig);
+ return arrow::Status::Invalid(error_message);
+ }
+ return ret;
+}
+
+arrow::Result<jmethodID> GetStaticMethodID(JNIEnv* env, jclass this_class,
+ const char* name, const char* sig) {
+ jmethodID ret = env->GetStaticMethodID(this_class, name, sig);
+ if (ret == nullptr) {
+ std::string error_message = "Unable to find static method " + std::string(name) +
+ " within signature" + std::string(sig);
+ return arrow::Status::Invalid(error_message);
+ }
+ return ret;
+}
+
+std::shared_ptr<arrow::Schema> SchemaFromColumnNames(
+ const std::shared_ptr<arrow::Schema>& input,
+ const std::vector<std::string>& column_names) {
+ std::vector<std::shared_ptr<arrow::Field>> columns;
+ for (const auto& name : column_names) {
+ columns.push_back(input->GetFieldByName(name));
+ }
+ return std::make_shared<arrow::Schema>(columns);
+}
+
+arrow::Result<std::shared_ptr<arrow::dataset::FileFormat>> GetFileFormat(jint id) {
+ switch (id) {
+ case 0:
+ return std::make_shared<arrow::dataset::ParquetFileFormat>();
+ default:
+ std::string error_message = "illegal file format id: " + std::to_string(id);
+ return arrow::Status::Invalid(error_message);
+ }
+}
+
+std::string JStringToCString(JNIEnv* env, jstring string) {
+ if (string == nullptr) {
+ return std::string();
+ }
+ jboolean copied;
+ const char* chars = env->GetStringUTFChars(string, &copied);
+ std::string ret = strdup(chars);
+ env->ReleaseStringUTFChars(string, chars);
+ return ret;
+}
+
+std::vector<std::string> ToStringVector(JNIEnv* env, jobjectArray& str_array) {
+ int length = env->GetArrayLength(str_array);
+ std::vector<std::string> vector;
+ for (int i = 0; i < length; i++) {
+ auto string = (jstring)(env->GetObjectArrayElement(str_array, i));
+ vector.push_back(JStringToCString(env, string));
+ }
+ return vector;
+}
+
+template <typename T>
+jlong CreateNativeRef(std::shared_ptr<T> t) {
+ std::shared_ptr<T>* retained_ptr = new std::shared_ptr<T>(t);
+ return reinterpret_cast<jlong>(retained_ptr);
+}
+
+template <typename T>
+std::shared_ptr<T> RetrieveNativeInstance(jlong ref) {
+ std::shared_ptr<T>* retrieved_ptr = reinterpret_cast<std::shared_ptr<T>*>(ref);
+ return *retrieved_ptr;
+}
+
+template <typename T>
+void ReleaseNativeRef(jlong ref) {
+ std::shared_ptr<T>* retrieved_ptr = reinterpret_cast<std::shared_ptr<T>*>(ref);
+ delete retrieved_ptr;
+}
+
+arrow::Result<jbyteArray> ToSchemaByteArray(JNIEnv* env,
+ std::shared_ptr<arrow::Schema> schema) {
+ ARROW_ASSIGN_OR_RAISE(
+ std::shared_ptr<arrow::Buffer> buffer,
+ arrow::ipc::SerializeSchema(*schema, arrow::default_memory_pool()))
+
+ jbyteArray out = env->NewByteArray(buffer->size());
+ auto src = reinterpret_cast<const jbyte*>(buffer->data());
+ env->SetByteArrayRegion(out, 0, buffer->size(), src);
+ return out;
+}
+
+arrow::Result<std::shared_ptr<arrow::Schema>> FromSchemaByteArray(
+ JNIEnv* env, jbyteArray schemaBytes) {
+ arrow::ipc::DictionaryMemo in_memo;
+ int schemaBytes_len = env->GetArrayLength(schemaBytes);
+ jbyte* schemaBytes_data = env->GetByteArrayElements(schemaBytes, nullptr);
+ auto serialized_schema = std::make_shared<arrow::Buffer>(
+ reinterpret_cast<uint8_t*>(schemaBytes_data), schemaBytes_len);
+ arrow::io::BufferReader buf_reader(serialized_schema);
+ ARROW_ASSIGN_OR_RAISE(std::shared_ptr<arrow::Schema> schema,
+ arrow::ipc::ReadSchema(&buf_reader, &in_memo))
+ env->ReleaseByteArrayElements(schemaBytes, schemaBytes_data, JNI_ABORT);
+ return schema;
+}
+
+/// Listener to act on reservations/unreservations from ReservationListenableMemoryPool.
+///
+/// Note the memory pool will call this listener only on block-level memory
+/// reservation/unreservation is granted. So the invocation parameter "size" is always
+/// multiple of block size (by default, 512k) specified in memory pool.
+class ReservationListener {
+ public:
+ virtual ~ReservationListener() = default;
+
+ virtual arrow::Status OnReservation(int64_t size) = 0;
+ virtual arrow::Status OnRelease(int64_t size) = 0;
+
+ protected:
+ ReservationListener() = default;
+};
+
+class ReservationListenableMemoryPoolImpl {
+ public:
+ explicit ReservationListenableMemoryPoolImpl(
+ arrow::MemoryPool* pool, std::shared_ptr<ReservationListener> listener,
+ int64_t block_size)
+ : pool_(pool),
+ listener_(listener),
+ block_size_(block_size),
+ blocks_reserved_(0),
+ bytes_reserved_(0) {}
+
+ arrow::Status Allocate(int64_t size, uint8_t** out) {
+ RETURN_NOT_OK(UpdateReservation(size));
+ arrow::Status error = pool_->Allocate(size, out);
+ if (!error.ok()) {
+ RETURN_NOT_OK(UpdateReservation(-size));
+ return error;
+ }
+ return arrow::Status::OK();
+ }
+
+ arrow::Status Reallocate(int64_t old_size, int64_t new_size, uint8_t** ptr) {
+ bool reserved = false;
+ int64_t diff = new_size - old_size;
+ if (new_size >= old_size) {
+ // new_size >= old_size, pre-reserve bytes from listener before allocating
+ // from underlying pool
+ RETURN_NOT_OK(UpdateReservation(diff));
+ reserved = true;
+ }
+ arrow::Status error = pool_->Reallocate(old_size, new_size, ptr);
+ if (!error.ok()) {
+ if (reserved) {
+ // roll back reservations on error
+ RETURN_NOT_OK(UpdateReservation(-diff));
+ }
+ return error;
+ }
+ if (!reserved) {
+ // otherwise (e.g. new_size < old_size), make updates after calling underlying pool
+ RETURN_NOT_OK(UpdateReservation(diff));
+ }
+ return arrow::Status::OK();
+ }
+
+ void Free(uint8_t* buffer, int64_t size) {
+ pool_->Free(buffer, size);
+ // FIXME: See ARROW-11143, currently method ::Free doesn't allow Status return
+ arrow::Status s = UpdateReservation(-size);
+ if (!s.ok()) {
+ ARROW_LOG(FATAL) << "Failed to update reservation while freeing bytes: "
+ << s.message();
+ return;
+ }
+ }
+
+ arrow::Status UpdateReservation(int64_t diff) {
+ int64_t granted = Reserve(diff);
+ if (granted == 0) {
+ return arrow::Status::OK();
+ }
+ if (granted < 0) {
+ RETURN_NOT_OK(listener_->OnRelease(-granted));
+ return arrow::Status::OK();
+ }
+ RETURN_NOT_OK(listener_->OnReservation(granted));
+ return arrow::Status::OK();
+ }
+
+ int64_t Reserve(int64_t diff) {
+ std::lock_guard<std::mutex> lock(mutex_);
+ bytes_reserved_ += diff;
+ int64_t new_block_count;
+ if (bytes_reserved_ == 0) {
+ new_block_count = 0;
+ } else {
+ // ceil to get the required block number
+ new_block_count = (bytes_reserved_ - 1) / block_size_ + 1;
+ }
+ int64_t bytes_granted = (new_block_count - blocks_reserved_) * block_size_;
+ blocks_reserved_ = new_block_count;
+ return bytes_granted;
+ }
+
+ int64_t bytes_allocated() { return pool_->bytes_allocated(); }
+
+ int64_t max_memory() { return pool_->max_memory(); }
+
+ std::string backend_name() { return pool_->backend_name(); }
+
+ std::shared_ptr<ReservationListener> get_listener() { return listener_; }
+
+ private:
+ arrow::MemoryPool* pool_;
+ std::shared_ptr<ReservationListener> listener_;
+ int64_t block_size_;
+ int64_t blocks_reserved_;
+ int64_t bytes_reserved_;
+ std::mutex mutex_;
+};
+
+/// A memory pool implementation for pre-reserving memory blocks from a
Review comment:
I've made some refactors on `jni_wrapper.cpp`. Now there is a new source file `jni_util.cpp`/`jni_util.h` for storing util members in namespace `arrow::dataset::jni`.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] emkornfield commented on a change in pull request #7030: ARROW-7808: [Java][Dataset] Implement Dataset Java API by JNI to C++
Posted by GitBox <gi...@apache.org>.
emkornfield commented on a change in pull request #7030:
URL: https://github.com/apache/arrow/pull/7030#discussion_r567202084
##########
File path: cpp/src/jni/dataset/jni_wrapper.cpp
##########
@@ -0,0 +1,859 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include <mutex>
+
+#include <arrow/array.h>
+#include <arrow/dataset/api.h>
+#include <arrow/dataset/file_base.h>
+#include <arrow/filesystem/localfs.h>
+#include <arrow/io/api.h>
+#include <arrow/ipc/api.h>
+#include <arrow/util/iterator.h>
+#include <arrow/util/logging.h>
+
+#include "org_apache_arrow_dataset_file_JniWrapper.h"
+#include "org_apache_arrow_dataset_jni_JniWrapper.h"
+#include "org_apache_arrow_dataset_jni_NativeMemoryPool.h"
+
+namespace arrow {
+namespace dataset {
+namespace jni {
+
+static jclass illegal_access_exception_class;
+static jclass illegal_argument_exception_class;
+static jclass runtime_exception_class;
+
+static jclass record_batch_handle_class;
+static jclass record_batch_handle_field_class;
+static jclass record_batch_handle_buffer_class;
+static jclass java_reservation_listener_class;
+
+static jmethodID record_batch_handle_constructor;
+static jmethodID record_batch_handle_field_constructor;
+static jmethodID record_batch_handle_buffer_constructor;
+static jmethodID reserve_memory_method;
+static jmethodID unreserve_memory_method;
+
+static jlong default_memory_pool_id = -1L;
+
+static jint JNI_VERSION = JNI_VERSION_1_6;
+
+class JniPendingException : public std::runtime_error {
+ public:
+ explicit JniPendingException(const std::string& arg) : runtime_error(arg) {}
+};
+
+void ThrowPendingException(const std::string& message) {
+ throw JniPendingException(message);
+}
+
+template <typename T>
+T JniGetOrThrow(arrow::Result<T> result) {
+ if (!result.status().ok()) {
+ ThrowPendingException(result.status().message());
+ }
+ return std::move(result).ValueOrDie();
+}
+
+void JniAssertOkOrThrow(arrow::Status status) {
+ if (!status.ok()) {
+ ThrowPendingException(status.message());
+ }
+}
+
+void JniThrow(std::string message) { ThrowPendingException(message); }
+
+jclass CreateGlobalClassReference(JNIEnv* env, const char* class_name) {
+ jclass local_class = env->FindClass(class_name);
+ jclass global_class = (jclass)env->NewGlobalRef(local_class);
+ env->DeleteLocalRef(local_class);
+ return global_class;
+}
+
+arrow::Result<jmethodID> GetMethodID(JNIEnv* env, jclass this_class, const char* name,
+ const char* sig) {
+ jmethodID ret = env->GetMethodID(this_class, name, sig);
+ if (ret == nullptr) {
+ std::string error_message = "Unable to find method " + std::string(name) +
+ " within signature" + std::string(sig);
+ return arrow::Status::Invalid(error_message);
+ }
+ return ret;
+}
+
+arrow::Result<jmethodID> GetStaticMethodID(JNIEnv* env, jclass this_class,
+ const char* name, const char* sig) {
+ jmethodID ret = env->GetStaticMethodID(this_class, name, sig);
+ if (ret == nullptr) {
+ std::string error_message = "Unable to find static method " + std::string(name) +
+ " within signature" + std::string(sig);
+ return arrow::Status::Invalid(error_message);
+ }
+ return ret;
+}
+
+std::shared_ptr<arrow::Schema> SchemaFromColumnNames(
+ const std::shared_ptr<arrow::Schema>& input,
+ const std::vector<std::string>& column_names) {
+ std::vector<std::shared_ptr<arrow::Field>> columns;
+ for (const auto& name : column_names) {
+ columns.push_back(input->GetFieldByName(name));
+ }
+ return std::make_shared<arrow::Schema>(columns);
+}
+
+arrow::Result<std::shared_ptr<arrow::dataset::FileFormat>> GetFileFormat(jint id) {
+ switch (id) {
+ case 0:
+ return std::make_shared<arrow::dataset::ParquetFileFormat>();
+ default:
+ std::string error_message = "illegal file format id: " + std::to_string(id);
+ return arrow::Status::Invalid(error_message);
+ }
+}
+
+std::string JStringToCString(JNIEnv* env, jstring string) {
+ if (string == nullptr) {
+ return std::string();
+ }
+ jboolean copied;
+ const char* chars = env->GetStringUTFChars(string, &copied);
+ std::string ret = strdup(chars);
+ env->ReleaseStringUTFChars(string, chars);
+ return ret;
+}
+
+std::vector<std::string> ToStringVector(JNIEnv* env, jobjectArray& str_array) {
+ int length = env->GetArrayLength(str_array);
+ std::vector<std::string> vector;
+ for (int i = 0; i < length; i++) {
+ auto string = (jstring)(env->GetObjectArrayElement(str_array, i));
+ vector.push_back(JStringToCString(env, string));
+ }
+ return vector;
+}
+
+template <typename T>
+jlong CreateNativeRef(std::shared_ptr<T> t) {
+ std::shared_ptr<T>* retained_ptr = new std::shared_ptr<T>(t);
+ return reinterpret_cast<jlong>(retained_ptr);
+}
+
+template <typename T>
+std::shared_ptr<T> RetrieveNativeInstance(jlong ref) {
+ std::shared_ptr<T>* retrieved_ptr = reinterpret_cast<std::shared_ptr<T>*>(ref);
+ return *retrieved_ptr;
+}
+
+template <typename T>
+void ReleaseNativeRef(jlong ref) {
+ std::shared_ptr<T>* retrieved_ptr = reinterpret_cast<std::shared_ptr<T>*>(ref);
+ delete retrieved_ptr;
+}
+
+arrow::Result<jbyteArray> ToSchemaByteArray(JNIEnv* env,
+ std::shared_ptr<arrow::Schema> schema) {
+ ARROW_ASSIGN_OR_RAISE(
+ std::shared_ptr<arrow::Buffer> buffer,
+ arrow::ipc::SerializeSchema(*schema, arrow::default_memory_pool()))
+
+ jbyteArray out = env->NewByteArray(buffer->size());
+ auto src = reinterpret_cast<const jbyte*>(buffer->data());
+ env->SetByteArrayRegion(out, 0, buffer->size(), src);
+ return out;
+}
+
+arrow::Result<std::shared_ptr<arrow::Schema>> FromSchemaByteArray(
+ JNIEnv* env, jbyteArray schemaBytes) {
+ arrow::ipc::DictionaryMemo in_memo;
+ int schemaBytes_len = env->GetArrayLength(schemaBytes);
+ jbyte* schemaBytes_data = env->GetByteArrayElements(schemaBytes, nullptr);
+ auto serialized_schema = std::make_shared<arrow::Buffer>(
+ reinterpret_cast<uint8_t*>(schemaBytes_data), schemaBytes_len);
+ arrow::io::BufferReader buf_reader(serialized_schema);
+ ARROW_ASSIGN_OR_RAISE(std::shared_ptr<arrow::Schema> schema,
+ arrow::ipc::ReadSchema(&buf_reader, &in_memo))
+ env->ReleaseByteArrayElements(schemaBytes, schemaBytes_data, JNI_ABORT);
+ return schema;
+}
+
+/// Listener to act on reservations/unreservations from ReservationListenableMemoryPool.
+///
+/// Note the memory pool will call this listener only on block-level memory
+/// reservation/unreservation is granted. So the invocation parameter "size" is always
+/// multiple of block size (by default, 512k) specified in memory pool.
+class ReservationListener {
+ public:
+ virtual ~ReservationListener() = default;
+
+ virtual arrow::Status OnReservation(int64_t size) = 0;
+ virtual arrow::Status OnRelease(int64_t size) = 0;
+
+ protected:
+ ReservationListener() = default;
+};
+
+class ReservationListenableMemoryPoolImpl {
+ public:
+ explicit ReservationListenableMemoryPoolImpl(
+ arrow::MemoryPool* pool, std::shared_ptr<ReservationListener> listener,
+ int64_t block_size)
+ : pool_(pool),
+ listener_(listener),
+ block_size_(block_size),
+ blocks_reserved_(0),
+ bytes_reserved_(0) {}
+
+ arrow::Status Allocate(int64_t size, uint8_t** out) {
+ RETURN_NOT_OK(UpdateReservation(size));
+ arrow::Status error = pool_->Allocate(size, out);
+ if (!error.ok()) {
+ RETURN_NOT_OK(UpdateReservation(-size));
+ return error;
+ }
+ return arrow::Status::OK();
+ }
+
+ arrow::Status Reallocate(int64_t old_size, int64_t new_size, uint8_t** ptr) {
+ bool reserved = false;
+ int64_t diff = new_size - old_size;
+ if (new_size >= old_size) {
+ // new_size >= old_size, pre-reserve bytes from listener before allocating
+ // from underlying pool
+ RETURN_NOT_OK(UpdateReservation(diff));
+ reserved = true;
+ }
+ arrow::Status error = pool_->Reallocate(old_size, new_size, ptr);
+ if (!error.ok()) {
+ if (reserved) {
+ // roll back reservations on error
+ RETURN_NOT_OK(UpdateReservation(-diff));
+ }
+ return error;
+ }
+ if (!reserved) {
+ // otherwise (e.g. new_size < old_size), make updates after calling underlying pool
+ RETURN_NOT_OK(UpdateReservation(diff));
+ }
+ return arrow::Status::OK();
+ }
+
+ void Free(uint8_t* buffer, int64_t size) {
+ pool_->Free(buffer, size);
+ // FIXME: See ARROW-11143, currently method ::Free doesn't allow Status return
+ arrow::Status s = UpdateReservation(-size);
+ if (!s.ok()) {
+ ARROW_LOG(FATAL) << "Failed to update reservation while freeing bytes: "
+ << s.message();
+ return;
+ }
+ }
+
+ arrow::Status UpdateReservation(int64_t diff) {
+ int64_t granted = Reserve(diff);
+ if (granted == 0) {
+ return arrow::Status::OK();
+ }
+ if (granted < 0) {
+ RETURN_NOT_OK(listener_->OnRelease(-granted));
+ return arrow::Status::OK();
+ }
+ RETURN_NOT_OK(listener_->OnReservation(granted));
+ return arrow::Status::OK();
+ }
+
+ int64_t Reserve(int64_t diff) {
+ std::lock_guard<std::mutex> lock(mutex_);
+ bytes_reserved_ += diff;
+ int64_t new_block_count;
+ if (bytes_reserved_ == 0) {
+ new_block_count = 0;
+ } else {
+ // ceil to get the required block number
+ new_block_count = (bytes_reserved_ - 1) / block_size_ + 1;
+ }
+ int64_t bytes_granted = (new_block_count - blocks_reserved_) * block_size_;
+ blocks_reserved_ = new_block_count;
+ return bytes_granted;
+ }
+
+ int64_t bytes_allocated() { return pool_->bytes_allocated(); }
+
+ int64_t max_memory() { return pool_->max_memory(); }
+
+ std::string backend_name() { return pool_->backend_name(); }
+
+ std::shared_ptr<ReservationListener> get_listener() { return listener_; }
+
+ private:
+ arrow::MemoryPool* pool_;
+ std::shared_ptr<ReservationListener> listener_;
+ int64_t block_size_;
+ int64_t blocks_reserved_;
+ int64_t bytes_reserved_;
+ std::mutex mutex_;
+};
+
+/// A memory pool implementation for pre-reserving memory blocks from a
Review comment:
there is no reason for the pimpl pattern if both interface and implementation are bot in the .cpp file.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] zhztheplayer commented on pull request #7030: ARROW-7808: [Java][Dataset] Implement Datasets Java API by JNI to C++
Posted by GitBox <gi...@apache.org>.
zhztheplayer commented on pull request #7030:
URL: https://github.com/apache/arrow/pull/7030#issuecomment-700390598
Hi @Sebastiaan-Alvarez-Rodriguez ,
First of all, thanks for the detailed explanation about the issue and your suggestions. Sorry for the late reply.
Actually I don't mind to change approach from `System.load` to `System.loadLibrary` as long as it too works fine or even better. I am just worried about why `getResourceAsStream` returns null in your case.
Just saw your practice on compiling Java code:
> cd <project-root>/java/dataset
> mvn clean install -Dmaven.test.skip=true # skipping tests to let build succeed
Have you checked if arrow_dataset_jni.so was successfully installed within the jar `arrow-dataset-XXX.jar`? If not, specifying `arrow.cpp.build.dir` may helps.
> Maybe it is a good idea to pick one of the following:
> 1. Change the 'java.io.FileNotFoundException: libarrow_dataset_jni.so' to display the full path (so people know where your library > looks to find the shared object)
> 2. Make a small JNI_dataset_dev_install_note.md somewhere on getting this library to work.
> 3. Use System.loadLibrary(libraryToLoad);
> 4. Use some other way to find the location of a shared library and copy it to a tmpfile
I did a little try and it appears that `System.loadLibrary` requires for property `java.library.path` to be set to include library locations. If we change to use this way, do users have to do this every time before getting started?
I may intend to pick option 1 here. While we should iterate on multiple resource path roots and display several possible paths to let users find in.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] fsaintjacques commented on a change in pull request #7030: ARROW-7808: [Java][Dataset] Implement Datasets Java API by JNI to C++
Posted by GitBox <gi...@apache.org>.
fsaintjacques commented on a change in pull request #7030:
URL: https://github.com/apache/arrow/pull/7030#discussion_r430588931
##########
File path: cpp/.gitignore
##########
@@ -24,6 +24,8 @@ cmake_install.cmake
build/
*-build/
Testing/
+cmake-build-debug/
Review comment:
I didn't know this was the default behavior of CLion, in such case, your change is a sane default for developers. Do the following:
```
# build directories created by clion
cmake-build-*
```
##########
File path: cpp/src/jni/dataset/jni_wrapper.cpp
##########
@@ -0,0 +1,517 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include <arrow/dataset/api.h>
+#include <arrow/dataset/file_base.h>
+#include <arrow/filesystem/hdfs.h>
+#include <arrow/filesystem/localfs.h>
+#include <arrow/io/api.h>
+#include <arrow/ipc/api.h>
+#include <arrow/util/iterator.h>
+
+#include "org_apache_arrow_dataset_file_JniWrapper.h"
+#include "org_apache_arrow_dataset_jni_JniWrapper.h"
+
+static jclass illegal_access_exception_class;
+static jclass illegal_argument_exception_class;
+static jclass runtime_exception_class;
+
+static jclass record_batch_handle_class;
+static jclass record_batch_handle_field_class;
+static jclass record_batch_handle_buffer_class;
+
+static jmethodID record_batch_handle_constructor;
+static jmethodID record_batch_handle_field_constructor;
+static jmethodID record_batch_handle_buffer_constructor;
+
+static jint JNI_VERSION = JNI_VERSION_1_6;
+
+class JniPendingException : public std::runtime_error {
+ public:
+ explicit JniPendingException(const std::string& arg) : runtime_error(arg) {}
+};
+
+void ThrowPendingException(const std::string& message) {
+ throw JniPendingException(message);
+}
+
+template <typename T>
+T JniGetOrThrow(arrow::Result<T> result) {
+ if (!result.status().ok()) {
+ ThrowPendingException(result.status().message());
+ }
+ return std::move(result).ValueOrDie();
+}
+
+void JniAssertOkOrThrow(arrow::Status status) {
+ if (!status.ok()) {
+ ThrowPendingException(status.message());
+ }
+}
+
+void JniThrow(std::string message) { ThrowPendingException(message); }
+
+#define JNI_METHOD_START try {
+// macro ended
+
+#define JNI_METHOD_END(fallback_expr) \
+ } \
+ catch (JniPendingException & e) { \
+ env->ThrowNew(runtime_exception_class, e.what()); \
+ return fallback_expr; \
+ }
+// macro ended
+
+jclass CreateGlobalClassReference(JNIEnv* env, const char* class_name) {
+ jclass local_class = env->FindClass(class_name);
+ jclass global_class = (jclass)env->NewGlobalRef(local_class);
+ env->DeleteLocalRef(local_class);
+ return global_class;
+}
+
+arrow::Result<jmethodID> GetMethodID(JNIEnv* env, jclass this_class, const char* name,
+ const char* sig) {
+ jmethodID ret = env->GetMethodID(this_class, name, sig);
+ if (ret == nullptr) {
+ std::string error_message = "Unable to find method " + std::string(name) +
+ " within signature" + std::string(sig);
+ return arrow::Status::Invalid(error_message);
+ }
+ return ret;
+}
+
+jint JNI_OnLoad(JavaVM* vm, void* reserved) {
+ JNIEnv* env;
+ if (vm->GetEnv(reinterpret_cast<void**>(&env), JNI_VERSION) != JNI_OK) {
+ return JNI_ERR;
+ }
+ JNI_METHOD_START
+ illegal_access_exception_class =
+ CreateGlobalClassReference(env, "Ljava/lang/IllegalAccessException;");
+ illegal_argument_exception_class =
+ CreateGlobalClassReference(env, "Ljava/lang/IllegalArgumentException;");
+ runtime_exception_class =
+ CreateGlobalClassReference(env, "Ljava/lang/RuntimeException;");
+
+ record_batch_handle_class =
+ CreateGlobalClassReference(env,
+ "Lorg/apache/arrow/"
+ "dataset/jni/NativeRecordBatchHandle;");
+ record_batch_handle_field_class =
+ CreateGlobalClassReference(env,
+ "Lorg/apache/arrow/"
+ "dataset/jni/NativeRecordBatchHandle$Field;");
+ record_batch_handle_buffer_class =
+ CreateGlobalClassReference(env,
+ "Lorg/apache/arrow/"
+ "dataset/jni/NativeRecordBatchHandle$Buffer;");
+
+ record_batch_handle_constructor =
+ JniGetOrThrow(GetMethodID(env, record_batch_handle_class, "<init>",
+ "(J[Lorg/apache/arrow/dataset/"
+ "jni/NativeRecordBatchHandle$Field;"
+ "[Lorg/apache/arrow/dataset/"
+ "jni/NativeRecordBatchHandle$Buffer;)V"));
+ record_batch_handle_field_constructor =
+ JniGetOrThrow(GetMethodID(env, record_batch_handle_field_class, "<init>", "(JJ)V"));
+ record_batch_handle_buffer_constructor = JniGetOrThrow(
+ GetMethodID(env, record_batch_handle_buffer_class, "<init>", "(JJJJ)V"));
+
+ return JNI_VERSION;
+ JNI_METHOD_END(JNI_ERR)
+}
+
+void JNI_OnUnload(JavaVM* vm, void* reserved) {
+ JNIEnv* env;
+ vm->GetEnv(reinterpret_cast<void**>(&env), JNI_VERSION);
+ env->DeleteGlobalRef(illegal_access_exception_class);
+ env->DeleteGlobalRef(illegal_argument_exception_class);
+ env->DeleteGlobalRef(runtime_exception_class);
+ env->DeleteGlobalRef(record_batch_handle_class);
+ env->DeleteGlobalRef(record_batch_handle_field_class);
+ env->DeleteGlobalRef(record_batch_handle_buffer_class);
+}
+
+std::shared_ptr<arrow::Schema> SchemaFromColumnNames(
+ const std::shared_ptr<arrow::Schema>& input,
+ const std::vector<std::string>& column_names) {
+ std::vector<std::shared_ptr<arrow::Field>> columns;
+ for (const auto& name : column_names) {
+ columns.push_back(input->GetFieldByName(name));
+ }
+ return std::make_shared<arrow::Schema>(columns);
+}
+
+arrow::Result<std::shared_ptr<arrow::dataset::FileFormat>> GetFileFormat(jint id) {
+ switch (id) {
+ case 0:
+ return std::make_shared<arrow::dataset::ParquetFileFormat>();
+ default:
+ std::string error_message = "illegal file format id: " + std::to_string(id);
+ return arrow::Status::Invalid(error_message);
+ }
+}
+
+arrow::Result<std::shared_ptr<arrow::fs::FileSystem>> GetFileSystem(
+ jint id, std::string path, std::string* out_path) {
+ switch (id) {
+ case 0:
+ *out_path = path;
+ return std::make_shared<arrow::fs::LocalFileSystem>();
+ case 1: {
+ ARROW_ASSIGN_OR_RAISE(std::shared_ptr<arrow::fs::FileSystem> ret,
+ arrow::fs::FileSystemFromUri(path, out_path))
+ return ret;
+ }
+ default:
+ std::string error_message = "illegal file system id: " + std::to_string(id);
+ return arrow::Status::Invalid(error_message);
+ }
+}
+
+std::string JStringToCString(JNIEnv* env, jstring string) {
+ if (string == nullptr) {
+ return std::string();
+ }
+ jboolean copied;
+ int32_t length = env->GetStringUTFLength(string);
+ const char* chars = env->GetStringUTFChars(string, &copied);
+ std::string str = std::string(chars, length);
+ // fixme calling ReleaseStringUTFChars if memory leak faced
+ return str;
+}
+
+std::vector<std::string> ToStringVector(JNIEnv* env, jobjectArray& str_array) {
+ int length = env->GetArrayLength(str_array);
+ std::vector<std::string> vector;
+ for (int i = 0; i < length; i++) {
+ auto string = (jstring)(env->GetObjectArrayElement(str_array, i));
+ vector.push_back(JStringToCString(env, string));
+ }
+ return vector;
+}
+
+template <typename T>
+jlong CreateNativeRef(std::shared_ptr<T> t) {
+ std::shared_ptr<T>* retained_ptr = new std::shared_ptr<T>(t);
+ return reinterpret_cast<jlong>(retained_ptr);
+}
+
+template <typename T>
+std::shared_ptr<T> RetrieveNativeInstance(jlong ref) {
+ std::shared_ptr<T>* retrieved_ptr = reinterpret_cast<std::shared_ptr<T>*>(ref);
+ return *retrieved_ptr;
+}
+
+template <typename T>
+void ReleaseNativeRef(jlong ref) {
+ std::shared_ptr<T>* retrieved_ptr = reinterpret_cast<std::shared_ptr<T>*>(ref);
+ delete retrieved_ptr;
+}
+
+arrow::Result<jbyteArray> ToSchemaByteArray(JNIEnv* env,
+ std::shared_ptr<arrow::Schema> schema) {
+ ARROW_ASSIGN_OR_RAISE(
+ std::shared_ptr<arrow::Buffer> buffer,
+ arrow::ipc::SerializeSchema(*schema, nullptr, arrow::default_memory_pool()))
+
+ jbyteArray out = env->NewByteArray(buffer->size());
+ auto src = reinterpret_cast<const jbyte*>(buffer->data());
+ env->SetByteArrayRegion(out, 0, buffer->size(), src);
+ return out;
+}
+
+arrow::Result<std::shared_ptr<arrow::Schema>> FromSchemaByteArray(
+ JNIEnv* env, jbyteArray schemaBytes) {
+ arrow::ipc::DictionaryMemo in_memo;
+ int schemaBytes_len = env->GetArrayLength(schemaBytes);
+ jbyte* schemaBytes_data = env->GetByteArrayElements(schemaBytes, nullptr);
+ auto serialized_schema = std::make_shared<arrow::Buffer>(
+ reinterpret_cast<uint8_t*>(schemaBytes_data), schemaBytes_len);
+ arrow::io::BufferReader buf_reader(serialized_schema);
+ ARROW_ASSIGN_OR_RAISE(std::shared_ptr<arrow::Schema> schema,
+ arrow::ipc::ReadSchema(&buf_reader, &in_memo))
+ env->ReleaseByteArrayElements(schemaBytes, schemaBytes_data, JNI_ABORT);
+ return schema;
+}
+
+/*
+ * Class: org_apache_arrow_dataset_jni_JniWrapper
+ * Method: closeDatasetFactory
+ * Signature: (J)V
+ */
+JNIEXPORT void JNICALL Java_org_apache_arrow_dataset_jni_JniWrapper_closeDatasetFactory(
+ JNIEnv* env, jobject, jlong id) {
+ JNI_METHOD_START
+ ReleaseNativeRef<arrow::dataset::DatasetFactory>(id);
+ JNI_METHOD_END()
+}
+
+/*
+ * Class: org_apache_arrow_dataset_jni_JniWrapper
+ * Method: inspectSchema
+ * Signature: (J)[B
+ */
+JNIEXPORT jbyteArray JNICALL Java_org_apache_arrow_dataset_jni_JniWrapper_inspectSchema(
+ JNIEnv* env, jobject, jlong dataset_factor_id) {
+ JNI_METHOD_START
+ std::shared_ptr<arrow::dataset::DatasetFactory> d =
+ RetrieveNativeInstance<arrow::dataset::DatasetFactory>(dataset_factor_id);
+ std::shared_ptr<arrow::Schema> schema = JniGetOrThrow(d->Inspect());
+ return JniGetOrThrow(ToSchemaByteArray(env, schema));
+ JNI_METHOD_END(nullptr)
+}
+
+/*
+ * Class: org_apache_arrow_dataset_jni_JniWrapper
+ * Method: createDataset
+ * Signature: (J[B)J
+ */
+JNIEXPORT jlong JNICALL Java_org_apache_arrow_dataset_jni_JniWrapper_createDataset(
+ JNIEnv* env, jobject, jlong dataset_factory_id, jbyteArray schema_bytes) {
+ JNI_METHOD_START
+ std::shared_ptr<arrow::dataset::DatasetFactory> d =
+ RetrieveNativeInstance<arrow::dataset::DatasetFactory>(dataset_factory_id);
+ std::shared_ptr<arrow::Schema> schema;
+ schema = JniGetOrThrow(FromSchemaByteArray(env, schema_bytes));
+ std::shared_ptr<arrow::dataset::Dataset> dataset = JniGetOrThrow(d->Finish(schema));
+ return CreateNativeRef(dataset);
+ JNI_METHOD_END(-1L)
+}
+
+/*
+ * Class: org_apache_arrow_dataset_jni_JniWrapper
+ * Method: closeDataset
+ * Signature: (J)V
+ */
+JNIEXPORT void JNICALL Java_org_apache_arrow_dataset_jni_JniWrapper_closeDataset(
+ JNIEnv* env, jobject, jlong id) {
+ JNI_METHOD_START
+ ReleaseNativeRef<arrow::dataset::Dataset>(id);
+ JNI_METHOD_END()
+}
+
+/// \class DisposableScannerAdaptor
+/// \brief An adaptor that iterates over a Scanner instance then returns RecordBatches
+/// directly.
+///
+/// This lessens the complexity of the JNI bridge to make sure it to be easier to
+/// maintain. On Java-side, NativeScanner can only produces a single NativeScanTask
+/// instance during its whole lifecycle. Each task stands for a DisposableScannerAdaptor
+/// instance through JNI bridge.
+///
+class DisposableScannerAdaptor {
Review comment:
I think it's fine for now, I was just wondering what was your plan.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] fsaintjacques commented on pull request #7030: ARROW-7808: [Java][Dataset] Implement Datasets Java API by JNI to C++
Posted by GitBox <gi...@apache.org>.
fsaintjacques commented on pull request #7030:
URL: https://github.com/apache/arrow/pull/7030#issuecomment-644238932
Tests are still failing, looks like orc is failing, did you rebase with master?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] zhztheplayer commented on a change in pull request #7030: ARROW-7808: [Java][Dataset] Implement Datasets Java API by JNI to C++
Posted by GitBox <gi...@apache.org>.
zhztheplayer commented on a change in pull request #7030:
URL: https://github.com/apache/arrow/pull/7030#discussion_r429715403
##########
File path: cpp/src/arrow/dataset/discovery.h
##########
@@ -238,5 +238,27 @@ class ARROW_DS_EXPORT FileSystemDatasetFactory : public DatasetFactory {
util::optional<util::string_view> RemovePartitionBaseDir(util::string_view path);
};
+class ARROW_DS_EXPORT SingleFileDatasetFactory : public DatasetFactory {
Review comment:
One question: can we add a construction function `FileSystemDatasetFactory::Make(std::string uri, std::shared_ptr<FileFormat> format, FileSystemFactoryOptions options)` to `FileSystemDatasetFactory`, for reading a single file from an uri? This is similar to functionality of `SingleFileDataset`.
The original reason of creating `SingleFileDataset` is to avoid passing file system object via JNI. Passing such objects can make the bridge even complicated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] rymurr commented on a change in pull request #7030: ARROW-7808: [Java][Dataset] Implement Datasets Java API by JNI to C++
Posted by GitBox <gi...@apache.org>.
rymurr commented on a change in pull request #7030:
URL: https://github.com/apache/arrow/pull/7030#discussion_r446103501
##########
File path: java/dataset/src/test/resources/avroschema/user.avsc
##########
@@ -0,0 +1,26 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+{
Review comment:
I personally agree with @emkornfield on this one. While harder to read the code than json its more compact and isolated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] zhztheplayer commented on a change in pull request #7030: ARROW-7808: [Java][Dataset] Implement Dataset Java API by JNI to C++
Posted by GitBox <gi...@apache.org>.
zhztheplayer commented on a change in pull request #7030:
URL: https://github.com/apache/arrow/pull/7030#discussion_r551694880
##########
File path: cpp/src/arrow/memory_pool.h
##########
@@ -149,6 +149,43 @@ class ARROW_EXPORT ProxyMemoryPool : public MemoryPool {
std::unique_ptr<ProxyMemoryPoolImpl> impl_;
};
+class ARROW_EXPORT ReservationListener {
+ public:
+ virtual ~ReservationListener();
+
+ virtual Status OnReservation(int64_t size) = 0;
+ virtual Status OnRelease(int64_t size) = 0;
+
+ protected:
+ ReservationListener();
+};
+
+class ARROW_EXPORT ReservationListenableMemoryPool : public MemoryPool {
+ public:
+ explicit ReservationListenableMemoryPool(MemoryPool* pool,
+ std::shared_ptr<ReservationListener> listener,
Review comment:
Would you suggest `unique_ptr` here? I thought that ReservationListener instances may be able to share over pools. For example a listener simply for logging reservations to std out.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] emkornfield commented on a change in pull request #7030: ARROW-7808: [Java][Dataset] Implement Datasets Java API
Posted by GitBox <gi...@apache.org>.
emkornfield commented on a change in pull request #7030:
URL: https://github.com/apache/arrow/pull/7030#discussion_r414986841
##########
File path: java/dataset/src/main/java/org/apache/arrow/dataset/file/JniWrapper.java
##########
@@ -0,0 +1,39 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.arrow.dataset.file;
+
+import org.apache.arrow.dataset.jni.JniLoader;
+
+/**
+ * JniWrapper for filesystem based {@link org.apache.arrow.dataset.source.Dataset} implementations.
+ */
+public class JniWrapper {
+
Review comment:
private constructor?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] zhztheplayer commented on pull request #7030: ARROW-7808: [Java][Dataset] Implement Datasets Java API by JNI to C++
Posted by GitBox <gi...@apache.org>.
zhztheplayer commented on pull request #7030:
URL: https://github.com/apache/arrow/pull/7030#issuecomment-641936183
Hi @fsaintjacques, @emkornfield it would be much appreciated if you would like to continue reviewing this, here is a list for modifications since latest review comment:
- code rebase
- removed SingleFileDataset(Factory)
- added API `FileSystemDatasetFactory::Make(
std::string uri, std::shared_ptr<FileFormat> format,
FileSystemFactoryOptions options)`
- other minor changes (see commits after 6f4a914)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] zhztheplayer commented on a change in pull request #7030: ARROW-7808: [Java][Dataset] Implement Datasets Java API by JNI to C++
Posted by GitBox <gi...@apache.org>.
zhztheplayer commented on a change in pull request #7030:
URL: https://github.com/apache/arrow/pull/7030#discussion_r416409859
##########
File path: java/dataset/src/main/java/org/apache/arrow/dataset/scanner/ScanTask.java
##########
@@ -0,0 +1,42 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.arrow.dataset.scanner;
+
+import java.util.Iterator;
+
+import org.apache.arrow.vector.VectorSchemaRoot;
+
+/**
+ * Read record batches from a range of a single data fragment. A
+ * ScanTask is meant to be a unit of work to be dispatched. The implementation
+ * must be thread and concurrent safe.
+ */
+public interface ScanTask {
+
+ /**
+ * Creates and returns a {@link Itr} instance.
+ */
+ Itr scan();
+
+ /**
+ * The iterator implementation for {@link VectorSchemaRoot}s.
+ */
+ interface Itr extends Iterator<VectorSchemaRoot>, AutoCloseable {
+ // FIXME VectorSchemaRoot is not actually something ITERABLE. Using a reader convention instead.
Review comment:
Done. Now it is changed to iterate ArrowRecordBatch rather than VectorSchemaRoot.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] zhztheplayer commented on pull request #7030: ARROW-7808: [Java][Dataset] Implement Dataset Java API by JNI to C++
Posted by GitBox <gi...@apache.org>.
zhztheplayer commented on pull request #7030:
URL: https://github.com/apache/arrow/pull/7030#issuecomment-753840334
Hi @emkornfield Sorry for annoying you but do we have some updates on this? I saw we are bringing 3.0.0 onto schedule so I want to see if I can help to push this feature in it (if possible).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] zhztheplayer commented on pull request #7030: ARROW-7808: [Java][Dataset] Implement Dataset Java API by JNI to C++
Posted by GitBox <gi...@apache.org>.
zhztheplayer commented on pull request #7030:
URL: https://github.com/apache/arrow/pull/7030#issuecomment-766616108
Hi @emkornfield @pitrou
I think I have addressed existing comments so would you like to take another look now? Thanks a lot. Since this was submitted long time ago (before release 1.0.0 IIRC) and I've perform big rebase for times, If we could make progress I would like to help with it in any way.
I know to review a PR creating a new module is usually not trivial work, and current shape of the implementation might be not that perfect for everybody so I may want to suggest we can break up current/future goal to minor patches like we have already done on [the topic of filters](https://github.com/apache/arrow/pull/7030#discussion_r426990234) for this PR.
Based on some recent comments I think we may don't have to make 100% consensus on the current design of `ReservationListenableMemoryPool` or relevant things instantly, if this part is in a way controversial I could split out the part from this PR then create a new one for it after this get merged (should I? would you suggest that). Everything will be working fine enough without it based on our experience (yes we are relying on this feature in our own project for long time). That doesn't mean it is not a "problem" but we can put discussions in separate places to avoid this from being hard to review and merge. The review history is already too long to give every reviewer information he need yet.
And if you have other suggestions on the whole design or architecture of the basic functionality please be free to add comments and I am happy to make further refactors accordingly.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] wesm edited a comment on pull request #7030: ARROW-7808: [Java][Dataset] Implement Datasets Java API
Posted by GitBox <gi...@apache.org>.
wesm edited a comment on pull request #7030:
URL: https://github.com/apache/arrow/pull/7030#issuecomment-619660367
As a high level advisory: the Datasets C++ API should still be regarded as alpha-stage, so I recommend making the JNI bindings as minimal as possible (to satisfy required functionality) so that refactoring is not too painful.
Additionally, I recommend that you do not build JNI bindings for the Expression classes in arrow/dataset/filter.h -- this code is likely to change quite a bit as the associated query engine project gets off the ground this year.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] emkornfield commented on a change in pull request #7030: ARROW-7808: [Java][Dataset] Implement Dataset Java API by JNI to C++
Posted by GitBox <gi...@apache.org>.
emkornfield commented on a change in pull request #7030:
URL: https://github.com/apache/arrow/pull/7030#discussion_r567202908
##########
File path: java/dataset/src/main/java/org/apache/arrow/dataset/jni/DirectReservationListener.java
##########
@@ -0,0 +1,97 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.arrow.dataset.jni;
+
+import java.lang.reflect.Field;
+import java.lang.reflect.Method;
+import java.util.concurrent.atomic.AtomicLong;
+
+import org.apache.arrow.util.VisibleForTesting;
+
+/**
+ * Reserving Java direct memory bytes from java.nio.Bits. Used by Java Dataset API's C++ memory
+ * pool implementation. This makes memory allocated by the pool to be controlled by JVM option
+ * "-XX:MaxDirectMemorySize".
+ */
+public class DirectReservationListener implements ReservationListener {
+ private final Method methodReserve;
+ private final Method methodUnreserve;
+
+ private DirectReservationListener() {
+ try {
+ final Class<?> classBits = Class.forName("java.nio.Bits");
+ methodReserve = classBits.getDeclaredMethod("reserveMemory", long.class, int.class);
+ methodReserve.setAccessible(true);
+ methodUnreserve = classBits.getDeclaredMethod("unreserveMemory", long.class, int.class);
+ methodUnreserve.setAccessible(true);
+ } catch (Exception e) {
+ throw new RuntimeException(e);
+ }
+ }
+
+ private static final DirectReservationListener INSTANCE = new DirectReservationListener();
+
+ public static DirectReservationListener instance() {
+ return INSTANCE;
+ }
+
+ /**
+ * Reserve bytes by invoking java.nio.java.Bitjava.nio.Bitss#reserveMemory.
+ */
+ @Override
+ public void reserve(long size) {
+ try {
+ if (size > Integer.MAX_VALUE) {
+ throw new IllegalArgumentException("reserve size should be of type int");
Review comment:
this is misleading, it should be less then int32 max value.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] zhztheplayer commented on a change in pull request #7030: ARROW-7808: [Java][Dataset] Implement Datasets Java API by JNI to C++
Posted by GitBox <gi...@apache.org>.
zhztheplayer commented on a change in pull request #7030:
URL: https://github.com/apache/arrow/pull/7030#discussion_r417800919
##########
File path: java/dataset/src/main/java/org/apache/arrow/util/SchemaUtils.java
##########
@@ -0,0 +1,72 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.arrow.util;
+
+import java.io.ByteArrayOutputStream;
+import java.io.IOException;
+import java.nio.channels.Channels;
+
+import org.apache.arrow.memory.BufferAllocator;
+import org.apache.arrow.vector.ipc.ReadChannel;
+import org.apache.arrow.vector.ipc.WriteChannel;
+import org.apache.arrow.vector.ipc.message.MessageChannelReader;
+import org.apache.arrow.vector.ipc.message.MessageResult;
+import org.apache.arrow.vector.ipc.message.MessageSerializer;
+import org.apache.arrow.vector.types.pojo.Schema;
+import org.apache.arrow.vector.util.ByteArrayReadableSeekableByteChannel;
+
+/**
+ * Schema utility class including serialization and deserialization.
+ */
+public class SchemaUtils {
+ private static final SchemaUtils INSTANCE = new SchemaUtils();
+
+ public static SchemaUtils get() {
+ return INSTANCE;
+ }
+
+ private SchemaUtils() {
+
+ }
+
+ /**
+ * Deserialize Arrow schema from byte array.
+ */
+ public Schema deserialize(byte[] bytes, BufferAllocator allocator) throws IOException {
Review comment:
It's just the way I got used to. Singleton makes it easier to extend things. As I noticed we use static pattern in other places I have changed to static now.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] zhztheplayer commented on pull request #7030: ARROW-7808: [Java][Dataset] Implement Dataset Java API by JNI to C++
Posted by GitBox <gi...@apache.org>.
zhztheplayer commented on pull request #7030:
URL: https://github.com/apache/arrow/pull/7030#issuecomment-717205959
Hi @emkornfield @jacques-n , would you please have a review on the latest commit? I've squashed previous commit history into the a single base commit to ease rebasing.
Now there is a new memory pool[1] to reserve JVM direct memory(Bits.java) via JNI. Block size is defaulted to 64KB.
Besides, regarding direct memory maybe it's a thinkable practice to make UnsafeAllocationManager/NettyAllocationManager to depend on the same counter(Bits.java) too. But maybe we can do that in a separate PR.
[1] https://github.com/apache/arrow/pull/7030/commits/5a6be7a8443cafb971efae8e83e88878f6742c12#diff-2111aac8ee579e238fb15d4380f2ea1e2f3e2830da939ec3f07c61dc68038d1fR163-R185
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] zhztheplayer commented on a change in pull request #7030: ARROW-7808: [Java][Dataset] Implement Datasets Java API by JNI to C++
Posted by GitBox <gi...@apache.org>.
zhztheplayer commented on a change in pull request #7030:
URL: https://github.com/apache/arrow/pull/7030#discussion_r429552370
##########
File path: cpp/src/arrow/dataset/discovery.h
##########
@@ -238,5 +238,27 @@ class ARROW_DS_EXPORT FileSystemDatasetFactory : public DatasetFactory {
util::optional<util::string_view> RemovePartitionBaseDir(util::string_view path);
};
+class ARROW_DS_EXPORT SingleFileDatasetFactory : public DatasetFactory {
Review comment:
I was aware of the changes in ARROW-8318 but at this time I can remember what stops me from removing SingleFileDataset and migrating to FileSystemDataset. I'll have another try next week.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] zhztheplayer commented on a change in pull request #7030: ARROW-7808: [Java][Dataset] Implement Datasets Java API by JNI to C++
Posted by GitBox <gi...@apache.org>.
zhztheplayer commented on a change in pull request #7030:
URL: https://github.com/apache/arrow/pull/7030#discussion_r415483479
##########
File path: cpp/src/jni/dataset/jni_wrapper.cpp
##########
@@ -0,0 +1,577 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include <arrow/dataset/file_base.h>
+#include <arrow/dataset/api.h>
+#include <arrow/filesystem/localfs.h>
+#include <arrow/ipc/api.h>
+#include <arrow/util/iterator.h>
+#include <arrow/filesystem/hdfs.h>
+#include <arrow/io/api.h>
+#include <google/protobuf/io/coded_stream.h>
+#include <google/protobuf/message.h>
+#include "concurrent_map.h"
+#include "arrow/compute/kernel.h"
+#include "arrow/compute/kernels/cast.h"
+#include "arrow/compute/kernels/compare.h"
+#include "jni/dataset/Types.pb.h"
+
+#include "org_apache_arrow_dataset_jni_JniWrapper.h"
+#include "org_apache_arrow_dataset_file_JniWrapper.h"
+
+static jclass illegal_access_exception_class;
+static jclass illegal_argument_exception_class;
+static jclass runtime_exception_class;
+
+static jclass record_batch_handle_class;
+static jclass record_batch_handle_field_class;
+static jclass record_batch_handle_buffer_class;
+
+static jmethodID record_batch_handle_constructor;
+static jmethodID record_batch_handle_field_constructor;
+static jmethodID record_batch_handle_buffer_constructor;
+
+static jint JNI_VERSION = JNI_VERSION_1_6;
+
+using arrow::jni::ConcurrentMap;
+
+static ConcurrentMap<std::shared_ptr<arrow::dataset::DatasetFactory>> dataset_factory_holder_;
Review comment:
Will do. But I kindly like the current pattern - it benefits more or less when debugging/tracing (self-growing id, instance cache, etc.), right? Is there any other consideration making it bad?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] zhztheplayer commented on a change in pull request #7030: ARROW-7808: [Java][Dataset] Implement Datasets Java API by JNI to C++
Posted by GitBox <gi...@apache.org>.
zhztheplayer commented on a change in pull request #7030:
URL: https://github.com/apache/arrow/pull/7030#discussion_r439266993
##########
File path: java/dataset/src/main/java/org/apache/arrow/memory/Ownerships.java
##########
@@ -0,0 +1,47 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.arrow.memory;
+
+/**
+ * Utility class managing ownership's transferring between Native Arrow buffers and Java Arrow buffers.
+ */
+public class Ownerships {
+ private static final Ownerships INSTANCE = new Ownerships();
+
+ private Ownerships() {
+ }
+
+ public static Ownerships get() {
+ return INSTANCE;
+ }
+
+ /**
+ * Returned ledger's ref count is initialized or increased by 1.
+ */
+ public BufferLedger takeOwnership(BaseAllocator allocator, AllocationManager allocationManager) {
+ final BufferLedger ledger = allocationManager.associate(allocator);
+ long size = allocationManager.getSize();
+ boolean succeed = allocator.forceAllocate(size);
+ if (!succeed) {
+ throw new OutOfMemoryException("Target allocator is full");
+ }
+ ((BaseAllocator) allocationManager.getOwningLedger().getAllocator()).releaseBytes(size);
Review comment:
Yes calls to `forceAllocate` should be synchronized.
The fix appear to be in a separate topic because in Java API we don't yet give allocator the functionality to take over ownerless memory which requires for some slight refactors to allocation process. I'll open a ticket for that.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] pitrou commented on pull request #7030: ARROW-7808: [Java][Dataset] Implement Dataset Java API by JNI to C++
Posted by GitBox <gi...@apache.org>.
pitrou commented on pull request #7030:
URL: https://github.com/apache/arrow/pull/7030#issuecomment-755274666
I'll add that the JNI code doesn't seem to follow common guidelines and practices (for example, static variables in the root namespace), but I suppose that's not a new issue.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] emkornfield commented on a change in pull request #7030: ARROW-7808: [Java][Dataset] Implement Datasets Java API by JNI to C++
Posted by GitBox <gi...@apache.org>.
emkornfield commented on a change in pull request #7030:
URL: https://github.com/apache/arrow/pull/7030#discussion_r425563920
##########
File path: cpp/src/jni/dataset/proto/Types.proto
##########
@@ -0,0 +1,149 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+syntax = "proto2";
+package types;
+
+option java_package = "org.apache.arrow.dataset";
Review comment:
Can you start a thread on the mailing list about filter support. I'm concerned with a proto specific to the java implementation of datasets. Especially, because I think this essentially replicates what is in the Gandiva proto. We should ideally have one representation for all the languages that are going to bind to the same functionality.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] emkornfield commented on a change in pull request #7030: ARROW-7808: [Java][Dataset] Implement Dataset Java API by JNI to C++
Posted by GitBox <gi...@apache.org>.
emkornfield commented on a change in pull request #7030:
URL: https://github.com/apache/arrow/pull/7030#discussion_r571716025
##########
File path: java/vector/src/main/java/org/apache/arrow/vector/util/SchemaUtility.java
##########
@@ -0,0 +1,62 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.arrow.vector.util;
+
+import java.io.ByteArrayOutputStream;
+import java.io.IOException;
+import java.nio.channels.Channels;
+
+import org.apache.arrow.memory.BufferAllocator;
+import org.apache.arrow.vector.ipc.ReadChannel;
+import org.apache.arrow.vector.ipc.WriteChannel;
+import org.apache.arrow.vector.ipc.message.MessageChannelReader;
+import org.apache.arrow.vector.ipc.message.MessageResult;
+import org.apache.arrow.vector.ipc.message.MessageSerializer;
+import org.apache.arrow.vector.types.pojo.Schema;
+
+/**
+ * Schema utility class including serialization and deserialization.
+ */
+public class SchemaUtility {
+
Review comment:
private constructor to prevent initialization?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] zhztheplayer commented on a change in pull request #7030: ARROW-7808: [Java][Dataset] Implement Datasets Java API by JNI to C++
Posted by GitBox <gi...@apache.org>.
zhztheplayer commented on a change in pull request #7030:
URL: https://github.com/apache/arrow/pull/7030#discussion_r417791568
##########
File path: cpp/src/jni/dataset/jni_wrapper.cpp
##########
@@ -0,0 +1,577 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include <arrow/dataset/file_base.h>
+#include <arrow/dataset/api.h>
+#include <arrow/filesystem/localfs.h>
+#include <arrow/ipc/api.h>
+#include <arrow/util/iterator.h>
+#include <arrow/filesystem/hdfs.h>
+#include <arrow/io/api.h>
+#include <google/protobuf/io/coded_stream.h>
+#include <google/protobuf/message.h>
+#include "concurrent_map.h"
+#include "arrow/compute/kernel.h"
+#include "arrow/compute/kernels/cast.h"
+#include "arrow/compute/kernels/compare.h"
+#include "jni/dataset/Types.pb.h"
+
+#include "org_apache_arrow_dataset_jni_JniWrapper.h"
+#include "org_apache_arrow_dataset_file_JniWrapper.h"
+
+static jclass illegal_access_exception_class;
+static jclass illegal_argument_exception_class;
+static jclass runtime_exception_class;
+
+static jclass record_batch_handle_class;
+static jclass record_batch_handle_field_class;
+static jclass record_batch_handle_buffer_class;
+
+static jmethodID record_batch_handle_constructor;
+static jmethodID record_batch_handle_field_constructor;
+static jmethodID record_batch_handle_buffer_constructor;
+
+static jint JNI_VERSION = JNI_VERSION_1_6;
+
+using arrow::jni::ConcurrentMap;
+
+static ConcurrentMap<std::shared_ptr<arrow::dataset::DatasetFactory>> dataset_factory_holder_;
+static ConcurrentMap<std::shared_ptr<arrow::dataset::Dataset>> dataset_holder_;
+static ConcurrentMap<std::shared_ptr<arrow::dataset::ScanTask>> scan_task_holder_;
+static ConcurrentMap<std::shared_ptr<arrow::dataset::Scanner>> scanner_holder_;
+static ConcurrentMap<std::shared_ptr<arrow::RecordBatchIterator>> iterator_holder_;
+static ConcurrentMap<std::shared_ptr<arrow::Buffer>> buffer_holder_;
+
+#define JNI_ASSIGN_OR_THROW_NAME(x, y) ARROW_CONCAT(x, y)
+
+#define JNI_ASSIGN_OR_THROW_IMPL(t, lhs, rexpr) \
+ auto t = (rexpr); \
+ if (!t.status().ok()) { \
+ env->ThrowNew(runtime_exception_class, t.status().message().c_str()); \
+ } \
+ lhs = std::move(t).ValueOrDie();
+
+#define JNI_ASSIGN_OR_THROW(lhs, rexpr) \
+ JNI_ASSIGN_OR_THROW_IMPL(JNI_ASSIGN_OR_THROW_NAME(_tmp_var, __COUNTER__), lhs, rexpr)
+
+#define JNI_ASSERT_OK_OR_THROW(expr) \
+ do { \
+ auto _res = (expr); \
+ arrow::Status _st = ::arrow::internal::GenericToStatus(_res); \
+ if (!_st.ok()) { \
+ env->ThrowNew(runtime_exception_class, _st.message().c_str()); \
+ } \
+ } while (false);
+
+jclass CreateGlobalClassReference(JNIEnv* env, const char* class_name) {
+ jclass local_class = env->FindClass(class_name);
+ jclass global_class = (jclass)env->NewGlobalRef(local_class);
+ env->DeleteLocalRef(local_class);
+ return global_class;
+}
+
+jmethodID GetMethodID(JNIEnv* env, jclass this_class, const char* name, const char* sig) {
+ jmethodID ret = env->GetMethodID(this_class, name, sig);
+ if (ret == nullptr) {
+ std::string error_message = "Unable to find method " + std::string(name) +
+ " within signature" + std::string(sig);
+ env->ThrowNew(illegal_access_exception_class, error_message.c_str());
+ }
+ return ret;
+}
+
+jint JNI_OnLoad(JavaVM* vm, void* reserved) {
+ JNIEnv* env;
+ if (vm->GetEnv(reinterpret_cast<void**>(&env), JNI_VERSION) != JNI_OK) {
+ return JNI_ERR;
+ }
+
+ illegal_access_exception_class =
+ CreateGlobalClassReference(env, "Ljava/lang/IllegalAccessException;");
+ illegal_argument_exception_class =
+ CreateGlobalClassReference(env, "Ljava/lang/IllegalArgumentException;");
+ runtime_exception_class =
+ CreateGlobalClassReference(env, "Ljava/lang/RuntimeException;");
+
+ record_batch_handle_class =
+ CreateGlobalClassReference(env, "Lorg/apache/arrow/dataset/jni/NativeRecordBatchHandle;");
+ record_batch_handle_field_class =
+ CreateGlobalClassReference(env, "Lorg/apache/arrow/dataset/jni/NativeRecordBatchHandle$Field;");
+ record_batch_handle_buffer_class =
+ CreateGlobalClassReference(env, "Lorg/apache/arrow/dataset/jni/NativeRecordBatchHandle$Buffer;");
+
+ record_batch_handle_constructor = GetMethodID(env, record_batch_handle_class, "<init>",
+ "(J[Lorg/apache/arrow/dataset/jni/NativeRecordBatchHandle$Field;"
+ "[Lorg/apache/arrow/dataset/jni/NativeRecordBatchHandle$Buffer;)V");
+ record_batch_handle_field_constructor = GetMethodID(env, record_batch_handle_field_class, "<init>",
+ "(JJ)V");
+ record_batch_handle_buffer_constructor = GetMethodID(env, record_batch_handle_buffer_class, "<init>",
+ "(JJJJ)V");
+
+ env->ExceptionDescribe();
+
+ return JNI_VERSION;
+}
+
+
+void JNI_OnUnload(JavaVM* vm, void* reserved) {
+ JNIEnv* env;
+ vm->GetEnv(reinterpret_cast<void**>(&env), JNI_VERSION);
+ env->DeleteGlobalRef(illegal_access_exception_class);
+ env->DeleteGlobalRef(illegal_argument_exception_class);
+ env->DeleteGlobalRef(runtime_exception_class);
+ env->DeleteGlobalRef(record_batch_handle_class);
+ env->DeleteGlobalRef(record_batch_handle_field_class);
+ env->DeleteGlobalRef(record_batch_handle_buffer_class);
+
+ dataset_factory_holder_.Clear();
+ dataset_holder_.Clear();
+ scan_task_holder_.Clear();
+ scanner_holder_.Clear();
+ iterator_holder_.Clear();
+ buffer_holder_.Clear();
+}
+
+std::shared_ptr<arrow::Schema> SchemaFromColumnNames(
+ const std::shared_ptr<arrow::Schema>& input, const std::vector<std::string>& column_names) {
+ std::vector<std::shared_ptr<arrow::Field>> columns;
+ for (const auto& name : column_names) {
+ columns.push_back(input->GetFieldByName(name));
+ }
+ return std::make_shared<arrow::Schema>(columns);
+}
+
+std::shared_ptr<arrow::dataset::FileFormat> GetFileFormat(JNIEnv *env, jint id) {
+ switch (id) {
+ case 0:
+ return std::make_shared<arrow::dataset::ParquetFileFormat>();
+ default:
+ std::string error_message = "illegal file format id: " + std::to_string(id);
+ env->ThrowNew(illegal_argument_exception_class, error_message.c_str());
+ return nullptr; // unreachable
+ }
+}
+
+std::shared_ptr<arrow::fs::FileSystem> GetFileSystem(JNIEnv *env, jint id, std::string path,
+ std::string* out_path) {
+ switch (id) {
+ case 0:
+ *out_path = path;
+ return std::make_shared<arrow::fs::LocalFileSystem>();
+ case 1: {
+ JNI_ASSIGN_OR_THROW(std::shared_ptr<arrow::fs::FileSystem> ret, arrow::fs::FileSystemFromUri(path, out_path))
+ return ret;
+ }
+ default:std::string error_message = "illegal filesystem id: " + std::to_string(id);
+ env->ThrowNew(illegal_argument_exception_class, error_message.c_str());
+ return nullptr; // unreachable
+ }
+}
+
+std::string JStringToCString(JNIEnv* env, jstring string) {
+ jboolean copied;
+ int32_t length = env->GetStringUTFLength(string);
+ const char *chars = env->GetStringUTFChars(string, &copied);
+ std::string str = std::string(chars, length);
+ // fixme calling ReleaseStringUTFChars if memory leak faced
+ return str;
+}
+
+std::vector<std::string> ToStringVector(JNIEnv* env, jobjectArray& str_array) {
+ int length = env->GetArrayLength(str_array);
+ std::vector<std::string> vector;
+ for (int i = 0; i < length; i++) {
+ auto string = (jstring) (env->GetObjectArrayElement(str_array, i));
+ vector.push_back(JStringToCString(env, string));
+ }
+ return vector;
+}
+
+template <typename T>
+std::vector<T> collect(JNIEnv* env, arrow::Iterator<T> itr) {
+ std::vector<T> vector;
+ while(true) {
+ JNI_ASSIGN_OR_THROW(T t, itr.Next())
+ if (!t) {
+ break;
+ }
+ vector.push_back(t);
+ }
+ return vector;
+}
+
+// FIXME: COPIED FROM intel/master on which this branch is not rebased yet
+// FIXME: https://github.com/Intel-bigdata/arrow/blob/02502a4eb59834c2471dd629e77dbeed19559f68/cpp/src/jni/jni_common.h#L239-L254
+jbyteArray ToSchemaByteArray(JNIEnv* env, std::shared_ptr<arrow::Schema> schema) {
+ JNI_ASSIGN_OR_THROW(std::shared_ptr<arrow::Buffer> buffer,
+ arrow::ipc::SerializeSchema(*schema.get(), nullptr, arrow::default_memory_pool()))
+
+ jbyteArray out = env->NewByteArray(buffer->size());
+ auto src = reinterpret_cast<const jbyte*>(buffer->data());
+ env->SetByteArrayRegion(out, 0, buffer->size(), src);
+ return out;
+}
+
+// FIXME: COPIED FROM intel/master on which this branch is not rebased yet
+// FIXME: https://github.com/Intel-bigdata/arrow/blob/02502a4eb59834c2471dd629e77dbeed19559f68/cpp/src/jni/jni_common.h#L256-L272
+std::shared_ptr<arrow::Schema> FromSchemaByteArray(JNIEnv* env, jbyteArray schemaBytes) {
+ arrow::ipc::DictionaryMemo in_memo;
+ int schemaBytes_len = env->GetArrayLength(schemaBytes);
+ jbyte* schemaBytes_data = env->GetByteArrayElements(schemaBytes, 0);
+ auto serialized_schema =
+ std::make_shared<arrow::Buffer>((uint8_t*)schemaBytes_data, schemaBytes_len);
+ arrow::io::BufferReader buf_reader(serialized_schema);
+ JNI_ASSIGN_OR_THROW(std::shared_ptr<arrow::Schema> schema, arrow::ipc::ReadSchema(&buf_reader, &in_memo))
+ env->ReleaseByteArrayElements(schemaBytes, schemaBytes_data, JNI_ABORT);
+ return schema;
+}
+
+bool ParseProtobuf(uint8_t* buf, int bufLen, google::protobuf::Message* msg) {
+ google::protobuf::io::CodedInputStream cis(buf, bufLen);
+ cis.SetRecursionLimit(1000);
+ return msg->ParseFromCodedStream(&cis);
+}
+
+void releaseFilterInput(jbyteArray condition_arr, jbyte* condition_bytes, JNIEnv* env) {
+ env->ReleaseByteArrayElements(condition_arr, condition_bytes, JNI_ABORT);
+}
+
+// fixme in development. Not all node types considered.
+std::shared_ptr<arrow::dataset::Expression> translateNode(types::TreeNode node, JNIEnv* env) {
+ if (node.has_fieldnode()) {
+ const types::FieldNode& f_node = node.fieldnode();
+ const std::string& name = f_node.name();
+ return std::make_shared<arrow::dataset::FieldExpression>(name);
+ }
+ if (node.has_intnode()) {
+ const types::IntNode& int_node = node.intnode();
+ int32_t val = int_node.value();
+ return std::make_shared<arrow::dataset::ScalarExpression>(std::make_shared<arrow::Int32Scalar>(val));
+ }
+ if (node.has_longnode()) {
+ const types::LongNode& long_node = node.longnode();
+ int64_t val = long_node.value();
+ return std::make_shared<arrow::dataset::ScalarExpression>(std::make_shared<arrow::Int64Scalar>(val));
+ }
+ if (node.has_floatnode()) {
+ const types::FloatNode& float_node = node.floatnode();
+ float_t val = float_node.value();
+ return std::make_shared<arrow::dataset::ScalarExpression>(std::make_shared<arrow::FloatScalar>(val));
+ }
+ if (node.has_doublenode()) {
+ const types::DoubleNode& double_node = node.doublenode();
+ double_t val = double_node.value();
+ return std::make_shared<arrow::dataset::ScalarExpression>(std::make_shared<arrow::DoubleScalar>(val));
+ }
+ if (node.has_booleannode()) {
+ const types::BooleanNode& boolean_node = node.booleannode();
+ bool val = boolean_node.value();
+ return std::make_shared<arrow::dataset::ScalarExpression>(std::make_shared<arrow::BooleanScalar>(val));
+ }
+ if (node.has_andnode()) {
+ const types::AndNode& and_node = node.andnode();
+ const types::TreeNode& left_arg = and_node.leftarg();
+ const types::TreeNode& right_arg = and_node.rightarg();
+ return std::make_shared<arrow::dataset::AndExpression>(translateNode(left_arg, env), translateNode(right_arg, env));
+ }
+ if (node.has_ornode()) {
+ const types::OrNode& or_node = node.ornode();
+ const types::TreeNode& left_arg = or_node.leftarg();
+ const types::TreeNode& right_arg = or_node.rightarg();
+ return std::make_shared<arrow::dataset::OrExpression>(translateNode(left_arg, env), translateNode(right_arg, env));
+ }
+ if (node.has_cpnode()) {
+ const types::ComparisonNode& cp_node = node.cpnode();
+ const std::string& op_name = cp_node.opname();
+ arrow::compute::CompareOperator op;
+ if (op_name == "equal") {
+ op = arrow::compute::CompareOperator::EQUAL;
+ } else if (op_name == "greaterThan") {
+ op = arrow::compute::CompareOperator::GREATER;
+ } else if (op_name == "greaterThanOrEqual") {
+ op = arrow::compute::CompareOperator::GREATER_EQUAL;
+ } else if (op_name == "lessThan") {
+ op = arrow::compute::CompareOperator::LESS;
+ } else if (op_name == "lessThanOrEqual") {
+ op = arrow::compute::CompareOperator::LESS_EQUAL;
+ } else {
+ std::string error_message = "Unknown operation name in comparison node";
+ env->ThrowNew(illegal_argument_exception_class, error_message.c_str());
+ return nullptr; // unreachable
+ }
+ const types::TreeNode& left_arg = cp_node.leftarg();
+ const types::TreeNode& right_arg = cp_node.rightarg();
+ return std::make_shared<arrow::dataset::ComparisonExpression>(op,
+ translateNode(left_arg, env), translateNode(right_arg, env));
+ }
+ if (node.has_notnode()) {
+ const types::NotNode& not_node = node.notnode();
+ const ::types::TreeNode& child = not_node.args();
+ std::shared_ptr<arrow::dataset::Expression> translatedChild = translateNode(child, env);
+ return std::make_shared<arrow::dataset::NotExpression>(translatedChild);
+ }
+ if (node.has_isvalidnode()) {
+ const types::IsValidNode& is_valid_node = node.isvalidnode();
+ const ::types::TreeNode& child = is_valid_node.args();
+ std::shared_ptr<arrow::dataset::Expression> translatedChild = translateNode(child, env);
+ return std::make_shared<arrow::dataset::IsValidExpression>(translatedChild);
+ }
+ std::string error_message = "Unknown node type";
+ env->ThrowNew(illegal_argument_exception_class, error_message.c_str());
+ return nullptr; // unreachable
+}
+
+std::shared_ptr<arrow::dataset::Expression> translateFilter(types::Condition condition, JNIEnv* env) {
+ const types::TreeNode& tree_node = condition.root();
+ return translateNode(tree_node, env);
+}
+
+/*
+ * Class: org_apache_arrow_dataset_jni_JniWrapper
+ * Method: closeDatasetFactory
+ * Signature: (J)V
+ */
+JNIEXPORT void JNICALL Java_org_apache_arrow_dataset_jni_JniWrapper_closeDatasetFactory
+ (JNIEnv *, jobject, jlong id) {
+ dataset_factory_holder_.Erase(id);
+}
+
+/*
+ * Class: org_apache_arrow_dataset_jni_JniWrapper
+ * Method: inspectSchema
+ * Signature: (J)[B
+ */
+JNIEXPORT jbyteArray JNICALL Java_org_apache_arrow_dataset_jni_JniWrapper_inspectSchema
+ (JNIEnv* env, jobject, jlong dataset_factor_id) {
+ std::shared_ptr<arrow::dataset::DatasetFactory> d
+ = dataset_factory_holder_.Lookup(dataset_factor_id);
+ JNI_ASSIGN_OR_THROW(std::shared_ptr<arrow::Schema> schema, d->Inspect())
+ return ToSchemaByteArray(env, schema);
+}
+
+/*
+ * Class: org_apache_arrow_dataset_jni_JniWrapper
+ * Method: createDataset
+ * Signature: (J[B)J
+ */
+JNIEXPORT jlong JNICALL Java_org_apache_arrow_dataset_jni_JniWrapper_createDataset
+ (JNIEnv* env, jobject, jlong dataset_factory_id, jbyteArray schema_bytes) {
+ std::shared_ptr<arrow::dataset::DatasetFactory> d
+ = dataset_factory_holder_.Lookup(dataset_factory_id);
+ std::shared_ptr<arrow::Schema> schema;
+ schema = FromSchemaByteArray(env, schema_bytes);
+ JNI_ASSIGN_OR_THROW(std::shared_ptr<arrow::dataset::Dataset> dataset, d->Finish(schema))
+ return dataset_holder_.Insert(dataset);
+}
+
+/*
+ * Class: org_apache_arrow_dataset_jni_JniWrapper
+ * Method: closeDataset
+ * Signature: (J)V
+ */
+JNIEXPORT void JNICALL Java_org_apache_arrow_dataset_jni_JniWrapper_closeDataset
+ (JNIEnv *, jobject, jlong id) {
+ dataset_holder_.Erase(id);
+}
+
+/*
+ * Class: org_apache_arrow_dataset_jni_JniWrapper
+ * Method: createScanner
+ * Signature: (J[Ljava/lang/String;[BJ)J
+ */
+JNIEXPORT jlong JNICALL Java_org_apache_arrow_dataset_jni_JniWrapper_createScanner
+ (JNIEnv* env, jobject, jlong dataset_id, jobjectArray columns,
+ jbyteArray filter, jlong batch_size) {
+ std::shared_ptr<arrow::dataset::ScanContext> context = std::make_shared<arrow::dataset::ScanContext>();
+ std::shared_ptr<arrow::dataset::Dataset> dataset = dataset_holder_.Lookup(dataset_id);
+ JNI_ASSIGN_OR_THROW(std::shared_ptr<arrow::dataset::ScannerBuilder> scanner_builder, dataset->NewScan())
+
+ std::vector<std::string> column_vector = ToStringVector(env, columns);
+ JNI_ASSERT_OK_OR_THROW(scanner_builder->Project(column_vector));
+ JNI_ASSERT_OK_OR_THROW(scanner_builder->BatchSize(batch_size));
+
+ // initialize filters
+ jsize exprs_len = env->GetArrayLength(filter);
+ jbyte* exprs_bytes = env->GetByteArrayElements(filter, 0);
+ types::Condition condition;
+ if (!ParseProtobuf(reinterpret_cast<uint8_t*>(exprs_bytes), exprs_len, &condition)) {
+ releaseFilterInput(filter, exprs_bytes, env);
+ std::string error_message = "bad protobuf message";
+ env->ThrowNew(illegal_argument_exception_class, error_message.c_str());
+ }
+ if (condition.has_root()) {
+ JNI_ASSERT_OK_OR_THROW(scanner_builder->Filter(translateFilter(condition, env)));
+ }
+ JNI_ASSIGN_OR_THROW(auto scanner, scanner_builder->Finish())
+ jlong id = scanner_holder_.Insert(scanner);
+ releaseFilterInput(filter, exprs_bytes, env);
+ return id;
+}
+
+/*
+ * Class: org_apache_arrow_dataset_jni_JniWrapper
+ * Method: getSchemaFromScanner
+ * Signature: (J)[B
+ */
+JNIEXPORT jbyteArray JNICALL Java_org_apache_arrow_dataset_jni_JniWrapper_getSchemaFromScanner
+ (JNIEnv* env, jobject, jlong scanner_id) {
+ std::shared_ptr<arrow::Schema> schema = scanner_holder_.Lookup(scanner_id)->schema();
+ return ToSchemaByteArray(env, schema);
+}
+
+/*
+ * Class: org_apache_arrow_dataset_jni_JniWrapper
+ * Method: getScanTasksFromScanner
+ * Signature: (J)[J
+ */
+JNIEXPORT jlongArray JNICALL Java_org_apache_arrow_dataset_jni_JniWrapper_getScanTasksFromScanner
+ (JNIEnv* env, jobject, jlong scanner_id) {
+ std::shared_ptr<arrow::dataset::Scanner> scanner = scanner_holder_.Lookup(scanner_id);
+ JNI_ASSIGN_OR_THROW(arrow::dataset::ScanTaskIterator itr, scanner->Scan())
+ std::vector<std::shared_ptr<arrow::dataset::ScanTask>> vector = collect(env, std::move(itr));
Review comment:
The logic is now moved into DisposableScannerAdaptor, which doesn't collect these tasks anymore. Also removed the collect function.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] zhztheplayer commented on a change in pull request #7030: ARROW-7808: [Java][Dataset] Implement Datasets Java API
Posted by GitBox <gi...@apache.org>.
zhztheplayer commented on a change in pull request #7030:
URL: https://github.com/apache/arrow/pull/7030#discussion_r415256413
##########
File path: cpp/src/jni/dataset/concurrent_map.h
##########
@@ -0,0 +1,81 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ */
+
+#ifndef JNI_ID_TO_MODULE_MAP_H
+#define JNI_ID_TO_MODULE_MAP_H
+
+#include <memory>
+#include <mutex>
+#include <unordered_map>
+#include <utility>
+
+#include "jni.h"
+#include "arrow/util/macros.h"
+
+namespace arrow {
Review comment:
Maybe move concurrent_map.h out into `src/jni/`? By the way, should we somehow migrate `src/gandiva/jni` to `src/jni/gandiva`, or migrate `src/jni` to `arrow/jni`? It seems that currently they are organized inconsistently.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] zhztheplayer commented on a change in pull request #7030: ARROW-7808: [Java][Dataset] Implement Dataset Java API by JNI to C++
Posted by GitBox <gi...@apache.org>.
zhztheplayer commented on a change in pull request #7030:
URL: https://github.com/apache/arrow/pull/7030#discussion_r567768450
##########
File path: java/dataset/src/test/java/org/apache/arrow/dataset/file/TestFileSystemDataset.java
##########
@@ -0,0 +1,348 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.arrow.dataset.file;
+
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertTrue;
+import static org.junit.jupiter.api.Assertions.assertDoesNotThrow;
+import static org.junit.jupiter.api.Assertions.assertThrows;
+
+import java.util.Collections;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.Objects;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+import java.util.concurrent.atomic.AtomicLong;
+import java.util.stream.Collectors;
+
+import org.apache.arrow.dataset.jni.DirectReservationListener;
+import org.apache.arrow.dataset.jni.NativeDataset;
+import org.apache.arrow.dataset.jni.NativeInstanceReleasedException;
+import org.apache.arrow.dataset.jni.NativeMemoryPool;
+import org.apache.arrow.dataset.jni.NativeScanTask;
+import org.apache.arrow.dataset.jni.NativeScanner;
+import org.apache.arrow.dataset.jni.ReservationListener;
+import org.apache.arrow.dataset.jni.TestNativeDataset;
+import org.apache.arrow.dataset.scanner.ScanOptions;
+import org.apache.arrow.dataset.scanner.ScanTask;
+import org.apache.arrow.util.AutoCloseables;
+import org.apache.arrow.vector.FieldVector;
+import org.apache.arrow.vector.VectorLoader;
+import org.apache.arrow.vector.VectorSchemaRoot;
+import org.apache.arrow.vector.ipc.message.ArrowRecordBatch;
+import org.apache.arrow.vector.types.Types;
+import org.apache.arrow.vector.types.pojo.Schema;
+import org.apache.avro.generic.GenericRecord;
+import org.apache.avro.generic.GenericRecordBuilder;
+import org.junit.Assert;
+import org.junit.ClassRule;
+import org.junit.Test;
+import org.junit.jupiter.api.Assertions;
+import org.junit.rules.TemporaryFolder;
+
+public class TestFileSystemDataset extends TestNativeDataset {
+
+ @ClassRule
+ public static final TemporaryFolder TMP = new TemporaryFolder();
+
+ public static final String AVRO_SCHEMA_USER = "user.avsc";
+
+ /**
+ * The default block size of C++ ReservationListenableMemoryPool.
+ */
+ public static final long DEFAULT_NATIVE_MEMORY_POOL_BLOCK_SIZE = 512 * 1024;
+
+ @Test
+ public void testParquetRead() throws Exception {
+ ParquetWriteSupport writeSupport = ParquetWriteSupport.writeTempFile(AVRO_SCHEMA_USER, TMP.newFolder(), 1, "a");
+
+ FileSystemDatasetFactory factory = new FileSystemDatasetFactory(rootAllocator(), NativeMemoryPool.getDefault(),
+ FileFormat.PARQUET, writeSupport.getOutputURI());
+ ScanOptions options = new ScanOptions(new String[0], 100);
+ Schema schema = inferResultSchemaFromFactory(factory, options);
+ List<ArrowRecordBatch> datum = collectResultFromFactory(factory, options);
+
+ assertSingleTaskProduced(factory, options);
+ assertEquals(1, datum.size());
+ assertEquals(2, schema.getFields().size());
+ assertEquals("id", schema.getFields().get(0).getName());
+ assertEquals("name", schema.getFields().get(1).getName());
+ assertEquals(Types.MinorType.INT.getType(), schema.getFields().get(0).getType());
+ assertEquals(Types.MinorType.VARCHAR.getType(), schema.getFields().get(1).getType());
+ checkParquetReadResult(schema, writeSupport.getWrittenRecords(), datum);
+
+ AutoCloseables.close(datum);
+ }
+
+ @Test
+ public void testParquetProjector() throws Exception {
+ ParquetWriteSupport writeSupport = ParquetWriteSupport.writeTempFile(AVRO_SCHEMA_USER, TMP.newFolder(), 1, "a");
+
+ FileSystemDatasetFactory factory = new FileSystemDatasetFactory(rootAllocator(), NativeMemoryPool.getDefault(),
+ FileFormat.PARQUET, writeSupport.getOutputURI());
+ ScanOptions options = new ScanOptions(new String[]{"id"}, 100);
+ Schema schema = inferResultSchemaFromFactory(factory, options);
+ List<ArrowRecordBatch> datum = collectResultFromFactory(factory, options);
+ org.apache.avro.Schema expectedSchema = truncateAvroSchema(writeSupport.getAvroSchema(), 0, 1);
+
+ assertSingleTaskProduced(factory, options);
+ assertEquals(1, schema.getFields().size());
+ assertEquals("id", schema.getFields().get(0).getName());
+ assertEquals(Types.MinorType.INT.getType(), schema.getFields().get(0).getType());
+ assertEquals(1, datum.size());
+ checkParquetReadResult(schema,
+ Collections.singletonList(
+ new GenericRecordBuilder(
+ expectedSchema)
+ .set("id", 1)
+ .build()), datum);
+
+ AutoCloseables.close(datum);
+ }
+
+ @Test
+ public void testParquetBatchSize() throws Exception {
+ ParquetWriteSupport writeSupport = ParquetWriteSupport.writeTempFile(AVRO_SCHEMA_USER, TMP.newFolder(),
+ 1, "a", 2, "b", 3, "c");
+
+ ScanOptions options = new ScanOptions(new String[0], 1);
+ FileSystemDatasetFactory factory = new FileSystemDatasetFactory(rootAllocator(), NativeMemoryPool.getDefault(),
+ FileFormat.PARQUET, writeSupport.getOutputURI());
+ Schema schema = inferResultSchemaFromFactory(factory, options);
+ List<ArrowRecordBatch> datum = collectResultFromFactory(factory, options);
+
+ assertSingleTaskProduced(factory, options);
+ assertEquals(3, datum.size());
+ datum.forEach(batch -> assertEquals(1, batch.getLength()));
+ checkParquetReadResult(schema, writeSupport.getWrittenRecords(), datum);
+
+ AutoCloseables.close(datum);
+ }
+
+ @Test
+ public void testCloseAgain() throws Exception {
+ ParquetWriteSupport writeSupport = ParquetWriteSupport.writeTempFile(AVRO_SCHEMA_USER, TMP.newFolder(), 1, "a");
+
+ FileSystemDatasetFactory factory = new FileSystemDatasetFactory(rootAllocator(), NativeMemoryPool.getDefault(),
+ FileFormat.PARQUET, writeSupport.getOutputURI());
+
+ assertDoesNotThrow(() -> {
+ NativeDataset dataset = factory.finish();
+ dataset.close();
+ dataset.close();
+ });
+ }
+
+ @Test
+ public void testScanAgain() throws Exception {
+ ParquetWriteSupport writeSupport = ParquetWriteSupport.writeTempFile(AVRO_SCHEMA_USER, TMP.newFolder(), 1, "a");
+
+ FileSystemDatasetFactory factory = new FileSystemDatasetFactory(rootAllocator(), NativeMemoryPool.getDefault(),
+ FileFormat.PARQUET, writeSupport.getOutputURI());
+ NativeDataset dataset = factory.finish();
+ ScanOptions options = new ScanOptions(new String[0], 100);
+ NativeScanner scanner = dataset.newScan(options);
+ List<? extends NativeScanTask> taskList1 = collect(scanner.scan());
+ List<? extends NativeScanTask> taskList2 = collect(scanner.scan());
+ NativeScanTask task1 = taskList1.get(0);
+ NativeScanTask task2 = taskList2.get(0);
+ List<ArrowRecordBatch> datum = collect(task1.execute());
+
+ UnsupportedOperationException uoe = assertThrows(UnsupportedOperationException.class, task2::execute);
+ Assertions.assertEquals("NativeScanner cannot be executed more than once. Consider creating new scanner instead",
+ uoe.getMessage());
+
+ AutoCloseables.close(datum);
+ AutoCloseables.close(taskList1);
+ AutoCloseables.close(taskList2);
+ AutoCloseables.close(scanner, dataset, factory);
+ }
+
+ @Test
+ public void testScanInOtherThread() throws Exception {
+ ExecutorService executor = Executors.newSingleThreadExecutor();
+ ParquetWriteSupport writeSupport = ParquetWriteSupport.writeTempFile(AVRO_SCHEMA_USER, TMP.newFolder(), 1, "a");
+
+ FileSystemDatasetFactory factory = new FileSystemDatasetFactory(rootAllocator(), NativeMemoryPool.getDefault(),
+ FileFormat.PARQUET, writeSupport.getOutputURI());
+ NativeDataset dataset = factory.finish();
+ ScanOptions options = new ScanOptions(new String[0], 100);
+ NativeScanner scanner = dataset.newScan(options);
+ List<? extends NativeScanTask> taskList = collect(scanner.scan());
+ NativeScanTask task = taskList.get(0);
+ List<ArrowRecordBatch> datum = executor.submit(() -> collect(task.execute())).get();
+
+ AutoCloseables.close(datum);
+ AutoCloseables.close(taskList);
+ AutoCloseables.close(scanner, dataset, factory);
+ }
+
+ @Test
+ public void testScanAfterClose1() throws Exception {
+ ParquetWriteSupport writeSupport = ParquetWriteSupport.writeTempFile(AVRO_SCHEMA_USER, TMP.newFolder(), 1, "a");
+
+ FileSystemDatasetFactory factory = new FileSystemDatasetFactory(rootAllocator(), NativeMemoryPool.getDefault(),
+ FileFormat.PARQUET, writeSupport.getOutputURI());
+ NativeDataset dataset = factory.finish();
+ ScanOptions options = new ScanOptions(new String[0], 100);
+ NativeScanner scanner = dataset.newScan(options);
+ scanner.close();
+ assertThrows(NativeInstanceReleasedException.class, scanner::scan);
+ }
+
+ @Test
+ public void testScanAfterClose2() throws Exception {
+ ParquetWriteSupport writeSupport = ParquetWriteSupport.writeTempFile(AVRO_SCHEMA_USER, TMP.newFolder(), 1, "a");
+
+ FileSystemDatasetFactory factory = new FileSystemDatasetFactory(rootAllocator(), NativeMemoryPool.getDefault(),
+ FileFormat.PARQUET, writeSupport.getOutputURI());
+ NativeDataset dataset = factory.finish();
+ ScanOptions options = new ScanOptions(new String[0], 100);
+ NativeScanner scanner = dataset.newScan(options);
+ List<? extends NativeScanTask> tasks = collect(scanner.scan());
+ NativeScanTask task = tasks.get(0);
+ task.close();
+ assertThrows(NativeInstanceReleasedException.class, task::execute);
+ }
+
+ @Test
+ public void testScanAfterClose3() throws Exception {
+ ParquetWriteSupport writeSupport = ParquetWriteSupport.writeTempFile(AVRO_SCHEMA_USER, TMP.newFolder(), 1, "a");
+
+ FileSystemDatasetFactory factory = new FileSystemDatasetFactory(rootAllocator(), NativeMemoryPool.getDefault(),
+ FileFormat.PARQUET, writeSupport.getOutputURI());
+ NativeDataset dataset = factory.finish();
+ ScanOptions options = new ScanOptions(new String[0], 100);
+ NativeScanner scanner = dataset.newScan(options);
+ List<? extends NativeScanTask> tasks = collect(scanner.scan());
+ NativeScanTask task = tasks.get(0);
+ ScanTask.BatchIterator iterator = task.execute();
+ task.close();
+ assertThrows(NativeInstanceReleasedException.class, iterator::hasNext);
+ }
+
+ @Test
+ public void testAllocatorReservation() throws Exception {
+ ParquetWriteSupport writeSupport = ParquetWriteSupport.writeTempFile(AVRO_SCHEMA_USER, TMP.newFolder(), 1, "a");
+ FileSystemDatasetFactory factory = new FileSystemDatasetFactory(rootAllocator(), NativeMemoryPool.getDefault(),
+ FileFormat.PARQUET, writeSupport.getOutputURI());
+ ScanOptions options = new ScanOptions(new String[0], 100);
+ long initReservation = rootAllocator().getAllocatedMemory();
+ List<ArrowRecordBatch> datum = collectResultFromFactory(factory, options);
+ final long expected_diff = datum.stream()
+ .flatMapToLong(batch -> batch.getBuffers()
+ .stream()
+ .mapToLong(buf -> buf.getReferenceManager().getAccountedSize())).sum();
+ long reservation = rootAllocator().getAllocatedMemory();
+ AutoCloseables.close(datum);
+ long finalReservation = rootAllocator().getAllocatedMemory();
+ Assert.assertEquals(expected_diff, reservation - initReservation);
+ Assert.assertEquals(-expected_diff, finalReservation - reservation);
+ }
+
+ @Test
+ public void testDirectMemoryReservation() throws Exception {
+ ParquetWriteSupport writeSupport = ParquetWriteSupport.writeTempFile(AVRO_SCHEMA_USER, TMP.newFolder(), 1, "a");
+ NativeMemoryPool pool = NativeMemoryPool.createListenable(DirectReservationListener.instance());
+ FileSystemDatasetFactory factory = new FileSystemDatasetFactory(rootAllocator(),
+ pool, FileFormat.PARQUET,
+ writeSupport.getOutputURI());
+ ScanOptions options = new ScanOptions(new String[0], 100);
+ long initReservation = DirectReservationListener.instance().getCurrentDirectMemReservation();
+ List<ArrowRecordBatch> datum = collectResultFromFactory(factory, options);
+ long reservation = DirectReservationListener.instance().getCurrentDirectMemReservation();
+ AutoCloseables.close(datum);
+ AutoCloseables.close(pool);
+ long finalReservation = DirectReservationListener.instance().getCurrentDirectMemReservation();
+ final long expected_diff = DEFAULT_NATIVE_MEMORY_POOL_BLOCK_SIZE;
+ Assert.assertEquals(expected_diff, reservation - initReservation);
+ Assert.assertEquals(-expected_diff, finalReservation - reservation);
+ }
+
+ @Test
+ public void testCustomReservationListener() throws Exception {
Review comment:
done
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] zhztheplayer commented on a change in pull request #7030: ARROW-7808: [Java][Dataset] Implement Dataset Java API by JNI to C++
Posted by GitBox <gi...@apache.org>.
zhztheplayer commented on a change in pull request #7030:
URL: https://github.com/apache/arrow/pull/7030#discussion_r551694096
##########
File path: cpp/src/arrow/memory_pool.cc
##########
@@ -534,4 +535,139 @@ int64_t ProxyMemoryPool::max_memory() const { return impl_->max_memory(); }
std::string ProxyMemoryPool::backend_name() const { return impl_->backend_name(); }
+ReservationListener::~ReservationListener() {}
+
+ReservationListener::ReservationListener() {}
+
+class ReservationListenableMemoryPool::ReservationListenableMemoryPoolImpl {
+ public:
+ explicit ReservationListenableMemoryPoolImpl(
+ MemoryPool* pool, std::shared_ptr<ReservationListener> listener, int64_t block_size)
+ : pool_(pool),
+ listener_(listener),
+ block_size_(block_size),
+ blocks_reserved_(0),
+ bytes_reserved_(0) {}
+
+ Status Allocate(int64_t size, uint8_t** out) {
+ RETURN_NOT_OK(UpdateReservation(size));
+ Status error = pool_->Allocate(size, out);
+ if (!error.ok()) {
+ RETURN_NOT_OK(UpdateReservation(-size));
+ return error;
+ }
+ return Status::OK();
+ }
+
+ Status Reallocate(int64_t old_size, int64_t new_size, uint8_t** ptr) {
+ bool reserved = false;
+ int64_t diff = new_size - old_size;
+ if (new_size >= old_size) {
+ RETURN_NOT_OK(UpdateReservation(diff));
+ reserved = true;
Review comment:
Done, thanks for the suggestion.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] pitrou commented on a change in pull request #7030: ARROW-7808: [Java][Dataset] Implement Dataset Java API by JNI to C++
Posted by GitBox <gi...@apache.org>.
pitrou commented on a change in pull request #7030:
URL: https://github.com/apache/arrow/pull/7030#discussion_r552554266
##########
File path: cpp/src/arrow/memory_pool.h
##########
@@ -149,6 +149,43 @@ class ARROW_EXPORT ProxyMemoryPool : public MemoryPool {
std::unique_ptr<ProxyMemoryPoolImpl> impl_;
};
+class ARROW_EXPORT ReservationListener {
+ public:
+ virtual ~ReservationListener();
+
+ virtual Status OnReservation(int64_t size) = 0;
+ virtual Status OnRelease(int64_t size) = 0;
+
+ protected:
+ ReservationListener();
+};
+
+class ARROW_EXPORT ReservationListenableMemoryPool : public MemoryPool {
Review comment:
What does "ReservationListenableMemoryPool" mean? What does "pre-reserving" mean? How does this even work, since you're delegating actual allocations to a regular MemoryPool?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] zhztheplayer commented on a change in pull request #7030: ARROW-7808: [Java][Dataset] Implement Datasets Java API by JNI to C++
Posted by GitBox <gi...@apache.org>.
zhztheplayer commented on a change in pull request #7030:
URL: https://github.com/apache/arrow/pull/7030#discussion_r417790061
##########
File path: java/pom.xml
##########
@@ -369,24 +369,24 @@
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-compiler-plugin</artifactId>
<version>3.6.2</version>
- <configuration>
- <compilerArgs>
- <arg>-XDcompilePolicy=simple</arg>
- <arg>-Xplugin:ErrorProne</arg>
- </compilerArgs>
- <annotationProcessorPaths>
- <path>
- <groupId>com.google.errorprone</groupId>
- <artifactId>error_prone_core</artifactId>
- <version>2.3.3</version>
- </path>
- <path>
- <groupId>org.immutables</groupId>
- <artifactId>value</artifactId>
- <version>2.8.2</version>
- </path>
- </annotationProcessorPaths>
- </configuration>
+<!-- <configuration>-->
Review comment:
my mistake, reverted. Thanks!
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] emkornfield commented on pull request #7030: ARROW-7808: [Java][Dataset] Implement Datasets Java API by JNI to C++
Posted by GitBox <gi...@apache.org>.
emkornfield commented on pull request #7030:
URL: https://github.com/apache/arrow/pull/7030#issuecomment-632478673
@fsaintjacques it would be good to get a review from someone actively working on datasets to make sure bindings are at the right level, I can try to do a final pass after that.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] zhztheplayer commented on a change in pull request #7030: ARROW-7808: [Java][Dataset] Implement Datasets Java API by JNI to C++
Posted by GitBox <gi...@apache.org>.
zhztheplayer commented on a change in pull request #7030:
URL: https://github.com/apache/arrow/pull/7030#discussion_r429553399
##########
File path: cpp/src/jni/dataset/jni_wrapper.cpp
##########
@@ -0,0 +1,517 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include <arrow/dataset/api.h>
+#include <arrow/dataset/file_base.h>
+#include <arrow/filesystem/hdfs.h>
+#include <arrow/filesystem/localfs.h>
+#include <arrow/io/api.h>
+#include <arrow/ipc/api.h>
+#include <arrow/util/iterator.h>
+
+#include "org_apache_arrow_dataset_file_JniWrapper.h"
+#include "org_apache_arrow_dataset_jni_JniWrapper.h"
+
+static jclass illegal_access_exception_class;
+static jclass illegal_argument_exception_class;
+static jclass runtime_exception_class;
+
+static jclass record_batch_handle_class;
+static jclass record_batch_handle_field_class;
+static jclass record_batch_handle_buffer_class;
+
+static jmethodID record_batch_handle_constructor;
+static jmethodID record_batch_handle_field_constructor;
+static jmethodID record_batch_handle_buffer_constructor;
+
+static jint JNI_VERSION = JNI_VERSION_1_6;
+
+class JniPendingException : public std::runtime_error {
+ public:
+ explicit JniPendingException(const std::string& arg) : runtime_error(arg) {}
+};
+
+void ThrowPendingException(const std::string& message) {
+ throw JniPendingException(message);
+}
+
+template <typename T>
+T JniGetOrThrow(arrow::Result<T> result) {
+ if (!result.status().ok()) {
+ ThrowPendingException(result.status().message());
+ }
+ return std::move(result).ValueOrDie();
+}
+
+void JniAssertOkOrThrow(arrow::Status status) {
+ if (!status.ok()) {
+ ThrowPendingException(status.message());
+ }
+}
+
+void JniThrow(std::string message) { ThrowPendingException(message); }
+
+#define JNI_METHOD_START try {
+// macro ended
+
+#define JNI_METHOD_END(fallback_expr) \
+ } \
+ catch (JniPendingException & e) { \
+ env->ThrowNew(runtime_exception_class, e.what()); \
+ return fallback_expr; \
+ }
+// macro ended
+
+jclass CreateGlobalClassReference(JNIEnv* env, const char* class_name) {
+ jclass local_class = env->FindClass(class_name);
+ jclass global_class = (jclass)env->NewGlobalRef(local_class);
+ env->DeleteLocalRef(local_class);
+ return global_class;
+}
+
+arrow::Result<jmethodID> GetMethodID(JNIEnv* env, jclass this_class, const char* name,
+ const char* sig) {
+ jmethodID ret = env->GetMethodID(this_class, name, sig);
+ if (ret == nullptr) {
+ std::string error_message = "Unable to find method " + std::string(name) +
+ " within signature" + std::string(sig);
+ return arrow::Status::Invalid(error_message);
+ }
+ return ret;
+}
+
+jint JNI_OnLoad(JavaVM* vm, void* reserved) {
+ JNIEnv* env;
+ if (vm->GetEnv(reinterpret_cast<void**>(&env), JNI_VERSION) != JNI_OK) {
+ return JNI_ERR;
+ }
+ JNI_METHOD_START
+ illegal_access_exception_class =
+ CreateGlobalClassReference(env, "Ljava/lang/IllegalAccessException;");
+ illegal_argument_exception_class =
+ CreateGlobalClassReference(env, "Ljava/lang/IllegalArgumentException;");
+ runtime_exception_class =
+ CreateGlobalClassReference(env, "Ljava/lang/RuntimeException;");
+
+ record_batch_handle_class =
+ CreateGlobalClassReference(env,
+ "Lorg/apache/arrow/"
+ "dataset/jni/NativeRecordBatchHandle;");
+ record_batch_handle_field_class =
+ CreateGlobalClassReference(env,
+ "Lorg/apache/arrow/"
+ "dataset/jni/NativeRecordBatchHandle$Field;");
+ record_batch_handle_buffer_class =
+ CreateGlobalClassReference(env,
+ "Lorg/apache/arrow/"
+ "dataset/jni/NativeRecordBatchHandle$Buffer;");
+
+ record_batch_handle_constructor =
+ JniGetOrThrow(GetMethodID(env, record_batch_handle_class, "<init>",
+ "(J[Lorg/apache/arrow/dataset/"
+ "jni/NativeRecordBatchHandle$Field;"
+ "[Lorg/apache/arrow/dataset/"
+ "jni/NativeRecordBatchHandle$Buffer;)V"));
+ record_batch_handle_field_constructor =
+ JniGetOrThrow(GetMethodID(env, record_batch_handle_field_class, "<init>", "(JJ)V"));
+ record_batch_handle_buffer_constructor = JniGetOrThrow(
+ GetMethodID(env, record_batch_handle_buffer_class, "<init>", "(JJJJ)V"));
+
+ return JNI_VERSION;
+ JNI_METHOD_END(JNI_ERR)
+}
+
+void JNI_OnUnload(JavaVM* vm, void* reserved) {
+ JNIEnv* env;
+ vm->GetEnv(reinterpret_cast<void**>(&env), JNI_VERSION);
+ env->DeleteGlobalRef(illegal_access_exception_class);
+ env->DeleteGlobalRef(illegal_argument_exception_class);
+ env->DeleteGlobalRef(runtime_exception_class);
+ env->DeleteGlobalRef(record_batch_handle_class);
+ env->DeleteGlobalRef(record_batch_handle_field_class);
+ env->DeleteGlobalRef(record_batch_handle_buffer_class);
+}
+
+std::shared_ptr<arrow::Schema> SchemaFromColumnNames(
+ const std::shared_ptr<arrow::Schema>& input,
+ const std::vector<std::string>& column_names) {
+ std::vector<std::shared_ptr<arrow::Field>> columns;
+ for (const auto& name : column_names) {
+ columns.push_back(input->GetFieldByName(name));
+ }
+ return std::make_shared<arrow::Schema>(columns);
+}
+
+arrow::Result<std::shared_ptr<arrow::dataset::FileFormat>> GetFileFormat(jint id) {
+ switch (id) {
+ case 0:
+ return std::make_shared<arrow::dataset::ParquetFileFormat>();
+ default:
+ std::string error_message = "illegal file format id: " + std::to_string(id);
+ return arrow::Status::Invalid(error_message);
+ }
+}
+
+arrow::Result<std::shared_ptr<arrow::fs::FileSystem>> GetFileSystem(
+ jint id, std::string path, std::string* out_path) {
+ switch (id) {
+ case 0:
+ *out_path = path;
+ return std::make_shared<arrow::fs::LocalFileSystem>();
+ case 1: {
+ ARROW_ASSIGN_OR_RAISE(std::shared_ptr<arrow::fs::FileSystem> ret,
+ arrow::fs::FileSystemFromUri(path, out_path))
+ return ret;
+ }
+ default:
+ std::string error_message = "illegal file system id: " + std::to_string(id);
+ return arrow::Status::Invalid(error_message);
+ }
+}
+
+std::string JStringToCString(JNIEnv* env, jstring string) {
+ if (string == nullptr) {
+ return std::string();
+ }
+ jboolean copied;
+ int32_t length = env->GetStringUTFLength(string);
+ const char* chars = env->GetStringUTFChars(string, &copied);
+ std::string str = std::string(chars, length);
+ // fixme calling ReleaseStringUTFChars if memory leak faced
+ return str;
+}
+
+std::vector<std::string> ToStringVector(JNIEnv* env, jobjectArray& str_array) {
+ int length = env->GetArrayLength(str_array);
+ std::vector<std::string> vector;
+ for (int i = 0; i < length; i++) {
+ auto string = (jstring)(env->GetObjectArrayElement(str_array, i));
+ vector.push_back(JStringToCString(env, string));
+ }
+ return vector;
+}
+
+template <typename T>
+jlong CreateNativeRef(std::shared_ptr<T> t) {
+ std::shared_ptr<T>* retained_ptr = new std::shared_ptr<T>(t);
+ return reinterpret_cast<jlong>(retained_ptr);
+}
+
+template <typename T>
+std::shared_ptr<T> RetrieveNativeInstance(jlong ref) {
+ std::shared_ptr<T>* retrieved_ptr = reinterpret_cast<std::shared_ptr<T>*>(ref);
+ return *retrieved_ptr;
+}
+
+template <typename T>
+void ReleaseNativeRef(jlong ref) {
+ std::shared_ptr<T>* retrieved_ptr = reinterpret_cast<std::shared_ptr<T>*>(ref);
+ delete retrieved_ptr;
+}
+
+arrow::Result<jbyteArray> ToSchemaByteArray(JNIEnv* env,
+ std::shared_ptr<arrow::Schema> schema) {
+ ARROW_ASSIGN_OR_RAISE(
+ std::shared_ptr<arrow::Buffer> buffer,
+ arrow::ipc::SerializeSchema(*schema, nullptr, arrow::default_memory_pool()))
+
+ jbyteArray out = env->NewByteArray(buffer->size());
+ auto src = reinterpret_cast<const jbyte*>(buffer->data());
+ env->SetByteArrayRegion(out, 0, buffer->size(), src);
+ return out;
+}
+
+arrow::Result<std::shared_ptr<arrow::Schema>> FromSchemaByteArray(
+ JNIEnv* env, jbyteArray schemaBytes) {
+ arrow::ipc::DictionaryMemo in_memo;
+ int schemaBytes_len = env->GetArrayLength(schemaBytes);
+ jbyte* schemaBytes_data = env->GetByteArrayElements(schemaBytes, nullptr);
+ auto serialized_schema = std::make_shared<arrow::Buffer>(
+ reinterpret_cast<uint8_t*>(schemaBytes_data), schemaBytes_len);
+ arrow::io::BufferReader buf_reader(serialized_schema);
+ ARROW_ASSIGN_OR_RAISE(std::shared_ptr<arrow::Schema> schema,
+ arrow::ipc::ReadSchema(&buf_reader, &in_memo))
+ env->ReleaseByteArrayElements(schemaBytes, schemaBytes_data, JNI_ABORT);
+ return schema;
+}
+
+/*
+ * Class: org_apache_arrow_dataset_jni_JniWrapper
+ * Method: closeDatasetFactory
+ * Signature: (J)V
+ */
+JNIEXPORT void JNICALL Java_org_apache_arrow_dataset_jni_JniWrapper_closeDatasetFactory(
+ JNIEnv* env, jobject, jlong id) {
+ JNI_METHOD_START
+ ReleaseNativeRef<arrow::dataset::DatasetFactory>(id);
+ JNI_METHOD_END()
+}
+
+/*
+ * Class: org_apache_arrow_dataset_jni_JniWrapper
+ * Method: inspectSchema
+ * Signature: (J)[B
+ */
+JNIEXPORT jbyteArray JNICALL Java_org_apache_arrow_dataset_jni_JniWrapper_inspectSchema(
+ JNIEnv* env, jobject, jlong dataset_factor_id) {
+ JNI_METHOD_START
+ std::shared_ptr<arrow::dataset::DatasetFactory> d =
+ RetrieveNativeInstance<arrow::dataset::DatasetFactory>(dataset_factor_id);
+ std::shared_ptr<arrow::Schema> schema = JniGetOrThrow(d->Inspect());
+ return JniGetOrThrow(ToSchemaByteArray(env, schema));
+ JNI_METHOD_END(nullptr)
+}
+
+/*
+ * Class: org_apache_arrow_dataset_jni_JniWrapper
+ * Method: createDataset
+ * Signature: (J[B)J
+ */
+JNIEXPORT jlong JNICALL Java_org_apache_arrow_dataset_jni_JniWrapper_createDataset(
+ JNIEnv* env, jobject, jlong dataset_factory_id, jbyteArray schema_bytes) {
+ JNI_METHOD_START
+ std::shared_ptr<arrow::dataset::DatasetFactory> d =
+ RetrieveNativeInstance<arrow::dataset::DatasetFactory>(dataset_factory_id);
+ std::shared_ptr<arrow::Schema> schema;
+ schema = JniGetOrThrow(FromSchemaByteArray(env, schema_bytes));
+ std::shared_ptr<arrow::dataset::Dataset> dataset = JniGetOrThrow(d->Finish(schema));
+ return CreateNativeRef(dataset);
+ JNI_METHOD_END(-1L)
+}
+
+/*
+ * Class: org_apache_arrow_dataset_jni_JniWrapper
+ * Method: closeDataset
+ * Signature: (J)V
+ */
+JNIEXPORT void JNICALL Java_org_apache_arrow_dataset_jni_JniWrapper_closeDataset(
+ JNIEnv* env, jobject, jlong id) {
+ JNI_METHOD_START
+ ReleaseNativeRef<arrow::dataset::Dataset>(id);
+ JNI_METHOD_END()
+}
+
+/// \class DisposableScannerAdaptor
+/// \brief An adaptor that iterates over a Scanner instance then returns RecordBatches
+/// directly.
+///
+/// This lessens the complexity of the JNI bridge to make sure it to be easier to
+/// maintain. On Java-side, NativeScanner can only produces a single NativeScanTask
+/// instance during its whole lifecycle. Each task stands for a DisposableScannerAdaptor
+/// instance through JNI bridge.
+///
+class DisposableScannerAdaptor {
Review comment:
So let's just remove this and add a method `Iterator<ScanTask> Scanner.scan()` to JNI bridge? I have same concern with you and for me this code is temporary. I think we wanted to limited JNI bridge simple enough at the first implementation so that if C++ API get factored again in future we may save time from changing JNI code to make Java side work.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] emkornfield commented on a change in pull request #7030: ARROW-7808: [Java][Dataset] Implement Datasets Java API
Posted by GitBox <gi...@apache.org>.
emkornfield commented on a change in pull request #7030:
URL: https://github.com/apache/arrow/pull/7030#discussion_r414987154
##########
File path: java/dataset/src/main/java/org/apache/arrow/dataset/jni/JniWrapper.java
##########
@@ -0,0 +1,61 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.arrow.dataset.jni;
+
+/**
+ * JNI wrapper for Datasets API's native implementation.
+ */
+public class JniWrapper {
+
+ private static final JniWrapper INSTANCE = new JniWrapper();
+
+ public static JniWrapper get() {
+ return INSTANCE;
+ }
+
+ private JniWrapper() {
+ JniLoader.get().ensureLoaded();
+ }
+
+ public native void closeDatasetFactory(long datasetFactoryId);
Review comment:
these methods need javadoc.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] jacques-n commented on a change in pull request #7030: ARROW-7808: [Java][Dataset] Implement Datasets Java API by JNI to C++
Posted by GitBox <gi...@apache.org>.
jacques-n commented on a change in pull request #7030:
URL: https://github.com/apache/arrow/pull/7030#discussion_r449967588
##########
File path: java/dataset/src/main/java/org/apache/arrow/memory/NativeUnderlingMemory.java
##########
@@ -0,0 +1,65 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.arrow.memory;
+
+import org.apache.arrow.dataset.jni.JniWrapper;
+
+/**
+ * AllocationManager implementation for Native allocated memory.
+ */
+public class NativeUnderlingMemory extends AllocationManager {
Review comment:
Not necessarily. When you allocate and deallocate memory in c++ you can call (via reflection) java.nio.Bits.reserveMemory() and unreserveMemory(). If you wanted to be extra fancy you could actually just do the reads/writes to the underlying atomics. Alternatively, you could build a pooling impl for C++ that uses the JNIEnv. NewDirectByteBuffer.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] zhztheplayer commented on pull request #7030: ARROW-7808: [Java][Dataset] Implement Datasets Java API by JNI to C++
Posted by GitBox <gi...@apache.org>.
zhztheplayer commented on pull request #7030:
URL: https://github.com/apache/arrow/pull/7030#issuecomment-658605439
> I was suggesting a third approach: C++ memory pool that simply updates the direct memory info via Bits in java. It should be done chunk-wise to avoid excessive JNI boundary calls.
It looks a little bit more complicated: IIUC Netty doesn't seem to reserve from Bits internally, it has a individual counter that is limited to max direct mem amount. If we use Bits then we'll be able to exceed the overall usage by using up to 2x amount. My suggestion is to maintain our own mechanism to reserve Bits for every byte Java allocator managed, then ignore Netty's own direct mem counter. This way no JNI overhead will be brought. Would you think of this a feasible direction?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] zhztheplayer commented on a change in pull request #7030: ARROW-7808: [Java][Dataset] Implement Datasets Java API by JNI to C++
Posted by GitBox <gi...@apache.org>.
zhztheplayer commented on a change in pull request #7030:
URL: https://github.com/apache/arrow/pull/7030#discussion_r440075959
##########
File path: java/dataset/src/main/java/org/apache/arrow/memory/NativeUnderlingMemory.java
##########
@@ -0,0 +1,65 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.arrow.memory;
+
+import org.apache.arrow.dataset.jni.JniWrapper;
+
+/**
+ * AllocationManager implementation for Native allocated memory.
+ */
+public class NativeUnderlingMemory extends AllocationManager {
Review comment:
Hi @jacques-n,
Thanks for your suggestion but I think the "weaknesses" of current implementation might be somehow overrated. Regarding the weaknesses, it's of course urgent for orc adaptor to improve its memory allocation process because all of the allocated memory by orc is being escaped from Java allocator. However we don't have the same problem here in Datasets, allocator is going to manage all the native memory. The only issue we may have is that OOM error might be thrown after a buffer is allocated rather than before we create it. But because OOM will be finally thrown and over-allocated memory will be finally released, the impact will be very very small.
And I knew we can possibly change to allocate Java memory directly from C++ via JNI, but codes will become messy and performance will turn to worse. That is why I didn't propose such a solution in this PR.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] emkornfield commented on a change in pull request #7030: ARROW-7808: [Java][Dataset] Implement Datasets Java API by JNI to C++
Posted by GitBox <gi...@apache.org>.
emkornfield commented on a change in pull request #7030:
URL: https://github.com/apache/arrow/pull/7030#discussion_r425565223
##########
File path: java/dataset/src/main/java/org/apache/arrow/dataset/jni/NativeScanTask.java
##########
@@ -0,0 +1,110 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.arrow.dataset.jni;
+
+import java.util.ArrayList;
+import java.util.NoSuchElementException;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.stream.Collectors;
+
+import org.apache.arrow.dataset.scanner.ScanTask;
+import org.apache.arrow.memory.ArrowBuf;
+import org.apache.arrow.memory.BaseAllocator;
+import org.apache.arrow.memory.BufferLedger;
+import org.apache.arrow.memory.NativeUnderlingMemory;
+import org.apache.arrow.memory.Ownerships;
+import org.apache.arrow.vector.ipc.message.ArrowFieldNode;
+import org.apache.arrow.vector.ipc.message.ArrowRecordBatch;
+
+/**
+ * Native implementation of {@link ScanTask}. Currently RecordBatches are iterated directly by the scanner
+ * id via {@link JniWrapper}, thus we allow only one-time execution of method {@link #scan()}. If a re-scan
+ * operation is expected, call {@link NativeDataset#newScan} to create a new scanner instance.
+ */
+public class NativeScanTask implements ScanTask, AutoCloseable {
+ private final NativeScanner scanner;
+ private final AtomicBoolean executed = new AtomicBoolean(false);
+
+ /**
+ * Constructor.
+ */
+ public NativeScanTask(NativeScanner scanner) {
+ this.scanner = scanner;
+ }
+
+ @Override
+ public BatchIterator scan() {
+ if (!executed.compareAndSet(false, true)) {
+ throw new UnsupportedOperationException("Method scan() cannot be executed more than once if it is a " +
+ "native scan task");
+ }
+ return new BatchIterator() {
+ private ArrowRecordBatch peek = null;
+
+ @Override
+ public void close() throws Exception {
+ scanner.close();
+ }
+
+ @Override
+ public boolean hasNext() {
+ if (peek != null) {
+ return true;
+ }
+ NativeRecordBatchHandle handle = JniWrapper.get().nextRecordBatch(scanner.getId());
+ if (handle == null) {
+ return false;
+ }
+ final ArrayList<ArrowBuf> buffers = new ArrayList<>();
+ for (NativeRecordBatchHandle.Buffer buffer : handle.getBuffers()) {
+ final BaseAllocator allocator = scanner.getContext().getAllocator();
+ final NativeUnderlingMemory am = new NativeUnderlingMemory(allocator,
+ (int) buffer.size, buffer.nativeInstanceId, buffer.memoryAddress);
+ final BufferLedger ledger = Ownerships.get().takeOwnership(allocator, am);
+ ArrowBuf buf = new ArrowBuf(ledger, null, (int) buffer.size, buffer.memoryAddress, false);
+ buffers.add(buf);
+ }
+ try {
+ peek = new ArrowRecordBatch((int) handle.getNumRows(), handle.getFields().stream()
+ .map(field -> new ArrowFieldNode((int) field.length, (int) field.nullCount))
+ .collect(Collectors.toList()), buffers);
+ return true;
+ } finally {
+ buffers.forEach(buffer -> buffer.getReferenceManager().release());
+ }
+ }
+
+ @Override
+ public ArrowRecordBatch next() {
+ if (!hasNext()) {
+ throw new NoSuchElementException();
+ }
+ try {
Review comment:
nit: new line here would make this more readable.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] zhztheplayer commented on a change in pull request #7030: ARROW-7808: [Java][Dataset] Implement Datasets Java API by JNI to C++
Posted by GitBox <gi...@apache.org>.
zhztheplayer commented on a change in pull request #7030:
URL: https://github.com/apache/arrow/pull/7030#discussion_r430864740
##########
File path: cpp/src/jni/dataset/jni_wrapper.cpp
##########
@@ -0,0 +1,517 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include <arrow/dataset/api.h>
+#include <arrow/dataset/file_base.h>
+#include <arrow/filesystem/hdfs.h>
+#include <arrow/filesystem/localfs.h>
+#include <arrow/io/api.h>
+#include <arrow/ipc/api.h>
+#include <arrow/util/iterator.h>
+
+#include "org_apache_arrow_dataset_file_JniWrapper.h"
+#include "org_apache_arrow_dataset_jni_JniWrapper.h"
+
+static jclass illegal_access_exception_class;
+static jclass illegal_argument_exception_class;
+static jclass runtime_exception_class;
+
+static jclass record_batch_handle_class;
+static jclass record_batch_handle_field_class;
+static jclass record_batch_handle_buffer_class;
+
+static jmethodID record_batch_handle_constructor;
+static jmethodID record_batch_handle_field_constructor;
+static jmethodID record_batch_handle_buffer_constructor;
+
+static jint JNI_VERSION = JNI_VERSION_1_6;
+
+class JniPendingException : public std::runtime_error {
+ public:
+ explicit JniPendingException(const std::string& arg) : runtime_error(arg) {}
+};
+
+void ThrowPendingException(const std::string& message) {
+ throw JniPendingException(message);
+}
+
+template <typename T>
+T JniGetOrThrow(arrow::Result<T> result) {
+ if (!result.status().ok()) {
+ ThrowPendingException(result.status().message());
+ }
+ return std::move(result).ValueOrDie();
+}
+
+void JniAssertOkOrThrow(arrow::Status status) {
+ if (!status.ok()) {
+ ThrowPendingException(status.message());
+ }
+}
+
+void JniThrow(std::string message) { ThrowPendingException(message); }
+
+#define JNI_METHOD_START try {
+// macro ended
+
+#define JNI_METHOD_END(fallback_expr) \
+ } \
+ catch (JniPendingException & e) { \
+ env->ThrowNew(runtime_exception_class, e.what()); \
+ return fallback_expr; \
+ }
+// macro ended
+
+jclass CreateGlobalClassReference(JNIEnv* env, const char* class_name) {
+ jclass local_class = env->FindClass(class_name);
+ jclass global_class = (jclass)env->NewGlobalRef(local_class);
+ env->DeleteLocalRef(local_class);
+ return global_class;
+}
+
+arrow::Result<jmethodID> GetMethodID(JNIEnv* env, jclass this_class, const char* name,
+ const char* sig) {
+ jmethodID ret = env->GetMethodID(this_class, name, sig);
+ if (ret == nullptr) {
+ std::string error_message = "Unable to find method " + std::string(name) +
+ " within signature" + std::string(sig);
+ return arrow::Status::Invalid(error_message);
+ }
+ return ret;
+}
+
+jint JNI_OnLoad(JavaVM* vm, void* reserved) {
+ JNIEnv* env;
+ if (vm->GetEnv(reinterpret_cast<void**>(&env), JNI_VERSION) != JNI_OK) {
+ return JNI_ERR;
+ }
+ JNI_METHOD_START
+ illegal_access_exception_class =
+ CreateGlobalClassReference(env, "Ljava/lang/IllegalAccessException;");
+ illegal_argument_exception_class =
+ CreateGlobalClassReference(env, "Ljava/lang/IllegalArgumentException;");
+ runtime_exception_class =
+ CreateGlobalClassReference(env, "Ljava/lang/RuntimeException;");
+
+ record_batch_handle_class =
+ CreateGlobalClassReference(env,
+ "Lorg/apache/arrow/"
+ "dataset/jni/NativeRecordBatchHandle;");
+ record_batch_handle_field_class =
+ CreateGlobalClassReference(env,
+ "Lorg/apache/arrow/"
+ "dataset/jni/NativeRecordBatchHandle$Field;");
+ record_batch_handle_buffer_class =
+ CreateGlobalClassReference(env,
+ "Lorg/apache/arrow/"
+ "dataset/jni/NativeRecordBatchHandle$Buffer;");
+
+ record_batch_handle_constructor =
+ JniGetOrThrow(GetMethodID(env, record_batch_handle_class, "<init>",
+ "(J[Lorg/apache/arrow/dataset/"
+ "jni/NativeRecordBatchHandle$Field;"
+ "[Lorg/apache/arrow/dataset/"
+ "jni/NativeRecordBatchHandle$Buffer;)V"));
+ record_batch_handle_field_constructor =
+ JniGetOrThrow(GetMethodID(env, record_batch_handle_field_class, "<init>", "(JJ)V"));
+ record_batch_handle_buffer_constructor = JniGetOrThrow(
+ GetMethodID(env, record_batch_handle_buffer_class, "<init>", "(JJJJ)V"));
+
+ return JNI_VERSION;
+ JNI_METHOD_END(JNI_ERR)
+}
+
+void JNI_OnUnload(JavaVM* vm, void* reserved) {
+ JNIEnv* env;
+ vm->GetEnv(reinterpret_cast<void**>(&env), JNI_VERSION);
+ env->DeleteGlobalRef(illegal_access_exception_class);
+ env->DeleteGlobalRef(illegal_argument_exception_class);
+ env->DeleteGlobalRef(runtime_exception_class);
+ env->DeleteGlobalRef(record_batch_handle_class);
+ env->DeleteGlobalRef(record_batch_handle_field_class);
+ env->DeleteGlobalRef(record_batch_handle_buffer_class);
+}
+
+std::shared_ptr<arrow::Schema> SchemaFromColumnNames(
+ const std::shared_ptr<arrow::Schema>& input,
+ const std::vector<std::string>& column_names) {
+ std::vector<std::shared_ptr<arrow::Field>> columns;
+ for (const auto& name : column_names) {
+ columns.push_back(input->GetFieldByName(name));
+ }
+ return std::make_shared<arrow::Schema>(columns);
+}
+
+arrow::Result<std::shared_ptr<arrow::dataset::FileFormat>> GetFileFormat(jint id) {
+ switch (id) {
+ case 0:
+ return std::make_shared<arrow::dataset::ParquetFileFormat>();
+ default:
+ std::string error_message = "illegal file format id: " + std::to_string(id);
+ return arrow::Status::Invalid(error_message);
+ }
+}
+
+arrow::Result<std::shared_ptr<arrow::fs::FileSystem>> GetFileSystem(
+ jint id, std::string path, std::string* out_path) {
+ switch (id) {
+ case 0:
+ *out_path = path;
+ return std::make_shared<arrow::fs::LocalFileSystem>();
+ case 1: {
+ ARROW_ASSIGN_OR_RAISE(std::shared_ptr<arrow::fs::FileSystem> ret,
+ arrow::fs::FileSystemFromUri(path, out_path))
+ return ret;
+ }
+ default:
+ std::string error_message = "illegal file system id: " + std::to_string(id);
+ return arrow::Status::Invalid(error_message);
+ }
+}
+
+std::string JStringToCString(JNIEnv* env, jstring string) {
+ if (string == nullptr) {
+ return std::string();
+ }
+ jboolean copied;
+ int32_t length = env->GetStringUTFLength(string);
+ const char* chars = env->GetStringUTFChars(string, &copied);
+ std::string str = std::string(chars, length);
+ // fixme calling ReleaseStringUTFChars if memory leak faced
+ return str;
+}
+
+std::vector<std::string> ToStringVector(JNIEnv* env, jobjectArray& str_array) {
+ int length = env->GetArrayLength(str_array);
+ std::vector<std::string> vector;
+ for (int i = 0; i < length; i++) {
+ auto string = (jstring)(env->GetObjectArrayElement(str_array, i));
+ vector.push_back(JStringToCString(env, string));
+ }
+ return vector;
+}
+
+template <typename T>
+jlong CreateNativeRef(std::shared_ptr<T> t) {
+ std::shared_ptr<T>* retained_ptr = new std::shared_ptr<T>(t);
+ return reinterpret_cast<jlong>(retained_ptr);
+}
+
+template <typename T>
+std::shared_ptr<T> RetrieveNativeInstance(jlong ref) {
+ std::shared_ptr<T>* retrieved_ptr = reinterpret_cast<std::shared_ptr<T>*>(ref);
+ return *retrieved_ptr;
+}
+
+template <typename T>
+void ReleaseNativeRef(jlong ref) {
+ std::shared_ptr<T>* retrieved_ptr = reinterpret_cast<std::shared_ptr<T>*>(ref);
+ delete retrieved_ptr;
+}
+
+arrow::Result<jbyteArray> ToSchemaByteArray(JNIEnv* env,
+ std::shared_ptr<arrow::Schema> schema) {
+ ARROW_ASSIGN_OR_RAISE(
+ std::shared_ptr<arrow::Buffer> buffer,
+ arrow::ipc::SerializeSchema(*schema, nullptr, arrow::default_memory_pool()))
+
+ jbyteArray out = env->NewByteArray(buffer->size());
+ auto src = reinterpret_cast<const jbyte*>(buffer->data());
+ env->SetByteArrayRegion(out, 0, buffer->size(), src);
+ return out;
+}
+
+arrow::Result<std::shared_ptr<arrow::Schema>> FromSchemaByteArray(
+ JNIEnv* env, jbyteArray schemaBytes) {
+ arrow::ipc::DictionaryMemo in_memo;
+ int schemaBytes_len = env->GetArrayLength(schemaBytes);
+ jbyte* schemaBytes_data = env->GetByteArrayElements(schemaBytes, nullptr);
+ auto serialized_schema = std::make_shared<arrow::Buffer>(
+ reinterpret_cast<uint8_t*>(schemaBytes_data), schemaBytes_len);
+ arrow::io::BufferReader buf_reader(serialized_schema);
+ ARROW_ASSIGN_OR_RAISE(std::shared_ptr<arrow::Schema> schema,
+ arrow::ipc::ReadSchema(&buf_reader, &in_memo))
+ env->ReleaseByteArrayElements(schemaBytes, schemaBytes_data, JNI_ABORT);
+ return schema;
+}
+
+/*
+ * Class: org_apache_arrow_dataset_jni_JniWrapper
+ * Method: closeDatasetFactory
+ * Signature: (J)V
+ */
+JNIEXPORT void JNICALL Java_org_apache_arrow_dataset_jni_JniWrapper_closeDatasetFactory(
+ JNIEnv* env, jobject, jlong id) {
+ JNI_METHOD_START
+ ReleaseNativeRef<arrow::dataset::DatasetFactory>(id);
+ JNI_METHOD_END()
+}
+
+/*
+ * Class: org_apache_arrow_dataset_jni_JniWrapper
+ * Method: inspectSchema
+ * Signature: (J)[B
+ */
+JNIEXPORT jbyteArray JNICALL Java_org_apache_arrow_dataset_jni_JniWrapper_inspectSchema(
+ JNIEnv* env, jobject, jlong dataset_factor_id) {
+ JNI_METHOD_START
+ std::shared_ptr<arrow::dataset::DatasetFactory> d =
+ RetrieveNativeInstance<arrow::dataset::DatasetFactory>(dataset_factor_id);
+ std::shared_ptr<arrow::Schema> schema = JniGetOrThrow(d->Inspect());
+ return JniGetOrThrow(ToSchemaByteArray(env, schema));
+ JNI_METHOD_END(nullptr)
+}
+
+/*
+ * Class: org_apache_arrow_dataset_jni_JniWrapper
+ * Method: createDataset
+ * Signature: (J[B)J
+ */
+JNIEXPORT jlong JNICALL Java_org_apache_arrow_dataset_jni_JniWrapper_createDataset(
+ JNIEnv* env, jobject, jlong dataset_factory_id, jbyteArray schema_bytes) {
+ JNI_METHOD_START
+ std::shared_ptr<arrow::dataset::DatasetFactory> d =
+ RetrieveNativeInstance<arrow::dataset::DatasetFactory>(dataset_factory_id);
+ std::shared_ptr<arrow::Schema> schema;
+ schema = JniGetOrThrow(FromSchemaByteArray(env, schema_bytes));
+ std::shared_ptr<arrow::dataset::Dataset> dataset = JniGetOrThrow(d->Finish(schema));
+ return CreateNativeRef(dataset);
+ JNI_METHOD_END(-1L)
+}
+
+/*
+ * Class: org_apache_arrow_dataset_jni_JniWrapper
+ * Method: closeDataset
+ * Signature: (J)V
+ */
+JNIEXPORT void JNICALL Java_org_apache_arrow_dataset_jni_JniWrapper_closeDataset(
+ JNIEnv* env, jobject, jlong id) {
+ JNI_METHOD_START
+ ReleaseNativeRef<arrow::dataset::Dataset>(id);
+ JNI_METHOD_END()
+}
+
+/// \class DisposableScannerAdaptor
+/// \brief An adaptor that iterates over a Scanner instance then returns RecordBatches
+/// directly.
+///
+/// This lessens the complexity of the JNI bridge to make sure it to be easier to
+/// maintain. On Java-side, NativeScanner can only produces a single NativeScanTask
+/// instance during its whole lifecycle. Each task stands for a DisposableScannerAdaptor
+/// instance through JNI bridge.
+///
+class DisposableScannerAdaptor {
Review comment:
OK. So far the plan is to replace it with passing task iterator to jni bridge. After c++ things got stable.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] jacques-n commented on a change in pull request #7030: ARROW-7808: [Java][Dataset] Implement Datasets Java API by JNI to C++
Posted by GitBox <gi...@apache.org>.
jacques-n commented on a change in pull request #7030:
URL: https://github.com/apache/arrow/pull/7030#discussion_r449967588
##########
File path: java/dataset/src/main/java/org/apache/arrow/memory/NativeUnderlingMemory.java
##########
@@ -0,0 +1,65 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.arrow.memory;
+
+import org.apache.arrow.dataset.jni.JniWrapper;
+
+/**
+ * AllocationManager implementation for Native allocated memory.
+ */
+public class NativeUnderlingMemory extends AllocationManager {
Review comment:
Not necessarily. When you allocate and deallocate memory in c++ you can call (via reflection) java.nio.Bits.reserveMemory() and unreserveMemory(). If you wanted to be extra fancy you could actually just do the reads/writes to the underlying atomics. Alternatively, you could build a pooling impl for C++ that uses the JNIEnv.NewDirectByteBuffer().
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] zhztheplayer commented on a change in pull request #7030: ARROW-7808: [Java][Dataset] Implement Datasets Java API by JNI to C++
Posted by GitBox <gi...@apache.org>.
zhztheplayer commented on a change in pull request #7030:
URL: https://github.com/apache/arrow/pull/7030#discussion_r417034247
##########
File path: cpp/src/jni/dataset/proto/Types.proto
##########
@@ -0,0 +1,149 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+syntax = "proto2";
+package types;
+
+option java_package = "org.apache.arrow.dataset";
Review comment:
@wesm
> So if there are changes in the C++ API, then refactoring can take place only at the JNI C++ code and no changes will be required in Java.
Sorry I am not quite getting it. Is it really needed to change Java code if c++ filter changed when it comes to the current PR? I mean, now the "layers" of the approach are something like:
```
Java: DatasetTypes.Condition (ProtoBuf generated code)
|
Java: byte[]
|
JNI Bridge
|
C++: uint8_t*
|
C++ types::Condition (ProtoBuf generated code)
| <---- c++ function TranslateFilter(...)
C++ arrow::dataset::Expression (Dataset Filters)
```
The top 5 layers are all based on protobuf generated codes so we don't have to change any of them from Java-side when we want to make changes from C++. Even say if we want to change Java filter API we can also just modify file Types.proto. Or in future if we refactor C++ API arrow::dataset::Expression, we may have to make related changes to the function `TranslateFilter(...)` as well to make JNI things matched but we don't have to change Java codes anyway.
And the filter tree structures are not 1-1 mapped too, the definitions in Types.proto are much simpler than arrow::dataset::Expression so far, I didn't mean to make them strictly match. In future we can change C++ filters API without touching Java API and it is fine enough to me.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] zhztheplayer commented on a change in pull request #7030: ARROW-7808: [Java][Dataset] Implement Dataset Java API by JNI to C++
Posted by GitBox <gi...@apache.org>.
zhztheplayer commented on a change in pull request #7030:
URL: https://github.com/apache/arrow/pull/7030#discussion_r567760755
##########
File path: cpp/src/jni/dataset/jni_wrapper.cpp
##########
@@ -0,0 +1,859 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include <mutex>
+
+#include <arrow/array.h>
+#include <arrow/dataset/api.h>
+#include <arrow/dataset/file_base.h>
+#include <arrow/filesystem/localfs.h>
+#include <arrow/io/api.h>
+#include <arrow/ipc/api.h>
+#include <arrow/util/iterator.h>
+#include <arrow/util/logging.h>
+
+#include "org_apache_arrow_dataset_file_JniWrapper.h"
+#include "org_apache_arrow_dataset_jni_JniWrapper.h"
+#include "org_apache_arrow_dataset_jni_NativeMemoryPool.h"
+
+namespace arrow {
+namespace dataset {
+namespace jni {
+
+static jclass illegal_access_exception_class;
+static jclass illegal_argument_exception_class;
+static jclass runtime_exception_class;
+
+static jclass record_batch_handle_class;
+static jclass record_batch_handle_field_class;
+static jclass record_batch_handle_buffer_class;
+static jclass java_reservation_listener_class;
+
+static jmethodID record_batch_handle_constructor;
+static jmethodID record_batch_handle_field_constructor;
+static jmethodID record_batch_handle_buffer_constructor;
+static jmethodID reserve_memory_method;
+static jmethodID unreserve_memory_method;
+
+static jlong default_memory_pool_id = -1L;
+
+static jint JNI_VERSION = JNI_VERSION_1_6;
+
+class JniPendingException : public std::runtime_error {
+ public:
+ explicit JniPendingException(const std::string& arg) : runtime_error(arg) {}
+};
+
+void ThrowPendingException(const std::string& message) {
+ throw JniPendingException(message);
+}
+
+template <typename T>
+T JniGetOrThrow(arrow::Result<T> result) {
+ if (!result.status().ok()) {
+ ThrowPendingException(result.status().message());
+ }
+ return std::move(result).ValueOrDie();
+}
+
+void JniAssertOkOrThrow(arrow::Status status) {
+ if (!status.ok()) {
+ ThrowPendingException(status.message());
+ }
+}
+
+void JniThrow(std::string message) { ThrowPendingException(message); }
+
+jclass CreateGlobalClassReference(JNIEnv* env, const char* class_name) {
+ jclass local_class = env->FindClass(class_name);
+ jclass global_class = (jclass)env->NewGlobalRef(local_class);
+ env->DeleteLocalRef(local_class);
+ return global_class;
+}
+
+arrow::Result<jmethodID> GetMethodID(JNIEnv* env, jclass this_class, const char* name,
+ const char* sig) {
+ jmethodID ret = env->GetMethodID(this_class, name, sig);
+ if (ret == nullptr) {
+ std::string error_message = "Unable to find method " + std::string(name) +
+ " within signature" + std::string(sig);
+ return arrow::Status::Invalid(error_message);
+ }
+ return ret;
+}
+
+arrow::Result<jmethodID> GetStaticMethodID(JNIEnv* env, jclass this_class,
+ const char* name, const char* sig) {
+ jmethodID ret = env->GetStaticMethodID(this_class, name, sig);
+ if (ret == nullptr) {
+ std::string error_message = "Unable to find static method " + std::string(name) +
+ " within signature" + std::string(sig);
+ return arrow::Status::Invalid(error_message);
+ }
+ return ret;
+}
+
+std::shared_ptr<arrow::Schema> SchemaFromColumnNames(
+ const std::shared_ptr<arrow::Schema>& input,
+ const std::vector<std::string>& column_names) {
+ std::vector<std::shared_ptr<arrow::Field>> columns;
+ for (const auto& name : column_names) {
+ columns.push_back(input->GetFieldByName(name));
+ }
+ return std::make_shared<arrow::Schema>(columns);
+}
+
+arrow::Result<std::shared_ptr<arrow::dataset::FileFormat>> GetFileFormat(jint id) {
+ switch (id) {
+ case 0:
+ return std::make_shared<arrow::dataset::ParquetFileFormat>();
+ default:
+ std::string error_message = "illegal file format id: " + std::to_string(id);
+ return arrow::Status::Invalid(error_message);
+ }
+}
+
+std::string JStringToCString(JNIEnv* env, jstring string) {
+ if (string == nullptr) {
+ return std::string();
+ }
+ jboolean copied;
+ const char* chars = env->GetStringUTFChars(string, &copied);
+ std::string ret = strdup(chars);
+ env->ReleaseStringUTFChars(string, chars);
+ return ret;
+}
+
+std::vector<std::string> ToStringVector(JNIEnv* env, jobjectArray& str_array) {
+ int length = env->GetArrayLength(str_array);
+ std::vector<std::string> vector;
+ for (int i = 0; i < length; i++) {
+ auto string = (jstring)(env->GetObjectArrayElement(str_array, i));
Review comment:
Changed to reinterpret_cast. Correct me if I am wrong we cannot use arrow::internal::checked_cast (if you meant to use this one) here since class `_jobject` doesn't include a virtual members for dynamic_cast: https://github.com/AdoptOpenJDK/openjdk-jdk/blob/9a9add8825a040565051a09010b29b099c2e7d49/jdk/src/share/javavm/export/jni.h#L67-L80
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] zhztheplayer commented on a change in pull request #7030: ARROW-7808: [Java][Dataset] Implement Datasets Java API by JNI to C++
Posted by GitBox <gi...@apache.org>.
zhztheplayer commented on a change in pull request #7030:
URL: https://github.com/apache/arrow/pull/7030#discussion_r430839848
##########
File path: cpp/.gitignore
##########
@@ -24,6 +24,8 @@ cmake_install.cmake
build/
*-build/
Testing/
+cmake-build-debug/
Review comment:
Yes added it back.
done
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] zhztheplayer commented on a change in pull request #7030: ARROW-7808: [Java][Dataset] Implement Datasets Java API by JNI to C++
Posted by GitBox <gi...@apache.org>.
zhztheplayer commented on a change in pull request #7030:
URL: https://github.com/apache/arrow/pull/7030#discussion_r416295541
##########
File path: cpp/src/jni/dataset/proto/Types.proto
##########
@@ -0,0 +1,149 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+syntax = "proto2";
+package types;
+
+option java_package = "org.apache.arrow.dataset";
Review comment:
Thanks, as I knew some of the base design purposes of datasets API, dataframe API, and query engine system, so I think I would understand any of the concerns against a too detailed mapping. (actually currently Fragment is not mapped to Java-side so it's not yet a 1-1 mapping) I'll try remove more JNI stuffs.
As for filter, If I understand correctly it's OK to keep Java API but the JNI mapping for filters is considered fragile, right? I can remove the mapping anyway but when users read parquet files from Java they'll not be able to filter row groups to reduce I/O. Which is extremely important when low-selectivity filter is specified.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] zhztheplayer commented on a change in pull request #7030: ARROW-7808: [Java][Dataset] Implement Datasets Java API by JNI to C++
Posted by GitBox <gi...@apache.org>.
zhztheplayer commented on a change in pull request #7030:
URL: https://github.com/apache/arrow/pull/7030#discussion_r415667244
##########
File path: cpp/src/jni/dataset/jni_wrapper.cpp
##########
@@ -0,0 +1,577 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include <arrow/dataset/file_base.h>
+#include <arrow/dataset/api.h>
+#include <arrow/filesystem/localfs.h>
+#include <arrow/ipc/api.h>
+#include <arrow/util/iterator.h>
+#include <arrow/filesystem/hdfs.h>
+#include <arrow/io/api.h>
+#include <google/protobuf/io/coded_stream.h>
+#include <google/protobuf/message.h>
+#include "concurrent_map.h"
+#include "arrow/compute/kernel.h"
+#include "arrow/compute/kernels/cast.h"
+#include "arrow/compute/kernels/compare.h"
+#include "jni/dataset/Types.pb.h"
+
+#include "org_apache_arrow_dataset_jni_JniWrapper.h"
+#include "org_apache_arrow_dataset_file_JniWrapper.h"
+
+static jclass illegal_access_exception_class;
+static jclass illegal_argument_exception_class;
+static jclass runtime_exception_class;
+
+static jclass record_batch_handle_class;
+static jclass record_batch_handle_field_class;
+static jclass record_batch_handle_buffer_class;
+
+static jmethodID record_batch_handle_constructor;
+static jmethodID record_batch_handle_field_constructor;
+static jmethodID record_batch_handle_buffer_constructor;
+
+static jint JNI_VERSION = JNI_VERSION_1_6;
+
+using arrow::jni::ConcurrentMap;
+
+static ConcurrentMap<std::shared_ptr<arrow::dataset::DatasetFactory>> dataset_factory_holder_;
+static ConcurrentMap<std::shared_ptr<arrow::dataset::Dataset>> dataset_holder_;
+static ConcurrentMap<std::shared_ptr<arrow::dataset::ScanTask>> scan_task_holder_;
+static ConcurrentMap<std::shared_ptr<arrow::dataset::Scanner>> scanner_holder_;
+static ConcurrentMap<std::shared_ptr<arrow::RecordBatchIterator>> iterator_holder_;
+static ConcurrentMap<std::shared_ptr<arrow::Buffer>> buffer_holder_;
+
+#define JNI_ASSIGN_OR_THROW_NAME(x, y) ARROW_CONCAT(x, y)
+
+#define JNI_ASSIGN_OR_THROW_IMPL(t, lhs, rexpr) \
+ auto t = (rexpr); \
+ if (!t.status().ok()) { \
+ env->ThrowNew(runtime_exception_class, t.status().message().c_str()); \
+ } \
+ lhs = std::move(t).ValueOrDie();
+
+#define JNI_ASSIGN_OR_THROW(lhs, rexpr) \
+ JNI_ASSIGN_OR_THROW_IMPL(JNI_ASSIGN_OR_THROW_NAME(_tmp_var, __COUNTER__), lhs, rexpr)
+
+#define JNI_ASSERT_OK_OR_THROW(expr) \
+ do { \
+ auto _res = (expr); \
+ arrow::Status _st = ::arrow::internal::GenericToStatus(_res); \
+ if (!_st.ok()) { \
+ env->ThrowNew(runtime_exception_class, _st.message().c_str()); \
+ } \
+ } while (false);
+
+jclass CreateGlobalClassReference(JNIEnv* env, const char* class_name) {
+ jclass local_class = env->FindClass(class_name);
+ jclass global_class = (jclass)env->NewGlobalRef(local_class);
+ env->DeleteLocalRef(local_class);
+ return global_class;
+}
+
+jmethodID GetMethodID(JNIEnv* env, jclass this_class, const char* name, const char* sig) {
+ jmethodID ret = env->GetMethodID(this_class, name, sig);
+ if (ret == nullptr) {
+ std::string error_message = "Unable to find method " + std::string(name) +
+ " within signature" + std::string(sig);
+ env->ThrowNew(illegal_access_exception_class, error_message.c_str());
+ }
+ return ret;
+}
+
+jint JNI_OnLoad(JavaVM* vm, void* reserved) {
+ JNIEnv* env;
+ if (vm->GetEnv(reinterpret_cast<void**>(&env), JNI_VERSION) != JNI_OK) {
+ return JNI_ERR;
+ }
+
+ illegal_access_exception_class =
+ CreateGlobalClassReference(env, "Ljava/lang/IllegalAccessException;");
+ illegal_argument_exception_class =
+ CreateGlobalClassReference(env, "Ljava/lang/IllegalArgumentException;");
+ runtime_exception_class =
+ CreateGlobalClassReference(env, "Ljava/lang/RuntimeException;");
+
+ record_batch_handle_class =
+ CreateGlobalClassReference(env, "Lorg/apache/arrow/dataset/jni/NativeRecordBatchHandle;");
+ record_batch_handle_field_class =
+ CreateGlobalClassReference(env, "Lorg/apache/arrow/dataset/jni/NativeRecordBatchHandle$Field;");
+ record_batch_handle_buffer_class =
+ CreateGlobalClassReference(env, "Lorg/apache/arrow/dataset/jni/NativeRecordBatchHandle$Buffer;");
+
+ record_batch_handle_constructor = GetMethodID(env, record_batch_handle_class, "<init>",
+ "(J[Lorg/apache/arrow/dataset/jni/NativeRecordBatchHandle$Field;"
+ "[Lorg/apache/arrow/dataset/jni/NativeRecordBatchHandle$Buffer;)V");
+ record_batch_handle_field_constructor = GetMethodID(env, record_batch_handle_field_class, "<init>",
+ "(JJ)V");
+ record_batch_handle_buffer_constructor = GetMethodID(env, record_batch_handle_buffer_class, "<init>",
+ "(JJJJ)V");
+
+ env->ExceptionDescribe();
+
+ return JNI_VERSION;
+}
+
+
+void JNI_OnUnload(JavaVM* vm, void* reserved) {
+ JNIEnv* env;
+ vm->GetEnv(reinterpret_cast<void**>(&env), JNI_VERSION);
+ env->DeleteGlobalRef(illegal_access_exception_class);
+ env->DeleteGlobalRef(illegal_argument_exception_class);
+ env->DeleteGlobalRef(runtime_exception_class);
+ env->DeleteGlobalRef(record_batch_handle_class);
+ env->DeleteGlobalRef(record_batch_handle_field_class);
+ env->DeleteGlobalRef(record_batch_handle_buffer_class);
+
+ dataset_factory_holder_.Clear();
+ dataset_holder_.Clear();
+ scan_task_holder_.Clear();
+ scanner_holder_.Clear();
+ iterator_holder_.Clear();
+ buffer_holder_.Clear();
+}
+
+std::shared_ptr<arrow::Schema> SchemaFromColumnNames(
+ const std::shared_ptr<arrow::Schema>& input, const std::vector<std::string>& column_names) {
+ std::vector<std::shared_ptr<arrow::Field>> columns;
+ for (const auto& name : column_names) {
+ columns.push_back(input->GetFieldByName(name));
+ }
+ return std::make_shared<arrow::Schema>(columns);
+}
+
+std::shared_ptr<arrow::dataset::FileFormat> GetFileFormat(JNIEnv *env, jint id) {
+ switch (id) {
+ case 0:
+ return std::make_shared<arrow::dataset::ParquetFileFormat>();
+ default:
+ std::string error_message = "illegal file format id: " + std::to_string(id);
+ env->ThrowNew(illegal_argument_exception_class, error_message.c_str());
+ return nullptr; // unreachable
+ }
+}
+
+std::shared_ptr<arrow::fs::FileSystem> GetFileSystem(JNIEnv *env, jint id, std::string path,
+ std::string* out_path) {
+ switch (id) {
+ case 0:
+ *out_path = path;
+ return std::make_shared<arrow::fs::LocalFileSystem>();
+ case 1: {
+ JNI_ASSIGN_OR_THROW(std::shared_ptr<arrow::fs::FileSystem> ret, arrow::fs::FileSystemFromUri(path, out_path))
+ return ret;
+ }
+ default:std::string error_message = "illegal filesystem id: " + std::to_string(id);
+ env->ThrowNew(illegal_argument_exception_class, error_message.c_str());
+ return nullptr; // unreachable
+ }
+}
+
+std::string JStringToCString(JNIEnv* env, jstring string) {
+ jboolean copied;
+ int32_t length = env->GetStringUTFLength(string);
+ const char *chars = env->GetStringUTFChars(string, &copied);
+ std::string str = std::string(chars, length);
+ // fixme calling ReleaseStringUTFChars if memory leak faced
+ return str;
+}
+
+std::vector<std::string> ToStringVector(JNIEnv* env, jobjectArray& str_array) {
+ int length = env->GetArrayLength(str_array);
+ std::vector<std::string> vector;
+ for (int i = 0; i < length; i++) {
+ auto string = (jstring) (env->GetObjectArrayElement(str_array, i));
+ vector.push_back(JStringToCString(env, string));
+ }
+ return vector;
+}
+
+template <typename T>
+std::vector<T> collect(JNIEnv* env, arrow::Iterator<T> itr) {
+ std::vector<T> vector;
+ while(true) {
+ JNI_ASSIGN_OR_THROW(T t, itr.Next())
+ if (!t) {
+ break;
+ }
+ vector.push_back(t);
+ }
+ return vector;
+}
+
+// FIXME: COPIED FROM intel/master on which this branch is not rebased yet
+// FIXME: https://github.com/Intel-bigdata/arrow/blob/02502a4eb59834c2471dd629e77dbeed19559f68/cpp/src/jni/jni_common.h#L239-L254
+jbyteArray ToSchemaByteArray(JNIEnv* env, std::shared_ptr<arrow::Schema> schema) {
+ JNI_ASSIGN_OR_THROW(std::shared_ptr<arrow::Buffer> buffer,
+ arrow::ipc::SerializeSchema(*schema.get(), nullptr, arrow::default_memory_pool()))
+
+ jbyteArray out = env->NewByteArray(buffer->size());
+ auto src = reinterpret_cast<const jbyte*>(buffer->data());
+ env->SetByteArrayRegion(out, 0, buffer->size(), src);
+ return out;
+}
+
+// FIXME: COPIED FROM intel/master on which this branch is not rebased yet
+// FIXME: https://github.com/Intel-bigdata/arrow/blob/02502a4eb59834c2471dd629e77dbeed19559f68/cpp/src/jni/jni_common.h#L256-L272
+std::shared_ptr<arrow::Schema> FromSchemaByteArray(JNIEnv* env, jbyteArray schemaBytes) {
+ arrow::ipc::DictionaryMemo in_memo;
+ int schemaBytes_len = env->GetArrayLength(schemaBytes);
+ jbyte* schemaBytes_data = env->GetByteArrayElements(schemaBytes, 0);
+ auto serialized_schema =
+ std::make_shared<arrow::Buffer>((uint8_t*)schemaBytes_data, schemaBytes_len);
+ arrow::io::BufferReader buf_reader(serialized_schema);
+ JNI_ASSIGN_OR_THROW(std::shared_ptr<arrow::Schema> schema, arrow::ipc::ReadSchema(&buf_reader, &in_memo))
+ env->ReleaseByteArrayElements(schemaBytes, schemaBytes_data, JNI_ABORT);
+ return schema;
+}
+
+bool ParseProtobuf(uint8_t* buf, int bufLen, google::protobuf::Message* msg) {
+ google::protobuf::io::CodedInputStream cis(buf, bufLen);
+ cis.SetRecursionLimit(1000);
+ return msg->ParseFromCodedStream(&cis);
+}
+
+void releaseFilterInput(jbyteArray condition_arr, jbyte* condition_bytes, JNIEnv* env) {
+ env->ReleaseByteArrayElements(condition_arr, condition_bytes, JNI_ABORT);
+}
+
+// fixme in development. Not all node types considered.
+std::shared_ptr<arrow::dataset::Expression> translateNode(types::TreeNode node, JNIEnv* env) {
+ if (node.has_fieldnode()) {
+ const types::FieldNode& f_node = node.fieldnode();
+ const std::string& name = f_node.name();
+ return std::make_shared<arrow::dataset::FieldExpression>(name);
+ }
+ if (node.has_intnode()) {
+ const types::IntNode& int_node = node.intnode();
+ int32_t val = int_node.value();
+ return std::make_shared<arrow::dataset::ScalarExpression>(std::make_shared<arrow::Int32Scalar>(val));
+ }
+ if (node.has_longnode()) {
+ const types::LongNode& long_node = node.longnode();
+ int64_t val = long_node.value();
+ return std::make_shared<arrow::dataset::ScalarExpression>(std::make_shared<arrow::Int64Scalar>(val));
+ }
+ if (node.has_floatnode()) {
+ const types::FloatNode& float_node = node.floatnode();
+ float_t val = float_node.value();
+ return std::make_shared<arrow::dataset::ScalarExpression>(std::make_shared<arrow::FloatScalar>(val));
+ }
+ if (node.has_doublenode()) {
+ const types::DoubleNode& double_node = node.doublenode();
+ double_t val = double_node.value();
+ return std::make_shared<arrow::dataset::ScalarExpression>(std::make_shared<arrow::DoubleScalar>(val));
+ }
+ if (node.has_booleannode()) {
+ const types::BooleanNode& boolean_node = node.booleannode();
+ bool val = boolean_node.value();
+ return std::make_shared<arrow::dataset::ScalarExpression>(std::make_shared<arrow::BooleanScalar>(val));
+ }
+ if (node.has_andnode()) {
+ const types::AndNode& and_node = node.andnode();
+ const types::TreeNode& left_arg = and_node.leftarg();
+ const types::TreeNode& right_arg = and_node.rightarg();
+ return std::make_shared<arrow::dataset::AndExpression>(translateNode(left_arg, env), translateNode(right_arg, env));
+ }
+ if (node.has_ornode()) {
+ const types::OrNode& or_node = node.ornode();
+ const types::TreeNode& left_arg = or_node.leftarg();
+ const types::TreeNode& right_arg = or_node.rightarg();
+ return std::make_shared<arrow::dataset::OrExpression>(translateNode(left_arg, env), translateNode(right_arg, env));
+ }
+ if (node.has_cpnode()) {
+ const types::ComparisonNode& cp_node = node.cpnode();
+ const std::string& op_name = cp_node.opname();
+ arrow::compute::CompareOperator op;
+ if (op_name == "equal") {
+ op = arrow::compute::CompareOperator::EQUAL;
+ } else if (op_name == "greaterThan") {
+ op = arrow::compute::CompareOperator::GREATER;
+ } else if (op_name == "greaterThanOrEqual") {
+ op = arrow::compute::CompareOperator::GREATER_EQUAL;
+ } else if (op_name == "lessThan") {
+ op = arrow::compute::CompareOperator::LESS;
+ } else if (op_name == "lessThanOrEqual") {
+ op = arrow::compute::CompareOperator::LESS_EQUAL;
+ } else {
+ std::string error_message = "Unknown operation name in comparison node";
+ env->ThrowNew(illegal_argument_exception_class, error_message.c_str());
+ return nullptr; // unreachable
+ }
+ const types::TreeNode& left_arg = cp_node.leftarg();
+ const types::TreeNode& right_arg = cp_node.rightarg();
+ return std::make_shared<arrow::dataset::ComparisonExpression>(op,
+ translateNode(left_arg, env), translateNode(right_arg, env));
+ }
+ if (node.has_notnode()) {
+ const types::NotNode& not_node = node.notnode();
+ const ::types::TreeNode& child = not_node.args();
+ std::shared_ptr<arrow::dataset::Expression> translatedChild = translateNode(child, env);
+ return std::make_shared<arrow::dataset::NotExpression>(translatedChild);
+ }
+ if (node.has_isvalidnode()) {
+ const types::IsValidNode& is_valid_node = node.isvalidnode();
+ const ::types::TreeNode& child = is_valid_node.args();
+ std::shared_ptr<arrow::dataset::Expression> translatedChild = translateNode(child, env);
+ return std::make_shared<arrow::dataset::IsValidExpression>(translatedChild);
+ }
+ std::string error_message = "Unknown node type";
+ env->ThrowNew(illegal_argument_exception_class, error_message.c_str());
+ return nullptr; // unreachable
+}
+
+std::shared_ptr<arrow::dataset::Expression> translateFilter(types::Condition condition, JNIEnv* env) {
Review comment:
done but I'll remove the entire function
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] zhztheplayer commented on a change in pull request #7030: ARROW-7808: [Java][Dataset] Implement Datasets Java API by JNI to C++
Posted by GitBox <gi...@apache.org>.
zhztheplayer commented on a change in pull request #7030:
URL: https://github.com/apache/arrow/pull/7030#discussion_r415667244
##########
File path: cpp/src/jni/dataset/jni_wrapper.cpp
##########
@@ -0,0 +1,577 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include <arrow/dataset/file_base.h>
+#include <arrow/dataset/api.h>
+#include <arrow/filesystem/localfs.h>
+#include <arrow/ipc/api.h>
+#include <arrow/util/iterator.h>
+#include <arrow/filesystem/hdfs.h>
+#include <arrow/io/api.h>
+#include <google/protobuf/io/coded_stream.h>
+#include <google/protobuf/message.h>
+#include "concurrent_map.h"
+#include "arrow/compute/kernel.h"
+#include "arrow/compute/kernels/cast.h"
+#include "arrow/compute/kernels/compare.h"
+#include "jni/dataset/Types.pb.h"
+
+#include "org_apache_arrow_dataset_jni_JniWrapper.h"
+#include "org_apache_arrow_dataset_file_JniWrapper.h"
+
+static jclass illegal_access_exception_class;
+static jclass illegal_argument_exception_class;
+static jclass runtime_exception_class;
+
+static jclass record_batch_handle_class;
+static jclass record_batch_handle_field_class;
+static jclass record_batch_handle_buffer_class;
+
+static jmethodID record_batch_handle_constructor;
+static jmethodID record_batch_handle_field_constructor;
+static jmethodID record_batch_handle_buffer_constructor;
+
+static jint JNI_VERSION = JNI_VERSION_1_6;
+
+using arrow::jni::ConcurrentMap;
+
+static ConcurrentMap<std::shared_ptr<arrow::dataset::DatasetFactory>> dataset_factory_holder_;
+static ConcurrentMap<std::shared_ptr<arrow::dataset::Dataset>> dataset_holder_;
+static ConcurrentMap<std::shared_ptr<arrow::dataset::ScanTask>> scan_task_holder_;
+static ConcurrentMap<std::shared_ptr<arrow::dataset::Scanner>> scanner_holder_;
+static ConcurrentMap<std::shared_ptr<arrow::RecordBatchIterator>> iterator_holder_;
+static ConcurrentMap<std::shared_ptr<arrow::Buffer>> buffer_holder_;
+
+#define JNI_ASSIGN_OR_THROW_NAME(x, y) ARROW_CONCAT(x, y)
+
+#define JNI_ASSIGN_OR_THROW_IMPL(t, lhs, rexpr) \
+ auto t = (rexpr); \
+ if (!t.status().ok()) { \
+ env->ThrowNew(runtime_exception_class, t.status().message().c_str()); \
+ } \
+ lhs = std::move(t).ValueOrDie();
+
+#define JNI_ASSIGN_OR_THROW(lhs, rexpr) \
+ JNI_ASSIGN_OR_THROW_IMPL(JNI_ASSIGN_OR_THROW_NAME(_tmp_var, __COUNTER__), lhs, rexpr)
+
+#define JNI_ASSERT_OK_OR_THROW(expr) \
+ do { \
+ auto _res = (expr); \
+ arrow::Status _st = ::arrow::internal::GenericToStatus(_res); \
+ if (!_st.ok()) { \
+ env->ThrowNew(runtime_exception_class, _st.message().c_str()); \
+ } \
+ } while (false);
+
+jclass CreateGlobalClassReference(JNIEnv* env, const char* class_name) {
+ jclass local_class = env->FindClass(class_name);
+ jclass global_class = (jclass)env->NewGlobalRef(local_class);
+ env->DeleteLocalRef(local_class);
+ return global_class;
+}
+
+jmethodID GetMethodID(JNIEnv* env, jclass this_class, const char* name, const char* sig) {
+ jmethodID ret = env->GetMethodID(this_class, name, sig);
+ if (ret == nullptr) {
+ std::string error_message = "Unable to find method " + std::string(name) +
+ " within signature" + std::string(sig);
+ env->ThrowNew(illegal_access_exception_class, error_message.c_str());
+ }
+ return ret;
+}
+
+jint JNI_OnLoad(JavaVM* vm, void* reserved) {
+ JNIEnv* env;
+ if (vm->GetEnv(reinterpret_cast<void**>(&env), JNI_VERSION) != JNI_OK) {
+ return JNI_ERR;
+ }
+
+ illegal_access_exception_class =
+ CreateGlobalClassReference(env, "Ljava/lang/IllegalAccessException;");
+ illegal_argument_exception_class =
+ CreateGlobalClassReference(env, "Ljava/lang/IllegalArgumentException;");
+ runtime_exception_class =
+ CreateGlobalClassReference(env, "Ljava/lang/RuntimeException;");
+
+ record_batch_handle_class =
+ CreateGlobalClassReference(env, "Lorg/apache/arrow/dataset/jni/NativeRecordBatchHandle;");
+ record_batch_handle_field_class =
+ CreateGlobalClassReference(env, "Lorg/apache/arrow/dataset/jni/NativeRecordBatchHandle$Field;");
+ record_batch_handle_buffer_class =
+ CreateGlobalClassReference(env, "Lorg/apache/arrow/dataset/jni/NativeRecordBatchHandle$Buffer;");
+
+ record_batch_handle_constructor = GetMethodID(env, record_batch_handle_class, "<init>",
+ "(J[Lorg/apache/arrow/dataset/jni/NativeRecordBatchHandle$Field;"
+ "[Lorg/apache/arrow/dataset/jni/NativeRecordBatchHandle$Buffer;)V");
+ record_batch_handle_field_constructor = GetMethodID(env, record_batch_handle_field_class, "<init>",
+ "(JJ)V");
+ record_batch_handle_buffer_constructor = GetMethodID(env, record_batch_handle_buffer_class, "<init>",
+ "(JJJJ)V");
+
+ env->ExceptionDescribe();
+
+ return JNI_VERSION;
+}
+
+
+void JNI_OnUnload(JavaVM* vm, void* reserved) {
+ JNIEnv* env;
+ vm->GetEnv(reinterpret_cast<void**>(&env), JNI_VERSION);
+ env->DeleteGlobalRef(illegal_access_exception_class);
+ env->DeleteGlobalRef(illegal_argument_exception_class);
+ env->DeleteGlobalRef(runtime_exception_class);
+ env->DeleteGlobalRef(record_batch_handle_class);
+ env->DeleteGlobalRef(record_batch_handle_field_class);
+ env->DeleteGlobalRef(record_batch_handle_buffer_class);
+
+ dataset_factory_holder_.Clear();
+ dataset_holder_.Clear();
+ scan_task_holder_.Clear();
+ scanner_holder_.Clear();
+ iterator_holder_.Clear();
+ buffer_holder_.Clear();
+}
+
+std::shared_ptr<arrow::Schema> SchemaFromColumnNames(
+ const std::shared_ptr<arrow::Schema>& input, const std::vector<std::string>& column_names) {
+ std::vector<std::shared_ptr<arrow::Field>> columns;
+ for (const auto& name : column_names) {
+ columns.push_back(input->GetFieldByName(name));
+ }
+ return std::make_shared<arrow::Schema>(columns);
+}
+
+std::shared_ptr<arrow::dataset::FileFormat> GetFileFormat(JNIEnv *env, jint id) {
+ switch (id) {
+ case 0:
+ return std::make_shared<arrow::dataset::ParquetFileFormat>();
+ default:
+ std::string error_message = "illegal file format id: " + std::to_string(id);
+ env->ThrowNew(illegal_argument_exception_class, error_message.c_str());
+ return nullptr; // unreachable
+ }
+}
+
+std::shared_ptr<arrow::fs::FileSystem> GetFileSystem(JNIEnv *env, jint id, std::string path,
+ std::string* out_path) {
+ switch (id) {
+ case 0:
+ *out_path = path;
+ return std::make_shared<arrow::fs::LocalFileSystem>();
+ case 1: {
+ JNI_ASSIGN_OR_THROW(std::shared_ptr<arrow::fs::FileSystem> ret, arrow::fs::FileSystemFromUri(path, out_path))
+ return ret;
+ }
+ default:std::string error_message = "illegal filesystem id: " + std::to_string(id);
+ env->ThrowNew(illegal_argument_exception_class, error_message.c_str());
+ return nullptr; // unreachable
+ }
+}
+
+std::string JStringToCString(JNIEnv* env, jstring string) {
+ jboolean copied;
+ int32_t length = env->GetStringUTFLength(string);
+ const char *chars = env->GetStringUTFChars(string, &copied);
+ std::string str = std::string(chars, length);
+ // fixme calling ReleaseStringUTFChars if memory leak faced
+ return str;
+}
+
+std::vector<std::string> ToStringVector(JNIEnv* env, jobjectArray& str_array) {
+ int length = env->GetArrayLength(str_array);
+ std::vector<std::string> vector;
+ for (int i = 0; i < length; i++) {
+ auto string = (jstring) (env->GetObjectArrayElement(str_array, i));
+ vector.push_back(JStringToCString(env, string));
+ }
+ return vector;
+}
+
+template <typename T>
+std::vector<T> collect(JNIEnv* env, arrow::Iterator<T> itr) {
+ std::vector<T> vector;
+ while(true) {
+ JNI_ASSIGN_OR_THROW(T t, itr.Next())
+ if (!t) {
+ break;
+ }
+ vector.push_back(t);
+ }
+ return vector;
+}
+
+// FIXME: COPIED FROM intel/master on which this branch is not rebased yet
+// FIXME: https://github.com/Intel-bigdata/arrow/blob/02502a4eb59834c2471dd629e77dbeed19559f68/cpp/src/jni/jni_common.h#L239-L254
+jbyteArray ToSchemaByteArray(JNIEnv* env, std::shared_ptr<arrow::Schema> schema) {
+ JNI_ASSIGN_OR_THROW(std::shared_ptr<arrow::Buffer> buffer,
+ arrow::ipc::SerializeSchema(*schema.get(), nullptr, arrow::default_memory_pool()))
+
+ jbyteArray out = env->NewByteArray(buffer->size());
+ auto src = reinterpret_cast<const jbyte*>(buffer->data());
+ env->SetByteArrayRegion(out, 0, buffer->size(), src);
+ return out;
+}
+
+// FIXME: COPIED FROM intel/master on which this branch is not rebased yet
+// FIXME: https://github.com/Intel-bigdata/arrow/blob/02502a4eb59834c2471dd629e77dbeed19559f68/cpp/src/jni/jni_common.h#L256-L272
+std::shared_ptr<arrow::Schema> FromSchemaByteArray(JNIEnv* env, jbyteArray schemaBytes) {
+ arrow::ipc::DictionaryMemo in_memo;
+ int schemaBytes_len = env->GetArrayLength(schemaBytes);
+ jbyte* schemaBytes_data = env->GetByteArrayElements(schemaBytes, 0);
+ auto serialized_schema =
+ std::make_shared<arrow::Buffer>((uint8_t*)schemaBytes_data, schemaBytes_len);
+ arrow::io::BufferReader buf_reader(serialized_schema);
+ JNI_ASSIGN_OR_THROW(std::shared_ptr<arrow::Schema> schema, arrow::ipc::ReadSchema(&buf_reader, &in_memo))
+ env->ReleaseByteArrayElements(schemaBytes, schemaBytes_data, JNI_ABORT);
+ return schema;
+}
+
+bool ParseProtobuf(uint8_t* buf, int bufLen, google::protobuf::Message* msg) {
+ google::protobuf::io::CodedInputStream cis(buf, bufLen);
+ cis.SetRecursionLimit(1000);
+ return msg->ParseFromCodedStream(&cis);
+}
+
+void releaseFilterInput(jbyteArray condition_arr, jbyte* condition_bytes, JNIEnv* env) {
+ env->ReleaseByteArrayElements(condition_arr, condition_bytes, JNI_ABORT);
+}
+
+// fixme in development. Not all node types considered.
+std::shared_ptr<arrow::dataset::Expression> translateNode(types::TreeNode node, JNIEnv* env) {
+ if (node.has_fieldnode()) {
+ const types::FieldNode& f_node = node.fieldnode();
+ const std::string& name = f_node.name();
+ return std::make_shared<arrow::dataset::FieldExpression>(name);
+ }
+ if (node.has_intnode()) {
+ const types::IntNode& int_node = node.intnode();
+ int32_t val = int_node.value();
+ return std::make_shared<arrow::dataset::ScalarExpression>(std::make_shared<arrow::Int32Scalar>(val));
+ }
+ if (node.has_longnode()) {
+ const types::LongNode& long_node = node.longnode();
+ int64_t val = long_node.value();
+ return std::make_shared<arrow::dataset::ScalarExpression>(std::make_shared<arrow::Int64Scalar>(val));
+ }
+ if (node.has_floatnode()) {
+ const types::FloatNode& float_node = node.floatnode();
+ float_t val = float_node.value();
+ return std::make_shared<arrow::dataset::ScalarExpression>(std::make_shared<arrow::FloatScalar>(val));
+ }
+ if (node.has_doublenode()) {
+ const types::DoubleNode& double_node = node.doublenode();
+ double_t val = double_node.value();
+ return std::make_shared<arrow::dataset::ScalarExpression>(std::make_shared<arrow::DoubleScalar>(val));
+ }
+ if (node.has_booleannode()) {
+ const types::BooleanNode& boolean_node = node.booleannode();
+ bool val = boolean_node.value();
+ return std::make_shared<arrow::dataset::ScalarExpression>(std::make_shared<arrow::BooleanScalar>(val));
+ }
+ if (node.has_andnode()) {
+ const types::AndNode& and_node = node.andnode();
+ const types::TreeNode& left_arg = and_node.leftarg();
+ const types::TreeNode& right_arg = and_node.rightarg();
+ return std::make_shared<arrow::dataset::AndExpression>(translateNode(left_arg, env), translateNode(right_arg, env));
+ }
+ if (node.has_ornode()) {
+ const types::OrNode& or_node = node.ornode();
+ const types::TreeNode& left_arg = or_node.leftarg();
+ const types::TreeNode& right_arg = or_node.rightarg();
+ return std::make_shared<arrow::dataset::OrExpression>(translateNode(left_arg, env), translateNode(right_arg, env));
+ }
+ if (node.has_cpnode()) {
+ const types::ComparisonNode& cp_node = node.cpnode();
+ const std::string& op_name = cp_node.opname();
+ arrow::compute::CompareOperator op;
+ if (op_name == "equal") {
+ op = arrow::compute::CompareOperator::EQUAL;
+ } else if (op_name == "greaterThan") {
+ op = arrow::compute::CompareOperator::GREATER;
+ } else if (op_name == "greaterThanOrEqual") {
+ op = arrow::compute::CompareOperator::GREATER_EQUAL;
+ } else if (op_name == "lessThan") {
+ op = arrow::compute::CompareOperator::LESS;
+ } else if (op_name == "lessThanOrEqual") {
+ op = arrow::compute::CompareOperator::LESS_EQUAL;
+ } else {
+ std::string error_message = "Unknown operation name in comparison node";
+ env->ThrowNew(illegal_argument_exception_class, error_message.c_str());
+ return nullptr; // unreachable
+ }
+ const types::TreeNode& left_arg = cp_node.leftarg();
+ const types::TreeNode& right_arg = cp_node.rightarg();
+ return std::make_shared<arrow::dataset::ComparisonExpression>(op,
+ translateNode(left_arg, env), translateNode(right_arg, env));
+ }
+ if (node.has_notnode()) {
+ const types::NotNode& not_node = node.notnode();
+ const ::types::TreeNode& child = not_node.args();
+ std::shared_ptr<arrow::dataset::Expression> translatedChild = translateNode(child, env);
+ return std::make_shared<arrow::dataset::NotExpression>(translatedChild);
+ }
+ if (node.has_isvalidnode()) {
+ const types::IsValidNode& is_valid_node = node.isvalidnode();
+ const ::types::TreeNode& child = is_valid_node.args();
+ std::shared_ptr<arrow::dataset::Expression> translatedChild = translateNode(child, env);
+ return std::make_shared<arrow::dataset::IsValidExpression>(translatedChild);
+ }
+ std::string error_message = "Unknown node type";
+ env->ThrowNew(illegal_argument_exception_class, error_message.c_str());
+ return nullptr; // unreachable
+}
+
+std::shared_ptr<arrow::dataset::Expression> translateFilter(types::Condition condition, JNIEnv* env) {
Review comment:
~done but I'll remove the entire function~
##########
File path: cpp/src/jni/dataset/jni_wrapper.cpp
##########
@@ -0,0 +1,577 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include <arrow/dataset/file_base.h>
+#include <arrow/dataset/api.h>
+#include <arrow/filesystem/localfs.h>
+#include <arrow/ipc/api.h>
+#include <arrow/util/iterator.h>
+#include <arrow/filesystem/hdfs.h>
+#include <arrow/io/api.h>
+#include <google/protobuf/io/coded_stream.h>
+#include <google/protobuf/message.h>
+#include "concurrent_map.h"
+#include "arrow/compute/kernel.h"
+#include "arrow/compute/kernels/cast.h"
+#include "arrow/compute/kernels/compare.h"
+#include "jni/dataset/Types.pb.h"
+
+#include "org_apache_arrow_dataset_jni_JniWrapper.h"
+#include "org_apache_arrow_dataset_file_JniWrapper.h"
+
+static jclass illegal_access_exception_class;
+static jclass illegal_argument_exception_class;
+static jclass runtime_exception_class;
+
+static jclass record_batch_handle_class;
+static jclass record_batch_handle_field_class;
+static jclass record_batch_handle_buffer_class;
+
+static jmethodID record_batch_handle_constructor;
+static jmethodID record_batch_handle_field_constructor;
+static jmethodID record_batch_handle_buffer_constructor;
+
+static jint JNI_VERSION = JNI_VERSION_1_6;
+
+using arrow::jni::ConcurrentMap;
+
+static ConcurrentMap<std::shared_ptr<arrow::dataset::DatasetFactory>> dataset_factory_holder_;
+static ConcurrentMap<std::shared_ptr<arrow::dataset::Dataset>> dataset_holder_;
+static ConcurrentMap<std::shared_ptr<arrow::dataset::ScanTask>> scan_task_holder_;
+static ConcurrentMap<std::shared_ptr<arrow::dataset::Scanner>> scanner_holder_;
+static ConcurrentMap<std::shared_ptr<arrow::RecordBatchIterator>> iterator_holder_;
+static ConcurrentMap<std::shared_ptr<arrow::Buffer>> buffer_holder_;
+
+#define JNI_ASSIGN_OR_THROW_NAME(x, y) ARROW_CONCAT(x, y)
+
+#define JNI_ASSIGN_OR_THROW_IMPL(t, lhs, rexpr) \
+ auto t = (rexpr); \
+ if (!t.status().ok()) { \
+ env->ThrowNew(runtime_exception_class, t.status().message().c_str()); \
+ } \
+ lhs = std::move(t).ValueOrDie();
+
+#define JNI_ASSIGN_OR_THROW(lhs, rexpr) \
+ JNI_ASSIGN_OR_THROW_IMPL(JNI_ASSIGN_OR_THROW_NAME(_tmp_var, __COUNTER__), lhs, rexpr)
+
+#define JNI_ASSERT_OK_OR_THROW(expr) \
+ do { \
+ auto _res = (expr); \
+ arrow::Status _st = ::arrow::internal::GenericToStatus(_res); \
+ if (!_st.ok()) { \
+ env->ThrowNew(runtime_exception_class, _st.message().c_str()); \
+ } \
+ } while (false);
+
+jclass CreateGlobalClassReference(JNIEnv* env, const char* class_name) {
+ jclass local_class = env->FindClass(class_name);
+ jclass global_class = (jclass)env->NewGlobalRef(local_class);
+ env->DeleteLocalRef(local_class);
+ return global_class;
+}
+
+jmethodID GetMethodID(JNIEnv* env, jclass this_class, const char* name, const char* sig) {
+ jmethodID ret = env->GetMethodID(this_class, name, sig);
+ if (ret == nullptr) {
+ std::string error_message = "Unable to find method " + std::string(name) +
+ " within signature" + std::string(sig);
+ env->ThrowNew(illegal_access_exception_class, error_message.c_str());
+ }
+ return ret;
+}
+
+jint JNI_OnLoad(JavaVM* vm, void* reserved) {
+ JNIEnv* env;
+ if (vm->GetEnv(reinterpret_cast<void**>(&env), JNI_VERSION) != JNI_OK) {
+ return JNI_ERR;
+ }
+
+ illegal_access_exception_class =
+ CreateGlobalClassReference(env, "Ljava/lang/IllegalAccessException;");
+ illegal_argument_exception_class =
+ CreateGlobalClassReference(env, "Ljava/lang/IllegalArgumentException;");
+ runtime_exception_class =
+ CreateGlobalClassReference(env, "Ljava/lang/RuntimeException;");
+
+ record_batch_handle_class =
+ CreateGlobalClassReference(env, "Lorg/apache/arrow/dataset/jni/NativeRecordBatchHandle;");
+ record_batch_handle_field_class =
+ CreateGlobalClassReference(env, "Lorg/apache/arrow/dataset/jni/NativeRecordBatchHandle$Field;");
+ record_batch_handle_buffer_class =
+ CreateGlobalClassReference(env, "Lorg/apache/arrow/dataset/jni/NativeRecordBatchHandle$Buffer;");
+
+ record_batch_handle_constructor = GetMethodID(env, record_batch_handle_class, "<init>",
+ "(J[Lorg/apache/arrow/dataset/jni/NativeRecordBatchHandle$Field;"
+ "[Lorg/apache/arrow/dataset/jni/NativeRecordBatchHandle$Buffer;)V");
+ record_batch_handle_field_constructor = GetMethodID(env, record_batch_handle_field_class, "<init>",
+ "(JJ)V");
+ record_batch_handle_buffer_constructor = GetMethodID(env, record_batch_handle_buffer_class, "<init>",
+ "(JJJJ)V");
+
+ env->ExceptionDescribe();
+
+ return JNI_VERSION;
+}
+
+
+void JNI_OnUnload(JavaVM* vm, void* reserved) {
+ JNIEnv* env;
+ vm->GetEnv(reinterpret_cast<void**>(&env), JNI_VERSION);
+ env->DeleteGlobalRef(illegal_access_exception_class);
+ env->DeleteGlobalRef(illegal_argument_exception_class);
+ env->DeleteGlobalRef(runtime_exception_class);
+ env->DeleteGlobalRef(record_batch_handle_class);
+ env->DeleteGlobalRef(record_batch_handle_field_class);
+ env->DeleteGlobalRef(record_batch_handle_buffer_class);
+
+ dataset_factory_holder_.Clear();
+ dataset_holder_.Clear();
+ scan_task_holder_.Clear();
+ scanner_holder_.Clear();
+ iterator_holder_.Clear();
+ buffer_holder_.Clear();
+}
+
+std::shared_ptr<arrow::Schema> SchemaFromColumnNames(
+ const std::shared_ptr<arrow::Schema>& input, const std::vector<std::string>& column_names) {
+ std::vector<std::shared_ptr<arrow::Field>> columns;
+ for (const auto& name : column_names) {
+ columns.push_back(input->GetFieldByName(name));
+ }
+ return std::make_shared<arrow::Schema>(columns);
+}
+
+std::shared_ptr<arrow::dataset::FileFormat> GetFileFormat(JNIEnv *env, jint id) {
+ switch (id) {
+ case 0:
+ return std::make_shared<arrow::dataset::ParquetFileFormat>();
+ default:
+ std::string error_message = "illegal file format id: " + std::to_string(id);
+ env->ThrowNew(illegal_argument_exception_class, error_message.c_str());
+ return nullptr; // unreachable
+ }
+}
+
+std::shared_ptr<arrow::fs::FileSystem> GetFileSystem(JNIEnv *env, jint id, std::string path,
+ std::string* out_path) {
+ switch (id) {
+ case 0:
+ *out_path = path;
+ return std::make_shared<arrow::fs::LocalFileSystem>();
+ case 1: {
+ JNI_ASSIGN_OR_THROW(std::shared_ptr<arrow::fs::FileSystem> ret, arrow::fs::FileSystemFromUri(path, out_path))
+ return ret;
+ }
+ default:std::string error_message = "illegal filesystem id: " + std::to_string(id);
+ env->ThrowNew(illegal_argument_exception_class, error_message.c_str());
+ return nullptr; // unreachable
+ }
+}
+
+std::string JStringToCString(JNIEnv* env, jstring string) {
+ jboolean copied;
+ int32_t length = env->GetStringUTFLength(string);
+ const char *chars = env->GetStringUTFChars(string, &copied);
+ std::string str = std::string(chars, length);
+ // fixme calling ReleaseStringUTFChars if memory leak faced
+ return str;
+}
+
+std::vector<std::string> ToStringVector(JNIEnv* env, jobjectArray& str_array) {
+ int length = env->GetArrayLength(str_array);
+ std::vector<std::string> vector;
+ for (int i = 0; i < length; i++) {
+ auto string = (jstring) (env->GetObjectArrayElement(str_array, i));
+ vector.push_back(JStringToCString(env, string));
+ }
+ return vector;
+}
+
+template <typename T>
+std::vector<T> collect(JNIEnv* env, arrow::Iterator<T> itr) {
+ std::vector<T> vector;
+ while(true) {
+ JNI_ASSIGN_OR_THROW(T t, itr.Next())
+ if (!t) {
+ break;
+ }
+ vector.push_back(t);
+ }
+ return vector;
+}
+
+// FIXME: COPIED FROM intel/master on which this branch is not rebased yet
+// FIXME: https://github.com/Intel-bigdata/arrow/blob/02502a4eb59834c2471dd629e77dbeed19559f68/cpp/src/jni/jni_common.h#L239-L254
+jbyteArray ToSchemaByteArray(JNIEnv* env, std::shared_ptr<arrow::Schema> schema) {
+ JNI_ASSIGN_OR_THROW(std::shared_ptr<arrow::Buffer> buffer,
+ arrow::ipc::SerializeSchema(*schema.get(), nullptr, arrow::default_memory_pool()))
+
+ jbyteArray out = env->NewByteArray(buffer->size());
+ auto src = reinterpret_cast<const jbyte*>(buffer->data());
+ env->SetByteArrayRegion(out, 0, buffer->size(), src);
+ return out;
+}
+
+// FIXME: COPIED FROM intel/master on which this branch is not rebased yet
+// FIXME: https://github.com/Intel-bigdata/arrow/blob/02502a4eb59834c2471dd629e77dbeed19559f68/cpp/src/jni/jni_common.h#L256-L272
+std::shared_ptr<arrow::Schema> FromSchemaByteArray(JNIEnv* env, jbyteArray schemaBytes) {
+ arrow::ipc::DictionaryMemo in_memo;
+ int schemaBytes_len = env->GetArrayLength(schemaBytes);
+ jbyte* schemaBytes_data = env->GetByteArrayElements(schemaBytes, 0);
+ auto serialized_schema =
+ std::make_shared<arrow::Buffer>((uint8_t*)schemaBytes_data, schemaBytes_len);
+ arrow::io::BufferReader buf_reader(serialized_schema);
+ JNI_ASSIGN_OR_THROW(std::shared_ptr<arrow::Schema> schema, arrow::ipc::ReadSchema(&buf_reader, &in_memo))
+ env->ReleaseByteArrayElements(schemaBytes, schemaBytes_data, JNI_ABORT);
+ return schema;
+}
+
+bool ParseProtobuf(uint8_t* buf, int bufLen, google::protobuf::Message* msg) {
+ google::protobuf::io::CodedInputStream cis(buf, bufLen);
+ cis.SetRecursionLimit(1000);
+ return msg->ParseFromCodedStream(&cis);
+}
+
+void releaseFilterInput(jbyteArray condition_arr, jbyte* condition_bytes, JNIEnv* env) {
+ env->ReleaseByteArrayElements(condition_arr, condition_bytes, JNI_ABORT);
+}
+
+// fixme in development. Not all node types considered.
+std::shared_ptr<arrow::dataset::Expression> translateNode(types::TreeNode node, JNIEnv* env) {
+ if (node.has_fieldnode()) {
+ const types::FieldNode& f_node = node.fieldnode();
+ const std::string& name = f_node.name();
+ return std::make_shared<arrow::dataset::FieldExpression>(name);
+ }
+ if (node.has_intnode()) {
+ const types::IntNode& int_node = node.intnode();
+ int32_t val = int_node.value();
+ return std::make_shared<arrow::dataset::ScalarExpression>(std::make_shared<arrow::Int32Scalar>(val));
+ }
+ if (node.has_longnode()) {
+ const types::LongNode& long_node = node.longnode();
+ int64_t val = long_node.value();
+ return std::make_shared<arrow::dataset::ScalarExpression>(std::make_shared<arrow::Int64Scalar>(val));
+ }
+ if (node.has_floatnode()) {
+ const types::FloatNode& float_node = node.floatnode();
+ float_t val = float_node.value();
+ return std::make_shared<arrow::dataset::ScalarExpression>(std::make_shared<arrow::FloatScalar>(val));
+ }
+ if (node.has_doublenode()) {
+ const types::DoubleNode& double_node = node.doublenode();
+ double_t val = double_node.value();
+ return std::make_shared<arrow::dataset::ScalarExpression>(std::make_shared<arrow::DoubleScalar>(val));
+ }
+ if (node.has_booleannode()) {
+ const types::BooleanNode& boolean_node = node.booleannode();
+ bool val = boolean_node.value();
+ return std::make_shared<arrow::dataset::ScalarExpression>(std::make_shared<arrow::BooleanScalar>(val));
+ }
+ if (node.has_andnode()) {
+ const types::AndNode& and_node = node.andnode();
+ const types::TreeNode& left_arg = and_node.leftarg();
+ const types::TreeNode& right_arg = and_node.rightarg();
+ return std::make_shared<arrow::dataset::AndExpression>(translateNode(left_arg, env), translateNode(right_arg, env));
+ }
+ if (node.has_ornode()) {
+ const types::OrNode& or_node = node.ornode();
+ const types::TreeNode& left_arg = or_node.leftarg();
+ const types::TreeNode& right_arg = or_node.rightarg();
+ return std::make_shared<arrow::dataset::OrExpression>(translateNode(left_arg, env), translateNode(right_arg, env));
+ }
+ if (node.has_cpnode()) {
+ const types::ComparisonNode& cp_node = node.cpnode();
+ const std::string& op_name = cp_node.opname();
+ arrow::compute::CompareOperator op;
+ if (op_name == "equal") {
+ op = arrow::compute::CompareOperator::EQUAL;
+ } else if (op_name == "greaterThan") {
+ op = arrow::compute::CompareOperator::GREATER;
+ } else if (op_name == "greaterThanOrEqual") {
+ op = arrow::compute::CompareOperator::GREATER_EQUAL;
+ } else if (op_name == "lessThan") {
+ op = arrow::compute::CompareOperator::LESS;
+ } else if (op_name == "lessThanOrEqual") {
+ op = arrow::compute::CompareOperator::LESS_EQUAL;
+ } else {
+ std::string error_message = "Unknown operation name in comparison node";
+ env->ThrowNew(illegal_argument_exception_class, error_message.c_str());
+ return nullptr; // unreachable
+ }
+ const types::TreeNode& left_arg = cp_node.leftarg();
+ const types::TreeNode& right_arg = cp_node.rightarg();
+ return std::make_shared<arrow::dataset::ComparisonExpression>(op,
+ translateNode(left_arg, env), translateNode(right_arg, env));
+ }
+ if (node.has_notnode()) {
+ const types::NotNode& not_node = node.notnode();
+ const ::types::TreeNode& child = not_node.args();
+ std::shared_ptr<arrow::dataset::Expression> translatedChild = translateNode(child, env);
+ return std::make_shared<arrow::dataset::NotExpression>(translatedChild);
+ }
+ if (node.has_isvalidnode()) {
+ const types::IsValidNode& is_valid_node = node.isvalidnode();
+ const ::types::TreeNode& child = is_valid_node.args();
+ std::shared_ptr<arrow::dataset::Expression> translatedChild = translateNode(child, env);
+ return std::make_shared<arrow::dataset::IsValidExpression>(translatedChild);
+ }
+ std::string error_message = "Unknown node type";
+ env->ThrowNew(illegal_argument_exception_class, error_message.c_str());
+ return nullptr; // unreachable
+}
+
+std::shared_ptr<arrow::dataset::Expression> translateFilter(types::Condition condition, JNIEnv* env) {
Review comment:
renamed
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] zhztheplayer commented on a change in pull request #7030: ARROW-7808: [Java][Dataset] Implement Datasets Java API by JNI to C++
Posted by GitBox <gi...@apache.org>.
zhztheplayer commented on a change in pull request #7030:
URL: https://github.com/apache/arrow/pull/7030#discussion_r429551637
##########
File path: cpp/.gitignore
##########
@@ -24,6 +24,8 @@ cmake_install.cmake
build/
*-build/
Testing/
+cmake-build-debug/
Review comment:
OK. Actually I added these ignore terms because they are the default build dir names used by Clion.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] emkornfield commented on pull request #7030: ARROW-7808: [Java][Dataset] Implement Dataset Java API by JNI to C++
Posted by GitBox <gi...@apache.org>.
emkornfield commented on pull request #7030:
URL: https://github.com/apache/arrow/pull/7030#issuecomment-754270017
@zhztheplayer I need to take a closer look at the JNI changes since this was last approved, and will take another look at the memory stuff once you've added some docs.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] emkornfield commented on a change in pull request #7030: ARROW-7808: [Java][Dataset] Implement Datasets Java API
Posted by GitBox <gi...@apache.org>.
emkornfield commented on a change in pull request #7030:
URL: https://github.com/apache/arrow/pull/7030#discussion_r414988474
##########
File path: java/dataset/src/main/java/org/apache/arrow/util/SchemaUtils.java
##########
@@ -0,0 +1,72 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.arrow.util;
+
+import java.io.ByteArrayOutputStream;
+import java.io.IOException;
+import java.nio.channels.Channels;
+
+import org.apache.arrow.memory.BufferAllocator;
+import org.apache.arrow.vector.ipc.ReadChannel;
+import org.apache.arrow.vector.ipc.WriteChannel;
+import org.apache.arrow.vector.ipc.message.MessageChannelReader;
+import org.apache.arrow.vector.ipc.message.MessageResult;
+import org.apache.arrow.vector.ipc.message.MessageSerializer;
+import org.apache.arrow.vector.types.pojo.Schema;
+import org.apache.arrow.vector.util.ByteArrayReadableSeekableByteChannel;
+
+/**
+ * Schema utility class including serialization and deserialization.
+ */
+public class SchemaUtils {
+ private static final SchemaUtils INSTANCE = new SchemaUtils();
+
+ public static SchemaUtils get() {
+ return INSTANCE;
+ }
+
+ private SchemaUtils() {
+
+ }
+
+ /**
+ * Deserialize Arrow schema from byte array.
+ */
+ public Schema deserialize(byte[] bytes, BufferAllocator allocator) throws IOException {
Review comment:
why not make these static methods? instead of using a singleton pattern?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] zhztheplayer edited a comment on pull request #7030: ARROW-7808: [Java][Dataset] Implement Datasets Java API by JNI to C++
Posted by GitBox <gi...@apache.org>.
zhztheplayer edited a comment on pull request #7030:
URL: https://github.com/apache/arrow/pull/7030#issuecomment-700390598
Hi @Sebastiaan-Alvarez-Rodriguez ,
First of all, thanks for the detailed explanation about the issue and your suggestions. Sorry for the late reply.
Actually I don't mind to change approach from `System.load` to `System.loadLibrary` as long as it too works fine or even better. I am just worried about why `getResourceAsStream` returns null in your case.
Just saw your practice on compiling Java code:
> cd <project-root>/java/dataset
> mvn clean install -Dmaven.test.skip=true # skipping tests to let build succeed
Have you checked if arrow_dataset_jni.so was successfully installed within the jar `arrow-dataset-XXX.jar`? If not, specifying `arrow.cpp.build.dir` may help.
> Maybe it is a good idea to pick one of the following:
> 1. Change the 'java.io.FileNotFoundException: libarrow_dataset_jni.so' to display the full path (so people know where your library > looks to find the shared object)
> 2. Make a small JNI_dataset_dev_install_note.md somewhere on getting this library to work.
> 3. Use System.loadLibrary(libraryToLoad);
> 4. Use some other way to find the location of a shared library and copy it to a tmpfile
I did a little try and it appears that `System.loadLibrary` requires for property `java.library.path` to be set to include library locations. If we change to use this way, do users have to do this every time before getting started?
I may intend to pick option 1 here. While we should iterate on multiple resource path roots and display several possible paths to let users find in.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] zhztheplayer commented on a change in pull request #7030: ARROW-7808: [Java][Dataset] Implement Datasets Java API by JNI to C++
Posted by GitBox <gi...@apache.org>.
zhztheplayer commented on a change in pull request #7030:
URL: https://github.com/apache/arrow/pull/7030#discussion_r439266993
##########
File path: java/dataset/src/main/java/org/apache/arrow/memory/Ownerships.java
##########
@@ -0,0 +1,47 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.arrow.memory;
+
+/**
+ * Utility class managing ownership's transferring between Native Arrow buffers and Java Arrow buffers.
+ */
+public class Ownerships {
+ private static final Ownerships INSTANCE = new Ownerships();
+
+ private Ownerships() {
+ }
+
+ public static Ownerships get() {
+ return INSTANCE;
+ }
+
+ /**
+ * Returned ledger's ref count is initialized or increased by 1.
+ */
+ public BufferLedger takeOwnership(BaseAllocator allocator, AllocationManager allocationManager) {
+ final BufferLedger ledger = allocationManager.associate(allocator);
+ long size = allocationManager.getSize();
+ boolean succeed = allocator.forceAllocate(size);
+ if (!succeed) {
+ throw new OutOfMemoryException("Target allocator is full");
+ }
+ ((BaseAllocator) allocationManager.getOwningLedger().getAllocator()).releaseBytes(size);
Review comment:
Yes calls to `forceAllocate` should be synchronized.
The fix appear to be in a separate topic because in Java API we don't yet provide give allocator the functionality to take over ownerless memory which requires for some slight refactors to allocation process. I'll open a ticket for that.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] emkornfield commented on a change in pull request #7030: ARROW-7808: [Java][Dataset] Implement Dataset Java API by JNI to C++
Posted by GitBox <gi...@apache.org>.
emkornfield commented on a change in pull request #7030:
URL: https://github.com/apache/arrow/pull/7030#discussion_r567200527
##########
File path: cpp/src/jni/dataset/jni_wrapper.cpp
##########
@@ -0,0 +1,859 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include <mutex>
+
+#include <arrow/array.h>
+#include <arrow/dataset/api.h>
+#include <arrow/dataset/file_base.h>
+#include <arrow/filesystem/localfs.h>
+#include <arrow/io/api.h>
+#include <arrow/ipc/api.h>
+#include <arrow/util/iterator.h>
+#include <arrow/util/logging.h>
+
+#include "org_apache_arrow_dataset_file_JniWrapper.h"
+#include "org_apache_arrow_dataset_jni_JniWrapper.h"
+#include "org_apache_arrow_dataset_jni_NativeMemoryPool.h"
+
+namespace arrow {
+namespace dataset {
+namespace jni {
+
+static jclass illegal_access_exception_class;
+static jclass illegal_argument_exception_class;
+static jclass runtime_exception_class;
+
+static jclass record_batch_handle_class;
+static jclass record_batch_handle_field_class;
+static jclass record_batch_handle_buffer_class;
+static jclass java_reservation_listener_class;
+
+static jmethodID record_batch_handle_constructor;
+static jmethodID record_batch_handle_field_constructor;
+static jmethodID record_batch_handle_buffer_constructor;
+static jmethodID reserve_memory_method;
+static jmethodID unreserve_memory_method;
+
+static jlong default_memory_pool_id = -1L;
+
+static jint JNI_VERSION = JNI_VERSION_1_6;
+
+class JniPendingException : public std::runtime_error {
+ public:
+ explicit JniPendingException(const std::string& arg) : runtime_error(arg) {}
+};
+
+void ThrowPendingException(const std::string& message) {
+ throw JniPendingException(message);
+}
+
+template <typename T>
+T JniGetOrThrow(arrow::Result<T> result) {
+ if (!result.status().ok()) {
+ ThrowPendingException(result.status().message());
+ }
+ return std::move(result).ValueOrDie();
+}
+
+void JniAssertOkOrThrow(arrow::Status status) {
+ if (!status.ok()) {
+ ThrowPendingException(status.message());
+ }
+}
+
+void JniThrow(std::string message) { ThrowPendingException(message); }
+
+jclass CreateGlobalClassReference(JNIEnv* env, const char* class_name) {
+ jclass local_class = env->FindClass(class_name);
+ jclass global_class = (jclass)env->NewGlobalRef(local_class);
+ env->DeleteLocalRef(local_class);
+ return global_class;
+}
+
+arrow::Result<jmethodID> GetMethodID(JNIEnv* env, jclass this_class, const char* name,
+ const char* sig) {
+ jmethodID ret = env->GetMethodID(this_class, name, sig);
+ if (ret == nullptr) {
+ std::string error_message = "Unable to find method " + std::string(name) +
+ " within signature" + std::string(sig);
+ return arrow::Status::Invalid(error_message);
+ }
+ return ret;
+}
+
+arrow::Result<jmethodID> GetStaticMethodID(JNIEnv* env, jclass this_class,
+ const char* name, const char* sig) {
+ jmethodID ret = env->GetStaticMethodID(this_class, name, sig);
+ if (ret == nullptr) {
+ std::string error_message = "Unable to find static method " + std::string(name) +
+ " within signature" + std::string(sig);
+ return arrow::Status::Invalid(error_message);
+ }
+ return ret;
+}
+
+std::shared_ptr<arrow::Schema> SchemaFromColumnNames(
+ const std::shared_ptr<arrow::Schema>& input,
+ const std::vector<std::string>& column_names) {
+ std::vector<std::shared_ptr<arrow::Field>> columns;
+ for (const auto& name : column_names) {
+ columns.push_back(input->GetFieldByName(name));
+ }
+ return std::make_shared<arrow::Schema>(columns);
+}
+
+arrow::Result<std::shared_ptr<arrow::dataset::FileFormat>> GetFileFormat(jint id) {
Review comment:
nit: format_id
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] emkornfield commented on a change in pull request #7030: ARROW-7808: [Java][Dataset] Implement Dataset Java API by JNI to C++
Posted by GitBox <gi...@apache.org>.
emkornfield commented on a change in pull request #7030:
URL: https://github.com/apache/arrow/pull/7030#discussion_r567201346
##########
File path: cpp/src/jni/dataset/jni_wrapper.cpp
##########
@@ -0,0 +1,859 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include <mutex>
+
+#include <arrow/array.h>
+#include <arrow/dataset/api.h>
+#include <arrow/dataset/file_base.h>
+#include <arrow/filesystem/localfs.h>
+#include <arrow/io/api.h>
+#include <arrow/ipc/api.h>
+#include <arrow/util/iterator.h>
+#include <arrow/util/logging.h>
+
+#include "org_apache_arrow_dataset_file_JniWrapper.h"
+#include "org_apache_arrow_dataset_jni_JniWrapper.h"
+#include "org_apache_arrow_dataset_jni_NativeMemoryPool.h"
+
+namespace arrow {
+namespace dataset {
+namespace jni {
+
+static jclass illegal_access_exception_class;
+static jclass illegal_argument_exception_class;
+static jclass runtime_exception_class;
+
+static jclass record_batch_handle_class;
+static jclass record_batch_handle_field_class;
+static jclass record_batch_handle_buffer_class;
+static jclass java_reservation_listener_class;
+
+static jmethodID record_batch_handle_constructor;
+static jmethodID record_batch_handle_field_constructor;
+static jmethodID record_batch_handle_buffer_constructor;
+static jmethodID reserve_memory_method;
+static jmethodID unreserve_memory_method;
+
+static jlong default_memory_pool_id = -1L;
+
+static jint JNI_VERSION = JNI_VERSION_1_6;
+
+class JniPendingException : public std::runtime_error {
+ public:
+ explicit JniPendingException(const std::string& arg) : runtime_error(arg) {}
+};
+
+void ThrowPendingException(const std::string& message) {
+ throw JniPendingException(message);
+}
+
+template <typename T>
+T JniGetOrThrow(arrow::Result<T> result) {
+ if (!result.status().ok()) {
+ ThrowPendingException(result.status().message());
+ }
+ return std::move(result).ValueOrDie();
+}
+
+void JniAssertOkOrThrow(arrow::Status status) {
+ if (!status.ok()) {
+ ThrowPendingException(status.message());
+ }
+}
+
+void JniThrow(std::string message) { ThrowPendingException(message); }
+
+jclass CreateGlobalClassReference(JNIEnv* env, const char* class_name) {
+ jclass local_class = env->FindClass(class_name);
+ jclass global_class = (jclass)env->NewGlobalRef(local_class);
+ env->DeleteLocalRef(local_class);
+ return global_class;
+}
+
+arrow::Result<jmethodID> GetMethodID(JNIEnv* env, jclass this_class, const char* name,
+ const char* sig) {
+ jmethodID ret = env->GetMethodID(this_class, name, sig);
+ if (ret == nullptr) {
+ std::string error_message = "Unable to find method " + std::string(name) +
+ " within signature" + std::string(sig);
+ return arrow::Status::Invalid(error_message);
+ }
+ return ret;
+}
+
+arrow::Result<jmethodID> GetStaticMethodID(JNIEnv* env, jclass this_class,
+ const char* name, const char* sig) {
+ jmethodID ret = env->GetStaticMethodID(this_class, name, sig);
+ if (ret == nullptr) {
+ std::string error_message = "Unable to find static method " + std::string(name) +
+ " within signature" + std::string(sig);
+ return arrow::Status::Invalid(error_message);
+ }
+ return ret;
+}
+
+std::shared_ptr<arrow::Schema> SchemaFromColumnNames(
+ const std::shared_ptr<arrow::Schema>& input,
+ const std::vector<std::string>& column_names) {
+ std::vector<std::shared_ptr<arrow::Field>> columns;
+ for (const auto& name : column_names) {
+ columns.push_back(input->GetFieldByName(name));
+ }
+ return std::make_shared<arrow::Schema>(columns);
+}
+
+arrow::Result<std::shared_ptr<arrow::dataset::FileFormat>> GetFileFormat(jint id) {
+ switch (id) {
+ case 0:
+ return std::make_shared<arrow::dataset::ParquetFileFormat>();
+ default:
+ std::string error_message = "illegal file format id: " + std::to_string(id);
+ return arrow::Status::Invalid(error_message);
+ }
+}
+
+std::string JStringToCString(JNIEnv* env, jstring string) {
+ if (string == nullptr) {
+ return std::string();
+ }
+ jboolean copied;
+ const char* chars = env->GetStringUTFChars(string, &copied);
+ std::string ret = strdup(chars);
+ env->ReleaseStringUTFChars(string, chars);
+ return ret;
+}
+
+std::vector<std::string> ToStringVector(JNIEnv* env, jobjectArray& str_array) {
+ int length = env->GetArrayLength(str_array);
+ std::vector<std::string> vector;
+ for (int i = 0; i < length; i++) {
+ auto string = (jstring)(env->GetObjectArrayElement(str_array, i));
Review comment:
static_cast
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] emkornfield commented on a change in pull request #7030: ARROW-7808: [Java][Dataset] Implement Datasets Java API
Posted by GitBox <gi...@apache.org>.
emkornfield commented on a change in pull request #7030:
URL: https://github.com/apache/arrow/pull/7030#discussion_r414986335
##########
File path: cpp/src/jni/dataset/proto/Types.proto
##########
@@ -0,0 +1,149 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+syntax = "proto2";
+package types;
+
+option java_package = "org.apache.arrow.dataset";
Review comment:
why is this file being introduced?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] emkornfield commented on pull request #7030: ARROW-7808: [Java][Dataset] Implement Dataset Java API by JNI to C++
Posted by GitBox <gi...@apache.org>.
emkornfield commented on pull request #7030:
URL: https://github.com/apache/arrow/pull/7030#issuecomment-735549511
Thanks, I will try to rereview this week. @jacques-n did you want to take a look to see if your concerns were addressed?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] Sebastiaan-Alvarez-Rodriguez commented on pull request #7030: ARROW-7808: [Java][Dataset] Implement Datasets Java API by JNI to C++
Posted by GitBox <gi...@apache.org>.
Sebastiaan-Alvarez-Rodriguez commented on pull request #7030:
URL: https://github.com/apache/arrow/pull/7030#issuecomment-694925466
hi there,
Could you please add a few compilation/installation instructions to your branch?
I just completed a long debug session to find out why the JNI runtime did not work.
### CPP
Compiled `<project-root>/cpp/` project using:
```bash
cd <project-root>/cpp
cmake -DARROW_DATASET=ON -DARROW_JNI=ON -DARROW_PARQUET=ON -DARROW_IPC=ON .
sudo make install
```
Confirmed shared libs are indeed visible using `whereis libarrow_dataset_jni`.
### Java
Compiled `<project-root>/java/dataset/` using
```bash
cd <project-root>/java/dataset
mvn clean install -Dmaven.test.skip=true # skipping tests to let build succeed
```
To test whether it works, successfully ran a sample program:
```java
public class test {
public static void main(String[] args) {
System.loadLibrary("arrow_dataset_jni");
System.out.println("Own classLoader works fine");
}
}
```
So, my sample program's classloader can access `libarrow_dataset_jni.so` just fine.
However, the following does not work:
```java
import org.apache.arrow.memory.RootAllocator;
import org.apache.arrow.dataset.file.FileFormat;
import org.apache.arrow.dataset.file.FileSystemDatasetFactory;
public class test {
private static FileSystemDatasetFactory getDatasetFactory() {
RootAllocator allocator = new RootAllocator(Long.MAX_VALUE);
return new FileSystemDatasetFactory(allocator, FileFormat.PARQUET, "/path/to/pq.parquet"); //crash
}
public static void main(String[] args) {
System.loadLibrary("arrow_dataset_jni");
System.out.println("Own classLoader works fine");
FileSystemDatasetFactory test = getDatasetFactory();
System.out.println("Not so good! I crash before I can print this");
}
}
```
The relevant part of the crash is:
```
Caused by: java.lang.IllegalStateException: error loading native libraries: java.io.FileNotFoundException: libarrow_dataset_jni.so
at org.apache.arrow.dataset.jni.JniLoader.load(JniLoader.java:91)
at org.apache.arrow.dataset.jni.JniLoader.loadRemaining(JniLoader.java:73)
at org.apache.arrow.dataset.jni.JniLoader.ensureLoaded(JniLoader.java:60)
at org.apache.arrow.dataset.file.JniWrapper.<init>(JniWrapper.java:34)
```
The source code of the Dataset Java library causing the error is found [here](https://github.com/zhztheplayer/arrow-1/blob/fd163e199a10a7225765b0c30fbf60d8df8d20db/java/dataset/src/main/java/org/apache/arrow/dataset/jni/JniLoader.java#L78-L94).
The problem occurs because `JniWrapper.class.getClassLoader().getResourceAsStream(libraryToLoad)` always returns `null`.
Only when adding a symlink (`ln -s /usr/local/lib/libarrow_dataset_jni.so libarrow_dataset_jni.so`) to the package directory, it is able to find the correct file.
Why do you create a temporary file, store the contents of the shared library in it, and then call `System.load(tmpfile)` and not just directly call it using `System.loadLibrary(libraryToLoad);`?
Maybe it is a good idea to pick one of the following:
1. Change the `'java.io.FileNotFoundException: libarrow_dataset_jni.so'` to display the full path (so people know where your library looks to find the shared object)
2. Make a small `JNI_dataset_dev_install_note.md` somewhere on getting this library to work.
3. Use `System.loadLibrary(libraryToLoad);`
4. Use some other way to find the location of a shared library and copy it to a tmpfile
I believe it would help many people who are not familiar with `getClassLoader().getResourceAsStream(...)` (like myself)
Have a nice day,
Sebastiaan
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] zhztheplayer commented on a change in pull request #7030: ARROW-7808: [Java][Dataset] Implement Datasets Java API by JNI to C++
Posted by GitBox <gi...@apache.org>.
zhztheplayer commented on a change in pull request #7030:
URL: https://github.com/apache/arrow/pull/7030#discussion_r440098238
##########
File path: java/dataset/src/main/java/org/apache/arrow/dataset/jni/NativeScanner.java
##########
@@ -0,0 +1,143 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.arrow.dataset.jni;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.NoSuchElementException;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.stream.Collectors;
+
+import org.apache.arrow.dataset.scanner.ScanTask;
+import org.apache.arrow.dataset.scanner.Scanner;
+import org.apache.arrow.memory.ArrowBuf;
+import org.apache.arrow.memory.BufferAllocator;
+import org.apache.arrow.memory.BufferLedger;
+import org.apache.arrow.memory.NativeUnderlingMemory;
+import org.apache.arrow.memory.Ownerships;
+import org.apache.arrow.vector.ipc.message.ArrowFieldNode;
+import org.apache.arrow.vector.ipc.message.ArrowRecordBatch;
+import org.apache.arrow.vector.types.pojo.Schema;
+import org.apache.arrow.vector.util.SchemaUtility;
+
+/**
+ * Native implementation of {@link Scanner}. Note that it currently emits only a single scan task of type
+ * {@link NativeScanTask}, which is internally a combination of all scan task instances returned by the
+ * native scanner.
+ */
+public class NativeScanner implements Scanner {
+
+ private final AtomicBoolean closed = new AtomicBoolean(false);
+ private final AtomicBoolean executed = new AtomicBoolean(false);
+ private final NativeContext context;
+ private final long scannerId;
+
+ public NativeScanner(NativeContext context, long scannerId) {
+ this.context = context;
+ this.scannerId = scannerId;
+ }
+
+ ScanTask.BatchIterator execute() {
+ if (closed.get()) {
+ throw new NativeInstanceClosedException();
+ }
+ if (!executed.compareAndSet(false, true)) {
+ throw new UnsupportedOperationException("NativeScanner cannot be executed more than once. Consider creating " +
+ "new scanner instead");
+ }
+ return new ScanTask.BatchIterator() {
+ private ArrowRecordBatch peek = null;
+
+ @Override
+ public void close() {
+ NativeScanner.this.close();
+ }
+
+ @Override
+ public boolean hasNext() {
+ if (closed.get()) {
Review comment:
Thanks for catching! I've pushed a mutex solution.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] emkornfield commented on a change in pull request #7030: ARROW-7808: [Java][Dataset] Implement Datasets Java API by JNI to C++
Posted by GitBox <gi...@apache.org>.
emkornfield commented on a change in pull request #7030:
URL: https://github.com/apache/arrow/pull/7030#discussion_r425564123
##########
File path: java/dataset/src/main/java/org/apache/arrow/dataset/file/JniWrapper.java
##########
@@ -0,0 +1,39 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.arrow.dataset.file;
+
+import org.apache.arrow.dataset.jni.JniLoader;
+
+/**
+ * JniWrapper for filesystem based {@link org.apache.arrow.dataset.source.Dataset} implementations.
+ */
+public class JniWrapper {
+
Review comment:
ahh missed that.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] emkornfield commented on a change in pull request #7030: ARROW-7808: [Java][Dataset] Implement Dataset Java API by JNI to C++
Posted by GitBox <gi...@apache.org>.
emkornfield commented on a change in pull request #7030:
URL: https://github.com/apache/arrow/pull/7030#discussion_r567203372
##########
File path: java/dataset/src/test/java/org/apache/arrow/dataset/file/TestFileSystemDataset.java
##########
@@ -0,0 +1,348 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.arrow.dataset.file;
+
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertTrue;
+import static org.junit.jupiter.api.Assertions.assertDoesNotThrow;
+import static org.junit.jupiter.api.Assertions.assertThrows;
+
+import java.util.Collections;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.Objects;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+import java.util.concurrent.atomic.AtomicLong;
+import java.util.stream.Collectors;
+
+import org.apache.arrow.dataset.jni.DirectReservationListener;
+import org.apache.arrow.dataset.jni.NativeDataset;
+import org.apache.arrow.dataset.jni.NativeInstanceReleasedException;
+import org.apache.arrow.dataset.jni.NativeMemoryPool;
+import org.apache.arrow.dataset.jni.NativeScanTask;
+import org.apache.arrow.dataset.jni.NativeScanner;
+import org.apache.arrow.dataset.jni.ReservationListener;
+import org.apache.arrow.dataset.jni.TestNativeDataset;
+import org.apache.arrow.dataset.scanner.ScanOptions;
+import org.apache.arrow.dataset.scanner.ScanTask;
+import org.apache.arrow.util.AutoCloseables;
+import org.apache.arrow.vector.FieldVector;
+import org.apache.arrow.vector.VectorLoader;
+import org.apache.arrow.vector.VectorSchemaRoot;
+import org.apache.arrow.vector.ipc.message.ArrowRecordBatch;
+import org.apache.arrow.vector.types.Types;
+import org.apache.arrow.vector.types.pojo.Schema;
+import org.apache.avro.generic.GenericRecord;
+import org.apache.avro.generic.GenericRecordBuilder;
+import org.junit.Assert;
+import org.junit.ClassRule;
+import org.junit.Test;
+import org.junit.jupiter.api.Assertions;
+import org.junit.rules.TemporaryFolder;
+
+public class TestFileSystemDataset extends TestNativeDataset {
+
+ @ClassRule
+ public static final TemporaryFolder TMP = new TemporaryFolder();
+
+ public static final String AVRO_SCHEMA_USER = "user.avsc";
+
+ /**
+ * The default block size of C++ ReservationListenableMemoryPool.
+ */
+ public static final long DEFAULT_NATIVE_MEMORY_POOL_BLOCK_SIZE = 512 * 1024;
+
+ @Test
+ public void testParquetRead() throws Exception {
+ ParquetWriteSupport writeSupport = ParquetWriteSupport.writeTempFile(AVRO_SCHEMA_USER, TMP.newFolder(), 1, "a");
+
+ FileSystemDatasetFactory factory = new FileSystemDatasetFactory(rootAllocator(), NativeMemoryPool.getDefault(),
+ FileFormat.PARQUET, writeSupport.getOutputURI());
+ ScanOptions options = new ScanOptions(new String[0], 100);
+ Schema schema = inferResultSchemaFromFactory(factory, options);
+ List<ArrowRecordBatch> datum = collectResultFromFactory(factory, options);
+
+ assertSingleTaskProduced(factory, options);
+ assertEquals(1, datum.size());
+ assertEquals(2, schema.getFields().size());
+ assertEquals("id", schema.getFields().get(0).getName());
+ assertEquals("name", schema.getFields().get(1).getName());
+ assertEquals(Types.MinorType.INT.getType(), schema.getFields().get(0).getType());
+ assertEquals(Types.MinorType.VARCHAR.getType(), schema.getFields().get(1).getType());
+ checkParquetReadResult(schema, writeSupport.getWrittenRecords(), datum);
+
+ AutoCloseables.close(datum);
+ }
+
+ @Test
+ public void testParquetProjector() throws Exception {
+ ParquetWriteSupport writeSupport = ParquetWriteSupport.writeTempFile(AVRO_SCHEMA_USER, TMP.newFolder(), 1, "a");
+
+ FileSystemDatasetFactory factory = new FileSystemDatasetFactory(rootAllocator(), NativeMemoryPool.getDefault(),
+ FileFormat.PARQUET, writeSupport.getOutputURI());
+ ScanOptions options = new ScanOptions(new String[]{"id"}, 100);
+ Schema schema = inferResultSchemaFromFactory(factory, options);
+ List<ArrowRecordBatch> datum = collectResultFromFactory(factory, options);
+ org.apache.avro.Schema expectedSchema = truncateAvroSchema(writeSupport.getAvroSchema(), 0, 1);
+
+ assertSingleTaskProduced(factory, options);
+ assertEquals(1, schema.getFields().size());
+ assertEquals("id", schema.getFields().get(0).getName());
+ assertEquals(Types.MinorType.INT.getType(), schema.getFields().get(0).getType());
+ assertEquals(1, datum.size());
+ checkParquetReadResult(schema,
+ Collections.singletonList(
+ new GenericRecordBuilder(
+ expectedSchema)
+ .set("id", 1)
+ .build()), datum);
+
+ AutoCloseables.close(datum);
+ }
+
+ @Test
+ public void testParquetBatchSize() throws Exception {
+ ParquetWriteSupport writeSupport = ParquetWriteSupport.writeTempFile(AVRO_SCHEMA_USER, TMP.newFolder(),
+ 1, "a", 2, "b", 3, "c");
+
+ ScanOptions options = new ScanOptions(new String[0], 1);
+ FileSystemDatasetFactory factory = new FileSystemDatasetFactory(rootAllocator(), NativeMemoryPool.getDefault(),
+ FileFormat.PARQUET, writeSupport.getOutputURI());
+ Schema schema = inferResultSchemaFromFactory(factory, options);
+ List<ArrowRecordBatch> datum = collectResultFromFactory(factory, options);
+
+ assertSingleTaskProduced(factory, options);
+ assertEquals(3, datum.size());
+ datum.forEach(batch -> assertEquals(1, batch.getLength()));
+ checkParquetReadResult(schema, writeSupport.getWrittenRecords(), datum);
+
+ AutoCloseables.close(datum);
+ }
+
+ @Test
+ public void testCloseAgain() throws Exception {
+ ParquetWriteSupport writeSupport = ParquetWriteSupport.writeTempFile(AVRO_SCHEMA_USER, TMP.newFolder(), 1, "a");
+
+ FileSystemDatasetFactory factory = new FileSystemDatasetFactory(rootAllocator(), NativeMemoryPool.getDefault(),
+ FileFormat.PARQUET, writeSupport.getOutputURI());
+
+ assertDoesNotThrow(() -> {
+ NativeDataset dataset = factory.finish();
+ dataset.close();
+ dataset.close();
+ });
+ }
+
+ @Test
+ public void testScanAgain() throws Exception {
+ ParquetWriteSupport writeSupport = ParquetWriteSupport.writeTempFile(AVRO_SCHEMA_USER, TMP.newFolder(), 1, "a");
+
+ FileSystemDatasetFactory factory = new FileSystemDatasetFactory(rootAllocator(), NativeMemoryPool.getDefault(),
+ FileFormat.PARQUET, writeSupport.getOutputURI());
+ NativeDataset dataset = factory.finish();
+ ScanOptions options = new ScanOptions(new String[0], 100);
+ NativeScanner scanner = dataset.newScan(options);
+ List<? extends NativeScanTask> taskList1 = collect(scanner.scan());
+ List<? extends NativeScanTask> taskList2 = collect(scanner.scan());
+ NativeScanTask task1 = taskList1.get(0);
+ NativeScanTask task2 = taskList2.get(0);
+ List<ArrowRecordBatch> datum = collect(task1.execute());
+
+ UnsupportedOperationException uoe = assertThrows(UnsupportedOperationException.class, task2::execute);
+ Assertions.assertEquals("NativeScanner cannot be executed more than once. Consider creating new scanner instead",
+ uoe.getMessage());
+
+ AutoCloseables.close(datum);
+ AutoCloseables.close(taskList1);
+ AutoCloseables.close(taskList2);
+ AutoCloseables.close(scanner, dataset, factory);
+ }
+
+ @Test
+ public void testScanInOtherThread() throws Exception {
+ ExecutorService executor = Executors.newSingleThreadExecutor();
+ ParquetWriteSupport writeSupport = ParquetWriteSupport.writeTempFile(AVRO_SCHEMA_USER, TMP.newFolder(), 1, "a");
+
+ FileSystemDatasetFactory factory = new FileSystemDatasetFactory(rootAllocator(), NativeMemoryPool.getDefault(),
+ FileFormat.PARQUET, writeSupport.getOutputURI());
+ NativeDataset dataset = factory.finish();
+ ScanOptions options = new ScanOptions(new String[0], 100);
+ NativeScanner scanner = dataset.newScan(options);
+ List<? extends NativeScanTask> taskList = collect(scanner.scan());
+ NativeScanTask task = taskList.get(0);
+ List<ArrowRecordBatch> datum = executor.submit(() -> collect(task.execute())).get();
+
+ AutoCloseables.close(datum);
+ AutoCloseables.close(taskList);
+ AutoCloseables.close(scanner, dataset, factory);
+ }
+
+ @Test
+ public void testScanAfterClose1() throws Exception {
+ ParquetWriteSupport writeSupport = ParquetWriteSupport.writeTempFile(AVRO_SCHEMA_USER, TMP.newFolder(), 1, "a");
+
+ FileSystemDatasetFactory factory = new FileSystemDatasetFactory(rootAllocator(), NativeMemoryPool.getDefault(),
+ FileFormat.PARQUET, writeSupport.getOutputURI());
+ NativeDataset dataset = factory.finish();
+ ScanOptions options = new ScanOptions(new String[0], 100);
+ NativeScanner scanner = dataset.newScan(options);
+ scanner.close();
+ assertThrows(NativeInstanceReleasedException.class, scanner::scan);
+ }
+
+ @Test
+ public void testScanAfterClose2() throws Exception {
+ ParquetWriteSupport writeSupport = ParquetWriteSupport.writeTempFile(AVRO_SCHEMA_USER, TMP.newFolder(), 1, "a");
+
+ FileSystemDatasetFactory factory = new FileSystemDatasetFactory(rootAllocator(), NativeMemoryPool.getDefault(),
+ FileFormat.PARQUET, writeSupport.getOutputURI());
+ NativeDataset dataset = factory.finish();
+ ScanOptions options = new ScanOptions(new String[0], 100);
+ NativeScanner scanner = dataset.newScan(options);
+ List<? extends NativeScanTask> tasks = collect(scanner.scan());
+ NativeScanTask task = tasks.get(0);
+ task.close();
+ assertThrows(NativeInstanceReleasedException.class, task::execute);
+ }
+
+ @Test
+ public void testScanAfterClose3() throws Exception {
+ ParquetWriteSupport writeSupport = ParquetWriteSupport.writeTempFile(AVRO_SCHEMA_USER, TMP.newFolder(), 1, "a");
+
+ FileSystemDatasetFactory factory = new FileSystemDatasetFactory(rootAllocator(), NativeMemoryPool.getDefault(),
+ FileFormat.PARQUET, writeSupport.getOutputURI());
+ NativeDataset dataset = factory.finish();
+ ScanOptions options = new ScanOptions(new String[0], 100);
+ NativeScanner scanner = dataset.newScan(options);
+ List<? extends NativeScanTask> tasks = collect(scanner.scan());
+ NativeScanTask task = tasks.get(0);
+ ScanTask.BatchIterator iterator = task.execute();
+ task.close();
+ assertThrows(NativeInstanceReleasedException.class, iterator::hasNext);
+ }
+
+ @Test
+ public void testAllocatorReservation() throws Exception {
+ ParquetWriteSupport writeSupport = ParquetWriteSupport.writeTempFile(AVRO_SCHEMA_USER, TMP.newFolder(), 1, "a");
+ FileSystemDatasetFactory factory = new FileSystemDatasetFactory(rootAllocator(), NativeMemoryPool.getDefault(),
+ FileFormat.PARQUET, writeSupport.getOutputURI());
+ ScanOptions options = new ScanOptions(new String[0], 100);
+ long initReservation = rootAllocator().getAllocatedMemory();
+ List<ArrowRecordBatch> datum = collectResultFromFactory(factory, options);
+ final long expected_diff = datum.stream()
+ .flatMapToLong(batch -> batch.getBuffers()
+ .stream()
+ .mapToLong(buf -> buf.getReferenceManager().getAccountedSize())).sum();
+ long reservation = rootAllocator().getAllocatedMemory();
+ AutoCloseables.close(datum);
+ long finalReservation = rootAllocator().getAllocatedMemory();
+ Assert.assertEquals(expected_diff, reservation - initReservation);
+ Assert.assertEquals(-expected_diff, finalReservation - reservation);
+ }
+
+ @Test
+ public void testDirectMemoryReservation() throws Exception {
+ ParquetWriteSupport writeSupport = ParquetWriteSupport.writeTempFile(AVRO_SCHEMA_USER, TMP.newFolder(), 1, "a");
+ NativeMemoryPool pool = NativeMemoryPool.createListenable(DirectReservationListener.instance());
+ FileSystemDatasetFactory factory = new FileSystemDatasetFactory(rootAllocator(),
+ pool, FileFormat.PARQUET,
+ writeSupport.getOutputURI());
+ ScanOptions options = new ScanOptions(new String[0], 100);
+ long initReservation = DirectReservationListener.instance().getCurrentDirectMemReservation();
+ List<ArrowRecordBatch> datum = collectResultFromFactory(factory, options);
+ long reservation = DirectReservationListener.instance().getCurrentDirectMemReservation();
+ AutoCloseables.close(datum);
+ AutoCloseables.close(pool);
+ long finalReservation = DirectReservationListener.instance().getCurrentDirectMemReservation();
+ final long expected_diff = DEFAULT_NATIVE_MEMORY_POOL_BLOCK_SIZE;
+ Assert.assertEquals(expected_diff, reservation - initReservation);
+ Assert.assertEquals(-expected_diff, finalReservation - reservation);
+ }
+
+ @Test
+ public void testCustomReservationListener() throws Exception {
Review comment:
could we separate out allocation tests to their own file?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] zhztheplayer commented on a change in pull request #7030: ARROW-7808: [Java][Dataset] Implement Dataset Java API by JNI to C++
Posted by GitBox <gi...@apache.org>.
zhztheplayer commented on a change in pull request #7030:
URL: https://github.com/apache/arrow/pull/7030#discussion_r551693725
##########
File path: cpp/src/arrow/memory_pool.h
##########
@@ -149,6 +149,43 @@ class ARROW_EXPORT ProxyMemoryPool : public MemoryPool {
std::unique_ptr<ProxyMemoryPoolImpl> impl_;
};
+class ARROW_EXPORT ReservationListener {
Review comment:
added
##########
File path: cpp/src/arrow/memory_pool.h
##########
@@ -149,6 +149,43 @@ class ARROW_EXPORT ProxyMemoryPool : public MemoryPool {
std::unique_ptr<ProxyMemoryPoolImpl> impl_;
};
+class ARROW_EXPORT ReservationListener {
+ public:
+ virtual ~ReservationListener();
Review comment:
Done. Thanks for the suggestion.
##########
File path: cpp/src/arrow/memory_pool.h
##########
@@ -149,6 +149,43 @@ class ARROW_EXPORT ProxyMemoryPool : public MemoryPool {
std::unique_ptr<ProxyMemoryPoolImpl> impl_;
};
+class ARROW_EXPORT ReservationListener {
+ public:
+ virtual ~ReservationListener();
+
+ virtual Status OnReservation(int64_t size) = 0;
+ virtual Status OnRelease(int64_t size) = 0;
+
+ protected:
+ ReservationListener();
Review comment:
done
##########
File path: cpp/src/arrow/memory_pool.h
##########
@@ -149,6 +149,43 @@ class ARROW_EXPORT ProxyMemoryPool : public MemoryPool {
std::unique_ptr<ProxyMemoryPoolImpl> impl_;
};
+class ARROW_EXPORT ReservationListener {
+ public:
+ virtual ~ReservationListener();
+
+ virtual Status OnReservation(int64_t size) = 0;
+ virtual Status OnRelease(int64_t size) = 0;
+
+ protected:
+ ReservationListener();
+};
+
+class ARROW_EXPORT ReservationListenableMemoryPool : public MemoryPool {
Review comment:
added
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] emkornfield commented on a change in pull request #7030: ARROW-7808: [Java][Dataset] Implement Dataset Java API by JNI to C++
Posted by GitBox <gi...@apache.org>.
emkornfield commented on a change in pull request #7030:
URL: https://github.com/apache/arrow/pull/7030#discussion_r567201346
##########
File path: cpp/src/jni/dataset/jni_wrapper.cpp
##########
@@ -0,0 +1,859 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include <mutex>
+
+#include <arrow/array.h>
+#include <arrow/dataset/api.h>
+#include <arrow/dataset/file_base.h>
+#include <arrow/filesystem/localfs.h>
+#include <arrow/io/api.h>
+#include <arrow/ipc/api.h>
+#include <arrow/util/iterator.h>
+#include <arrow/util/logging.h>
+
+#include "org_apache_arrow_dataset_file_JniWrapper.h"
+#include "org_apache_arrow_dataset_jni_JniWrapper.h"
+#include "org_apache_arrow_dataset_jni_NativeMemoryPool.h"
+
+namespace arrow {
+namespace dataset {
+namespace jni {
+
+static jclass illegal_access_exception_class;
+static jclass illegal_argument_exception_class;
+static jclass runtime_exception_class;
+
+static jclass record_batch_handle_class;
+static jclass record_batch_handle_field_class;
+static jclass record_batch_handle_buffer_class;
+static jclass java_reservation_listener_class;
+
+static jmethodID record_batch_handle_constructor;
+static jmethodID record_batch_handle_field_constructor;
+static jmethodID record_batch_handle_buffer_constructor;
+static jmethodID reserve_memory_method;
+static jmethodID unreserve_memory_method;
+
+static jlong default_memory_pool_id = -1L;
+
+static jint JNI_VERSION = JNI_VERSION_1_6;
+
+class JniPendingException : public std::runtime_error {
+ public:
+ explicit JniPendingException(const std::string& arg) : runtime_error(arg) {}
+};
+
+void ThrowPendingException(const std::string& message) {
+ throw JniPendingException(message);
+}
+
+template <typename T>
+T JniGetOrThrow(arrow::Result<T> result) {
+ if (!result.status().ok()) {
+ ThrowPendingException(result.status().message());
+ }
+ return std::move(result).ValueOrDie();
+}
+
+void JniAssertOkOrThrow(arrow::Status status) {
+ if (!status.ok()) {
+ ThrowPendingException(status.message());
+ }
+}
+
+void JniThrow(std::string message) { ThrowPendingException(message); }
+
+jclass CreateGlobalClassReference(JNIEnv* env, const char* class_name) {
+ jclass local_class = env->FindClass(class_name);
+ jclass global_class = (jclass)env->NewGlobalRef(local_class);
+ env->DeleteLocalRef(local_class);
+ return global_class;
+}
+
+arrow::Result<jmethodID> GetMethodID(JNIEnv* env, jclass this_class, const char* name,
+ const char* sig) {
+ jmethodID ret = env->GetMethodID(this_class, name, sig);
+ if (ret == nullptr) {
+ std::string error_message = "Unable to find method " + std::string(name) +
+ " within signature" + std::string(sig);
+ return arrow::Status::Invalid(error_message);
+ }
+ return ret;
+}
+
+arrow::Result<jmethodID> GetStaticMethodID(JNIEnv* env, jclass this_class,
+ const char* name, const char* sig) {
+ jmethodID ret = env->GetStaticMethodID(this_class, name, sig);
+ if (ret == nullptr) {
+ std::string error_message = "Unable to find static method " + std::string(name) +
+ " within signature" + std::string(sig);
+ return arrow::Status::Invalid(error_message);
+ }
+ return ret;
+}
+
+std::shared_ptr<arrow::Schema> SchemaFromColumnNames(
+ const std::shared_ptr<arrow::Schema>& input,
+ const std::vector<std::string>& column_names) {
+ std::vector<std::shared_ptr<arrow::Field>> columns;
+ for (const auto& name : column_names) {
+ columns.push_back(input->GetFieldByName(name));
+ }
+ return std::make_shared<arrow::Schema>(columns);
+}
+
+arrow::Result<std::shared_ptr<arrow::dataset::FileFormat>> GetFileFormat(jint id) {
+ switch (id) {
+ case 0:
+ return std::make_shared<arrow::dataset::ParquetFileFormat>();
+ default:
+ std::string error_message = "illegal file format id: " + std::to_string(id);
+ return arrow::Status::Invalid(error_message);
+ }
+}
+
+std::string JStringToCString(JNIEnv* env, jstring string) {
+ if (string == nullptr) {
+ return std::string();
+ }
+ jboolean copied;
+ const char* chars = env->GetStringUTFChars(string, &copied);
+ std::string ret = strdup(chars);
+ env->ReleaseStringUTFChars(string, chars);
+ return ret;
+}
+
+std::vector<std::string> ToStringVector(JNIEnv* env, jobjectArray& str_array) {
+ int length = env->GetArrayLength(str_array);
+ std::vector<std::string> vector;
+ for (int i = 0; i < length; i++) {
+ auto string = (jstring)(env->GetObjectArrayElement(str_array, i));
Review comment:
checked_cast
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] zhztheplayer commented on a change in pull request #7030: ARROW-7808: [Java][Dataset] Implement Dataset Java API by JNI to C++
Posted by GitBox <gi...@apache.org>.
zhztheplayer commented on a change in pull request #7030:
URL: https://github.com/apache/arrow/pull/7030#discussion_r567768269
##########
File path: java/dataset/src/main/java/org/apache/arrow/dataset/jni/DirectReservationListener.java
##########
@@ -0,0 +1,97 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.arrow.dataset.jni;
+
+import java.lang.reflect.Field;
+import java.lang.reflect.Method;
+import java.util.concurrent.atomic.AtomicLong;
+
+import org.apache.arrow.util.VisibleForTesting;
+
+/**
+ * Reserving Java direct memory bytes from java.nio.Bits. Used by Java Dataset API's C++ memory
+ * pool implementation. This makes memory allocated by the pool to be controlled by JVM option
+ * "-XX:MaxDirectMemorySize".
+ */
+public class DirectReservationListener implements ReservationListener {
+ private final Method methodReserve;
+ private final Method methodUnreserve;
+
+ private DirectReservationListener() {
+ try {
+ final Class<?> classBits = Class.forName("java.nio.Bits");
+ methodReserve = classBits.getDeclaredMethod("reserveMemory", long.class, int.class);
+ methodReserve.setAccessible(true);
+ methodUnreserve = classBits.getDeclaredMethod("unreserveMemory", long.class, int.class);
+ methodUnreserve.setAccessible(true);
+ } catch (Exception e) {
+ throw new RuntimeException(e);
+ }
+ }
+
+ private static final DirectReservationListener INSTANCE = new DirectReservationListener();
+
+ public static DirectReservationListener instance() {
+ return INSTANCE;
+ }
+
+ /**
+ * Reserve bytes by invoking java.nio.java.Bitjava.nio.Bitss#reserveMemory.
+ */
+ @Override
+ public void reserve(long size) {
+ try {
+ if (size > Integer.MAX_VALUE) {
+ throw new IllegalArgumentException("reserve size should be of type int");
Review comment:
Done, thanks.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] emkornfield commented on a change in pull request #7030: ARROW-7808: [Java][Dataset] Implement Dataset Java API by JNI to C++
Posted by GitBox <gi...@apache.org>.
emkornfield commented on a change in pull request #7030:
URL: https://github.com/apache/arrow/pull/7030#discussion_r551609196
##########
File path: cpp/src/arrow/memory_pool.h
##########
@@ -149,6 +149,43 @@ class ARROW_EXPORT ProxyMemoryPool : public MemoryPool {
std::unique_ptr<ProxyMemoryPoolImpl> impl_;
};
+class ARROW_EXPORT ReservationListener {
Review comment:
documentation please.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] zhztheplayer commented on a change in pull request #7030: ARROW-7808: [Java][Dataset] Implement Datasets Java API by JNI to C++
Posted by GitBox <gi...@apache.org>.
zhztheplayer commented on a change in pull request #7030:
URL: https://github.com/apache/arrow/pull/7030#discussion_r426990234
##########
File path: cpp/src/jni/dataset/proto/Types.proto
##########
@@ -0,0 +1,149 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+syntax = "proto2";
+package types;
+
+option java_package = "org.apache.arrow.dataset";
Review comment:
Yes I prefer removing them from this PR as I think of the implementation immature too. I can open ticket/discussion later.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] emkornfield commented on pull request #7030: ARROW-7808: [Java][Dataset] Implement Dataset Java API by JNI to C++
Posted by GitBox <gi...@apache.org>.
emkornfield commented on pull request #7030:
URL: https://github.com/apache/arrow/pull/7030#issuecomment-754171906
Sorry for the delay, I should have time to look at it today.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] zhztheplayer commented on pull request #7030: ARROW-7808: [Java][Dataset] Implement Dataset Java API by JNI to C++
Posted by GitBox <gi...@apache.org>.
zhztheplayer commented on pull request #7030:
URL: https://github.com/apache/arrow/pull/7030#issuecomment-772537663
Hi @emkornfield. I've addressed the new comments within the latest commit, please review. Previous commits has been merged into a single one.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] pitrou commented on pull request #7030: ARROW-7808: [Java][Dataset] Implement Dataset Java API by JNI to C++
Posted by GitBox <gi...@apache.org>.
pitrou commented on pull request #7030:
URL: https://github.com/apache/arrow/pull/7030#issuecomment-755957029
@zhztheplayer Yes, all Arrow C++ code should be inside our own namespaces to avoid potential name clashes.
Also, for private functions and classes it is recommended to put them inside the anonymous namespace (in addition to the `arrow::...` namespace) to hide them from other modules being linked.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] pitrou commented on pull request #7030: ARROW-7808: [Java][Dataset] Implement Dataset Java API by JNI to C++
Posted by GitBox <gi...@apache.org>.
pitrou commented on pull request #7030:
URL: https://github.com/apache/arrow/pull/7030#issuecomment-755274004
I suggest you put `ReservationListenableMemoryPool` (whatever it is) in the JNI code, so that you can do what you want with it :-)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] emkornfield commented on a change in pull request #7030: ARROW-7808: [Java][Dataset] Implement Dataset Java API by JNI to C++
Posted by GitBox <gi...@apache.org>.
emkornfield commented on a change in pull request #7030:
URL: https://github.com/apache/arrow/pull/7030#discussion_r567202461
##########
File path: cpp/src/jni/dataset/jni_wrapper.cpp
##########
@@ -0,0 +1,859 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include <mutex>
+
+#include <arrow/array.h>
+#include <arrow/dataset/api.h>
+#include <arrow/dataset/file_base.h>
+#include <arrow/filesystem/localfs.h>
+#include <arrow/io/api.h>
+#include <arrow/ipc/api.h>
+#include <arrow/util/iterator.h>
+#include <arrow/util/logging.h>
+
+#include "org_apache_arrow_dataset_file_JniWrapper.h"
+#include "org_apache_arrow_dataset_jni_JniWrapper.h"
+#include "org_apache_arrow_dataset_jni_NativeMemoryPool.h"
+
+namespace arrow {
+namespace dataset {
+namespace jni {
+
+static jclass illegal_access_exception_class;
+static jclass illegal_argument_exception_class;
+static jclass runtime_exception_class;
+
+static jclass record_batch_handle_class;
+static jclass record_batch_handle_field_class;
+static jclass record_batch_handle_buffer_class;
+static jclass java_reservation_listener_class;
+
+static jmethodID record_batch_handle_constructor;
+static jmethodID record_batch_handle_field_constructor;
+static jmethodID record_batch_handle_buffer_constructor;
+static jmethodID reserve_memory_method;
+static jmethodID unreserve_memory_method;
+
+static jlong default_memory_pool_id = -1L;
+
+static jint JNI_VERSION = JNI_VERSION_1_6;
+
+class JniPendingException : public std::runtime_error {
+ public:
+ explicit JniPendingException(const std::string& arg) : runtime_error(arg) {}
+};
+
+void ThrowPendingException(const std::string& message) {
+ throw JniPendingException(message);
+}
+
+template <typename T>
+T JniGetOrThrow(arrow::Result<T> result) {
+ if (!result.status().ok()) {
+ ThrowPendingException(result.status().message());
+ }
+ return std::move(result).ValueOrDie();
+}
+
+void JniAssertOkOrThrow(arrow::Status status) {
+ if (!status.ok()) {
+ ThrowPendingException(status.message());
+ }
+}
+
+void JniThrow(std::string message) { ThrowPendingException(message); }
+
+jclass CreateGlobalClassReference(JNIEnv* env, const char* class_name) {
+ jclass local_class = env->FindClass(class_name);
+ jclass global_class = (jclass)env->NewGlobalRef(local_class);
+ env->DeleteLocalRef(local_class);
+ return global_class;
+}
+
+arrow::Result<jmethodID> GetMethodID(JNIEnv* env, jclass this_class, const char* name,
+ const char* sig) {
+ jmethodID ret = env->GetMethodID(this_class, name, sig);
+ if (ret == nullptr) {
+ std::string error_message = "Unable to find method " + std::string(name) +
+ " within signature" + std::string(sig);
+ return arrow::Status::Invalid(error_message);
+ }
+ return ret;
+}
+
+arrow::Result<jmethodID> GetStaticMethodID(JNIEnv* env, jclass this_class,
+ const char* name, const char* sig) {
+ jmethodID ret = env->GetStaticMethodID(this_class, name, sig);
+ if (ret == nullptr) {
+ std::string error_message = "Unable to find static method " + std::string(name) +
+ " within signature" + std::string(sig);
+ return arrow::Status::Invalid(error_message);
+ }
+ return ret;
+}
+
+std::shared_ptr<arrow::Schema> SchemaFromColumnNames(
+ const std::shared_ptr<arrow::Schema>& input,
+ const std::vector<std::string>& column_names) {
+ std::vector<std::shared_ptr<arrow::Field>> columns;
+ for (const auto& name : column_names) {
+ columns.push_back(input->GetFieldByName(name));
+ }
+ return std::make_shared<arrow::Schema>(columns);
+}
+
+arrow::Result<std::shared_ptr<arrow::dataset::FileFormat>> GetFileFormat(jint id) {
+ switch (id) {
+ case 0:
+ return std::make_shared<arrow::dataset::ParquetFileFormat>();
+ default:
+ std::string error_message = "illegal file format id: " + std::to_string(id);
+ return arrow::Status::Invalid(error_message);
+ }
+}
+
+std::string JStringToCString(JNIEnv* env, jstring string) {
+ if (string == nullptr) {
+ return std::string();
+ }
+ jboolean copied;
+ const char* chars = env->GetStringUTFChars(string, &copied);
+ std::string ret = strdup(chars);
+ env->ReleaseStringUTFChars(string, chars);
+ return ret;
+}
+
+std::vector<std::string> ToStringVector(JNIEnv* env, jobjectArray& str_array) {
+ int length = env->GetArrayLength(str_array);
+ std::vector<std::string> vector;
+ for (int i = 0; i < length; i++) {
+ auto string = (jstring)(env->GetObjectArrayElement(str_array, i));
+ vector.push_back(JStringToCString(env, string));
+ }
+ return vector;
+}
+
+template <typename T>
+jlong CreateNativeRef(std::shared_ptr<T> t) {
+ std::shared_ptr<T>* retained_ptr = new std::shared_ptr<T>(t);
+ return reinterpret_cast<jlong>(retained_ptr);
+}
+
+template <typename T>
+std::shared_ptr<T> RetrieveNativeInstance(jlong ref) {
+ std::shared_ptr<T>* retrieved_ptr = reinterpret_cast<std::shared_ptr<T>*>(ref);
+ return *retrieved_ptr;
+}
+
+template <typename T>
+void ReleaseNativeRef(jlong ref) {
+ std::shared_ptr<T>* retrieved_ptr = reinterpret_cast<std::shared_ptr<T>*>(ref);
+ delete retrieved_ptr;
+}
+
+arrow::Result<jbyteArray> ToSchemaByteArray(JNIEnv* env,
+ std::shared_ptr<arrow::Schema> schema) {
+ ARROW_ASSIGN_OR_RAISE(
+ std::shared_ptr<arrow::Buffer> buffer,
+ arrow::ipc::SerializeSchema(*schema, arrow::default_memory_pool()))
+
+ jbyteArray out = env->NewByteArray(buffer->size());
+ auto src = reinterpret_cast<const jbyte*>(buffer->data());
+ env->SetByteArrayRegion(out, 0, buffer->size(), src);
+ return out;
+}
+
+arrow::Result<std::shared_ptr<arrow::Schema>> FromSchemaByteArray(
+ JNIEnv* env, jbyteArray schemaBytes) {
+ arrow::ipc::DictionaryMemo in_memo;
+ int schemaBytes_len = env->GetArrayLength(schemaBytes);
+ jbyte* schemaBytes_data = env->GetByteArrayElements(schemaBytes, nullptr);
+ auto serialized_schema = std::make_shared<arrow::Buffer>(
+ reinterpret_cast<uint8_t*>(schemaBytes_data), schemaBytes_len);
+ arrow::io::BufferReader buf_reader(serialized_schema);
+ ARROW_ASSIGN_OR_RAISE(std::shared_ptr<arrow::Schema> schema,
+ arrow::ipc::ReadSchema(&buf_reader, &in_memo))
+ env->ReleaseByteArrayElements(schemaBytes, schemaBytes_data, JNI_ABORT);
+ return schema;
+}
+
+/// Listener to act on reservations/unreservations from ReservationListenableMemoryPool.
+///
+/// Note the memory pool will call this listener only on block-level memory
+/// reservation/unreservation is granted. So the invocation parameter "size" is always
+/// multiple of block size (by default, 512k) specified in memory pool.
+class ReservationListener {
+ public:
+ virtual ~ReservationListener() = default;
+
+ virtual arrow::Status OnReservation(int64_t size) = 0;
+ virtual arrow::Status OnRelease(int64_t size) = 0;
+
+ protected:
+ ReservationListener() = default;
+};
+
+class ReservationListenableMemoryPoolImpl {
+ public:
+ explicit ReservationListenableMemoryPoolImpl(
+ arrow::MemoryPool* pool, std::shared_ptr<ReservationListener> listener,
+ int64_t block_size)
+ : pool_(pool),
+ listener_(listener),
+ block_size_(block_size),
+ blocks_reserved_(0),
+ bytes_reserved_(0) {}
+
+ arrow::Status Allocate(int64_t size, uint8_t** out) {
+ RETURN_NOT_OK(UpdateReservation(size));
+ arrow::Status error = pool_->Allocate(size, out);
+ if (!error.ok()) {
+ RETURN_NOT_OK(UpdateReservation(-size));
+ return error;
+ }
+ return arrow::Status::OK();
+ }
+
+ arrow::Status Reallocate(int64_t old_size, int64_t new_size, uint8_t** ptr) {
+ bool reserved = false;
+ int64_t diff = new_size - old_size;
+ if (new_size >= old_size) {
+ // new_size >= old_size, pre-reserve bytes from listener before allocating
+ // from underlying pool
+ RETURN_NOT_OK(UpdateReservation(diff));
+ reserved = true;
+ }
+ arrow::Status error = pool_->Reallocate(old_size, new_size, ptr);
+ if (!error.ok()) {
+ if (reserved) {
+ // roll back reservations on error
+ RETURN_NOT_OK(UpdateReservation(-diff));
+ }
+ return error;
+ }
+ if (!reserved) {
+ // otherwise (e.g. new_size < old_size), make updates after calling underlying pool
+ RETURN_NOT_OK(UpdateReservation(diff));
+ }
+ return arrow::Status::OK();
+ }
+
+ void Free(uint8_t* buffer, int64_t size) {
+ pool_->Free(buffer, size);
+ // FIXME: See ARROW-11143, currently method ::Free doesn't allow Status return
+ arrow::Status s = UpdateReservation(-size);
+ if (!s.ok()) {
+ ARROW_LOG(FATAL) << "Failed to update reservation while freeing bytes: "
+ << s.message();
+ return;
+ }
+ }
+
+ arrow::Status UpdateReservation(int64_t diff) {
+ int64_t granted = Reserve(diff);
+ if (granted == 0) {
+ return arrow::Status::OK();
+ }
+ if (granted < 0) {
+ RETURN_NOT_OK(listener_->OnRelease(-granted));
+ return arrow::Status::OK();
+ }
+ RETURN_NOT_OK(listener_->OnReservation(granted));
+ return arrow::Status::OK();
+ }
+
+ int64_t Reserve(int64_t diff) {
+ std::lock_guard<std::mutex> lock(mutex_);
+ bytes_reserved_ += diff;
+ int64_t new_block_count;
+ if (bytes_reserved_ == 0) {
+ new_block_count = 0;
+ } else {
+ // ceil to get the required block number
+ new_block_count = (bytes_reserved_ - 1) / block_size_ + 1;
+ }
+ int64_t bytes_granted = (new_block_count - blocks_reserved_) * block_size_;
+ blocks_reserved_ = new_block_count;
+ return bytes_granted;
+ }
+
+ int64_t bytes_allocated() { return pool_->bytes_allocated(); }
+
+ int64_t max_memory() { return pool_->max_memory(); }
+
+ std::string backend_name() { return pool_->backend_name(); }
+
+ std::shared_ptr<ReservationListener> get_listener() { return listener_; }
+
+ private:
+ arrow::MemoryPool* pool_;
+ std::shared_ptr<ReservationListener> listener_;
+ int64_t block_size_;
+ int64_t blocks_reserved_;
+ int64_t bytes_reserved_;
+ std::mutex mutex_;
+};
+
+/// A memory pool implementation for pre-reserving memory blocks from a
Review comment:
but if there aren't any tests that test this functionality, then I would recommend splitting this out to another .h/.cc file under this directory so tests can be written.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] zhztheplayer commented on a change in pull request #7030: ARROW-7808: [Java][Dataset] Implement Dataset Java API by JNI to C++
Posted by GitBox <gi...@apache.org>.
zhztheplayer commented on a change in pull request #7030:
URL: https://github.com/apache/arrow/pull/7030#discussion_r567756483
##########
File path: cpp/src/jni/dataset/jni_wrapper.cpp
##########
@@ -0,0 +1,859 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include <mutex>
+
+#include <arrow/array.h>
+#include <arrow/dataset/api.h>
+#include <arrow/dataset/file_base.h>
+#include <arrow/filesystem/localfs.h>
+#include <arrow/io/api.h>
+#include <arrow/ipc/api.h>
+#include <arrow/util/iterator.h>
+#include <arrow/util/logging.h>
+
+#include "org_apache_arrow_dataset_file_JniWrapper.h"
+#include "org_apache_arrow_dataset_jni_JniWrapper.h"
+#include "org_apache_arrow_dataset_jni_NativeMemoryPool.h"
+
+namespace arrow {
+namespace dataset {
+namespace jni {
+
+static jclass illegal_access_exception_class;
+static jclass illegal_argument_exception_class;
+static jclass runtime_exception_class;
+
+static jclass record_batch_handle_class;
+static jclass record_batch_handle_field_class;
+static jclass record_batch_handle_buffer_class;
+static jclass java_reservation_listener_class;
+
+static jmethodID record_batch_handle_constructor;
+static jmethodID record_batch_handle_field_constructor;
+static jmethodID record_batch_handle_buffer_constructor;
+static jmethodID reserve_memory_method;
+static jmethodID unreserve_memory_method;
+
+static jlong default_memory_pool_id = -1L;
+
+static jint JNI_VERSION = JNI_VERSION_1_6;
+
+class JniPendingException : public std::runtime_error {
+ public:
+ explicit JniPendingException(const std::string& arg) : runtime_error(arg) {}
+};
+
+void ThrowPendingException(const std::string& message) {
+ throw JniPendingException(message);
+}
+
+template <typename T>
+T JniGetOrThrow(arrow::Result<T> result) {
+ if (!result.status().ok()) {
+ ThrowPendingException(result.status().message());
+ }
+ return std::move(result).ValueOrDie();
+}
+
+void JniAssertOkOrThrow(arrow::Status status) {
+ if (!status.ok()) {
+ ThrowPendingException(status.message());
+ }
+}
+
+void JniThrow(std::string message) { ThrowPendingException(message); }
+
+jclass CreateGlobalClassReference(JNIEnv* env, const char* class_name) {
+ jclass local_class = env->FindClass(class_name);
+ jclass global_class = (jclass)env->NewGlobalRef(local_class);
+ env->DeleteLocalRef(local_class);
+ return global_class;
+}
+
+arrow::Result<jmethodID> GetMethodID(JNIEnv* env, jclass this_class, const char* name,
+ const char* sig) {
+ jmethodID ret = env->GetMethodID(this_class, name, sig);
+ if (ret == nullptr) {
+ std::string error_message = "Unable to find method " + std::string(name) +
+ " within signature" + std::string(sig);
+ return arrow::Status::Invalid(error_message);
+ }
+ return ret;
+}
+
+arrow::Result<jmethodID> GetStaticMethodID(JNIEnv* env, jclass this_class,
+ const char* name, const char* sig) {
+ jmethodID ret = env->GetStaticMethodID(this_class, name, sig);
+ if (ret == nullptr) {
+ std::string error_message = "Unable to find static method " + std::string(name) +
+ " within signature" + std::string(sig);
+ return arrow::Status::Invalid(error_message);
+ }
+ return ret;
+}
+
+std::shared_ptr<arrow::Schema> SchemaFromColumnNames(
+ const std::shared_ptr<arrow::Schema>& input,
+ const std::vector<std::string>& column_names) {
+ std::vector<std::shared_ptr<arrow::Field>> columns;
+ for (const auto& name : column_names) {
+ columns.push_back(input->GetFieldByName(name));
+ }
+ return std::make_shared<arrow::Schema>(columns);
+}
+
+arrow::Result<std::shared_ptr<arrow::dataset::FileFormat>> GetFileFormat(jint id) {
+ switch (id) {
+ case 0:
+ return std::make_shared<arrow::dataset::ParquetFileFormat>();
+ default:
+ std::string error_message = "illegal file format id: " + std::to_string(id);
+ return arrow::Status::Invalid(error_message);
+ }
+}
+
+std::string JStringToCString(JNIEnv* env, jstring string) {
+ if (string == nullptr) {
+ return std::string();
+ }
+ jboolean copied;
+ const char* chars = env->GetStringUTFChars(string, &copied);
+ std::string ret = strdup(chars);
Review comment:
done
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] zhztheplayer commented on a change in pull request #7030: ARROW-7808: [Java][Dataset] Implement Datasets Java API by JNI to C++
Posted by GitBox <gi...@apache.org>.
zhztheplayer commented on a change in pull request #7030:
URL: https://github.com/apache/arrow/pull/7030#discussion_r416409003
##########
File path: cpp/src/jni/dataset/concurrent_map.h
##########
@@ -0,0 +1,81 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ */
+
+#ifndef JNI_ID_TO_MODULE_MAP_H
+#define JNI_ID_TO_MODULE_MAP_H
+
+#include <memory>
+#include <mutex>
+#include <unordered_map>
+#include <utility>
+
+#include "jni.h"
+#include "arrow/util/macros.h"
+
+namespace arrow {
Review comment:
the file has been removed
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] emkornfield commented on a change in pull request #7030: ARROW-7808: [Java][Dataset] Implement Datasets Java API by JNI to C++
Posted by GitBox <gi...@apache.org>.
emkornfield commented on a change in pull request #7030:
URL: https://github.com/apache/arrow/pull/7030#discussion_r425568786
##########
File path: java/dataset/src/test/java/org/apache/arrow/dataset/file/TestSingleFileDataset.java
##########
@@ -0,0 +1,237 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.arrow.dataset.file;
+
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertTrue;
+
+import java.util.Collections;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.Objects;
+import java.util.stream.Collectors;
+
+import org.apache.arrow.dataset.DatasetTypes;
+import org.apache.arrow.dataset.filter.Filter;
+import org.apache.arrow.dataset.jni.TestNativeDataset;
+import org.apache.arrow.dataset.scanner.ScanOptions;
+import org.apache.arrow.util.AutoCloseables;
+import org.apache.arrow.vector.FieldVector;
+import org.apache.arrow.vector.VectorLoader;
+import org.apache.arrow.vector.VectorSchemaRoot;
+import org.apache.arrow.vector.ipc.message.ArrowRecordBatch;
+import org.apache.arrow.vector.types.Types;
+import org.apache.arrow.vector.types.pojo.Schema;
+import org.apache.avro.generic.GenericData;
+import org.apache.avro.generic.GenericRecord;
+import org.apache.avro.generic.GenericRecordBuilder;
+import org.junit.ClassRule;
+import org.junit.Test;
+import org.junit.rules.TemporaryFolder;
+
+import com.google.common.collect.ImmutableList;
+
+public class TestSingleFileDataset extends TestNativeDataset {
+
+ @ClassRule
+ public static final TemporaryFolder TMP = new TemporaryFolder();
+
+ public static final String AVRO_SCHEMA_USER = "user.avsc";
+
+ @Test
+ public void testParquetRead() throws Exception {
+ ParquetWriteSupport writeSupport = new ParquetWriteSupport(AVRO_SCHEMA_USER, TMP.newFolder());
+ GenericRecord record = new GenericData.Record(writeSupport.getAvroSchema());
+ record.put("id", 1);
+ record.put("name", "a");
+ writeSupport.writeRecord(record);
+ writeSupport.close();
+
+ SingleFileDatasetFactory factory = new SingleFileDatasetFactory(rootAllocator(),
+ FileFormat.PARQUET, FileSystem.LOCAL, writeSupport.getOutputFile());
+ ScanOptions options = new ScanOptions(new String[0], Filter.EMPTY, 100);
+
+ Schema schema = inferResultSchemaFromFactory(factory, options);
+ assertEquals(2, schema.getFields().size());
+ assertEquals("id", schema.getFields().get(0).getName());
+ assertEquals("name", schema.getFields().get(1).getName());
+ assertEquals(Types.MinorType.INT.getType(), schema.getFields().get(0).getType());
+ assertEquals(Types.MinorType.VARCHAR.getType(), schema.getFields().get(1).getType());
+
+ assertSingleTaskProduced(factory, options);
+
+ List<ArrowRecordBatch> datum = collectResultFromFactory(factory, options);
+ assertEquals(1, datum.size());
+ checkParquetReadResult(schema, Collections.singletonList(record), datum);
+ AutoCloseables.close(datum);
+ }
+
+ @Test
+ public void testParquetProjector() throws Exception {
+ ParquetWriteSupport writeSupport = new ParquetWriteSupport(AVRO_SCHEMA_USER, TMP.newFolder());
Review comment:
I think it would remove a lot of boiler plate from tests if you had a method that looked something like.
ParquetWriteSupport writeSupport = UserIdNameParquetWriter.writeTempFile(/*id=*/1, /*name=*/"a", /*id=*/2, /*name=*/"b");
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] wesm commented on a change in pull request #7030: ARROW-7808: [Java][Dataset] Implement Datasets Java API by JNI to C++
Posted by GitBox <gi...@apache.org>.
wesm commented on a change in pull request #7030:
URL: https://github.com/apache/arrow/pull/7030#discussion_r417390692
##########
File path: cpp/src/jni/dataset/proto/Types.proto
##########
@@ -0,0 +1,149 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+syntax = "proto2";
+package types;
+
+option java_package = "org.apache.arrow.dataset";
Review comment:
I haven't reviewed the Java code, so it sounds like things are okay, I just wanted to warn you about the likely instability of the C++ API
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] zhztheplayer commented on pull request #7030: ARROW-7808: [Java][Dataset] Implement Datasets Java API by JNI to C++
Posted by GitBox <gi...@apache.org>.
zhztheplayer commented on pull request #7030:
URL: https://github.com/apache/arrow/pull/7030#issuecomment-678839259
@emkornfield Sorry for the late reply. And yes I was planing to try adding a `Bits.java` based implementation to this PR, but I may not be able to working on it instantly as I am occupied by some other projects these days. Also I suppose if we want to consider block level memory reservation then we should dive into jemalloc and use some thing like a hook API right? That will appear to take me sometime as I am new to jemalloc :)
If we'd like to see the feature in a release quickly maybe we can possibly resolve this one in advance and I could continue working on the above in a separated JIRA ticket. Either way is fine to me.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] wesm commented on pull request #7030: ARROW-7808: [Java][Dataset] Implement Datasets Java API by JNI to C++
Posted by GitBox <gi...@apache.org>.
wesm commented on pull request #7030:
URL: https://github.com/apache/arrow/pull/7030#issuecomment-656352612
I think @jacques-n may be on vacation, so responses could be delayed -- this needs nother rebase in the meantime, though
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] emkornfield commented on a change in pull request #7030: ARROW-7808: [Java][Dataset] Implement Datasets Java API
Posted by GitBox <gi...@apache.org>.
emkornfield commented on a change in pull request #7030:
URL: https://github.com/apache/arrow/pull/7030#discussion_r414987685
##########
File path: java/dataset/src/main/java/org/apache/arrow/dataset/scanner/ScanTask.java
##########
@@ -0,0 +1,42 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.arrow.dataset.scanner;
+
+import java.util.Iterator;
+
+import org.apache.arrow.vector.VectorSchemaRoot;
+
+/**
+ * Read record batches from a range of a single data fragment. A
+ * ScanTask is meant to be a unit of work to be dispatched. The implementation
+ * must be thread and concurrent safe.
+ */
+public interface ScanTask {
+
+ /**
+ * Creates and returns a {@link Itr} instance.
+ */
+ Itr scan();
+
+ /**
+ * The iterator implementation for {@link VectorSchemaRoot}s.
+ */
+ interface Itr extends Iterator<VectorSchemaRoot>, AutoCloseable {
Review comment:
can we call this "Iterator"?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] emkornfield commented on a change in pull request #7030: ARROW-7808: [Java][Dataset] Implement Datasets Java API
Posted by GitBox <gi...@apache.org>.
emkornfield commented on a change in pull request #7030:
URL: https://github.com/apache/arrow/pull/7030#discussion_r414983499
##########
File path: cpp/src/jni/dataset/concurrent_map.h
##########
@@ -0,0 +1,81 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ */
+
+#ifndef JNI_ID_TO_MODULE_MAP_H
+#define JNI_ID_TO_MODULE_MAP_H
+
+#include <memory>
+#include <mutex>
+#include <unordered_map>
+#include <utility>
+
+#include "jni.h"
+#include "arrow/util/macros.h"
+
+namespace arrow {
Review comment:
Please do not copy and paste this file, factor it out to a common place that can be used by all JNI implementations.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] zhztheplayer commented on a change in pull request #7030: ARROW-7808: [Java][Dataset] Implement Dataset Java API by JNI to C++
Posted by GitBox <gi...@apache.org>.
zhztheplayer commented on a change in pull request #7030:
URL: https://github.com/apache/arrow/pull/7030#discussion_r567768143
##########
File path: cpp/src/jni/dataset/jni_wrapper.cpp
##########
@@ -0,0 +1,859 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include <mutex>
+
+#include <arrow/array.h>
+#include <arrow/dataset/api.h>
+#include <arrow/dataset/file_base.h>
+#include <arrow/filesystem/localfs.h>
+#include <arrow/io/api.h>
+#include <arrow/ipc/api.h>
+#include <arrow/util/iterator.h>
+#include <arrow/util/logging.h>
+
+#include "org_apache_arrow_dataset_file_JniWrapper.h"
+#include "org_apache_arrow_dataset_jni_JniWrapper.h"
+#include "org_apache_arrow_dataset_jni_NativeMemoryPool.h"
+
+namespace arrow {
+namespace dataset {
+namespace jni {
+
+static jclass illegal_access_exception_class;
+static jclass illegal_argument_exception_class;
+static jclass runtime_exception_class;
+
+static jclass record_batch_handle_class;
+static jclass record_batch_handle_field_class;
+static jclass record_batch_handle_buffer_class;
+static jclass java_reservation_listener_class;
+
+static jmethodID record_batch_handle_constructor;
+static jmethodID record_batch_handle_field_constructor;
+static jmethodID record_batch_handle_buffer_constructor;
+static jmethodID reserve_memory_method;
+static jmethodID unreserve_memory_method;
+
+static jlong default_memory_pool_id = -1L;
+
+static jint JNI_VERSION = JNI_VERSION_1_6;
+
+class JniPendingException : public std::runtime_error {
+ public:
+ explicit JniPendingException(const std::string& arg) : runtime_error(arg) {}
+};
+
+void ThrowPendingException(const std::string& message) {
+ throw JniPendingException(message);
+}
+
+template <typename T>
+T JniGetOrThrow(arrow::Result<T> result) {
+ if (!result.status().ok()) {
+ ThrowPendingException(result.status().message());
+ }
+ return std::move(result).ValueOrDie();
+}
+
+void JniAssertOkOrThrow(arrow::Status status) {
+ if (!status.ok()) {
+ ThrowPendingException(status.message());
+ }
+}
+
+void JniThrow(std::string message) { ThrowPendingException(message); }
+
+jclass CreateGlobalClassReference(JNIEnv* env, const char* class_name) {
+ jclass local_class = env->FindClass(class_name);
+ jclass global_class = (jclass)env->NewGlobalRef(local_class);
+ env->DeleteLocalRef(local_class);
+ return global_class;
+}
+
+arrow::Result<jmethodID> GetMethodID(JNIEnv* env, jclass this_class, const char* name,
+ const char* sig) {
+ jmethodID ret = env->GetMethodID(this_class, name, sig);
+ if (ret == nullptr) {
+ std::string error_message = "Unable to find method " + std::string(name) +
+ " within signature" + std::string(sig);
+ return arrow::Status::Invalid(error_message);
+ }
+ return ret;
+}
+
+arrow::Result<jmethodID> GetStaticMethodID(JNIEnv* env, jclass this_class,
+ const char* name, const char* sig) {
+ jmethodID ret = env->GetStaticMethodID(this_class, name, sig);
+ if (ret == nullptr) {
+ std::string error_message = "Unable to find static method " + std::string(name) +
+ " within signature" + std::string(sig);
+ return arrow::Status::Invalid(error_message);
+ }
+ return ret;
+}
+
+std::shared_ptr<arrow::Schema> SchemaFromColumnNames(
+ const std::shared_ptr<arrow::Schema>& input,
+ const std::vector<std::string>& column_names) {
+ std::vector<std::shared_ptr<arrow::Field>> columns;
+ for (const auto& name : column_names) {
+ columns.push_back(input->GetFieldByName(name));
+ }
+ return std::make_shared<arrow::Schema>(columns);
+}
+
+arrow::Result<std::shared_ptr<arrow::dataset::FileFormat>> GetFileFormat(jint id) {
+ switch (id) {
+ case 0:
+ return std::make_shared<arrow::dataset::ParquetFileFormat>();
+ default:
+ std::string error_message = "illegal file format id: " + std::to_string(id);
+ return arrow::Status::Invalid(error_message);
+ }
+}
+
+std::string JStringToCString(JNIEnv* env, jstring string) {
+ if (string == nullptr) {
+ return std::string();
+ }
+ jboolean copied;
+ const char* chars = env->GetStringUTFChars(string, &copied);
+ std::string ret = strdup(chars);
+ env->ReleaseStringUTFChars(string, chars);
+ return ret;
+}
+
+std::vector<std::string> ToStringVector(JNIEnv* env, jobjectArray& str_array) {
+ int length = env->GetArrayLength(str_array);
+ std::vector<std::string> vector;
+ for (int i = 0; i < length; i++) {
+ auto string = (jstring)(env->GetObjectArrayElement(str_array, i));
+ vector.push_back(JStringToCString(env, string));
+ }
+ return vector;
+}
+
+template <typename T>
+jlong CreateNativeRef(std::shared_ptr<T> t) {
+ std::shared_ptr<T>* retained_ptr = new std::shared_ptr<T>(t);
+ return reinterpret_cast<jlong>(retained_ptr);
+}
+
+template <typename T>
+std::shared_ptr<T> RetrieveNativeInstance(jlong ref) {
+ std::shared_ptr<T>* retrieved_ptr = reinterpret_cast<std::shared_ptr<T>*>(ref);
+ return *retrieved_ptr;
+}
+
+template <typename T>
+void ReleaseNativeRef(jlong ref) {
+ std::shared_ptr<T>* retrieved_ptr = reinterpret_cast<std::shared_ptr<T>*>(ref);
+ delete retrieved_ptr;
+}
+
+arrow::Result<jbyteArray> ToSchemaByteArray(JNIEnv* env,
+ std::shared_ptr<arrow::Schema> schema) {
+ ARROW_ASSIGN_OR_RAISE(
+ std::shared_ptr<arrow::Buffer> buffer,
+ arrow::ipc::SerializeSchema(*schema, arrow::default_memory_pool()))
+
+ jbyteArray out = env->NewByteArray(buffer->size());
+ auto src = reinterpret_cast<const jbyte*>(buffer->data());
+ env->SetByteArrayRegion(out, 0, buffer->size(), src);
+ return out;
+}
+
+arrow::Result<std::shared_ptr<arrow::Schema>> FromSchemaByteArray(
+ JNIEnv* env, jbyteArray schemaBytes) {
+ arrow::ipc::DictionaryMemo in_memo;
+ int schemaBytes_len = env->GetArrayLength(schemaBytes);
+ jbyte* schemaBytes_data = env->GetByteArrayElements(schemaBytes, nullptr);
+ auto serialized_schema = std::make_shared<arrow::Buffer>(
+ reinterpret_cast<uint8_t*>(schemaBytes_data), schemaBytes_len);
+ arrow::io::BufferReader buf_reader(serialized_schema);
+ ARROW_ASSIGN_OR_RAISE(std::shared_ptr<arrow::Schema> schema,
+ arrow::ipc::ReadSchema(&buf_reader, &in_memo))
+ env->ReleaseByteArrayElements(schemaBytes, schemaBytes_data, JNI_ABORT);
+ return schema;
+}
+
+/// Listener to act on reservations/unreservations from ReservationListenableMemoryPool.
+///
+/// Note the memory pool will call this listener only on block-level memory
+/// reservation/unreservation is granted. So the invocation parameter "size" is always
+/// multiple of block size (by default, 512k) specified in memory pool.
+class ReservationListener {
+ public:
+ virtual ~ReservationListener() = default;
+
+ virtual arrow::Status OnReservation(int64_t size) = 0;
+ virtual arrow::Status OnRelease(int64_t size) = 0;
+
+ protected:
+ ReservationListener() = default;
+};
+
+class ReservationListenableMemoryPoolImpl {
+ public:
+ explicit ReservationListenableMemoryPoolImpl(
+ arrow::MemoryPool* pool, std::shared_ptr<ReservationListener> listener,
+ int64_t block_size)
+ : pool_(pool),
+ listener_(listener),
+ block_size_(block_size),
+ blocks_reserved_(0),
+ bytes_reserved_(0) {}
+
+ arrow::Status Allocate(int64_t size, uint8_t** out) {
+ RETURN_NOT_OK(UpdateReservation(size));
+ arrow::Status error = pool_->Allocate(size, out);
+ if (!error.ok()) {
+ RETURN_NOT_OK(UpdateReservation(-size));
+ return error;
+ }
+ return arrow::Status::OK();
+ }
+
+ arrow::Status Reallocate(int64_t old_size, int64_t new_size, uint8_t** ptr) {
+ bool reserved = false;
+ int64_t diff = new_size - old_size;
+ if (new_size >= old_size) {
+ // new_size >= old_size, pre-reserve bytes from listener before allocating
+ // from underlying pool
+ RETURN_NOT_OK(UpdateReservation(diff));
+ reserved = true;
+ }
+ arrow::Status error = pool_->Reallocate(old_size, new_size, ptr);
+ if (!error.ok()) {
+ if (reserved) {
+ // roll back reservations on error
+ RETURN_NOT_OK(UpdateReservation(-diff));
+ }
+ return error;
+ }
+ if (!reserved) {
+ // otherwise (e.g. new_size < old_size), make updates after calling underlying pool
+ RETURN_NOT_OK(UpdateReservation(diff));
+ }
+ return arrow::Status::OK();
+ }
+
+ void Free(uint8_t* buffer, int64_t size) {
+ pool_->Free(buffer, size);
+ // FIXME: See ARROW-11143, currently method ::Free doesn't allow Status return
+ arrow::Status s = UpdateReservation(-size);
+ if (!s.ok()) {
+ ARROW_LOG(FATAL) << "Failed to update reservation while freeing bytes: "
+ << s.message();
+ return;
+ }
+ }
+
+ arrow::Status UpdateReservation(int64_t diff) {
+ int64_t granted = Reserve(diff);
+ if (granted == 0) {
+ return arrow::Status::OK();
+ }
+ if (granted < 0) {
+ RETURN_NOT_OK(listener_->OnRelease(-granted));
+ return arrow::Status::OK();
+ }
+ RETURN_NOT_OK(listener_->OnReservation(granted));
+ return arrow::Status::OK();
+ }
+
+ int64_t Reserve(int64_t diff) {
+ std::lock_guard<std::mutex> lock(mutex_);
+ bytes_reserved_ += diff;
+ int64_t new_block_count;
+ if (bytes_reserved_ == 0) {
+ new_block_count = 0;
+ } else {
+ // ceil to get the required block number
+ new_block_count = (bytes_reserved_ - 1) / block_size_ + 1;
+ }
+ int64_t bytes_granted = (new_block_count - blocks_reserved_) * block_size_;
+ blocks_reserved_ = new_block_count;
+ return bytes_granted;
+ }
+
+ int64_t bytes_allocated() { return pool_->bytes_allocated(); }
+
+ int64_t max_memory() { return pool_->max_memory(); }
+
+ std::string backend_name() { return pool_->backend_name(); }
+
+ std::shared_ptr<ReservationListener> get_listener() { return listener_; }
+
+ private:
+ arrow::MemoryPool* pool_;
+ std::shared_ptr<ReservationListener> listener_;
+ int64_t block_size_;
+ int64_t blocks_reserved_;
+ int64_t bytes_reserved_;
+ std::mutex mutex_;
+};
+
+/// A memory pool implementation for pre-reserving memory blocks from a
+/// customizable listener. This will typically be used when memory allocations
+/// have to subject to another "virtual" resource manager, which just tracks or
+/// limits number of bytes of application's overall memory usage. The underlying
+/// memory pool will still be responsible for actual malloc/free operations.
+class ReservationListenableMemoryPool : public arrow::MemoryPool {
+ public:
+ /// \brief Constructor.
+ ///
+ /// \param[in] pool the underlying memory pool
+ /// \param[in] listener a listener for block-level reservations/releases.
+ /// \param[in] block_size size of each block to reserve from the listener
+ explicit ReservationListenableMemoryPool(MemoryPool* pool,
+ std::shared_ptr<ReservationListener> listener,
+ int64_t block_size = 512 * 1024) {
+ impl_.reset(new ReservationListenableMemoryPoolImpl(pool, listener, block_size));
+ }
+
+ arrow::Status Allocate(int64_t size, uint8_t** out) override {
+ return impl_->Allocate(size, out);
+ }
+
+ arrow::Status Reallocate(int64_t old_size, int64_t new_size, uint8_t** ptr) override {
+ return impl_->Reallocate(old_size, new_size, ptr);
+ }
+
+ void Free(uint8_t* buffer, int64_t size) override { return impl_->Free(buffer, size); }
+
+ int64_t bytes_allocated() const override { return impl_->bytes_allocated(); }
+
+ int64_t max_memory() const override { return impl_->max_memory(); }
+
+ std::string backend_name() const override { return impl_->backend_name(); }
+
+ std::shared_ptr<ReservationListener> get_listener() { return impl_->get_listener(); }
+
+ private:
+ std::unique_ptr<ReservationListenableMemoryPoolImpl> impl_;
+};
+
+class ReserveFromJava : public ReservationListener {
+ public:
+ ReserveFromJava(JavaVM* vm, jobject java_reservation_listener)
+ : vm_(vm), java_reservation_listener_(java_reservation_listener) {}
+
+ arrow::Status OnReservation(int64_t size) override {
+ JNIEnv* env;
+ if (vm_->GetEnv(reinterpret_cast<void**>(&env), JNI_VERSION) != JNI_OK) {
+ return arrow::Status::Invalid("JNIEnv was not attached to current thread");
+ }
+ env->CallObjectMethod(java_reservation_listener_, reserve_memory_method, size);
+ if (env->ExceptionCheck()) {
+ env->ExceptionDescribe();
+ env->ExceptionClear();
+ return arrow::Status::Invalid("Error calling Java side reservation listener");
+ }
+ return arrow::Status::OK();
+ }
+
+ arrow::Status OnRelease(int64_t size) override {
+ JNIEnv* env;
+ if (vm_->GetEnv(reinterpret_cast<void**>(&env), JNI_VERSION) != JNI_OK) {
+ return arrow::Status::Invalid("JNIEnv was not attached to current thread");
+ }
+ env->CallObjectMethod(java_reservation_listener_, unreserve_memory_method, size);
+ if (env->ExceptionCheck()) {
+ env->ExceptionDescribe();
+ env->ExceptionClear();
+ return arrow::Status::Invalid("Error calling Java side reservation listener");
+ }
+ return arrow::Status::OK();
+ }
+
+ jobject GetJavaReservationListener() { return java_reservation_listener_; }
+
+ private:
+ JavaVM* vm_;
+ jobject java_reservation_listener_;
+};
+
+/// \class DisposableScannerAdaptor
+/// \brief An adaptor that iterates over a Scanner instance then returns RecordBatches
+/// directly.
+///
+/// This lessens the complexity of the JNI bridge to make sure it to be easier to
+/// maintain. On Java-side, NativeScanner can only produces a single NativeScanTask
+/// instance during its whole lifecycle. Each task stands for a DisposableScannerAdaptor
+/// instance through JNI bridge.
+///
+class DisposableScannerAdaptor {
+ public:
+ DisposableScannerAdaptor(std::shared_ptr<arrow::dataset::Scanner> scanner,
+ arrow::dataset::ScanTaskIterator task_itr) {
+ this->scanner_ = std::move(scanner);
+ this->task_itr_ = std::move(task_itr);
+ }
+
+ static arrow::Result<std::shared_ptr<DisposableScannerAdaptor>> Create(
+ std::shared_ptr<arrow::dataset::Scanner> scanner) {
+ ARROW_ASSIGN_OR_RAISE(arrow::dataset::ScanTaskIterator task_itr, scanner->Scan())
+ return std::make_shared<DisposableScannerAdaptor>(scanner, std::move(task_itr));
+ }
+
+ arrow::Result<std::shared_ptr<arrow::RecordBatch>> Next() {
+ do {
+ ARROW_ASSIGN_OR_RAISE(std::shared_ptr<arrow::RecordBatch> batch, NextBatch())
+ if (batch != nullptr) {
+ return batch;
+ }
+ // batch is null, current task is fully consumed
+ ARROW_ASSIGN_OR_RAISE(bool has_next_task, NextTask())
+ if (!has_next_task) {
+ // no more tasks
+ return nullptr;
+ }
+ // new task appended, read again
+ } while (true);
+ }
+
+ const std::shared_ptr<arrow::dataset::Scanner>& GetScanner() const { return scanner_; }
+
+ protected:
Review comment:
done
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] emkornfield commented on a change in pull request #7030: ARROW-7808: [Java][Dataset] Implement Dataset Java API by JNI to C++
Posted by GitBox <gi...@apache.org>.
emkornfield commented on a change in pull request #7030:
URL: https://github.com/apache/arrow/pull/7030#discussion_r573059320
##########
File path: cpp/src/jni/dataset/jni_util.h
##########
@@ -0,0 +1,143 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#pragma once
+
+#include "arrow/array.h"
+#include "arrow/io/api.h"
+#include "arrow/ipc/api.h"
+#include "arrow/memory_pool.h"
+#include "arrow/result.h"
+#include "arrow/type.h"
+
+#include <jni.h>
+
+namespace arrow {
+namespace dataset {
+namespace jni {
+
+class JniPendingException : public std::runtime_error {
Review comment:
yeah, I think this is a good thought and it would be great to see some refactoring here to share common code. Thank you for taking the time to address all the concerns around this.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] zhztheplayer commented on a change in pull request #7030: ARROW-7808: [Java][Dataset] Implement Dataset Java API by JNI to C++
Posted by GitBox <gi...@apache.org>.
zhztheplayer commented on a change in pull request #7030:
URL: https://github.com/apache/arrow/pull/7030#discussion_r569376564
##########
File path: cpp/src/jni/dataset/jni_wrapper.cpp
##########
@@ -0,0 +1,859 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include <mutex>
+
+#include <arrow/array.h>
+#include <arrow/dataset/api.h>
+#include <arrow/dataset/file_base.h>
+#include <arrow/filesystem/localfs.h>
+#include <arrow/io/api.h>
+#include <arrow/ipc/api.h>
+#include <arrow/util/iterator.h>
+#include <arrow/util/logging.h>
+
+#include "org_apache_arrow_dataset_file_JniWrapper.h"
+#include "org_apache_arrow_dataset_jni_JniWrapper.h"
+#include "org_apache_arrow_dataset_jni_NativeMemoryPool.h"
+
+namespace arrow {
+namespace dataset {
+namespace jni {
+
+static jclass illegal_access_exception_class;
+static jclass illegal_argument_exception_class;
+static jclass runtime_exception_class;
+
+static jclass record_batch_handle_class;
+static jclass record_batch_handle_field_class;
+static jclass record_batch_handle_buffer_class;
+static jclass java_reservation_listener_class;
+
+static jmethodID record_batch_handle_constructor;
+static jmethodID record_batch_handle_field_constructor;
+static jmethodID record_batch_handle_buffer_constructor;
+static jmethodID reserve_memory_method;
+static jmethodID unreserve_memory_method;
+
+static jlong default_memory_pool_id = -1L;
+
+static jint JNI_VERSION = JNI_VERSION_1_6;
+
+class JniPendingException : public std::runtime_error {
+ public:
+ explicit JniPendingException(const std::string& arg) : runtime_error(arg) {}
+};
+
+void ThrowPendingException(const std::string& message) {
+ throw JniPendingException(message);
+}
+
+template <typename T>
+T JniGetOrThrow(arrow::Result<T> result) {
+ if (!result.status().ok()) {
+ ThrowPendingException(result.status().message());
+ }
+ return std::move(result).ValueOrDie();
+}
+
+void JniAssertOkOrThrow(arrow::Status status) {
+ if (!status.ok()) {
+ ThrowPendingException(status.message());
+ }
+}
+
+void JniThrow(std::string message) { ThrowPendingException(message); }
+
+jclass CreateGlobalClassReference(JNIEnv* env, const char* class_name) {
+ jclass local_class = env->FindClass(class_name);
+ jclass global_class = (jclass)env->NewGlobalRef(local_class);
+ env->DeleteLocalRef(local_class);
+ return global_class;
+}
+
+arrow::Result<jmethodID> GetMethodID(JNIEnv* env, jclass this_class, const char* name,
+ const char* sig) {
+ jmethodID ret = env->GetMethodID(this_class, name, sig);
+ if (ret == nullptr) {
+ std::string error_message = "Unable to find method " + std::string(name) +
+ " within signature" + std::string(sig);
+ return arrow::Status::Invalid(error_message);
+ }
+ return ret;
+}
+
+arrow::Result<jmethodID> GetStaticMethodID(JNIEnv* env, jclass this_class,
+ const char* name, const char* sig) {
+ jmethodID ret = env->GetStaticMethodID(this_class, name, sig);
+ if (ret == nullptr) {
+ std::string error_message = "Unable to find static method " + std::string(name) +
+ " within signature" + std::string(sig);
+ return arrow::Status::Invalid(error_message);
+ }
+ return ret;
+}
+
+std::shared_ptr<arrow::Schema> SchemaFromColumnNames(
+ const std::shared_ptr<arrow::Schema>& input,
+ const std::vector<std::string>& column_names) {
+ std::vector<std::shared_ptr<arrow::Field>> columns;
+ for (const auto& name : column_names) {
+ columns.push_back(input->GetFieldByName(name));
+ }
+ return std::make_shared<arrow::Schema>(columns);
+}
+
+arrow::Result<std::shared_ptr<arrow::dataset::FileFormat>> GetFileFormat(jint id) {
+ switch (id) {
+ case 0:
+ return std::make_shared<arrow::dataset::ParquetFileFormat>();
+ default:
+ std::string error_message = "illegal file format id: " + std::to_string(id);
+ return arrow::Status::Invalid(error_message);
+ }
+}
+
+std::string JStringToCString(JNIEnv* env, jstring string) {
+ if (string == nullptr) {
+ return std::string();
+ }
+ jboolean copied;
+ const char* chars = env->GetStringUTFChars(string, &copied);
+ std::string ret = strdup(chars);
+ env->ReleaseStringUTFChars(string, chars);
+ return ret;
+}
+
+std::vector<std::string> ToStringVector(JNIEnv* env, jobjectArray& str_array) {
+ int length = env->GetArrayLength(str_array);
+ std::vector<std::string> vector;
+ for (int i = 0; i < length; i++) {
+ auto string = (jstring)(env->GetObjectArrayElement(str_array, i));
+ vector.push_back(JStringToCString(env, string));
+ }
+ return vector;
+}
+
+template <typename T>
+jlong CreateNativeRef(std::shared_ptr<T> t) {
+ std::shared_ptr<T>* retained_ptr = new std::shared_ptr<T>(t);
+ return reinterpret_cast<jlong>(retained_ptr);
+}
+
+template <typename T>
+std::shared_ptr<T> RetrieveNativeInstance(jlong ref) {
+ std::shared_ptr<T>* retrieved_ptr = reinterpret_cast<std::shared_ptr<T>*>(ref);
+ return *retrieved_ptr;
+}
+
+template <typename T>
+void ReleaseNativeRef(jlong ref) {
+ std::shared_ptr<T>* retrieved_ptr = reinterpret_cast<std::shared_ptr<T>*>(ref);
+ delete retrieved_ptr;
+}
+
+arrow::Result<jbyteArray> ToSchemaByteArray(JNIEnv* env,
+ std::shared_ptr<arrow::Schema> schema) {
+ ARROW_ASSIGN_OR_RAISE(
+ std::shared_ptr<arrow::Buffer> buffer,
+ arrow::ipc::SerializeSchema(*schema, arrow::default_memory_pool()))
+
+ jbyteArray out = env->NewByteArray(buffer->size());
+ auto src = reinterpret_cast<const jbyte*>(buffer->data());
+ env->SetByteArrayRegion(out, 0, buffer->size(), src);
+ return out;
+}
+
+arrow::Result<std::shared_ptr<arrow::Schema>> FromSchemaByteArray(
+ JNIEnv* env, jbyteArray schemaBytes) {
+ arrow::ipc::DictionaryMemo in_memo;
+ int schemaBytes_len = env->GetArrayLength(schemaBytes);
+ jbyte* schemaBytes_data = env->GetByteArrayElements(schemaBytes, nullptr);
+ auto serialized_schema = std::make_shared<arrow::Buffer>(
+ reinterpret_cast<uint8_t*>(schemaBytes_data), schemaBytes_len);
+ arrow::io::BufferReader buf_reader(serialized_schema);
+ ARROW_ASSIGN_OR_RAISE(std::shared_ptr<arrow::Schema> schema,
+ arrow::ipc::ReadSchema(&buf_reader, &in_memo))
+ env->ReleaseByteArrayElements(schemaBytes, schemaBytes_data, JNI_ABORT);
+ return schema;
+}
+
+/// Listener to act on reservations/unreservations from ReservationListenableMemoryPool.
+///
+/// Note the memory pool will call this listener only on block-level memory
+/// reservation/unreservation is granted. So the invocation parameter "size" is always
+/// multiple of block size (by default, 512k) specified in memory pool.
+class ReservationListener {
+ public:
+ virtual ~ReservationListener() = default;
+
+ virtual arrow::Status OnReservation(int64_t size) = 0;
+ virtual arrow::Status OnRelease(int64_t size) = 0;
+
+ protected:
+ ReservationListener() = default;
+};
+
+class ReservationListenableMemoryPoolImpl {
+ public:
+ explicit ReservationListenableMemoryPoolImpl(
+ arrow::MemoryPool* pool, std::shared_ptr<ReservationListener> listener,
+ int64_t block_size)
+ : pool_(pool),
+ listener_(listener),
+ block_size_(block_size),
+ blocks_reserved_(0),
+ bytes_reserved_(0) {}
+
+ arrow::Status Allocate(int64_t size, uint8_t** out) {
+ RETURN_NOT_OK(UpdateReservation(size));
+ arrow::Status error = pool_->Allocate(size, out);
+ if (!error.ok()) {
+ RETURN_NOT_OK(UpdateReservation(-size));
+ return error;
+ }
+ return arrow::Status::OK();
+ }
+
+ arrow::Status Reallocate(int64_t old_size, int64_t new_size, uint8_t** ptr) {
+ bool reserved = false;
+ int64_t diff = new_size - old_size;
+ if (new_size >= old_size) {
+ // new_size >= old_size, pre-reserve bytes from listener before allocating
+ // from underlying pool
+ RETURN_NOT_OK(UpdateReservation(diff));
+ reserved = true;
+ }
+ arrow::Status error = pool_->Reallocate(old_size, new_size, ptr);
+ if (!error.ok()) {
+ if (reserved) {
+ // roll back reservations on error
+ RETURN_NOT_OK(UpdateReservation(-diff));
+ }
+ return error;
+ }
+ if (!reserved) {
+ // otherwise (e.g. new_size < old_size), make updates after calling underlying pool
+ RETURN_NOT_OK(UpdateReservation(diff));
+ }
+ return arrow::Status::OK();
+ }
+
+ void Free(uint8_t* buffer, int64_t size) {
+ pool_->Free(buffer, size);
+ // FIXME: See ARROW-11143, currently method ::Free doesn't allow Status return
+ arrow::Status s = UpdateReservation(-size);
+ if (!s.ok()) {
+ ARROW_LOG(FATAL) << "Failed to update reservation while freeing bytes: "
+ << s.message();
+ return;
+ }
+ }
+
+ arrow::Status UpdateReservation(int64_t diff) {
+ int64_t granted = Reserve(diff);
+ if (granted == 0) {
+ return arrow::Status::OK();
+ }
+ if (granted < 0) {
+ RETURN_NOT_OK(listener_->OnRelease(-granted));
+ return arrow::Status::OK();
+ }
+ RETURN_NOT_OK(listener_->OnReservation(granted));
+ return arrow::Status::OK();
+ }
+
+ int64_t Reserve(int64_t diff) {
+ std::lock_guard<std::mutex> lock(mutex_);
+ bytes_reserved_ += diff;
+ int64_t new_block_count;
+ if (bytes_reserved_ == 0) {
+ new_block_count = 0;
+ } else {
+ // ceil to get the required block number
+ new_block_count = (bytes_reserved_ - 1) / block_size_ + 1;
+ }
+ int64_t bytes_granted = (new_block_count - blocks_reserved_) * block_size_;
+ blocks_reserved_ = new_block_count;
+ return bytes_granted;
+ }
+
+ int64_t bytes_allocated() { return pool_->bytes_allocated(); }
+
+ int64_t max_memory() { return pool_->max_memory(); }
+
+ std::string backend_name() { return pool_->backend_name(); }
+
+ std::shared_ptr<ReservationListener> get_listener() { return listener_; }
+
+ private:
+ arrow::MemoryPool* pool_;
+ std::shared_ptr<ReservationListener> listener_;
+ int64_t block_size_;
+ int64_t blocks_reserved_;
+ int64_t bytes_reserved_;
+ std::mutex mutex_;
+};
+
+/// A memory pool implementation for pre-reserving memory blocks from a
Review comment:
Added some unit tests
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] zhztheplayer commented on a change in pull request #7030: ARROW-7808: [Java][Dataset] Implement Datasets Java API by JNI to C++
Posted by GitBox <gi...@apache.org>.
zhztheplayer commented on a change in pull request #7030:
URL: https://github.com/apache/arrow/pull/7030#discussion_r440076921
##########
File path: java/dataset/src/main/java/org/apache/arrow/dataset/jni/NativeScanner.java
##########
@@ -0,0 +1,143 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.arrow.dataset.jni;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.NoSuchElementException;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.stream.Collectors;
+
+import org.apache.arrow.dataset.scanner.ScanTask;
+import org.apache.arrow.dataset.scanner.Scanner;
+import org.apache.arrow.memory.ArrowBuf;
+import org.apache.arrow.memory.BufferAllocator;
+import org.apache.arrow.memory.BufferLedger;
+import org.apache.arrow.memory.NativeUnderlingMemory;
+import org.apache.arrow.memory.Ownerships;
+import org.apache.arrow.vector.ipc.message.ArrowFieldNode;
+import org.apache.arrow.vector.ipc.message.ArrowRecordBatch;
+import org.apache.arrow.vector.types.pojo.Schema;
+import org.apache.arrow.vector.util.SchemaUtility;
+
+/**
+ * Native implementation of {@link Scanner}. Note that it currently emits only a single scan task of type
+ * {@link NativeScanTask}, which is internally a combination of all scan task instances returned by the
+ * native scanner.
+ */
+public class NativeScanner implements Scanner {
+
+ private final AtomicBoolean closed = new AtomicBoolean(false);
+ private final AtomicBoolean executed = new AtomicBoolean(false);
+ private final NativeContext context;
+ private final long scannerId;
+
+ public NativeScanner(NativeContext context, long scannerId) {
+ this.context = context;
+ this.scannerId = scannerId;
+ }
+
+ ScanTask.BatchIterator execute() {
+ if (closed.get()) {
+ throw new NativeInstanceClosedException();
+ }
+ if (!executed.compareAndSet(false, true)) {
+ throw new UnsupportedOperationException("NativeScanner cannot be executed more than once. Consider creating " +
+ "new scanner instead");
+ }
+ return new ScanTask.BatchIterator() {
+ private ArrowRecordBatch peek = null;
+
+ @Override
+ public void close() {
+ NativeScanner.this.close();
+ }
+
+ @Override
+ public boolean hasNext() {
+ if (closed.get()) {
+ throw new NativeInstanceClosedException();
+ }
+ if (peek != null) {
+ return true;
+ }
+ NativeRecordBatchHandle handle = JniWrapper.get().nextRecordBatch(scannerId);
+ if (handle == null) {
+ return false;
+ }
+ final ArrayList<ArrowBuf> buffers = new ArrayList<>();
+ for (NativeRecordBatchHandle.Buffer buffer : handle.getBuffers()) {
+ final BufferAllocator allocator = context.getAllocator();
+ final NativeUnderlingMemory am = NativeUnderlingMemory.create(allocator,
+ (int) buffer.size, buffer.nativeInstanceId, buffer.memoryAddress);
Review comment:
done
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] emkornfield commented on a change in pull request #7030: ARROW-7808: [Java][Dataset] Implement Datasets Java API by JNI to C++
Posted by GitBox <gi...@apache.org>.
emkornfield commented on a change in pull request #7030:
URL: https://github.com/apache/arrow/pull/7030#discussion_r425567714
##########
File path: java/dataset/src/test/java/org/apache/arrow/dataset/file/TestSingleFileDatasetFactory.java
##########
@@ -0,0 +1,36 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.arrow.dataset.file;
+
+import org.apache.arrow.memory.RootAllocator;
+import org.junit.Test;
+
+public class TestSingleFileDatasetFactory {
+
+ @Test(expected = RuntimeException.class)
+ public void testErrorHandling1() {
+ new SingleFileDatasetFactory(new RootAllocator(Long.MAX_VALUE),
+ FileFormat.NONE, FileSystem.LOCAL, "NON_EXIST_FILE");
+ }
+
+ @Test(expected = RuntimeException.class)
Review comment:
use assertThrows instead
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] emkornfield commented on a change in pull request #7030: ARROW-7808: [Java][Dataset] Implement Datasets Java API
Posted by GitBox <gi...@apache.org>.
emkornfield commented on a change in pull request #7030:
URL: https://github.com/apache/arrow/pull/7030#discussion_r414988831
##########
File path: java/pom.xml
##########
@@ -369,24 +369,24 @@
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-compiler-plugin</artifactId>
<version>3.6.2</version>
- <configuration>
- <compilerArgs>
- <arg>-XDcompilePolicy=simple</arg>
- <arg>-Xplugin:ErrorProne</arg>
- </compilerArgs>
- <annotationProcessorPaths>
- <path>
- <groupId>com.google.errorprone</groupId>
- <artifactId>error_prone_core</artifactId>
- <version>2.3.3</version>
- </path>
- <path>
- <groupId>org.immutables</groupId>
- <artifactId>value</artifactId>
- <version>2.8.2</version>
- </path>
- </annotationProcessorPaths>
- </configuration>
+<!-- <configuration>-->
Review comment:
revert this?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] zhztheplayer commented on a change in pull request #7030: ARROW-7808: [Java][Dataset] Implement Dataset Java API by JNI to C++
Posted by GitBox <gi...@apache.org>.
zhztheplayer commented on a change in pull request #7030:
URL: https://github.com/apache/arrow/pull/7030#discussion_r572093499
##########
File path: cpp/src/jni/dataset/jni_util.h
##########
@@ -0,0 +1,143 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#pragma once
+
+#include "arrow/array.h"
+#include "arrow/io/api.h"
+#include "arrow/ipc/api.h"
+#include "arrow/memory_pool.h"
+#include "arrow/result.h"
+#include "arrow/type.h"
+
+#include <jni.h>
+
+namespace arrow {
+namespace dataset {
+namespace jni {
+
+class JniPendingException : public std::runtime_error {
Review comment:
Moved these error handling tools back to jni_wrapper.cc.
I was thinking of preparing some common utilities for both `jni/dataset` `jni/orc` (or maybe `gandiva/jni`). But we can actually do this later in some other tickets. The utilities also can not work well without the macros in `jni_wrapper.cc`.
Added some docs to other interfaces in this header.
##########
File path: java/vector/src/main/java/org/apache/arrow/vector/util/SchemaUtility.java
##########
@@ -0,0 +1,62 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.arrow.vector.util;
+
+import java.io.ByteArrayOutputStream;
+import java.io.IOException;
+import java.nio.channels.Channels;
+
+import org.apache.arrow.memory.BufferAllocator;
+import org.apache.arrow.vector.ipc.ReadChannel;
+import org.apache.arrow.vector.ipc.WriteChannel;
+import org.apache.arrow.vector.ipc.message.MessageChannelReader;
+import org.apache.arrow.vector.ipc.message.MessageResult;
+import org.apache.arrow.vector.ipc.message.MessageSerializer;
+import org.apache.arrow.vector.types.pojo.Schema;
+
+/**
+ * Schema utility class including serialization and deserialization.
+ */
+public class SchemaUtility {
+
Review comment:
done
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] emkornfield commented on a change in pull request #7030: ARROW-7808: [Java][Dataset] Implement Datasets Java API by JNI to C++
Posted by GitBox <gi...@apache.org>.
emkornfield commented on a change in pull request #7030:
URL: https://github.com/apache/arrow/pull/7030#discussion_r425565876
##########
File path: java/dataset/src/main/java/org/apache/arrow/memory/NativeUnderlingMemory.java
##########
@@ -0,0 +1,65 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.arrow.memory;
+
+import org.apache.arrow.dataset.jni.JniWrapper;
+
+/**
+ * AllocationManager implementation for Native allocated memory.
+ */
+public class NativeUnderlingMemory extends AllocationManager {
Review comment:
sounds ok to me.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] emkornfield commented on a change in pull request #7030: ARROW-7808: [Java][Dataset] Implement Datasets Java API by JNI to C++
Posted by GitBox <gi...@apache.org>.
emkornfield commented on a change in pull request #7030:
URL: https://github.com/apache/arrow/pull/7030#discussion_r425569039
##########
File path: java/dataset/src/test/resources/avroschema/user.avsc
##########
@@ -0,0 +1,26 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+{
Review comment:
Given the use of only one schema, it might be easier to create this in Java? so readers don't need to look into two places?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] zhztheplayer commented on pull request #7030: ARROW-7808: [Java][Dataset] Implement Datasets Java API by JNI to C++
Posted by GitBox <gi...@apache.org>.
zhztheplayer commented on pull request #7030:
URL: https://github.com/apache/arrow/pull/7030#issuecomment-658593410
@jacques-n
Addressing a previous comment:
> We need to avoid this entirely. If you need some functionality, let's figure out what should be exposed. BaseAllocator should not be referenced.
I've opened [ARROW-9475](https://issues.apache.org/jira/browse/ARROW-9475) and #7768 . We can eliminate package-hackings in this PR by solving that one first.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] zhztheplayer commented on a change in pull request #7030: ARROW-7808: [Java][Dataset] Implement Dataset Java API by JNI to C++
Posted by GitBox <gi...@apache.org>.
zhztheplayer commented on a change in pull request #7030:
URL: https://github.com/apache/arrow/pull/7030#discussion_r567590469
##########
File path: cpp/src/jni/dataset/jni_wrapper.cpp
##########
@@ -0,0 +1,859 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include <mutex>
+
+#include <arrow/array.h>
+#include <arrow/dataset/api.h>
+#include <arrow/dataset/file_base.h>
+#include <arrow/filesystem/localfs.h>
+#include <arrow/io/api.h>
+#include <arrow/ipc/api.h>
+#include <arrow/util/iterator.h>
+#include <arrow/util/logging.h>
+
+#include "org_apache_arrow_dataset_file_JniWrapper.h"
+#include "org_apache_arrow_dataset_jni_JniWrapper.h"
+#include "org_apache_arrow_dataset_jni_NativeMemoryPool.h"
+
+namespace arrow {
+namespace dataset {
+namespace jni {
+
+static jclass illegal_access_exception_class;
+static jclass illegal_argument_exception_class;
+static jclass runtime_exception_class;
+
+static jclass record_batch_handle_class;
+static jclass record_batch_handle_field_class;
+static jclass record_batch_handle_buffer_class;
+static jclass java_reservation_listener_class;
+
+static jmethodID record_batch_handle_constructor;
+static jmethodID record_batch_handle_field_constructor;
+static jmethodID record_batch_handle_buffer_constructor;
+static jmethodID reserve_memory_method;
+static jmethodID unreserve_memory_method;
+
+static jlong default_memory_pool_id = -1L;
+
+static jint JNI_VERSION = JNI_VERSION_1_6;
+
+class JniPendingException : public std::runtime_error {
+ public:
+ explicit JniPendingException(const std::string& arg) : runtime_error(arg) {}
+};
+
+void ThrowPendingException(const std::string& message) {
+ throw JniPendingException(message);
+}
+
+template <typename T>
+T JniGetOrThrow(arrow::Result<T> result) {
+ if (!result.status().ok()) {
+ ThrowPendingException(result.status().message());
+ }
+ return std::move(result).ValueOrDie();
+}
+
+void JniAssertOkOrThrow(arrow::Status status) {
+ if (!status.ok()) {
+ ThrowPendingException(status.message());
+ }
+}
+
+void JniThrow(std::string message) { ThrowPendingException(message); }
+
+jclass CreateGlobalClassReference(JNIEnv* env, const char* class_name) {
+ jclass local_class = env->FindClass(class_name);
+ jclass global_class = (jclass)env->NewGlobalRef(local_class);
+ env->DeleteLocalRef(local_class);
+ return global_class;
+}
+
+arrow::Result<jmethodID> GetMethodID(JNIEnv* env, jclass this_class, const char* name,
+ const char* sig) {
+ jmethodID ret = env->GetMethodID(this_class, name, sig);
+ if (ret == nullptr) {
+ std::string error_message = "Unable to find method " + std::string(name) +
+ " within signature" + std::string(sig);
+ return arrow::Status::Invalid(error_message);
+ }
+ return ret;
+}
+
+arrow::Result<jmethodID> GetStaticMethodID(JNIEnv* env, jclass this_class,
+ const char* name, const char* sig) {
+ jmethodID ret = env->GetStaticMethodID(this_class, name, sig);
+ if (ret == nullptr) {
+ std::string error_message = "Unable to find static method " + std::string(name) +
+ " within signature" + std::string(sig);
+ return arrow::Status::Invalid(error_message);
+ }
+ return ret;
+}
+
+std::shared_ptr<arrow::Schema> SchemaFromColumnNames(
+ const std::shared_ptr<arrow::Schema>& input,
+ const std::vector<std::string>& column_names) {
+ std::vector<std::shared_ptr<arrow::Field>> columns;
+ for (const auto& name : column_names) {
+ columns.push_back(input->GetFieldByName(name));
+ }
+ return std::make_shared<arrow::Schema>(columns);
+}
+
+arrow::Result<std::shared_ptr<arrow::dataset::FileFormat>> GetFileFormat(jint id) {
+ switch (id) {
+ case 0:
+ return std::make_shared<arrow::dataset::ParquetFileFormat>();
+ default:
+ std::string error_message = "illegal file format id: " + std::to_string(id);
+ return arrow::Status::Invalid(error_message);
+ }
+}
+
+std::string JStringToCString(JNIEnv* env, jstring string) {
+ if (string == nullptr) {
+ return std::string();
+ }
+ jboolean copied;
+ const char* chars = env->GetStringUTFChars(string, &copied);
+ std::string ret = strdup(chars);
Review comment:
Thanks. Didn't know the constructor implies a copy.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] emkornfield commented on a change in pull request #7030: ARROW-7808: [Java][Dataset] Implement Dataset Java API by JNI to C++
Posted by GitBox <gi...@apache.org>.
emkornfield commented on a change in pull request #7030:
URL: https://github.com/apache/arrow/pull/7030#discussion_r551610553
##########
File path: cpp/src/arrow/memory_pool.cc
##########
@@ -534,4 +535,139 @@ int64_t ProxyMemoryPool::max_memory() const { return impl_->max_memory(); }
std::string ProxyMemoryPool::backend_name() const { return impl_->backend_name(); }
+ReservationListener::~ReservationListener() {}
+
+ReservationListener::ReservationListener() {}
+
+class ReservationListenableMemoryPool::ReservationListenableMemoryPoolImpl {
+ public:
+ explicit ReservationListenableMemoryPoolImpl(
+ MemoryPool* pool, std::shared_ptr<ReservationListener> listener, int64_t block_size)
+ : pool_(pool),
+ listener_(listener),
+ block_size_(block_size),
+ blocks_reserved_(0),
+ bytes_reserved_(0) {}
+
+ Status Allocate(int64_t size, uint8_t** out) {
+ RETURN_NOT_OK(UpdateReservation(size));
+ Status error = pool_->Allocate(size, out);
+ if (!error.ok()) {
+ RETURN_NOT_OK(UpdateReservation(-size));
+ return error;
+ }
+ return Status::OK();
+ }
+
+ Status Reallocate(int64_t old_size, int64_t new_size, uint8_t** ptr) {
+ bool reserved = false;
+ int64_t diff = new_size - old_size;
+ if (new_size >= old_size) {
+ RETURN_NOT_OK(UpdateReservation(diff));
+ reserved = true;
Review comment:
some docs in this method could help readability.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] jacques-n commented on pull request #7030: ARROW-7808: [Java][Dataset] Implement Datasets Java API by JNI to C++
Posted by GitBox <gi...@apache.org>.
jacques-n commented on pull request #7030:
URL: https://github.com/apache/arrow/pull/7030#issuecomment-679358407
> The logic might warrant its own interface instead of a std::Function.
I believe we already have this functionality in Gandiva. Let's see if we can use a consistent pattern (whether that is use what Gandiva has or make sure Gandiva uses whatever is proposed here--either way).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] emkornfield commented on a change in pull request #7030: ARROW-7808: [Java][Dataset] Implement Datasets Java API
Posted by GitBox <gi...@apache.org>.
emkornfield commented on a change in pull request #7030:
URL: https://github.com/apache/arrow/pull/7030#discussion_r414984380
##########
File path: cpp/src/jni/dataset/jni_wrapper.cpp
##########
@@ -0,0 +1,577 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include <arrow/dataset/file_base.h>
+#include <arrow/dataset/api.h>
+#include <arrow/filesystem/localfs.h>
+#include <arrow/ipc/api.h>
+#include <arrow/util/iterator.h>
+#include <arrow/filesystem/hdfs.h>
+#include <arrow/io/api.h>
+#include <google/protobuf/io/coded_stream.h>
+#include <google/protobuf/message.h>
+#include "concurrent_map.h"
+#include "arrow/compute/kernel.h"
+#include "arrow/compute/kernels/cast.h"
+#include "arrow/compute/kernels/compare.h"
+#include "jni/dataset/Types.pb.h"
+
+#include "org_apache_arrow_dataset_jni_JniWrapper.h"
+#include "org_apache_arrow_dataset_file_JniWrapper.h"
+
+static jclass illegal_access_exception_class;
+static jclass illegal_argument_exception_class;
+static jclass runtime_exception_class;
+
+static jclass record_batch_handle_class;
+static jclass record_batch_handle_field_class;
+static jclass record_batch_handle_buffer_class;
+
+static jmethodID record_batch_handle_constructor;
+static jmethodID record_batch_handle_field_constructor;
+static jmethodID record_batch_handle_buffer_constructor;
+
+static jint JNI_VERSION = JNI_VERSION_1_6;
+
+using arrow::jni::ConcurrentMap;
+
+static ConcurrentMap<std::shared_ptr<arrow::dataset::DatasetFactory>> dataset_factory_holder_;
+static ConcurrentMap<std::shared_ptr<arrow::dataset::Dataset>> dataset_holder_;
+static ConcurrentMap<std::shared_ptr<arrow::dataset::ScanTask>> scan_task_holder_;
+static ConcurrentMap<std::shared_ptr<arrow::dataset::Scanner>> scanner_holder_;
+static ConcurrentMap<std::shared_ptr<arrow::RecordBatchIterator>> iterator_holder_;
+static ConcurrentMap<std::shared_ptr<arrow::Buffer>> buffer_holder_;
+
+#define JNI_ASSIGN_OR_THROW_NAME(x, y) ARROW_CONCAT(x, y)
Review comment:
I don't think these macros are necessary. you should be able to use a normal (templated) function since you throw an exception instead of return.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] zhztheplayer commented on a change in pull request #7030: ARROW-7808: [Java][Dataset] Implement Dataset Java API by JNI to C++
Posted by GitBox <gi...@apache.org>.
zhztheplayer commented on a change in pull request #7030:
URL: https://github.com/apache/arrow/pull/7030#discussion_r551693971
##########
File path: cpp/src/arrow/memory_pool.cc
##########
@@ -534,4 +535,139 @@ int64_t ProxyMemoryPool::max_memory() const { return impl_->max_memory(); }
std::string ProxyMemoryPool::backend_name() const { return impl_->backend_name(); }
+ReservationListener::~ReservationListener() {}
Review comment:
done
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] jacques-n commented on a change in pull request #7030: ARROW-7808: [Java][Dataset] Implement Datasets Java API by JNI to C++
Posted by GitBox <gi...@apache.org>.
jacques-n commented on a change in pull request #7030:
URL: https://github.com/apache/arrow/pull/7030#discussion_r446237545
##########
File path: java/dataset/src/main/java/org/apache/arrow/memory/NativeUnderlingMemory.java
##########
@@ -0,0 +1,65 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.arrow.memory;
+
+import org.apache.arrow.dataset.jni.JniWrapper;
+
+/**
+ * AllocationManager implementation for Native allocated memory.
+ */
+public class NativeUnderlingMemory extends AllocationManager {
Review comment:
In reality, this is not at all the case. There are two ways that you run out of memory in Java: you run out of available allocation space OR you hit the limit at the JVM level. The limit at the JVM level can be influenced by things like fragmentation so in nearly all cases you hit that limit before the allocator managed limit. In your patch, you only are respecting the allocator set limit, not the jvm limit. This means a user could configure java to allow X of memory and the process actually will use 2X of memory, leading to issues in many systems.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] zhztheplayer commented on pull request #7030: ARROW-7808: [Java][Dataset] Implement Dataset Java API by JNI to C++
Posted by GitBox <gi...@apache.org>.
zhztheplayer commented on pull request #7030:
URL: https://github.com/apache/arrow/pull/7030#issuecomment-766616108
Hi @emkornfield @pitrou
I think I have addressed existing comments so would you like to take another look now? Thanks a lot. Since this was submitted long time ago (before release 1.0.0 IIRC) and I've perform big rebase for times, If we could make progress I would like to help with it in any way.
I know to review a PR creating a new module is usually not trivial work, and current shape of the implementation might be not that perfect for everybody so I may want to suggest we can break up current/future goal to minor patches like we have already done on [the topic of filters](https://github.com/apache/arrow/pull/7030#discussion_r426990234) for this PR.
Based on some recent comments I think we may don't have to make 100% consensus on the current design of `ReservationListenableMemoryPool` or relevant things instantly, if this part is in a way controversial I could split out the part from this PR then create a new one for it after this get merged (should I? would you suggest that). Everything will be working fine enough without it based on our experience (yes we are relying on this feature in our own project for long time). That doesn't mean it is not a "problem" but we can put discussions in separate places to avoid this from being hard to review and merge. The review history is already too long to give every reviewer information he need yet.
And if you have other suggestions on the whole design or architecture of the basic functionality please be free to add comments and I am happy to make further refactors accordingly.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] zhztheplayer commented on a change in pull request #7030: ARROW-7808: [Java][Dataset] Implement Dataset Java API by JNI to C++
Posted by GitBox <gi...@apache.org>.
zhztheplayer commented on a change in pull request #7030:
URL: https://github.com/apache/arrow/pull/7030#discussion_r567757119
##########
File path: cpp/src/jni/dataset/jni_wrapper.cpp
##########
@@ -0,0 +1,859 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include <mutex>
+
+#include <arrow/array.h>
+#include <arrow/dataset/api.h>
+#include <arrow/dataset/file_base.h>
+#include <arrow/filesystem/localfs.h>
+#include <arrow/io/api.h>
+#include <arrow/ipc/api.h>
+#include <arrow/util/iterator.h>
+#include <arrow/util/logging.h>
+
+#include "org_apache_arrow_dataset_file_JniWrapper.h"
+#include "org_apache_arrow_dataset_jni_JniWrapper.h"
+#include "org_apache_arrow_dataset_jni_NativeMemoryPool.h"
+
+namespace arrow {
+namespace dataset {
+namespace jni {
+
+static jclass illegal_access_exception_class;
+static jclass illegal_argument_exception_class;
+static jclass runtime_exception_class;
+
+static jclass record_batch_handle_class;
+static jclass record_batch_handle_field_class;
+static jclass record_batch_handle_buffer_class;
+static jclass java_reservation_listener_class;
+
+static jmethodID record_batch_handle_constructor;
+static jmethodID record_batch_handle_field_constructor;
+static jmethodID record_batch_handle_buffer_constructor;
+static jmethodID reserve_memory_method;
+static jmethodID unreserve_memory_method;
+
+static jlong default_memory_pool_id = -1L;
+
+static jint JNI_VERSION = JNI_VERSION_1_6;
+
+class JniPendingException : public std::runtime_error {
+ public:
+ explicit JniPendingException(const std::string& arg) : runtime_error(arg) {}
+};
+
+void ThrowPendingException(const std::string& message) {
+ throw JniPendingException(message);
+}
+
+template <typename T>
+T JniGetOrThrow(arrow::Result<T> result) {
+ if (!result.status().ok()) {
+ ThrowPendingException(result.status().message());
+ }
+ return std::move(result).ValueOrDie();
+}
+
+void JniAssertOkOrThrow(arrow::Status status) {
+ if (!status.ok()) {
+ ThrowPendingException(status.message());
+ }
+}
+
+void JniThrow(std::string message) { ThrowPendingException(message); }
+
+jclass CreateGlobalClassReference(JNIEnv* env, const char* class_name) {
+ jclass local_class = env->FindClass(class_name);
+ jclass global_class = (jclass)env->NewGlobalRef(local_class);
+ env->DeleteLocalRef(local_class);
+ return global_class;
+}
+
+arrow::Result<jmethodID> GetMethodID(JNIEnv* env, jclass this_class, const char* name,
+ const char* sig) {
+ jmethodID ret = env->GetMethodID(this_class, name, sig);
+ if (ret == nullptr) {
+ std::string error_message = "Unable to find method " + std::string(name) +
+ " within signature" + std::string(sig);
+ return arrow::Status::Invalid(error_message);
+ }
+ return ret;
+}
+
+arrow::Result<jmethodID> GetStaticMethodID(JNIEnv* env, jclass this_class,
+ const char* name, const char* sig) {
+ jmethodID ret = env->GetStaticMethodID(this_class, name, sig);
+ if (ret == nullptr) {
+ std::string error_message = "Unable to find static method " + std::string(name) +
+ " within signature" + std::string(sig);
+ return arrow::Status::Invalid(error_message);
+ }
+ return ret;
+}
+
+std::shared_ptr<arrow::Schema> SchemaFromColumnNames(
+ const std::shared_ptr<arrow::Schema>& input,
+ const std::vector<std::string>& column_names) {
+ std::vector<std::shared_ptr<arrow::Field>> columns;
+ for (const auto& name : column_names) {
+ columns.push_back(input->GetFieldByName(name));
+ }
+ return std::make_shared<arrow::Schema>(columns);
+}
+
+arrow::Result<std::shared_ptr<arrow::dataset::FileFormat>> GetFileFormat(jint id) {
+ switch (id) {
+ case 0:
+ return std::make_shared<arrow::dataset::ParquetFileFormat>();
+ default:
+ std::string error_message = "illegal file format id: " + std::to_string(id);
+ return arrow::Status::Invalid(error_message);
+ }
+}
+
+std::string JStringToCString(JNIEnv* env, jstring string) {
+ if (string == nullptr) {
+ return std::string();
+ }
+ jboolean copied;
+ const char* chars = env->GetStringUTFChars(string, &copied);
Review comment:
Right. Done
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] zhztheplayer commented on a change in pull request #7030: ARROW-7808: [Java][Dataset] Implement Datasets Java API by JNI to C++
Posted by GitBox <gi...@apache.org>.
zhztheplayer commented on a change in pull request #7030:
URL: https://github.com/apache/arrow/pull/7030#discussion_r429715403
##########
File path: cpp/src/arrow/dataset/discovery.h
##########
@@ -238,5 +238,27 @@ class ARROW_DS_EXPORT FileSystemDatasetFactory : public DatasetFactory {
util::optional<util::string_view> RemovePartitionBaseDir(util::string_view path);
};
+class ARROW_DS_EXPORT SingleFileDatasetFactory : public DatasetFactory {
Review comment:
One question: can we add a construction function (see the newest changes) `FileSystemDatasetFactory::Make(std::string uri, std::shared_ptr<FileFormat> format, FileSystemFactoryOptions options)` to `FileSystemDatasetFactory`, for reading a single file from an uri? This is similar to functionality of `SingleFileDataset`.
The original reason of creating `SingleFileDataset` is to avoid passing file system object via JNI. Passing such objects can make the bridge even complicated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] emkornfield commented on a change in pull request #7030: ARROW-7808: [Java][Dataset] Implement Dataset Java API by JNI to C++
Posted by GitBox <gi...@apache.org>.
emkornfield commented on a change in pull request #7030:
URL: https://github.com/apache/arrow/pull/7030#discussion_r567203505
##########
File path: java/dataset/src/test/resources/avroschema/user.avsc
##########
@@ -0,0 +1,26 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+{
Review comment:
still think it would be a good idea to inline this in java it would make the tests overall easier to follow because everything is in one place.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] zhztheplayer commented on a change in pull request #7030: ARROW-7808: [Java][Dataset] Implement Datasets Java API by JNI to C++
Posted by GitBox <gi...@apache.org>.
zhztheplayer commented on a change in pull request #7030:
URL: https://github.com/apache/arrow/pull/7030#discussion_r426394944
##########
File path: java/dataset/src/test/java/org/apache/arrow/dataset/file/TestSingleFileDataset.java
##########
@@ -0,0 +1,237 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.arrow.dataset.file;
+
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertTrue;
+
+import java.util.Collections;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.Objects;
+import java.util.stream.Collectors;
+
+import org.apache.arrow.dataset.DatasetTypes;
+import org.apache.arrow.dataset.filter.Filter;
+import org.apache.arrow.dataset.jni.TestNativeDataset;
+import org.apache.arrow.dataset.scanner.ScanOptions;
+import org.apache.arrow.util.AutoCloseables;
+import org.apache.arrow.vector.FieldVector;
+import org.apache.arrow.vector.VectorLoader;
+import org.apache.arrow.vector.VectorSchemaRoot;
+import org.apache.arrow.vector.ipc.message.ArrowRecordBatch;
+import org.apache.arrow.vector.types.Types;
+import org.apache.arrow.vector.types.pojo.Schema;
+import org.apache.avro.generic.GenericData;
+import org.apache.avro.generic.GenericRecord;
+import org.apache.avro.generic.GenericRecordBuilder;
+import org.junit.ClassRule;
+import org.junit.Test;
+import org.junit.rules.TemporaryFolder;
+
+import com.google.common.collect.ImmutableList;
+
+public class TestSingleFileDataset extends TestNativeDataset {
+
+ @ClassRule
+ public static final TemporaryFolder TMP = new TemporaryFolder();
+
+ public static final String AVRO_SCHEMA_USER = "user.avsc";
+
+ @Test
+ public void testParquetRead() throws Exception {
+ ParquetWriteSupport writeSupport = new ParquetWriteSupport(AVRO_SCHEMA_USER, TMP.newFolder());
+ GenericRecord record = new GenericData.Record(writeSupport.getAvroSchema());
+ record.put("id", 1);
+ record.put("name", "a");
+ writeSupport.writeRecord(record);
+ writeSupport.close();
+
+ SingleFileDatasetFactory factory = new SingleFileDatasetFactory(rootAllocator(),
+ FileFormat.PARQUET, FileSystem.LOCAL, writeSupport.getOutputFile());
+ ScanOptions options = new ScanOptions(new String[0], Filter.EMPTY, 100);
+
+ Schema schema = inferResultSchemaFromFactory(factory, options);
+ assertEquals(2, schema.getFields().size());
+ assertEquals("id", schema.getFields().get(0).getName());
+ assertEquals("name", schema.getFields().get(1).getName());
+ assertEquals(Types.MinorType.INT.getType(), schema.getFields().get(0).getType());
+ assertEquals(Types.MinorType.VARCHAR.getType(), schema.getFields().get(1).getType());
+
+ assertSingleTaskProduced(factory, options);
+
+ List<ArrowRecordBatch> datum = collectResultFromFactory(factory, options);
+ assertEquals(1, datum.size());
+ checkParquetReadResult(schema, Collections.singletonList(record), datum);
+ AutoCloseables.close(datum);
+ }
+
+ @Test
+ public void testParquetProjector() throws Exception {
+ ParquetWriteSupport writeSupport = new ParquetWriteSupport(AVRO_SCHEMA_USER, TMP.newFolder());
+ GenericRecord record = new GenericData.Record(writeSupport.getAvroSchema());
+ record.put("id", 1);
+ record.put("name", "a");
+ writeSupport.writeRecord(record);
+ writeSupport.close();
+
+ SingleFileDatasetFactory factory = new SingleFileDatasetFactory(rootAllocator(),
+ FileFormat.PARQUET, FileSystem.LOCAL, writeSupport.getOutputFile());
+ ScanOptions options = new ScanOptions(new String[]{"id"}, Filter.EMPTY, 100);
+
+ Schema schema = inferResultSchemaFromFactory(factory, options);
+ assertEquals(1, schema.getFields().size());
+ assertEquals("id", schema.getFields().get(0).getName());
+ assertEquals(Types.MinorType.INT.getType(), schema.getFields().get(0).getType());
+
+ assertSingleTaskProduced(factory, options);
+
+ List<ArrowRecordBatch> datum = collectResultFromFactory(factory, options);
+ assertEquals(1, datum.size());
+ org.apache.avro.Schema expectedSchema = truncateAvroSchema(writeSupport.getAvroSchema(), 0, 1);
+ checkParquetReadResult(schema,
+ Collections.singletonList(
+ new GenericRecordBuilder(
+ expectedSchema)
+ .set("id", 1)
+ .build()), datum);
+ AutoCloseables.close(datum);
+ }
+
+ @Test
+ public void testParquetFilter() throws Exception {
+ ParquetWriteSupport writeSupport = new ParquetWriteSupport(AVRO_SCHEMA_USER, TMP.newFolder());
Review comment:
done. optimized with spacings
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] zhztheplayer edited a comment on pull request #7030: ARROW-7808: [Java][Dataset] Implement Datasets Java API by JNI to C++
Posted by GitBox <gi...@apache.org>.
zhztheplayer edited a comment on pull request #7030:
URL: https://github.com/apache/arrow/pull/7030#issuecomment-644494295
@fsaintjacques Rebased now.
As we have a unresolved [discussion](https://github.com/apache/arrow/pull/7030#discussion_r440075959) about memory management of native buffers. Should we clarify a bit more before writing any fixes? (also ping @jacques-n , @emkornfield)
The question is what's the major problem now? I list some that I can see in comments:
1. We are package-hacking `org.apache.arrow.memory` in module `arrow-dataset`.
Yes `Ownerships.java` uses some hacks, so I've removed the class in latest commits (the whole class is not necessary for current use case). Now only `NativeUnderlyingMemory.java` is still in the package `org.apache.arrow.memory`.
2. We lose java native heap caps.
Can we clarify this? I think the explanation can be either we lose caps from JVM direct memory, or we lose caps from BufferAllocator. For the latter one I think buffers are already registered to allocator.
3. OOM-check is proceed after buffer is created.
Yes. But is this really a big problem? see [comment](https://github.com/apache/arrow/pull/7030#discussion_r440075959).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] zhztheplayer commented on a change in pull request #7030: ARROW-7808: [Java][Dataset] Implement Datasets Java API by JNI to C++
Posted by GitBox <gi...@apache.org>.
zhztheplayer commented on a change in pull request #7030:
URL: https://github.com/apache/arrow/pull/7030#discussion_r429553399
##########
File path: cpp/src/jni/dataset/jni_wrapper.cpp
##########
@@ -0,0 +1,517 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include <arrow/dataset/api.h>
+#include <arrow/dataset/file_base.h>
+#include <arrow/filesystem/hdfs.h>
+#include <arrow/filesystem/localfs.h>
+#include <arrow/io/api.h>
+#include <arrow/ipc/api.h>
+#include <arrow/util/iterator.h>
+
+#include "org_apache_arrow_dataset_file_JniWrapper.h"
+#include "org_apache_arrow_dataset_jni_JniWrapper.h"
+
+static jclass illegal_access_exception_class;
+static jclass illegal_argument_exception_class;
+static jclass runtime_exception_class;
+
+static jclass record_batch_handle_class;
+static jclass record_batch_handle_field_class;
+static jclass record_batch_handle_buffer_class;
+
+static jmethodID record_batch_handle_constructor;
+static jmethodID record_batch_handle_field_constructor;
+static jmethodID record_batch_handle_buffer_constructor;
+
+static jint JNI_VERSION = JNI_VERSION_1_6;
+
+class JniPendingException : public std::runtime_error {
+ public:
+ explicit JniPendingException(const std::string& arg) : runtime_error(arg) {}
+};
+
+void ThrowPendingException(const std::string& message) {
+ throw JniPendingException(message);
+}
+
+template <typename T>
+T JniGetOrThrow(arrow::Result<T> result) {
+ if (!result.status().ok()) {
+ ThrowPendingException(result.status().message());
+ }
+ return std::move(result).ValueOrDie();
+}
+
+void JniAssertOkOrThrow(arrow::Status status) {
+ if (!status.ok()) {
+ ThrowPendingException(status.message());
+ }
+}
+
+void JniThrow(std::string message) { ThrowPendingException(message); }
+
+#define JNI_METHOD_START try {
+// macro ended
+
+#define JNI_METHOD_END(fallback_expr) \
+ } \
+ catch (JniPendingException & e) { \
+ env->ThrowNew(runtime_exception_class, e.what()); \
+ return fallback_expr; \
+ }
+// macro ended
+
+jclass CreateGlobalClassReference(JNIEnv* env, const char* class_name) {
+ jclass local_class = env->FindClass(class_name);
+ jclass global_class = (jclass)env->NewGlobalRef(local_class);
+ env->DeleteLocalRef(local_class);
+ return global_class;
+}
+
+arrow::Result<jmethodID> GetMethodID(JNIEnv* env, jclass this_class, const char* name,
+ const char* sig) {
+ jmethodID ret = env->GetMethodID(this_class, name, sig);
+ if (ret == nullptr) {
+ std::string error_message = "Unable to find method " + std::string(name) +
+ " within signature" + std::string(sig);
+ return arrow::Status::Invalid(error_message);
+ }
+ return ret;
+}
+
+jint JNI_OnLoad(JavaVM* vm, void* reserved) {
+ JNIEnv* env;
+ if (vm->GetEnv(reinterpret_cast<void**>(&env), JNI_VERSION) != JNI_OK) {
+ return JNI_ERR;
+ }
+ JNI_METHOD_START
+ illegal_access_exception_class =
+ CreateGlobalClassReference(env, "Ljava/lang/IllegalAccessException;");
+ illegal_argument_exception_class =
+ CreateGlobalClassReference(env, "Ljava/lang/IllegalArgumentException;");
+ runtime_exception_class =
+ CreateGlobalClassReference(env, "Ljava/lang/RuntimeException;");
+
+ record_batch_handle_class =
+ CreateGlobalClassReference(env,
+ "Lorg/apache/arrow/"
+ "dataset/jni/NativeRecordBatchHandle;");
+ record_batch_handle_field_class =
+ CreateGlobalClassReference(env,
+ "Lorg/apache/arrow/"
+ "dataset/jni/NativeRecordBatchHandle$Field;");
+ record_batch_handle_buffer_class =
+ CreateGlobalClassReference(env,
+ "Lorg/apache/arrow/"
+ "dataset/jni/NativeRecordBatchHandle$Buffer;");
+
+ record_batch_handle_constructor =
+ JniGetOrThrow(GetMethodID(env, record_batch_handle_class, "<init>",
+ "(J[Lorg/apache/arrow/dataset/"
+ "jni/NativeRecordBatchHandle$Field;"
+ "[Lorg/apache/arrow/dataset/"
+ "jni/NativeRecordBatchHandle$Buffer;)V"));
+ record_batch_handle_field_constructor =
+ JniGetOrThrow(GetMethodID(env, record_batch_handle_field_class, "<init>", "(JJ)V"));
+ record_batch_handle_buffer_constructor = JniGetOrThrow(
+ GetMethodID(env, record_batch_handle_buffer_class, "<init>", "(JJJJ)V"));
+
+ return JNI_VERSION;
+ JNI_METHOD_END(JNI_ERR)
+}
+
+void JNI_OnUnload(JavaVM* vm, void* reserved) {
+ JNIEnv* env;
+ vm->GetEnv(reinterpret_cast<void**>(&env), JNI_VERSION);
+ env->DeleteGlobalRef(illegal_access_exception_class);
+ env->DeleteGlobalRef(illegal_argument_exception_class);
+ env->DeleteGlobalRef(runtime_exception_class);
+ env->DeleteGlobalRef(record_batch_handle_class);
+ env->DeleteGlobalRef(record_batch_handle_field_class);
+ env->DeleteGlobalRef(record_batch_handle_buffer_class);
+}
+
+std::shared_ptr<arrow::Schema> SchemaFromColumnNames(
+ const std::shared_ptr<arrow::Schema>& input,
+ const std::vector<std::string>& column_names) {
+ std::vector<std::shared_ptr<arrow::Field>> columns;
+ for (const auto& name : column_names) {
+ columns.push_back(input->GetFieldByName(name));
+ }
+ return std::make_shared<arrow::Schema>(columns);
+}
+
+arrow::Result<std::shared_ptr<arrow::dataset::FileFormat>> GetFileFormat(jint id) {
+ switch (id) {
+ case 0:
+ return std::make_shared<arrow::dataset::ParquetFileFormat>();
+ default:
+ std::string error_message = "illegal file format id: " + std::to_string(id);
+ return arrow::Status::Invalid(error_message);
+ }
+}
+
+arrow::Result<std::shared_ptr<arrow::fs::FileSystem>> GetFileSystem(
+ jint id, std::string path, std::string* out_path) {
+ switch (id) {
+ case 0:
+ *out_path = path;
+ return std::make_shared<arrow::fs::LocalFileSystem>();
+ case 1: {
+ ARROW_ASSIGN_OR_RAISE(std::shared_ptr<arrow::fs::FileSystem> ret,
+ arrow::fs::FileSystemFromUri(path, out_path))
+ return ret;
+ }
+ default:
+ std::string error_message = "illegal file system id: " + std::to_string(id);
+ return arrow::Status::Invalid(error_message);
+ }
+}
+
+std::string JStringToCString(JNIEnv* env, jstring string) {
+ if (string == nullptr) {
+ return std::string();
+ }
+ jboolean copied;
+ int32_t length = env->GetStringUTFLength(string);
+ const char* chars = env->GetStringUTFChars(string, &copied);
+ std::string str = std::string(chars, length);
+ // fixme calling ReleaseStringUTFChars if memory leak faced
+ return str;
+}
+
+std::vector<std::string> ToStringVector(JNIEnv* env, jobjectArray& str_array) {
+ int length = env->GetArrayLength(str_array);
+ std::vector<std::string> vector;
+ for (int i = 0; i < length; i++) {
+ auto string = (jstring)(env->GetObjectArrayElement(str_array, i));
+ vector.push_back(JStringToCString(env, string));
+ }
+ return vector;
+}
+
+template <typename T>
+jlong CreateNativeRef(std::shared_ptr<T> t) {
+ std::shared_ptr<T>* retained_ptr = new std::shared_ptr<T>(t);
+ return reinterpret_cast<jlong>(retained_ptr);
+}
+
+template <typename T>
+std::shared_ptr<T> RetrieveNativeInstance(jlong ref) {
+ std::shared_ptr<T>* retrieved_ptr = reinterpret_cast<std::shared_ptr<T>*>(ref);
+ return *retrieved_ptr;
+}
+
+template <typename T>
+void ReleaseNativeRef(jlong ref) {
+ std::shared_ptr<T>* retrieved_ptr = reinterpret_cast<std::shared_ptr<T>*>(ref);
+ delete retrieved_ptr;
+}
+
+arrow::Result<jbyteArray> ToSchemaByteArray(JNIEnv* env,
+ std::shared_ptr<arrow::Schema> schema) {
+ ARROW_ASSIGN_OR_RAISE(
+ std::shared_ptr<arrow::Buffer> buffer,
+ arrow::ipc::SerializeSchema(*schema, nullptr, arrow::default_memory_pool()))
+
+ jbyteArray out = env->NewByteArray(buffer->size());
+ auto src = reinterpret_cast<const jbyte*>(buffer->data());
+ env->SetByteArrayRegion(out, 0, buffer->size(), src);
+ return out;
+}
+
+arrow::Result<std::shared_ptr<arrow::Schema>> FromSchemaByteArray(
+ JNIEnv* env, jbyteArray schemaBytes) {
+ arrow::ipc::DictionaryMemo in_memo;
+ int schemaBytes_len = env->GetArrayLength(schemaBytes);
+ jbyte* schemaBytes_data = env->GetByteArrayElements(schemaBytes, nullptr);
+ auto serialized_schema = std::make_shared<arrow::Buffer>(
+ reinterpret_cast<uint8_t*>(schemaBytes_data), schemaBytes_len);
+ arrow::io::BufferReader buf_reader(serialized_schema);
+ ARROW_ASSIGN_OR_RAISE(std::shared_ptr<arrow::Schema> schema,
+ arrow::ipc::ReadSchema(&buf_reader, &in_memo))
+ env->ReleaseByteArrayElements(schemaBytes, schemaBytes_data, JNI_ABORT);
+ return schema;
+}
+
+/*
+ * Class: org_apache_arrow_dataset_jni_JniWrapper
+ * Method: closeDatasetFactory
+ * Signature: (J)V
+ */
+JNIEXPORT void JNICALL Java_org_apache_arrow_dataset_jni_JniWrapper_closeDatasetFactory(
+ JNIEnv* env, jobject, jlong id) {
+ JNI_METHOD_START
+ ReleaseNativeRef<arrow::dataset::DatasetFactory>(id);
+ JNI_METHOD_END()
+}
+
+/*
+ * Class: org_apache_arrow_dataset_jni_JniWrapper
+ * Method: inspectSchema
+ * Signature: (J)[B
+ */
+JNIEXPORT jbyteArray JNICALL Java_org_apache_arrow_dataset_jni_JniWrapper_inspectSchema(
+ JNIEnv* env, jobject, jlong dataset_factor_id) {
+ JNI_METHOD_START
+ std::shared_ptr<arrow::dataset::DatasetFactory> d =
+ RetrieveNativeInstance<arrow::dataset::DatasetFactory>(dataset_factor_id);
+ std::shared_ptr<arrow::Schema> schema = JniGetOrThrow(d->Inspect());
+ return JniGetOrThrow(ToSchemaByteArray(env, schema));
+ JNI_METHOD_END(nullptr)
+}
+
+/*
+ * Class: org_apache_arrow_dataset_jni_JniWrapper
+ * Method: createDataset
+ * Signature: (J[B)J
+ */
+JNIEXPORT jlong JNICALL Java_org_apache_arrow_dataset_jni_JniWrapper_createDataset(
+ JNIEnv* env, jobject, jlong dataset_factory_id, jbyteArray schema_bytes) {
+ JNI_METHOD_START
+ std::shared_ptr<arrow::dataset::DatasetFactory> d =
+ RetrieveNativeInstance<arrow::dataset::DatasetFactory>(dataset_factory_id);
+ std::shared_ptr<arrow::Schema> schema;
+ schema = JniGetOrThrow(FromSchemaByteArray(env, schema_bytes));
+ std::shared_ptr<arrow::dataset::Dataset> dataset = JniGetOrThrow(d->Finish(schema));
+ return CreateNativeRef(dataset);
+ JNI_METHOD_END(-1L)
+}
+
+/*
+ * Class: org_apache_arrow_dataset_jni_JniWrapper
+ * Method: closeDataset
+ * Signature: (J)V
+ */
+JNIEXPORT void JNICALL Java_org_apache_arrow_dataset_jni_JniWrapper_closeDataset(
+ JNIEnv* env, jobject, jlong id) {
+ JNI_METHOD_START
+ ReleaseNativeRef<arrow::dataset::Dataset>(id);
+ JNI_METHOD_END()
+}
+
+/// \class DisposableScannerAdaptor
+/// \brief An adaptor that iterates over a Scanner instance then returns RecordBatches
+/// directly.
+///
+/// This lessens the complexity of the JNI bridge to make sure it to be easier to
+/// maintain. On Java-side, NativeScanner can only produces a single NativeScanTask
+/// instance during its whole lifecycle. Each task stands for a DisposableScannerAdaptor
+/// instance through JNI bridge.
+///
+class DisposableScannerAdaptor {
Review comment:
So let's just remove this and add a method `Iterator<ScanTask> Scanner.scan()` to JNI bridge? I have same concern with you and for me this code is temporary. I think we wanted to keep JNI bridge simple enough at the first implementation so that if C++ API get factored again in future we may save time from changing JNI code to make Java side work.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] zhztheplayer commented on a change in pull request #7030: ARROW-7808: [Java][Dataset] Implement Datasets Java API by JNI to C++
Posted by GitBox <gi...@apache.org>.
zhztheplayer commented on a change in pull request #7030:
URL: https://github.com/apache/arrow/pull/7030#discussion_r416412538
##########
File path: java/dataset/src/main/java/org/apache/arrow/dataset/scanner/ScanTask.java
##########
@@ -0,0 +1,42 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.arrow.dataset.scanner;
+
+import java.util.Iterator;
+
+import org.apache.arrow.vector.VectorSchemaRoot;
+
+/**
+ * Read record batches from a range of a single data fragment. A
+ * ScanTask is meant to be a unit of work to be dispatched. The implementation
+ * must be thread and concurrent safe.
+ */
+public interface ScanTask {
+
+ /**
+ * Creates and returns a {@link Itr} instance.
+ */
+ Itr scan();
+
+ /**
+ * The iterator implementation for {@link VectorSchemaRoot}s.
+ */
+ interface Itr extends Iterator<VectorSchemaRoot>, AutoCloseable {
Review comment:
Using Iterator is OK but it may collide with java.util.Iterator in scenarios. I we rename it should it be "BatchIterator" or something?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] wesm commented on pull request #7030: ARROW-7808: [Java][Dataset] Implement Datasets Java API
Posted by GitBox <gi...@apache.org>.
wesm commented on pull request #7030:
URL: https://github.com/apache/arrow/pull/7030#issuecomment-619660367
As a high level advisory: the Datasets C++ API should still be regarded as alpha-stage, so I recommend making the JNI bindings as minimal as possible so that refactoring is not too painful.
Additionally, I recommend that you do not build JNI bindings for the Expression classes in arrow/dataset/filter.h -- this code is likely to change quite a bit as the associated query engine project gets off the ground this year.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] emkornfield commented on pull request #7030: ARROW-7808: [Java][Dataset] Implement Datasets Java API by JNI to C++
Posted by GitBox <gi...@apache.org>.
emkornfield commented on pull request #7030:
URL: https://github.com/apache/arrow/pull/7030#issuecomment-643059101
@zhztheplayer it still looks like JNI is failing?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] zhztheplayer commented on a change in pull request #7030: ARROW-7808: [Java][Dataset] Implement Datasets Java API by JNI to C++
Posted by GitBox <gi...@apache.org>.
zhztheplayer commented on a change in pull request #7030:
URL: https://github.com/apache/arrow/pull/7030#discussion_r440053169
##########
File path: java/dataset/src/main/java/org/apache/arrow/memory/Ownerships.java
##########
@@ -0,0 +1,47 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.arrow.memory;
+
+/**
+ * Utility class managing ownership's transferring between Native Arrow buffers and Java Arrow buffers.
+ */
+public class Ownerships {
+ private static final Ownerships INSTANCE = new Ownerships();
+
+ private Ownerships() {
+ }
+
+ public static Ownerships get() {
+ return INSTANCE;
+ }
+
+ /**
+ * Returned ledger's ref count is initialized or increased by 1.
+ */
+ public BufferLedger takeOwnership(BaseAllocator allocator, AllocationManager allocationManager) {
+ final BufferLedger ledger = allocationManager.associate(allocator);
+ long size = allocationManager.getSize();
+ boolean succeed = allocator.forceAllocate(size);
+ if (!succeed) {
+ throw new OutOfMemoryException("Target allocator is full");
+ }
+ ((BaseAllocator) allocationManager.getOwningLedger().getAllocator()).releaseBytes(size);
Review comment:
After going though codes again I think it should be enough to use Reservation.reserve(int) to pre-allocate these the bytes needed (see newest commit). And it's not necessary to provide the transferring functionality independently as we don't have to use different allocators during creating the buffer.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] emkornfield commented on pull request #7030: ARROW-7808: [Java][Dataset] Implement Datasets Java API by JNI to C++
Posted by GitBox <gi...@apache.org>.
emkornfield commented on pull request #7030:
URL: https://github.com/apache/arrow/pull/7030#issuecomment-679355492
The logic might warrant its own interface instead of a std::Function.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] zhztheplayer commented on pull request #7030: ARROW-7808: [Java][Dataset] Implement Datasets Java API by JNI to C++
Posted by GitBox <gi...@apache.org>.
zhztheplayer commented on pull request #7030:
URL: https://github.com/apache/arrow/pull/7030#issuecomment-654004147
@jacques-n
I think we have several choices: we can try implement a c++ memory pool which is backed by a Java Allocator, actually I have a PoC branch for that: https://github.com/zhztheplayer/arrow-1/tree/ARROW-7808-wip-bridged-allocator.
Or we can continue hacking Java API to reserve direct memory after buffer is registered. The reservation should happen inside Netty API because Netty has its own direct memory counter: https://github.com/netty/netty/blob/cbe238a95bab238455e3ff1849d450a9836f39ef/common/src/main/java/io/netty/util/internal/PlatformDependent.java#L749-L766
Which one do you suggest or maybe there's a better one than both?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] zhztheplayer commented on pull request #7030: ARROW-7808: [Java][Dataset] Implement Dataset Java API by JNI to C++
Posted by GitBox <gi...@apache.org>.
zhztheplayer commented on pull request #7030:
URL: https://github.com/apache/arrow/pull/7030#issuecomment-775220292
Thanks @emkornfield. I've made updates accordingly.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] zhztheplayer commented on pull request #7030: ARROW-7808: [Java][Dataset] Implement Datasets Java API by JNI to C++
Posted by GitBox <gi...@apache.org>.
zhztheplayer commented on pull request #7030:
URL: https://github.com/apache/arrow/pull/7030#issuecomment-706836000
> > The logic might warrant its own interface instead of a std::Function.
>
> I believe we already have this functionality in Gandiva. Let's see if we can use a consistent pattern (whether that is use what Gandiva has or make sure Gandiva uses whatever is proposed here--either way).
Hi @jacques-n , I already noticed that Gandiva has a JavaResizableBuffer implementation for resizing Java-allocated buffers from C++ via JNI. Did you mean this one by the comment?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] jacques-n commented on a change in pull request #7030: ARROW-7808: [Java][Dataset] Implement Datasets Java API by JNI to C++
Posted by GitBox <gi...@apache.org>.
jacques-n commented on a change in pull request #7030:
URL: https://github.com/apache/arrow/pull/7030#discussion_r439206598
##########
File path: java/dataset/src/main/java/org/apache/arrow/memory/Ownerships.java
##########
@@ -0,0 +1,47 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.arrow.memory;
+
+/**
+ * Utility class managing ownership's transferring between Native Arrow buffers and Java Arrow buffers.
+ */
+public class Ownerships {
+ private static final Ownerships INSTANCE = new Ownerships();
+
+ private Ownerships() {
+ }
+
+ public static Ownerships get() {
+ return INSTANCE;
+ }
+
+ /**
+ * Returned ledger's ref count is initialized or increased by 1.
+ */
+ public BufferLedger takeOwnership(BaseAllocator allocator, AllocationManager allocationManager) {
+ final BufferLedger ledger = allocationManager.associate(allocator);
+ long size = allocationManager.getSize();
+ boolean succeed = allocator.forceAllocate(size);
+ if (!succeed) {
+ throw new OutOfMemoryException("Target allocator is full");
+ }
+ ((BaseAllocator) allocationManager.getOwningLedger().getAllocator()).releaseBytes(size);
Review comment:
We need to address this requirement another way. Package hacking isn't the right solution and I believe you're violating the thread safety of the allocator classes.
##########
File path: java/dataset/src/main/java/org/apache/arrow/memory/NativeUnderlingMemory.java
##########
@@ -0,0 +1,65 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.arrow.memory;
+
+import org.apache.arrow.dataset.jni.JniWrapper;
+
+/**
+ * AllocationManager implementation for Native allocated memory.
+ */
+public class NativeUnderlingMemory extends AllocationManager {
Review comment:
I think we should probably hold off on merging until we resolve this. Just because another part of the codebase has debt or weaknesses doesn't mean we should introduce more of that.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] jacques-n commented on pull request #7030: ARROW-7808: [Java][Dataset] Implement Datasets Java API by JNI to C++
Posted by GitBox <gi...@apache.org>.
jacques-n commented on pull request #7030:
URL: https://github.com/apache/arrow/pull/7030#issuecomment-650227890
> 1. We are package-hacking `org.apache.arrow.memory` in module `arrow-dataset`.
> Yes `Ownerships.java` uses some hacks, so I've removed the class in latest commits (the whole class is not necessary for current use case). Now only `NativeUnderlyingMemory.java` is still in the package `org.apache.arrow.memory`.
We need to avoid this entirely. If you need some functionality, let's figure out what should be exposed. BaseAllocator should not be referenced.
> 2. We lose java native heap caps.
> Can we clarify this? I think the explanation can be either we lose caps from JVM direct memory, or we lose caps from BufferAllocator. For the latter one I think buffers are already registered to allocator.
JVM should respect the max direct memory setting. I added more comments on your other comment to that end.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] emkornfield commented on a change in pull request #7030: ARROW-7808: [Java][Dataset] Implement Dataset Java API by JNI to C++
Posted by GitBox <gi...@apache.org>.
emkornfield commented on a change in pull request #7030:
URL: https://github.com/apache/arrow/pull/7030#discussion_r551610076
##########
File path: cpp/src/arrow/memory_pool.cc
##########
@@ -534,4 +535,139 @@ int64_t ProxyMemoryPool::max_memory() const { return impl_->max_memory(); }
std::string ProxyMemoryPool::backend_name() const { return impl_->backend_name(); }
+ReservationListener::~ReservationListener() {}
+
+ReservationListener::ReservationListener() {}
+
+class ReservationListenableMemoryPool::ReservationListenableMemoryPoolImpl {
+ public:
+ explicit ReservationListenableMemoryPoolImpl(
+ MemoryPool* pool, std::shared_ptr<ReservationListener> listener, int64_t block_size)
+ : pool_(pool),
+ listener_(listener),
+ block_size_(block_size),
+ blocks_reserved_(0),
+ bytes_reserved_(0) {}
+
+ Status Allocate(int64_t size, uint8_t** out) {
+ RETURN_NOT_OK(UpdateReservation(size));
+ Status error = pool_->Allocate(size, out);
+ if (!error.ok()) {
+ RETURN_NOT_OK(UpdateReservation(-size));
+ return error;
+ }
+ return Status::OK();
+ }
+
+ Status Reallocate(int64_t old_size, int64_t new_size, uint8_t** ptr) {
+ bool reserved = false;
+ int64_t diff = new_size - old_size;
+ if (new_size >= old_size) {
+ RETURN_NOT_OK(UpdateReservation(diff));
+ reserved = true;
+ }
+ Status error = pool_->Reallocate(old_size, new_size, ptr);
+ if (!error.ok()) {
+ if (reserved) {
+ RETURN_NOT_OK(UpdateReservation(-diff));
+ }
+ return error;
+ }
+ if (!reserved) {
+ RETURN_NOT_OK(UpdateReservation(diff));
+ }
+ return Status::OK();
+ }
+
+ void Free(uint8_t* buffer, int64_t size) {
+ pool_->Free(buffer, size);
+ // fixme currently method ::Free doesn't allow Status return
Review comment:
please open a JIRA or e-mail the mailing list for discussion.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] emkornfield commented on a change in pull request #7030: ARROW-7808: [Java][Dataset] Implement Datasets Java API
Posted by GitBox <gi...@apache.org>.
emkornfield commented on a change in pull request #7030:
URL: https://github.com/apache/arrow/pull/7030#discussion_r414989937
##########
File path: cpp/src/jni/dataset/jni_wrapper.cpp
##########
@@ -0,0 +1,577 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include <arrow/dataset/file_base.h>
+#include <arrow/dataset/api.h>
+#include <arrow/filesystem/localfs.h>
+#include <arrow/ipc/api.h>
+#include <arrow/util/iterator.h>
+#include <arrow/filesystem/hdfs.h>
+#include <arrow/io/api.h>
+#include <google/protobuf/io/coded_stream.h>
+#include <google/protobuf/message.h>
+#include "concurrent_map.h"
+#include "arrow/compute/kernel.h"
+#include "arrow/compute/kernels/cast.h"
+#include "arrow/compute/kernels/compare.h"
+#include "jni/dataset/Types.pb.h"
+
+#include "org_apache_arrow_dataset_jni_JniWrapper.h"
+#include "org_apache_arrow_dataset_file_JniWrapper.h"
+
+static jclass illegal_access_exception_class;
+static jclass illegal_argument_exception_class;
+static jclass runtime_exception_class;
+
+static jclass record_batch_handle_class;
+static jclass record_batch_handle_field_class;
+static jclass record_batch_handle_buffer_class;
+
+static jmethodID record_batch_handle_constructor;
+static jmethodID record_batch_handle_field_constructor;
+static jmethodID record_batch_handle_buffer_constructor;
+
+static jint JNI_VERSION = JNI_VERSION_1_6;
+
+using arrow::jni::ConcurrentMap;
+
+static ConcurrentMap<std::shared_ptr<arrow::dataset::DatasetFactory>> dataset_factory_holder_;
+static ConcurrentMap<std::shared_ptr<arrow::dataset::Dataset>> dataset_holder_;
+static ConcurrentMap<std::shared_ptr<arrow::dataset::ScanTask>> scan_task_holder_;
+static ConcurrentMap<std::shared_ptr<arrow::dataset::Scanner>> scanner_holder_;
+static ConcurrentMap<std::shared_ptr<arrow::RecordBatchIterator>> iterator_holder_;
+static ConcurrentMap<std::shared_ptr<arrow::Buffer>> buffer_holder_;
+
+#define JNI_ASSIGN_OR_THROW_NAME(x, y) ARROW_CONCAT(x, y)
+
+#define JNI_ASSIGN_OR_THROW_IMPL(t, lhs, rexpr) \
+ auto t = (rexpr); \
+ if (!t.status().ok()) { \
+ env->ThrowNew(runtime_exception_class, t.status().message().c_str()); \
+ } \
+ lhs = std::move(t).ValueOrDie();
+
+#define JNI_ASSIGN_OR_THROW(lhs, rexpr) \
+ JNI_ASSIGN_OR_THROW_IMPL(JNI_ASSIGN_OR_THROW_NAME(_tmp_var, __COUNTER__), lhs, rexpr)
+
+#define JNI_ASSERT_OK_OR_THROW(expr) \
+ do { \
+ auto _res = (expr); \
+ arrow::Status _st = ::arrow::internal::GenericToStatus(_res); \
+ if (!_st.ok()) { \
+ env->ThrowNew(runtime_exception_class, _st.message().c_str()); \
+ } \
+ } while (false);
+
+jclass CreateGlobalClassReference(JNIEnv* env, const char* class_name) {
+ jclass local_class = env->FindClass(class_name);
+ jclass global_class = (jclass)env->NewGlobalRef(local_class);
+ env->DeleteLocalRef(local_class);
+ return global_class;
+}
+
+jmethodID GetMethodID(JNIEnv* env, jclass this_class, const char* name, const char* sig) {
+ jmethodID ret = env->GetMethodID(this_class, name, sig);
+ if (ret == nullptr) {
+ std::string error_message = "Unable to find method " + std::string(name) +
+ " within signature" + std::string(sig);
+ env->ThrowNew(illegal_access_exception_class, error_message.c_str());
+ }
+ return ret;
+}
+
+jint JNI_OnLoad(JavaVM* vm, void* reserved) {
+ JNIEnv* env;
+ if (vm->GetEnv(reinterpret_cast<void**>(&env), JNI_VERSION) != JNI_OK) {
+ return JNI_ERR;
+ }
+
+ illegal_access_exception_class =
+ CreateGlobalClassReference(env, "Ljava/lang/IllegalAccessException;");
+ illegal_argument_exception_class =
+ CreateGlobalClassReference(env, "Ljava/lang/IllegalArgumentException;");
+ runtime_exception_class =
+ CreateGlobalClassReference(env, "Ljava/lang/RuntimeException;");
+
+ record_batch_handle_class =
+ CreateGlobalClassReference(env, "Lorg/apache/arrow/dataset/jni/NativeRecordBatchHandle;");
+ record_batch_handle_field_class =
+ CreateGlobalClassReference(env, "Lorg/apache/arrow/dataset/jni/NativeRecordBatchHandle$Field;");
+ record_batch_handle_buffer_class =
+ CreateGlobalClassReference(env, "Lorg/apache/arrow/dataset/jni/NativeRecordBatchHandle$Buffer;");
+
+ record_batch_handle_constructor = GetMethodID(env, record_batch_handle_class, "<init>",
+ "(J[Lorg/apache/arrow/dataset/jni/NativeRecordBatchHandle$Field;"
+ "[Lorg/apache/arrow/dataset/jni/NativeRecordBatchHandle$Buffer;)V");
+ record_batch_handle_field_constructor = GetMethodID(env, record_batch_handle_field_class, "<init>",
+ "(JJ)V");
+ record_batch_handle_buffer_constructor = GetMethodID(env, record_batch_handle_buffer_class, "<init>",
+ "(JJJJ)V");
+
+ env->ExceptionDescribe();
+
+ return JNI_VERSION;
+}
+
+
+void JNI_OnUnload(JavaVM* vm, void* reserved) {
+ JNIEnv* env;
+ vm->GetEnv(reinterpret_cast<void**>(&env), JNI_VERSION);
+ env->DeleteGlobalRef(illegal_access_exception_class);
+ env->DeleteGlobalRef(illegal_argument_exception_class);
+ env->DeleteGlobalRef(runtime_exception_class);
+ env->DeleteGlobalRef(record_batch_handle_class);
+ env->DeleteGlobalRef(record_batch_handle_field_class);
+ env->DeleteGlobalRef(record_batch_handle_buffer_class);
+
+ dataset_factory_holder_.Clear();
+ dataset_holder_.Clear();
+ scan_task_holder_.Clear();
+ scanner_holder_.Clear();
+ iterator_holder_.Clear();
+ buffer_holder_.Clear();
+}
+
+std::shared_ptr<arrow::Schema> SchemaFromColumnNames(
+ const std::shared_ptr<arrow::Schema>& input, const std::vector<std::string>& column_names) {
+ std::vector<std::shared_ptr<arrow::Field>> columns;
+ for (const auto& name : column_names) {
+ columns.push_back(input->GetFieldByName(name));
+ }
+ return std::make_shared<arrow::Schema>(columns);
+}
+
+std::shared_ptr<arrow::dataset::FileFormat> GetFileFormat(JNIEnv *env, jint id) {
+ switch (id) {
+ case 0:
+ return std::make_shared<arrow::dataset::ParquetFileFormat>();
+ default:
+ std::string error_message = "illegal file format id: " + std::to_string(id);
+ env->ThrowNew(illegal_argument_exception_class, error_message.c_str());
+ return nullptr; // unreachable
+ }
+}
+
+std::shared_ptr<arrow::fs::FileSystem> GetFileSystem(JNIEnv *env, jint id, std::string path,
+ std::string* out_path) {
+ switch (id) {
+ case 0:
+ *out_path = path;
+ return std::make_shared<arrow::fs::LocalFileSystem>();
+ case 1: {
+ JNI_ASSIGN_OR_THROW(std::shared_ptr<arrow::fs::FileSystem> ret, arrow::fs::FileSystemFromUri(path, out_path))
+ return ret;
+ }
+ default:std::string error_message = "illegal filesystem id: " + std::to_string(id);
+ env->ThrowNew(illegal_argument_exception_class, error_message.c_str());
+ return nullptr; // unreachable
+ }
+}
+
+std::string JStringToCString(JNIEnv* env, jstring string) {
+ jboolean copied;
+ int32_t length = env->GetStringUTFLength(string);
+ const char *chars = env->GetStringUTFChars(string, &copied);
+ std::string str = std::string(chars, length);
+ // fixme calling ReleaseStringUTFChars if memory leak faced
+ return str;
+}
+
+std::vector<std::string> ToStringVector(JNIEnv* env, jobjectArray& str_array) {
+ int length = env->GetArrayLength(str_array);
+ std::vector<std::string> vector;
+ for (int i = 0; i < length; i++) {
+ auto string = (jstring) (env->GetObjectArrayElement(str_array, i));
+ vector.push_back(JStringToCString(env, string));
+ }
+ return vector;
+}
+
+template <typename T>
+std::vector<T> collect(JNIEnv* env, arrow::Iterator<T> itr) {
+ std::vector<T> vector;
+ while(true) {
+ JNI_ASSIGN_OR_THROW(T t, itr.Next())
+ if (!t) {
+ break;
+ }
+ vector.push_back(t);
+ }
+ return vector;
+}
+
+// FIXME: COPIED FROM intel/master on which this branch is not rebased yet
+// FIXME: https://github.com/Intel-bigdata/arrow/blob/02502a4eb59834c2471dd629e77dbeed19559f68/cpp/src/jni/jni_common.h#L239-L254
+jbyteArray ToSchemaByteArray(JNIEnv* env, std::shared_ptr<arrow::Schema> schema) {
+ JNI_ASSIGN_OR_THROW(std::shared_ptr<arrow::Buffer> buffer,
+ arrow::ipc::SerializeSchema(*schema.get(), nullptr, arrow::default_memory_pool()))
+
+ jbyteArray out = env->NewByteArray(buffer->size());
+ auto src = reinterpret_cast<const jbyte*>(buffer->data());
+ env->SetByteArrayRegion(out, 0, buffer->size(), src);
+ return out;
+}
+
+// FIXME: COPIED FROM intel/master on which this branch is not rebased yet
+// FIXME: https://github.com/Intel-bigdata/arrow/blob/02502a4eb59834c2471dd629e77dbeed19559f68/cpp/src/jni/jni_common.h#L256-L272
+std::shared_ptr<arrow::Schema> FromSchemaByteArray(JNIEnv* env, jbyteArray schemaBytes) {
+ arrow::ipc::DictionaryMemo in_memo;
+ int schemaBytes_len = env->GetArrayLength(schemaBytes);
+ jbyte* schemaBytes_data = env->GetByteArrayElements(schemaBytes, 0);
+ auto serialized_schema =
+ std::make_shared<arrow::Buffer>((uint8_t*)schemaBytes_data, schemaBytes_len);
+ arrow::io::BufferReader buf_reader(serialized_schema);
+ JNI_ASSIGN_OR_THROW(std::shared_ptr<arrow::Schema> schema, arrow::ipc::ReadSchema(&buf_reader, &in_memo))
+ env->ReleaseByteArrayElements(schemaBytes, schemaBytes_data, JNI_ABORT);
+ return schema;
+}
+
+bool ParseProtobuf(uint8_t* buf, int bufLen, google::protobuf::Message* msg) {
+ google::protobuf::io::CodedInputStream cis(buf, bufLen);
+ cis.SetRecursionLimit(1000);
+ return msg->ParseFromCodedStream(&cis);
+}
+
+void releaseFilterInput(jbyteArray condition_arr, jbyte* condition_bytes, JNIEnv* env) {
+ env->ReleaseByteArrayElements(condition_arr, condition_bytes, JNI_ABORT);
+}
+
+// fixme in development. Not all node types considered.
+std::shared_ptr<arrow::dataset::Expression> translateNode(types::TreeNode node, JNIEnv* env) {
+ if (node.has_fieldnode()) {
+ const types::FieldNode& f_node = node.fieldnode();
+ const std::string& name = f_node.name();
+ return std::make_shared<arrow::dataset::FieldExpression>(name);
+ }
+ if (node.has_intnode()) {
+ const types::IntNode& int_node = node.intnode();
+ int32_t val = int_node.value();
+ return std::make_shared<arrow::dataset::ScalarExpression>(std::make_shared<arrow::Int32Scalar>(val));
+ }
+ if (node.has_longnode()) {
+ const types::LongNode& long_node = node.longnode();
+ int64_t val = long_node.value();
+ return std::make_shared<arrow::dataset::ScalarExpression>(std::make_shared<arrow::Int64Scalar>(val));
+ }
+ if (node.has_floatnode()) {
+ const types::FloatNode& float_node = node.floatnode();
+ float_t val = float_node.value();
+ return std::make_shared<arrow::dataset::ScalarExpression>(std::make_shared<arrow::FloatScalar>(val));
+ }
+ if (node.has_doublenode()) {
+ const types::DoubleNode& double_node = node.doublenode();
+ double_t val = double_node.value();
+ return std::make_shared<arrow::dataset::ScalarExpression>(std::make_shared<arrow::DoubleScalar>(val));
+ }
+ if (node.has_booleannode()) {
+ const types::BooleanNode& boolean_node = node.booleannode();
+ bool val = boolean_node.value();
+ return std::make_shared<arrow::dataset::ScalarExpression>(std::make_shared<arrow::BooleanScalar>(val));
+ }
+ if (node.has_andnode()) {
+ const types::AndNode& and_node = node.andnode();
+ const types::TreeNode& left_arg = and_node.leftarg();
+ const types::TreeNode& right_arg = and_node.rightarg();
+ return std::make_shared<arrow::dataset::AndExpression>(translateNode(left_arg, env), translateNode(right_arg, env));
+ }
+ if (node.has_ornode()) {
+ const types::OrNode& or_node = node.ornode();
+ const types::TreeNode& left_arg = or_node.leftarg();
+ const types::TreeNode& right_arg = or_node.rightarg();
+ return std::make_shared<arrow::dataset::OrExpression>(translateNode(left_arg, env), translateNode(right_arg, env));
+ }
+ if (node.has_cpnode()) {
+ const types::ComparisonNode& cp_node = node.cpnode();
+ const std::string& op_name = cp_node.opname();
+ arrow::compute::CompareOperator op;
+ if (op_name == "equal") {
+ op = arrow::compute::CompareOperator::EQUAL;
+ } else if (op_name == "greaterThan") {
+ op = arrow::compute::CompareOperator::GREATER;
+ } else if (op_name == "greaterThanOrEqual") {
+ op = arrow::compute::CompareOperator::GREATER_EQUAL;
+ } else if (op_name == "lessThan") {
+ op = arrow::compute::CompareOperator::LESS;
+ } else if (op_name == "lessThanOrEqual") {
+ op = arrow::compute::CompareOperator::LESS_EQUAL;
+ } else {
+ std::string error_message = "Unknown operation name in comparison node";
+ env->ThrowNew(illegal_argument_exception_class, error_message.c_str());
+ return nullptr; // unreachable
+ }
+ const types::TreeNode& left_arg = cp_node.leftarg();
+ const types::TreeNode& right_arg = cp_node.rightarg();
+ return std::make_shared<arrow::dataset::ComparisonExpression>(op,
+ translateNode(left_arg, env), translateNode(right_arg, env));
+ }
+ if (node.has_notnode()) {
+ const types::NotNode& not_node = node.notnode();
+ const ::types::TreeNode& child = not_node.args();
+ std::shared_ptr<arrow::dataset::Expression> translatedChild = translateNode(child, env);
+ return std::make_shared<arrow::dataset::NotExpression>(translatedChild);
+ }
+ if (node.has_isvalidnode()) {
+ const types::IsValidNode& is_valid_node = node.isvalidnode();
+ const ::types::TreeNode& child = is_valid_node.args();
+ std::shared_ptr<arrow::dataset::Expression> translatedChild = translateNode(child, env);
+ return std::make_shared<arrow::dataset::IsValidExpression>(translatedChild);
+ }
+ std::string error_message = "Unknown node type";
+ env->ThrowNew(illegal_argument_exception_class, error_message.c_str());
+ return nullptr; // unreachable
+}
+
+std::shared_ptr<arrow::dataset::Expression> translateFilter(types::Condition condition, JNIEnv* env) {
+ const types::TreeNode& tree_node = condition.root();
+ return translateNode(tree_node, env);
+}
+
+/*
+ * Class: org_apache_arrow_dataset_jni_JniWrapper
+ * Method: closeDatasetFactory
+ * Signature: (J)V
+ */
+JNIEXPORT void JNICALL Java_org_apache_arrow_dataset_jni_JniWrapper_closeDatasetFactory
+ (JNIEnv *, jobject, jlong id) {
+ dataset_factory_holder_.Erase(id);
+}
+
+/*
+ * Class: org_apache_arrow_dataset_jni_JniWrapper
+ * Method: inspectSchema
+ * Signature: (J)[B
+ */
+JNIEXPORT jbyteArray JNICALL Java_org_apache_arrow_dataset_jni_JniWrapper_inspectSchema
+ (JNIEnv* env, jobject, jlong dataset_factor_id) {
+ std::shared_ptr<arrow::dataset::DatasetFactory> d
+ = dataset_factory_holder_.Lookup(dataset_factor_id);
+ JNI_ASSIGN_OR_THROW(std::shared_ptr<arrow::Schema> schema, d->Inspect())
+ return ToSchemaByteArray(env, schema);
+}
+
+/*
+ * Class: org_apache_arrow_dataset_jni_JniWrapper
+ * Method: createDataset
+ * Signature: (J[B)J
+ */
+JNIEXPORT jlong JNICALL Java_org_apache_arrow_dataset_jni_JniWrapper_createDataset
+ (JNIEnv* env, jobject, jlong dataset_factory_id, jbyteArray schema_bytes) {
+ std::shared_ptr<arrow::dataset::DatasetFactory> d
+ = dataset_factory_holder_.Lookup(dataset_factory_id);
+ std::shared_ptr<arrow::Schema> schema;
+ schema = FromSchemaByteArray(env, schema_bytes);
+ JNI_ASSIGN_OR_THROW(std::shared_ptr<arrow::dataset::Dataset> dataset, d->Finish(schema))
+ return dataset_holder_.Insert(dataset);
+}
+
+/*
+ * Class: org_apache_arrow_dataset_jni_JniWrapper
+ * Method: closeDataset
+ * Signature: (J)V
+ */
+JNIEXPORT void JNICALL Java_org_apache_arrow_dataset_jni_JniWrapper_closeDataset
+ (JNIEnv *, jobject, jlong id) {
+ dataset_holder_.Erase(id);
+}
+
+/*
+ * Class: org_apache_arrow_dataset_jni_JniWrapper
+ * Method: createScanner
+ * Signature: (J[Ljava/lang/String;[BJ)J
+ */
+JNIEXPORT jlong JNICALL Java_org_apache_arrow_dataset_jni_JniWrapper_createScanner
+ (JNIEnv* env, jobject, jlong dataset_id, jobjectArray columns,
+ jbyteArray filter, jlong batch_size) {
+ std::shared_ptr<arrow::dataset::ScanContext> context = std::make_shared<arrow::dataset::ScanContext>();
+ std::shared_ptr<arrow::dataset::Dataset> dataset = dataset_holder_.Lookup(dataset_id);
+ JNI_ASSIGN_OR_THROW(std::shared_ptr<arrow::dataset::ScannerBuilder> scanner_builder, dataset->NewScan())
+
+ std::vector<std::string> column_vector = ToStringVector(env, columns);
+ JNI_ASSERT_OK_OR_THROW(scanner_builder->Project(column_vector));
+ JNI_ASSERT_OK_OR_THROW(scanner_builder->BatchSize(batch_size));
+
+ // initialize filters
+ jsize exprs_len = env->GetArrayLength(filter);
+ jbyte* exprs_bytes = env->GetByteArrayElements(filter, 0);
+ types::Condition condition;
+ if (!ParseProtobuf(reinterpret_cast<uint8_t*>(exprs_bytes), exprs_len, &condition)) {
+ releaseFilterInput(filter, exprs_bytes, env);
+ std::string error_message = "bad protobuf message";
+ env->ThrowNew(illegal_argument_exception_class, error_message.c_str());
+ }
+ if (condition.has_root()) {
+ JNI_ASSERT_OK_OR_THROW(scanner_builder->Filter(translateFilter(condition, env)));
+ }
+ JNI_ASSIGN_OR_THROW(auto scanner, scanner_builder->Finish())
+ jlong id = scanner_holder_.Insert(scanner);
+ releaseFilterInput(filter, exprs_bytes, env);
+ return id;
+}
+
+/*
+ * Class: org_apache_arrow_dataset_jni_JniWrapper
+ * Method: getSchemaFromScanner
+ * Signature: (J)[B
+ */
+JNIEXPORT jbyteArray JNICALL Java_org_apache_arrow_dataset_jni_JniWrapper_getSchemaFromScanner
+ (JNIEnv* env, jobject, jlong scanner_id) {
+ std::shared_ptr<arrow::Schema> schema = scanner_holder_.Lookup(scanner_id)->schema();
+ return ToSchemaByteArray(env, schema);
+}
+
+/*
+ * Class: org_apache_arrow_dataset_jni_JniWrapper
+ * Method: getScanTasksFromScanner
+ * Signature: (J)[J
+ */
+JNIEXPORT jlongArray JNICALL Java_org_apache_arrow_dataset_jni_JniWrapper_getScanTasksFromScanner
+ (JNIEnv* env, jobject, jlong scanner_id) {
+ std::shared_ptr<arrow::dataset::Scanner> scanner = scanner_holder_.Lookup(scanner_id);
+ JNI_ASSIGN_OR_THROW(arrow::dataset::ScanTaskIterator itr, scanner->Scan())
+ std::vector<std::shared_ptr<arrow::dataset::ScanTask>> vector = collect(env, std::move(itr));
Review comment:
it seems like a a bad idea to collect an iterator if the intention is for the iterator to provide laziness?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] emkornfield commented on a change in pull request #7030: ARROW-7808: [Java][Dataset] Implement Datasets Java API
Posted by GitBox <gi...@apache.org>.
emkornfield commented on a change in pull request #7030:
URL: https://github.com/apache/arrow/pull/7030#discussion_r414988443
##########
File path: java/dataset/src/main/java/org/apache/arrow/util/SchemaUtils.java
##########
@@ -0,0 +1,72 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.arrow.util;
+
+import java.io.ByteArrayOutputStream;
+import java.io.IOException;
+import java.nio.channels.Channels;
+
+import org.apache.arrow.memory.BufferAllocator;
+import org.apache.arrow.vector.ipc.ReadChannel;
+import org.apache.arrow.vector.ipc.WriteChannel;
+import org.apache.arrow.vector.ipc.message.MessageChannelReader;
+import org.apache.arrow.vector.ipc.message.MessageResult;
+import org.apache.arrow.vector.ipc.message.MessageSerializer;
+import org.apache.arrow.vector.types.pojo.Schema;
+import org.apache.arrow.vector.util.ByteArrayReadableSeekableByteChannel;
+
+/**
+ * Schema utility class including serialization and deserialization.
+ */
+public class SchemaUtils {
Review comment:
this probably belongs in the vector package.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] emkornfield commented on pull request #7030: ARROW-7808: [Java][Dataset] Implement Dataset Java API by JNI to C++
Posted by GitBox <gi...@apache.org>.
emkornfield commented on pull request #7030:
URL: https://github.com/apache/arrow/pull/7030#issuecomment-726557997
@zhztheplayer I think I saw you requested a re-review is that accurate? Is the CI failure related to your changes?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] emkornfield commented on a change in pull request #7030: ARROW-7808: [Java][Dataset] Implement Datasets Java API by JNI to C++
Posted by GitBox <gi...@apache.org>.
emkornfield commented on a change in pull request #7030:
URL: https://github.com/apache/arrow/pull/7030#discussion_r425568045
##########
File path: java/dataset/src/test/java/org/apache/arrow/dataset/file/TestSingleFileDataset.java
##########
@@ -0,0 +1,237 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.arrow.dataset.file;
+
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertTrue;
+
+import java.util.Collections;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.Objects;
+import java.util.stream.Collectors;
+
+import org.apache.arrow.dataset.DatasetTypes;
+import org.apache.arrow.dataset.filter.Filter;
+import org.apache.arrow.dataset.jni.TestNativeDataset;
+import org.apache.arrow.dataset.scanner.ScanOptions;
+import org.apache.arrow.util.AutoCloseables;
+import org.apache.arrow.vector.FieldVector;
+import org.apache.arrow.vector.VectorLoader;
+import org.apache.arrow.vector.VectorSchemaRoot;
+import org.apache.arrow.vector.ipc.message.ArrowRecordBatch;
+import org.apache.arrow.vector.types.Types;
+import org.apache.arrow.vector.types.pojo.Schema;
+import org.apache.avro.generic.GenericData;
+import org.apache.avro.generic.GenericRecord;
+import org.apache.avro.generic.GenericRecordBuilder;
+import org.junit.ClassRule;
+import org.junit.Test;
+import org.junit.rules.TemporaryFolder;
+
+import com.google.common.collect.ImmutableList;
+
+public class TestSingleFileDataset extends TestNativeDataset {
+
+ @ClassRule
+ public static final TemporaryFolder TMP = new TemporaryFolder();
+
+ public static final String AVRO_SCHEMA_USER = "user.avsc";
+
+ @Test
+ public void testParquetRead() throws Exception {
+ ParquetWriteSupport writeSupport = new ParquetWriteSupport(AVRO_SCHEMA_USER, TMP.newFolder());
+ GenericRecord record = new GenericData.Record(writeSupport.getAvroSchema());
+ record.put("id", 1);
+ record.put("name", "a");
+ writeSupport.writeRecord(record);
+ writeSupport.close();
+
+ SingleFileDatasetFactory factory = new SingleFileDatasetFactory(rootAllocator(),
+ FileFormat.PARQUET, FileSystem.LOCAL, writeSupport.getOutputFile());
+ ScanOptions options = new ScanOptions(new String[0], Filter.EMPTY, 100);
+
+ Schema schema = inferResultSchemaFromFactory(factory, options);
+ assertEquals(2, schema.getFields().size());
+ assertEquals("id", schema.getFields().get(0).getName());
+ assertEquals("name", schema.getFields().get(1).getName());
+ assertEquals(Types.MinorType.INT.getType(), schema.getFields().get(0).getType());
+ assertEquals(Types.MinorType.VARCHAR.getType(), schema.getFields().get(1).getType());
+
+ assertSingleTaskProduced(factory, options);
+
+ List<ArrowRecordBatch> datum = collectResultFromFactory(factory, options);
+ assertEquals(1, datum.size());
+ checkParquetReadResult(schema, Collections.singletonList(record), datum);
+ AutoCloseables.close(datum);
+ }
+
+ @Test
+ public void testParquetProjector() throws Exception {
+ ParquetWriteSupport writeSupport = new ParquetWriteSupport(AVRO_SCHEMA_USER, TMP.newFolder());
+ GenericRecord record = new GenericData.Record(writeSupport.getAvroSchema());
+ record.put("id", 1);
+ record.put("name", "a");
+ writeSupport.writeRecord(record);
+ writeSupport.close();
+
+ SingleFileDatasetFactory factory = new SingleFileDatasetFactory(rootAllocator(),
+ FileFormat.PARQUET, FileSystem.LOCAL, writeSupport.getOutputFile());
+ ScanOptions options = new ScanOptions(new String[]{"id"}, Filter.EMPTY, 100);
+
+ Schema schema = inferResultSchemaFromFactory(factory, options);
+ assertEquals(1, schema.getFields().size());
+ assertEquals("id", schema.getFields().get(0).getName());
+ assertEquals(Types.MinorType.INT.getType(), schema.getFields().get(0).getType());
+
+ assertSingleTaskProduced(factory, options);
+
+ List<ArrowRecordBatch> datum = collectResultFromFactory(factory, options);
+ assertEquals(1, datum.size());
+ org.apache.avro.Schema expectedSchema = truncateAvroSchema(writeSupport.getAvroSchema(), 0, 1);
+ checkParquetReadResult(schema,
+ Collections.singletonList(
+ new GenericRecordBuilder(
+ expectedSchema)
+ .set("id", 1)
+ .build()), datum);
+ AutoCloseables.close(datum);
+ }
+
+ @Test
+ public void testParquetFilter() throws Exception {
+ ParquetWriteSupport writeSupport = new ParquetWriteSupport(AVRO_SCHEMA_USER, TMP.newFolder());
Review comment:
please try to break tests into: setup, execute and verify sections. lack of spacing/comments make these hard to read.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] emkornfield commented on a change in pull request #7030: ARROW-7808: [Java][Dataset] Implement Datasets Java API by JNI to C++
Posted by GitBox <gi...@apache.org>.
emkornfield commented on a change in pull request #7030:
URL: https://github.com/apache/arrow/pull/7030#discussion_r417074480
##########
File path: cpp/src/jni/dataset/jni_wrapper.cpp
##########
@@ -0,0 +1,577 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include <arrow/dataset/file_base.h>
+#include <arrow/dataset/api.h>
+#include <arrow/filesystem/localfs.h>
+#include <arrow/ipc/api.h>
+#include <arrow/util/iterator.h>
+#include <arrow/filesystem/hdfs.h>
+#include <arrow/io/api.h>
+#include <google/protobuf/io/coded_stream.h>
+#include <google/protobuf/message.h>
+#include "concurrent_map.h"
+#include "arrow/compute/kernel.h"
+#include "arrow/compute/kernels/cast.h"
+#include "arrow/compute/kernels/compare.h"
+#include "jni/dataset/Types.pb.h"
+
+#include "org_apache_arrow_dataset_jni_JniWrapper.h"
+#include "org_apache_arrow_dataset_file_JniWrapper.h"
+
+static jclass illegal_access_exception_class;
+static jclass illegal_argument_exception_class;
+static jclass runtime_exception_class;
+
+static jclass record_batch_handle_class;
+static jclass record_batch_handle_field_class;
+static jclass record_batch_handle_buffer_class;
+
+static jmethodID record_batch_handle_constructor;
+static jmethodID record_batch_handle_field_constructor;
+static jmethodID record_batch_handle_buffer_constructor;
+
+static jint JNI_VERSION = JNI_VERSION_1_6;
+
+using arrow::jni::ConcurrentMap;
+
+static ConcurrentMap<std::shared_ptr<arrow::dataset::DatasetFactory>> dataset_factory_holder_;
Review comment:
extra memory and resolution overhead. The instance cache should be handled by the underlying allocator which should be highly optimized for returning pointer sized values.
Could you explain how a self-growing ID is useful for debugging purposes? If it is required would a DEBUG log with only an atomic int associated with each new ID be sufficient?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] emkornfield commented on a change in pull request #7030: ARROW-7808: [Java][Dataset] Implement Dataset Java API by JNI to C++
Posted by GitBox <gi...@apache.org>.
emkornfield commented on a change in pull request #7030:
URL: https://github.com/apache/arrow/pull/7030#discussion_r567202095
##########
File path: cpp/src/jni/dataset/jni_wrapper.cpp
##########
@@ -0,0 +1,859 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include <mutex>
+
+#include <arrow/array.h>
+#include <arrow/dataset/api.h>
+#include <arrow/dataset/file_base.h>
+#include <arrow/filesystem/localfs.h>
+#include <arrow/io/api.h>
+#include <arrow/ipc/api.h>
+#include <arrow/util/iterator.h>
+#include <arrow/util/logging.h>
+
+#include "org_apache_arrow_dataset_file_JniWrapper.h"
+#include "org_apache_arrow_dataset_jni_JniWrapper.h"
+#include "org_apache_arrow_dataset_jni_NativeMemoryPool.h"
+
+namespace arrow {
+namespace dataset {
+namespace jni {
+
+static jclass illegal_access_exception_class;
+static jclass illegal_argument_exception_class;
+static jclass runtime_exception_class;
+
+static jclass record_batch_handle_class;
+static jclass record_batch_handle_field_class;
+static jclass record_batch_handle_buffer_class;
+static jclass java_reservation_listener_class;
+
+static jmethodID record_batch_handle_constructor;
+static jmethodID record_batch_handle_field_constructor;
+static jmethodID record_batch_handle_buffer_constructor;
+static jmethodID reserve_memory_method;
+static jmethodID unreserve_memory_method;
+
+static jlong default_memory_pool_id = -1L;
+
+static jint JNI_VERSION = JNI_VERSION_1_6;
+
+class JniPendingException : public std::runtime_error {
+ public:
+ explicit JniPendingException(const std::string& arg) : runtime_error(arg) {}
+};
+
+void ThrowPendingException(const std::string& message) {
+ throw JniPendingException(message);
+}
+
+template <typename T>
+T JniGetOrThrow(arrow::Result<T> result) {
+ if (!result.status().ok()) {
+ ThrowPendingException(result.status().message());
+ }
+ return std::move(result).ValueOrDie();
+}
+
+void JniAssertOkOrThrow(arrow::Status status) {
+ if (!status.ok()) {
+ ThrowPendingException(status.message());
+ }
+}
+
+void JniThrow(std::string message) { ThrowPendingException(message); }
+
+jclass CreateGlobalClassReference(JNIEnv* env, const char* class_name) {
+ jclass local_class = env->FindClass(class_name);
+ jclass global_class = (jclass)env->NewGlobalRef(local_class);
+ env->DeleteLocalRef(local_class);
+ return global_class;
+}
+
+arrow::Result<jmethodID> GetMethodID(JNIEnv* env, jclass this_class, const char* name,
+ const char* sig) {
+ jmethodID ret = env->GetMethodID(this_class, name, sig);
+ if (ret == nullptr) {
+ std::string error_message = "Unable to find method " + std::string(name) +
+ " within signature" + std::string(sig);
+ return arrow::Status::Invalid(error_message);
+ }
+ return ret;
+}
+
+arrow::Result<jmethodID> GetStaticMethodID(JNIEnv* env, jclass this_class,
+ const char* name, const char* sig) {
+ jmethodID ret = env->GetStaticMethodID(this_class, name, sig);
+ if (ret == nullptr) {
+ std::string error_message = "Unable to find static method " + std::string(name) +
+ " within signature" + std::string(sig);
+ return arrow::Status::Invalid(error_message);
+ }
+ return ret;
+}
+
+std::shared_ptr<arrow::Schema> SchemaFromColumnNames(
+ const std::shared_ptr<arrow::Schema>& input,
+ const std::vector<std::string>& column_names) {
+ std::vector<std::shared_ptr<arrow::Field>> columns;
+ for (const auto& name : column_names) {
+ columns.push_back(input->GetFieldByName(name));
+ }
+ return std::make_shared<arrow::Schema>(columns);
+}
+
+arrow::Result<std::shared_ptr<arrow::dataset::FileFormat>> GetFileFormat(jint id) {
+ switch (id) {
+ case 0:
+ return std::make_shared<arrow::dataset::ParquetFileFormat>();
+ default:
+ std::string error_message = "illegal file format id: " + std::to_string(id);
+ return arrow::Status::Invalid(error_message);
+ }
+}
+
+std::string JStringToCString(JNIEnv* env, jstring string) {
+ if (string == nullptr) {
+ return std::string();
+ }
+ jboolean copied;
+ const char* chars = env->GetStringUTFChars(string, &copied);
+ std::string ret = strdup(chars);
+ env->ReleaseStringUTFChars(string, chars);
+ return ret;
+}
+
+std::vector<std::string> ToStringVector(JNIEnv* env, jobjectArray& str_array) {
+ int length = env->GetArrayLength(str_array);
+ std::vector<std::string> vector;
+ for (int i = 0; i < length; i++) {
+ auto string = (jstring)(env->GetObjectArrayElement(str_array, i));
+ vector.push_back(JStringToCString(env, string));
+ }
+ return vector;
+}
+
+template <typename T>
+jlong CreateNativeRef(std::shared_ptr<T> t) {
+ std::shared_ptr<T>* retained_ptr = new std::shared_ptr<T>(t);
+ return reinterpret_cast<jlong>(retained_ptr);
+}
+
+template <typename T>
+std::shared_ptr<T> RetrieveNativeInstance(jlong ref) {
+ std::shared_ptr<T>* retrieved_ptr = reinterpret_cast<std::shared_ptr<T>*>(ref);
+ return *retrieved_ptr;
+}
+
+template <typename T>
+void ReleaseNativeRef(jlong ref) {
+ std::shared_ptr<T>* retrieved_ptr = reinterpret_cast<std::shared_ptr<T>*>(ref);
+ delete retrieved_ptr;
+}
+
+arrow::Result<jbyteArray> ToSchemaByteArray(JNIEnv* env,
+ std::shared_ptr<arrow::Schema> schema) {
+ ARROW_ASSIGN_OR_RAISE(
+ std::shared_ptr<arrow::Buffer> buffer,
+ arrow::ipc::SerializeSchema(*schema, arrow::default_memory_pool()))
+
+ jbyteArray out = env->NewByteArray(buffer->size());
+ auto src = reinterpret_cast<const jbyte*>(buffer->data());
+ env->SetByteArrayRegion(out, 0, buffer->size(), src);
+ return out;
+}
+
+arrow::Result<std::shared_ptr<arrow::Schema>> FromSchemaByteArray(
+ JNIEnv* env, jbyteArray schemaBytes) {
+ arrow::ipc::DictionaryMemo in_memo;
+ int schemaBytes_len = env->GetArrayLength(schemaBytes);
+ jbyte* schemaBytes_data = env->GetByteArrayElements(schemaBytes, nullptr);
+ auto serialized_schema = std::make_shared<arrow::Buffer>(
+ reinterpret_cast<uint8_t*>(schemaBytes_data), schemaBytes_len);
+ arrow::io::BufferReader buf_reader(serialized_schema);
+ ARROW_ASSIGN_OR_RAISE(std::shared_ptr<arrow::Schema> schema,
+ arrow::ipc::ReadSchema(&buf_reader, &in_memo))
+ env->ReleaseByteArrayElements(schemaBytes, schemaBytes_data, JNI_ABORT);
+ return schema;
+}
+
+/// Listener to act on reservations/unreservations from ReservationListenableMemoryPool.
+///
+/// Note the memory pool will call this listener only on block-level memory
+/// reservation/unreservation is granted. So the invocation parameter "size" is always
+/// multiple of block size (by default, 512k) specified in memory pool.
+class ReservationListener {
+ public:
+ virtual ~ReservationListener() = default;
+
+ virtual arrow::Status OnReservation(int64_t size) = 0;
+ virtual arrow::Status OnRelease(int64_t size) = 0;
+
+ protected:
+ ReservationListener() = default;
+};
+
+class ReservationListenableMemoryPoolImpl {
Review comment:
nit: Ithink this can be an anonymous namespace?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] emkornfield commented on a change in pull request #7030: ARROW-7808: [Java][Dataset] Implement Dataset Java API by JNI to C++
Posted by GitBox <gi...@apache.org>.
emkornfield commented on a change in pull request #7030:
URL: https://github.com/apache/arrow/pull/7030#discussion_r571715852
##########
File path: cpp/src/jni/dataset/jni_util.h
##########
@@ -0,0 +1,143 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#pragma once
+
+#include "arrow/array.h"
+#include "arrow/io/api.h"
+#include "arrow/ipc/api.h"
+#include "arrow/memory_pool.h"
+#include "arrow/result.h"
+#include "arrow/type.h"
+
+#include <jni.h>
+
+namespace arrow {
+namespace dataset {
+namespace jni {
+
+class JniPendingException : public std::runtime_error {
Review comment:
does this need to be in the header? Please add docs?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] zhztheplayer commented on a change in pull request #7030: ARROW-7808: [Java][Dataset] Implement Datasets Java API by JNI to C++
Posted by GitBox <gi...@apache.org>.
zhztheplayer commented on a change in pull request #7030:
URL: https://github.com/apache/arrow/pull/7030#discussion_r417798498
##########
File path: java/dataset/src/main/java/org/apache/arrow/util/SchemaUtils.java
##########
@@ -0,0 +1,72 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.arrow.util;
+
+import java.io.ByteArrayOutputStream;
+import java.io.IOException;
+import java.nio.channels.Channels;
+
+import org.apache.arrow.memory.BufferAllocator;
+import org.apache.arrow.vector.ipc.ReadChannel;
+import org.apache.arrow.vector.ipc.WriteChannel;
+import org.apache.arrow.vector.ipc.message.MessageChannelReader;
+import org.apache.arrow.vector.ipc.message.MessageResult;
+import org.apache.arrow.vector.ipc.message.MessageSerializer;
+import org.apache.arrow.vector.types.pojo.Schema;
+import org.apache.arrow.vector.util.ByteArrayReadableSeekableByteChannel;
+
+/**
+ * Schema utility class including serialization and deserialization.
+ */
+public class SchemaUtils {
Review comment:
done
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] zhztheplayer edited a comment on pull request #7030: ARROW-7808: [Java][Dataset] Implement Datasets Java API by JNI to C++
Posted by GitBox <gi...@apache.org>.
zhztheplayer edited a comment on pull request #7030:
URL: https://github.com/apache/arrow/pull/7030#issuecomment-654004147
@jacques-n
I think we have several choices: we can try implement a c++ memory pool which is backed by a Java Allocator, actually I have a PoC branch for that: https://github.com/zhztheplayer/arrow-1/tree/ARROW-7808-wip-bridged-allocator.
Or we can continue hacking Java API to reserve direct memory after buffer is registered (performance impact will be smaller). The reservation should happen inside Netty API because Netty has its own direct memory counter: https://github.com/netty/netty/blob/cbe238a95bab238455e3ff1849d450a9836f39ef/common/src/main/java/io/netty/util/internal/PlatformDependent.java#L749-L766.
Which one do you suggest or maybe there's a better one than both?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] zhztheplayer commented on a change in pull request #7030: ARROW-7808: [Java][Dataset] Implement Dataset Java API by JNI to C++
Posted by GitBox <gi...@apache.org>.
zhztheplayer commented on a change in pull request #7030:
URL: https://github.com/apache/arrow/pull/7030#discussion_r567756409
##########
File path: cpp/src/jni/dataset/jni_wrapper.cpp
##########
@@ -0,0 +1,859 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include <mutex>
+
+#include <arrow/array.h>
+#include <arrow/dataset/api.h>
+#include <arrow/dataset/file_base.h>
+#include <arrow/filesystem/localfs.h>
+#include <arrow/io/api.h>
+#include <arrow/ipc/api.h>
+#include <arrow/util/iterator.h>
+#include <arrow/util/logging.h>
+
+#include "org_apache_arrow_dataset_file_JniWrapper.h"
+#include "org_apache_arrow_dataset_jni_JniWrapper.h"
+#include "org_apache_arrow_dataset_jni_NativeMemoryPool.h"
+
+namespace arrow {
+namespace dataset {
+namespace jni {
+
+static jclass illegal_access_exception_class;
+static jclass illegal_argument_exception_class;
+static jclass runtime_exception_class;
+
+static jclass record_batch_handle_class;
+static jclass record_batch_handle_field_class;
+static jclass record_batch_handle_buffer_class;
+static jclass java_reservation_listener_class;
+
+static jmethodID record_batch_handle_constructor;
+static jmethodID record_batch_handle_field_constructor;
+static jmethodID record_batch_handle_buffer_constructor;
+static jmethodID reserve_memory_method;
+static jmethodID unreserve_memory_method;
+
+static jlong default_memory_pool_id = -1L;
+
+static jint JNI_VERSION = JNI_VERSION_1_6;
+
+class JniPendingException : public std::runtime_error {
+ public:
+ explicit JniPendingException(const std::string& arg) : runtime_error(arg) {}
+};
+
+void ThrowPendingException(const std::string& message) {
+ throw JniPendingException(message);
+}
+
+template <typename T>
+T JniGetOrThrow(arrow::Result<T> result) {
+ if (!result.status().ok()) {
+ ThrowPendingException(result.status().message());
+ }
+ return std::move(result).ValueOrDie();
+}
+
+void JniAssertOkOrThrow(arrow::Status status) {
+ if (!status.ok()) {
+ ThrowPendingException(status.message());
+ }
+}
+
+void JniThrow(std::string message) { ThrowPendingException(message); }
+
+jclass CreateGlobalClassReference(JNIEnv* env, const char* class_name) {
+ jclass local_class = env->FindClass(class_name);
+ jclass global_class = (jclass)env->NewGlobalRef(local_class);
+ env->DeleteLocalRef(local_class);
+ return global_class;
+}
+
+arrow::Result<jmethodID> GetMethodID(JNIEnv* env, jclass this_class, const char* name,
+ const char* sig) {
+ jmethodID ret = env->GetMethodID(this_class, name, sig);
+ if (ret == nullptr) {
+ std::string error_message = "Unable to find method " + std::string(name) +
+ " within signature" + std::string(sig);
+ return arrow::Status::Invalid(error_message);
+ }
+ return ret;
+}
+
+arrow::Result<jmethodID> GetStaticMethodID(JNIEnv* env, jclass this_class,
+ const char* name, const char* sig) {
+ jmethodID ret = env->GetStaticMethodID(this_class, name, sig);
+ if (ret == nullptr) {
+ std::string error_message = "Unable to find static method " + std::string(name) +
+ " within signature" + std::string(sig);
+ return arrow::Status::Invalid(error_message);
+ }
+ return ret;
+}
+
+std::shared_ptr<arrow::Schema> SchemaFromColumnNames(
+ const std::shared_ptr<arrow::Schema>& input,
+ const std::vector<std::string>& column_names) {
+ std::vector<std::shared_ptr<arrow::Field>> columns;
+ for (const auto& name : column_names) {
+ columns.push_back(input->GetFieldByName(name));
+ }
+ return std::make_shared<arrow::Schema>(columns);
+}
+
+arrow::Result<std::shared_ptr<arrow::dataset::FileFormat>> GetFileFormat(jint id) {
Review comment:
done
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] zhztheplayer commented on a change in pull request #7030: ARROW-7808: [Java][Dataset] Implement Datasets Java API by JNI to C++
Posted by GitBox <gi...@apache.org>.
zhztheplayer commented on a change in pull request #7030:
URL: https://github.com/apache/arrow/pull/7030#discussion_r439266993
##########
File path: java/dataset/src/main/java/org/apache/arrow/memory/Ownerships.java
##########
@@ -0,0 +1,47 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.arrow.memory;
+
+/**
+ * Utility class managing ownership's transferring between Native Arrow buffers and Java Arrow buffers.
+ */
+public class Ownerships {
+ private static final Ownerships INSTANCE = new Ownerships();
+
+ private Ownerships() {
+ }
+
+ public static Ownerships get() {
+ return INSTANCE;
+ }
+
+ /**
+ * Returned ledger's ref count is initialized or increased by 1.
+ */
+ public BufferLedger takeOwnership(BaseAllocator allocator, AllocationManager allocationManager) {
+ final BufferLedger ledger = allocationManager.associate(allocator);
+ long size = allocationManager.getSize();
+ boolean succeed = allocator.forceAllocate(size);
+ if (!succeed) {
+ throw new OutOfMemoryException("Target allocator is full");
+ }
+ ((BaseAllocator) allocationManager.getOwningLedger().getAllocator()).releaseBytes(size);
Review comment:
Yes calls to `forceAllocate` should be synchronized.
The fix appear to be in a separate topic because in Java API we don't yet give allocator the functionality to take over ownerless memory. Some slight refactors to allocation process is required then. I'll open a ticket for that.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] zhztheplayer commented on a change in pull request #7030: ARROW-7808: [Java][Dataset] Implement Datasets Java API by JNI to C++
Posted by GitBox <gi...@apache.org>.
zhztheplayer commented on a change in pull request #7030:
URL: https://github.com/apache/arrow/pull/7030#discussion_r416295541
##########
File path: cpp/src/jni/dataset/proto/Types.proto
##########
@@ -0,0 +1,149 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+syntax = "proto2";
+package types;
+
+option java_package = "org.apache.arrow.dataset";
Review comment:
Thanks, I think I would understand any of the concerns against a too detailed mapping. (actually currently Fragment is not mapped to Java-side so it's not yet a 1-1 mapping) I'll try remove more JNI stuffs.
As for filter, If I understand correctly it's OK to keep Java API but the JNI mapping for filters is considered fragile, right? I can remove the mapping anyway but when users read parquet files from Java they'll not be able to filter row groups to reduce I/O. Which is extremely important when low-selectivity filter is specified.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] zhztheplayer commented on a change in pull request #7030: ARROW-7808: [Java][Dataset] Implement Datasets Java API by JNI to C++
Posted by GitBox <gi...@apache.org>.
zhztheplayer commented on a change in pull request #7030:
URL: https://github.com/apache/arrow/pull/7030#discussion_r426376484
##########
File path: java/dataset/src/main/java/org/apache/arrow/dataset/source/DatasetFactory.java
##########
@@ -0,0 +1,51 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.arrow.dataset.source;
+
+import org.apache.arrow.vector.types.pojo.Schema;
+
+/**
+ * DataSourceDiscovery provides a way to inspect a DataSource potential
Review comment:
done, thanks for catching.
##########
File path: java/dataset/src/test/java/org/apache/arrow/dataset/file/TestSingleFileDatasetFactory.java
##########
@@ -0,0 +1,36 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.arrow.dataset.file;
+
+import org.apache.arrow.memory.RootAllocator;
+import org.junit.Test;
+
+public class TestSingleFileDatasetFactory {
+
+ @Test(expected = RuntimeException.class)
+ public void testErrorHandling1() {
+ new SingleFileDatasetFactory(new RootAllocator(Long.MAX_VALUE),
+ FileFormat.NONE, FileSystem.LOCAL, "NON_EXIST_FILE");
+ }
+
+ @Test(expected = RuntimeException.class)
Review comment:
done
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] zhztheplayer commented on a change in pull request #7030: ARROW-7808: [Java][Dataset] Implement Datasets Java API by JNI to C++
Posted by GitBox <gi...@apache.org>.
zhztheplayer commented on a change in pull request #7030:
URL: https://github.com/apache/arrow/pull/7030#discussion_r415664616
##########
File path: cpp/src/jni/dataset/proto/Types.proto
##########
@@ -0,0 +1,149 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+syntax = "proto2";
+package types;
+
+option java_package = "org.apache.arrow.dataset";
Review comment:
Currently it is for passing datasets filter via JNI. I see Wes has recommended not to include filter bindings (in the bottom comment) in this version. I may consider removing it in a later commit.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] emkornfield commented on a change in pull request #7030: ARROW-7808: [Java][Dataset] Implement Datasets Java API
Posted by GitBox <gi...@apache.org>.
emkornfield commented on a change in pull request #7030:
URL: https://github.com/apache/arrow/pull/7030#discussion_r415417277
##########
File path: cpp/src/jni/dataset/concurrent_map.h
##########
@@ -0,0 +1,81 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ */
+
+#ifndef JNI_ID_TO_MODULE_MAP_H
+#define JNI_ID_TO_MODULE_MAP_H
+
+#include <memory>
+#include <mutex>
+#include <unordered_map>
+#include <utility>
+
+#include "jni.h"
+#include "arrow/util/macros.h"
+
+namespace arrow {
Review comment:
see comments below, I don't think we should be using concurrent_map in general, especially for new use cases so we can drop it for now.
> By the way, should we somehow migrate src/gandiva/jni to src/jni/gandiva, or migrate src/jni to src/arrow/jni? It seems that currently they are organized inconsistently.
I don't have a strong opinion here probably best to discuss this on the mailing list.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] emkornfield commented on pull request #7030: ARROW-7808: [Java][Dataset] Implement Datasets Java API by JNI to C++
Posted by GitBox <gi...@apache.org>.
emkornfield commented on pull request #7030:
URL: https://github.com/apache/arrow/pull/7030#issuecomment-675576763
@zhztheplayer based on ML discussion I think the consensus is to use Bits.java to limit allocation (ideally with some sort of pre-reservation to limit JNI calls). Do you think you will be able to add that to this PR?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] Sebastiaan-Alvarez-Rodriguez commented on pull request #7030: ARROW-7808: [Java][Dataset] Implement Datasets Java API by JNI to C++
Posted by GitBox <gi...@apache.org>.
Sebastiaan-Alvarez-Rodriguez commented on pull request #7030:
URL: https://github.com/apache/arrow/pull/7030#issuecomment-700644439
Hi there @zhztheplayer,
thanks for the response!
On building the project:
> Have you checked if arrow_dataset_jni.so was successfully installed within the jar arrow-dataset-XXX.jar? If not, specifying arrow.cpp.build.dir may help.
I checked the file `/java/dataset/target/arrow-dataset-1.0.0-SNAPSHOT.jar`, and it indeed does not contain the `.so` file you mentioned. When I compile using
`mvn clean install -P arrow-jni -am -Darrow.cpp.build.dir=../../cpp/build/release -Dmaven.test.skip=true`
the `.so` file is added. I took these extra parameters from `java/README.md`. I don't know how I could have missed that.
> I did a little try and it appears that System.loadLibrary requires for property java.library.path to be set to include library locations. If we change to use this way, do users have to do this every time before getting started?
Good point! My IDE handles runtime parameters for me, including `-Djava.library.path=...`.
Although there is no consensus on the internet, it would seem that not specifying the `java.library.path` will make use of an operating system-dependent default value.
E.g. for most Linux distros, the default value would be the LD_LIBRARY_PATH environment variable.
What you could do, is making sure your `.so` files are always installed in standard locations for each operating system, and then rely on the default value for `java.library.path`. However, it is unsafe to rely on the default value for `java.library.path` while there is no (official) documentation to back it (as far as I could find).
With this in mind, I think it would indeed be best to include the `.so` in the jar and use your original strategy for loading it.
Thank you for your time and have a nice day!
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] emkornfield commented on a change in pull request #7030: ARROW-7808: [Java][Dataset] Implement Dataset Java API by JNI to C++
Posted by GitBox <gi...@apache.org>.
emkornfield commented on a change in pull request #7030:
URL: https://github.com/apache/arrow/pull/7030#discussion_r551609465
##########
File path: cpp/src/arrow/memory_pool.h
##########
@@ -149,6 +149,43 @@ class ARROW_EXPORT ProxyMemoryPool : public MemoryPool {
std::unique_ptr<ProxyMemoryPoolImpl> impl_;
};
+class ARROW_EXPORT ReservationListener {
+ public:
+ virtual ~ReservationListener();
+
+ virtual Status OnReservation(int64_t size) = 0;
+ virtual Status OnRelease(int64_t size) = 0;
+
+ protected:
+ ReservationListener();
Review comment:
```suggestion
ReservationListener() = default;
```
##########
File path: cpp/src/arrow/memory_pool.h
##########
@@ -149,6 +149,43 @@ class ARROW_EXPORT ProxyMemoryPool : public MemoryPool {
std::unique_ptr<ProxyMemoryPoolImpl> impl_;
};
+class ARROW_EXPORT ReservationListener {
+ public:
+ virtual ~ReservationListener();
+
+ virtual Status OnReservation(int64_t size) = 0;
+ virtual Status OnRelease(int64_t size) = 0;
+
+ protected:
+ ReservationListener();
+};
+
+class ARROW_EXPORT ReservationListenableMemoryPool : public MemoryPool {
Review comment:
documentation please.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] emkornfield commented on a change in pull request #7030: ARROW-7808: [Java][Dataset] Implement Dataset Java API by JNI to C++
Posted by GitBox <gi...@apache.org>.
emkornfield commented on a change in pull request #7030:
URL: https://github.com/apache/arrow/pull/7030#discussion_r567202661
##########
File path: cpp/src/jni/dataset/jni_wrapper.cpp
##########
@@ -0,0 +1,859 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include <mutex>
+
+#include <arrow/array.h>
+#include <arrow/dataset/api.h>
+#include <arrow/dataset/file_base.h>
+#include <arrow/filesystem/localfs.h>
+#include <arrow/io/api.h>
+#include <arrow/ipc/api.h>
+#include <arrow/util/iterator.h>
+#include <arrow/util/logging.h>
+
+#include "org_apache_arrow_dataset_file_JniWrapper.h"
+#include "org_apache_arrow_dataset_jni_JniWrapper.h"
+#include "org_apache_arrow_dataset_jni_NativeMemoryPool.h"
+
+namespace arrow {
+namespace dataset {
+namespace jni {
+
+static jclass illegal_access_exception_class;
+static jclass illegal_argument_exception_class;
+static jclass runtime_exception_class;
+
+static jclass record_batch_handle_class;
+static jclass record_batch_handle_field_class;
+static jclass record_batch_handle_buffer_class;
+static jclass java_reservation_listener_class;
+
+static jmethodID record_batch_handle_constructor;
+static jmethodID record_batch_handle_field_constructor;
+static jmethodID record_batch_handle_buffer_constructor;
+static jmethodID reserve_memory_method;
+static jmethodID unreserve_memory_method;
+
+static jlong default_memory_pool_id = -1L;
+
+static jint JNI_VERSION = JNI_VERSION_1_6;
+
+class JniPendingException : public std::runtime_error {
+ public:
+ explicit JniPendingException(const std::string& arg) : runtime_error(arg) {}
+};
+
+void ThrowPendingException(const std::string& message) {
+ throw JniPendingException(message);
+}
+
+template <typename T>
+T JniGetOrThrow(arrow::Result<T> result) {
+ if (!result.status().ok()) {
+ ThrowPendingException(result.status().message());
+ }
+ return std::move(result).ValueOrDie();
+}
+
+void JniAssertOkOrThrow(arrow::Status status) {
+ if (!status.ok()) {
+ ThrowPendingException(status.message());
+ }
+}
+
+void JniThrow(std::string message) { ThrowPendingException(message); }
+
+jclass CreateGlobalClassReference(JNIEnv* env, const char* class_name) {
+ jclass local_class = env->FindClass(class_name);
+ jclass global_class = (jclass)env->NewGlobalRef(local_class);
+ env->DeleteLocalRef(local_class);
+ return global_class;
+}
+
+arrow::Result<jmethodID> GetMethodID(JNIEnv* env, jclass this_class, const char* name,
+ const char* sig) {
+ jmethodID ret = env->GetMethodID(this_class, name, sig);
+ if (ret == nullptr) {
+ std::string error_message = "Unable to find method " + std::string(name) +
+ " within signature" + std::string(sig);
+ return arrow::Status::Invalid(error_message);
+ }
+ return ret;
+}
+
+arrow::Result<jmethodID> GetStaticMethodID(JNIEnv* env, jclass this_class,
+ const char* name, const char* sig) {
+ jmethodID ret = env->GetStaticMethodID(this_class, name, sig);
+ if (ret == nullptr) {
+ std::string error_message = "Unable to find static method " + std::string(name) +
+ " within signature" + std::string(sig);
+ return arrow::Status::Invalid(error_message);
+ }
+ return ret;
+}
+
+std::shared_ptr<arrow::Schema> SchemaFromColumnNames(
+ const std::shared_ptr<arrow::Schema>& input,
+ const std::vector<std::string>& column_names) {
+ std::vector<std::shared_ptr<arrow::Field>> columns;
+ for (const auto& name : column_names) {
+ columns.push_back(input->GetFieldByName(name));
+ }
+ return std::make_shared<arrow::Schema>(columns);
+}
+
+arrow::Result<std::shared_ptr<arrow::dataset::FileFormat>> GetFileFormat(jint id) {
+ switch (id) {
+ case 0:
+ return std::make_shared<arrow::dataset::ParquetFileFormat>();
+ default:
+ std::string error_message = "illegal file format id: " + std::to_string(id);
+ return arrow::Status::Invalid(error_message);
+ }
+}
+
+std::string JStringToCString(JNIEnv* env, jstring string) {
+ if (string == nullptr) {
+ return std::string();
+ }
+ jboolean copied;
+ const char* chars = env->GetStringUTFChars(string, &copied);
+ std::string ret = strdup(chars);
+ env->ReleaseStringUTFChars(string, chars);
+ return ret;
+}
+
+std::vector<std::string> ToStringVector(JNIEnv* env, jobjectArray& str_array) {
+ int length = env->GetArrayLength(str_array);
+ std::vector<std::string> vector;
+ for (int i = 0; i < length; i++) {
+ auto string = (jstring)(env->GetObjectArrayElement(str_array, i));
+ vector.push_back(JStringToCString(env, string));
+ }
+ return vector;
+}
+
+template <typename T>
+jlong CreateNativeRef(std::shared_ptr<T> t) {
+ std::shared_ptr<T>* retained_ptr = new std::shared_ptr<T>(t);
+ return reinterpret_cast<jlong>(retained_ptr);
+}
+
+template <typename T>
+std::shared_ptr<T> RetrieveNativeInstance(jlong ref) {
+ std::shared_ptr<T>* retrieved_ptr = reinterpret_cast<std::shared_ptr<T>*>(ref);
+ return *retrieved_ptr;
+}
+
+template <typename T>
+void ReleaseNativeRef(jlong ref) {
+ std::shared_ptr<T>* retrieved_ptr = reinterpret_cast<std::shared_ptr<T>*>(ref);
+ delete retrieved_ptr;
+}
+
+arrow::Result<jbyteArray> ToSchemaByteArray(JNIEnv* env,
+ std::shared_ptr<arrow::Schema> schema) {
+ ARROW_ASSIGN_OR_RAISE(
+ std::shared_ptr<arrow::Buffer> buffer,
+ arrow::ipc::SerializeSchema(*schema, arrow::default_memory_pool()))
+
+ jbyteArray out = env->NewByteArray(buffer->size());
+ auto src = reinterpret_cast<const jbyte*>(buffer->data());
+ env->SetByteArrayRegion(out, 0, buffer->size(), src);
+ return out;
+}
+
+arrow::Result<std::shared_ptr<arrow::Schema>> FromSchemaByteArray(
+ JNIEnv* env, jbyteArray schemaBytes) {
+ arrow::ipc::DictionaryMemo in_memo;
+ int schemaBytes_len = env->GetArrayLength(schemaBytes);
+ jbyte* schemaBytes_data = env->GetByteArrayElements(schemaBytes, nullptr);
+ auto serialized_schema = std::make_shared<arrow::Buffer>(
+ reinterpret_cast<uint8_t*>(schemaBytes_data), schemaBytes_len);
+ arrow::io::BufferReader buf_reader(serialized_schema);
+ ARROW_ASSIGN_OR_RAISE(std::shared_ptr<arrow::Schema> schema,
+ arrow::ipc::ReadSchema(&buf_reader, &in_memo))
+ env->ReleaseByteArrayElements(schemaBytes, schemaBytes_data, JNI_ABORT);
+ return schema;
+}
+
+/// Listener to act on reservations/unreservations from ReservationListenableMemoryPool.
+///
+/// Note the memory pool will call this listener only on block-level memory
+/// reservation/unreservation is granted. So the invocation parameter "size" is always
+/// multiple of block size (by default, 512k) specified in memory pool.
+class ReservationListener {
+ public:
+ virtual ~ReservationListener() = default;
+
+ virtual arrow::Status OnReservation(int64_t size) = 0;
+ virtual arrow::Status OnRelease(int64_t size) = 0;
+
+ protected:
+ ReservationListener() = default;
+};
+
+class ReservationListenableMemoryPoolImpl {
+ public:
+ explicit ReservationListenableMemoryPoolImpl(
+ arrow::MemoryPool* pool, std::shared_ptr<ReservationListener> listener,
+ int64_t block_size)
+ : pool_(pool),
+ listener_(listener),
+ block_size_(block_size),
+ blocks_reserved_(0),
+ bytes_reserved_(0) {}
+
+ arrow::Status Allocate(int64_t size, uint8_t** out) {
+ RETURN_NOT_OK(UpdateReservation(size));
+ arrow::Status error = pool_->Allocate(size, out);
+ if (!error.ok()) {
+ RETURN_NOT_OK(UpdateReservation(-size));
+ return error;
+ }
+ return arrow::Status::OK();
+ }
+
+ arrow::Status Reallocate(int64_t old_size, int64_t new_size, uint8_t** ptr) {
+ bool reserved = false;
+ int64_t diff = new_size - old_size;
+ if (new_size >= old_size) {
+ // new_size >= old_size, pre-reserve bytes from listener before allocating
+ // from underlying pool
+ RETURN_NOT_OK(UpdateReservation(diff));
+ reserved = true;
+ }
+ arrow::Status error = pool_->Reallocate(old_size, new_size, ptr);
+ if (!error.ok()) {
+ if (reserved) {
+ // roll back reservations on error
+ RETURN_NOT_OK(UpdateReservation(-diff));
+ }
+ return error;
+ }
+ if (!reserved) {
+ // otherwise (e.g. new_size < old_size), make updates after calling underlying pool
+ RETURN_NOT_OK(UpdateReservation(diff));
+ }
+ return arrow::Status::OK();
+ }
+
+ void Free(uint8_t* buffer, int64_t size) {
+ pool_->Free(buffer, size);
+ // FIXME: See ARROW-11143, currently method ::Free doesn't allow Status return
+ arrow::Status s = UpdateReservation(-size);
+ if (!s.ok()) {
+ ARROW_LOG(FATAL) << "Failed to update reservation while freeing bytes: "
+ << s.message();
+ return;
+ }
+ }
+
+ arrow::Status UpdateReservation(int64_t diff) {
+ int64_t granted = Reserve(diff);
+ if (granted == 0) {
+ return arrow::Status::OK();
+ }
+ if (granted < 0) {
+ RETURN_NOT_OK(listener_->OnRelease(-granted));
+ return arrow::Status::OK();
+ }
+ RETURN_NOT_OK(listener_->OnReservation(granted));
+ return arrow::Status::OK();
+ }
+
+ int64_t Reserve(int64_t diff) {
+ std::lock_guard<std::mutex> lock(mutex_);
+ bytes_reserved_ += diff;
+ int64_t new_block_count;
+ if (bytes_reserved_ == 0) {
+ new_block_count = 0;
+ } else {
+ // ceil to get the required block number
+ new_block_count = (bytes_reserved_ - 1) / block_size_ + 1;
+ }
+ int64_t bytes_granted = (new_block_count - blocks_reserved_) * block_size_;
+ blocks_reserved_ = new_block_count;
+ return bytes_granted;
+ }
+
+ int64_t bytes_allocated() { return pool_->bytes_allocated(); }
+
+ int64_t max_memory() { return pool_->max_memory(); }
+
+ std::string backend_name() { return pool_->backend_name(); }
+
+ std::shared_ptr<ReservationListener> get_listener() { return listener_; }
+
+ private:
+ arrow::MemoryPool* pool_;
+ std::shared_ptr<ReservationListener> listener_;
+ int64_t block_size_;
+ int64_t blocks_reserved_;
+ int64_t bytes_reserved_;
+ std::mutex mutex_;
+};
+
+/// A memory pool implementation for pre-reserving memory blocks from a
+/// customizable listener. This will typically be used when memory allocations
+/// have to subject to another "virtual" resource manager, which just tracks or
+/// limits number of bytes of application's overall memory usage. The underlying
+/// memory pool will still be responsible for actual malloc/free operations.
+class ReservationListenableMemoryPool : public arrow::MemoryPool {
+ public:
+ /// \brief Constructor.
+ ///
+ /// \param[in] pool the underlying memory pool
+ /// \param[in] listener a listener for block-level reservations/releases.
+ /// \param[in] block_size size of each block to reserve from the listener
+ explicit ReservationListenableMemoryPool(MemoryPool* pool,
+ std::shared_ptr<ReservationListener> listener,
+ int64_t block_size = 512 * 1024) {
+ impl_.reset(new ReservationListenableMemoryPoolImpl(pool, listener, block_size));
+ }
+
+ arrow::Status Allocate(int64_t size, uint8_t** out) override {
+ return impl_->Allocate(size, out);
+ }
+
+ arrow::Status Reallocate(int64_t old_size, int64_t new_size, uint8_t** ptr) override {
+ return impl_->Reallocate(old_size, new_size, ptr);
+ }
+
+ void Free(uint8_t* buffer, int64_t size) override { return impl_->Free(buffer, size); }
+
+ int64_t bytes_allocated() const override { return impl_->bytes_allocated(); }
+
+ int64_t max_memory() const override { return impl_->max_memory(); }
+
+ std::string backend_name() const override { return impl_->backend_name(); }
+
+ std::shared_ptr<ReservationListener> get_listener() { return impl_->get_listener(); }
+
+ private:
+ std::unique_ptr<ReservationListenableMemoryPoolImpl> impl_;
+};
+
+class ReserveFromJava : public ReservationListener {
+ public:
+ ReserveFromJava(JavaVM* vm, jobject java_reservation_listener)
+ : vm_(vm), java_reservation_listener_(java_reservation_listener) {}
+
+ arrow::Status OnReservation(int64_t size) override {
+ JNIEnv* env;
+ if (vm_->GetEnv(reinterpret_cast<void**>(&env), JNI_VERSION) != JNI_OK) {
+ return arrow::Status::Invalid("JNIEnv was not attached to current thread");
+ }
+ env->CallObjectMethod(java_reservation_listener_, reserve_memory_method, size);
+ if (env->ExceptionCheck()) {
+ env->ExceptionDescribe();
+ env->ExceptionClear();
+ return arrow::Status::Invalid("Error calling Java side reservation listener");
+ }
+ return arrow::Status::OK();
+ }
+
+ arrow::Status OnRelease(int64_t size) override {
+ JNIEnv* env;
+ if (vm_->GetEnv(reinterpret_cast<void**>(&env), JNI_VERSION) != JNI_OK) {
+ return arrow::Status::Invalid("JNIEnv was not attached to current thread");
+ }
+ env->CallObjectMethod(java_reservation_listener_, unreserve_memory_method, size);
+ if (env->ExceptionCheck()) {
+ env->ExceptionDescribe();
+ env->ExceptionClear();
+ return arrow::Status::Invalid("Error calling Java side reservation listener");
+ }
+ return arrow::Status::OK();
+ }
+
+ jobject GetJavaReservationListener() { return java_reservation_listener_; }
+
+ private:
+ JavaVM* vm_;
+ jobject java_reservation_listener_;
+};
+
+/// \class DisposableScannerAdaptor
+/// \brief An adaptor that iterates over a Scanner instance then returns RecordBatches
+/// directly.
+///
+/// This lessens the complexity of the JNI bridge to make sure it to be easier to
+/// maintain. On Java-side, NativeScanner can only produces a single NativeScanTask
+/// instance during its whole lifecycle. Each task stands for a DisposableScannerAdaptor
+/// instance through JNI bridge.
+///
+class DisposableScannerAdaptor {
+ public:
+ DisposableScannerAdaptor(std::shared_ptr<arrow::dataset::Scanner> scanner,
+ arrow::dataset::ScanTaskIterator task_itr) {
+ this->scanner_ = std::move(scanner);
+ this->task_itr_ = std::move(task_itr);
+ }
+
+ static arrow::Result<std::shared_ptr<DisposableScannerAdaptor>> Create(
+ std::shared_ptr<arrow::dataset::Scanner> scanner) {
+ ARROW_ASSIGN_OR_RAISE(arrow::dataset::ScanTaskIterator task_itr, scanner->Scan())
+ return std::make_shared<DisposableScannerAdaptor>(scanner, std::move(task_itr));
+ }
+
+ arrow::Result<std::shared_ptr<arrow::RecordBatch>> Next() {
+ do {
+ ARROW_ASSIGN_OR_RAISE(std::shared_ptr<arrow::RecordBatch> batch, NextBatch())
+ if (batch != nullptr) {
+ return batch;
+ }
+ // batch is null, current task is fully consumed
+ ARROW_ASSIGN_OR_RAISE(bool has_next_task, NextTask())
+ if (!has_next_task) {
+ // no more tasks
+ return nullptr;
+ }
+ // new task appended, read again
+ } while (true);
+ }
+
+ const std::shared_ptr<arrow::dataset::Scanner>& GetScanner() const { return scanner_; }
+
+ protected:
Review comment:
any reason for protected an not private?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] github-actions[bot] commented on pull request #7030: ARROW-7808: [Java][Dataset] Implement Datasets Java API
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on pull request #7030:
URL: https://github.com/apache/arrow/pull/7030#issuecomment-618919224
https://issues.apache.org/jira/browse/ARROW-7808
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] jacques-n commented on pull request #7030: ARROW-7808: [Java][Dataset] Implement Datasets Java API by JNI to C++
Posted by GitBox <gi...@apache.org>.
jacques-n commented on pull request #7030:
URL: https://github.com/apache/arrow/pull/7030#issuecomment-657096664
> @jacques-n
>
> I think we have several choices: we can try implement a c++ memory pool which is backed by a Java Allocator, actually I have a PoC branch for that: https://github.com/zhztheplayer/arrow-1/tree/ARROW-7808-wip-bridged-allocator.
>
> Or we can continue hacking Java API to reserve direct memory after buffer is registered (performance impact will be smaller). The reservation should happen inside Netty API because Netty has its own direct memory counter: https://github.com/netty/netty/blob/cbe238a95bab238455e3ff1849d450a9836f39ef/common/src/main/java/io/netty/util/internal/PlatformDependent.java#L749-L766.
>
> Which one do you suggest or maybe there's a better one than both?
If you use Netty, then Netty has a flag that already respects the direct memory limits (so no additional work is necessary).
I was suggesting a third approach: C++ memory pool that simply updates the direct memory info via Bits in java. It should be done chunk-wise to avoid excessive JNI boundary calls.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] zhztheplayer commented on a change in pull request #7030: ARROW-7808: [Java][Dataset] Implement Dataset Java API by JNI to C++
Posted by GitBox <gi...@apache.org>.
zhztheplayer commented on a change in pull request #7030:
URL: https://github.com/apache/arrow/pull/7030#discussion_r552529167
##########
File path: cpp/src/arrow/memory_pool.cc
##########
@@ -534,4 +535,139 @@ int64_t ProxyMemoryPool::max_memory() const { return impl_->max_memory(); }
std::string ProxyMemoryPool::backend_name() const { return impl_->backend_name(); }
+ReservationListener::~ReservationListener() {}
+
+ReservationListener::ReservationListener() {}
+
+class ReservationListenableMemoryPool::ReservationListenableMemoryPoolImpl {
+ public:
+ explicit ReservationListenableMemoryPoolImpl(
+ MemoryPool* pool, std::shared_ptr<ReservationListener> listener, int64_t block_size)
+ : pool_(pool),
+ listener_(listener),
+ block_size_(block_size),
+ blocks_reserved_(0),
+ bytes_reserved_(0) {}
+
+ Status Allocate(int64_t size, uint8_t** out) {
+ RETURN_NOT_OK(UpdateReservation(size));
+ Status error = pool_->Allocate(size, out);
+ if (!error.ok()) {
+ RETURN_NOT_OK(UpdateReservation(-size));
+ return error;
+ }
+ return Status::OK();
+ }
+
+ Status Reallocate(int64_t old_size, int64_t new_size, uint8_t** ptr) {
+ bool reserved = false;
+ int64_t diff = new_size - old_size;
+ if (new_size >= old_size) {
+ RETURN_NOT_OK(UpdateReservation(diff));
+ reserved = true;
+ }
+ Status error = pool_->Reallocate(old_size, new_size, ptr);
+ if (!error.ok()) {
+ if (reserved) {
+ RETURN_NOT_OK(UpdateReservation(-diff));
+ }
+ return error;
+ }
+ if (!reserved) {
+ RETURN_NOT_OK(UpdateReservation(diff));
+ }
+ return Status::OK();
+ }
+
+ void Free(uint8_t* buffer, int64_t size) {
+ pool_->Free(buffer, size);
+ // fixme currently method ::Free doesn't allow Status return
Review comment:
I opened https://issues.apache.org/jira/browse/ARROW-11143 for this. Let's discuss there. Actually I have tried changing the base interface to include contextual return but it doesn't seem like trivial change. It's better to make a new ticket.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] zhztheplayer commented on a change in pull request #7030: ARROW-7808: [Java][Dataset] Implement Datasets Java API by JNI to C++
Posted by GitBox <gi...@apache.org>.
zhztheplayer commented on a change in pull request #7030:
URL: https://github.com/apache/arrow/pull/7030#discussion_r429552370
##########
File path: cpp/src/arrow/dataset/discovery.h
##########
@@ -238,5 +238,27 @@ class ARROW_DS_EXPORT FileSystemDatasetFactory : public DatasetFactory {
util::optional<util::string_view> RemovePartitionBaseDir(util::string_view path);
};
+class ARROW_DS_EXPORT SingleFileDatasetFactory : public DatasetFactory {
Review comment:
I was aware of the changes in ARROW-8318 but at this time I can't remember what stops me from removing SingleFileDataset and migrating to FileSystemDataset. I'll have another try next week.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] emkornfield commented on a change in pull request #7030: ARROW-7808: [Java][Dataset] Implement Dataset Java API by JNI to C++
Posted by GitBox <gi...@apache.org>.
emkornfield commented on a change in pull request #7030:
URL: https://github.com/apache/arrow/pull/7030#discussion_r567201299
##########
File path: cpp/src/jni/dataset/jni_wrapper.cpp
##########
@@ -0,0 +1,859 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include <mutex>
+
+#include <arrow/array.h>
+#include <arrow/dataset/api.h>
+#include <arrow/dataset/file_base.h>
+#include <arrow/filesystem/localfs.h>
+#include <arrow/io/api.h>
+#include <arrow/ipc/api.h>
+#include <arrow/util/iterator.h>
+#include <arrow/util/logging.h>
+
+#include "org_apache_arrow_dataset_file_JniWrapper.h"
+#include "org_apache_arrow_dataset_jni_JniWrapper.h"
+#include "org_apache_arrow_dataset_jni_NativeMemoryPool.h"
+
+namespace arrow {
+namespace dataset {
+namespace jni {
+
+static jclass illegal_access_exception_class;
+static jclass illegal_argument_exception_class;
+static jclass runtime_exception_class;
+
+static jclass record_batch_handle_class;
+static jclass record_batch_handle_field_class;
+static jclass record_batch_handle_buffer_class;
+static jclass java_reservation_listener_class;
+
+static jmethodID record_batch_handle_constructor;
+static jmethodID record_batch_handle_field_constructor;
+static jmethodID record_batch_handle_buffer_constructor;
+static jmethodID reserve_memory_method;
+static jmethodID unreserve_memory_method;
+
+static jlong default_memory_pool_id = -1L;
+
+static jint JNI_VERSION = JNI_VERSION_1_6;
+
+class JniPendingException : public std::runtime_error {
+ public:
+ explicit JniPendingException(const std::string& arg) : runtime_error(arg) {}
+};
+
+void ThrowPendingException(const std::string& message) {
+ throw JniPendingException(message);
+}
+
+template <typename T>
+T JniGetOrThrow(arrow::Result<T> result) {
+ if (!result.status().ok()) {
+ ThrowPendingException(result.status().message());
+ }
+ return std::move(result).ValueOrDie();
+}
+
+void JniAssertOkOrThrow(arrow::Status status) {
+ if (!status.ok()) {
+ ThrowPendingException(status.message());
+ }
+}
+
+void JniThrow(std::string message) { ThrowPendingException(message); }
+
+jclass CreateGlobalClassReference(JNIEnv* env, const char* class_name) {
+ jclass local_class = env->FindClass(class_name);
+ jclass global_class = (jclass)env->NewGlobalRef(local_class);
+ env->DeleteLocalRef(local_class);
+ return global_class;
+}
+
+arrow::Result<jmethodID> GetMethodID(JNIEnv* env, jclass this_class, const char* name,
+ const char* sig) {
+ jmethodID ret = env->GetMethodID(this_class, name, sig);
+ if (ret == nullptr) {
+ std::string error_message = "Unable to find method " + std::string(name) +
+ " within signature" + std::string(sig);
+ return arrow::Status::Invalid(error_message);
+ }
+ return ret;
+}
+
+arrow::Result<jmethodID> GetStaticMethodID(JNIEnv* env, jclass this_class,
+ const char* name, const char* sig) {
+ jmethodID ret = env->GetStaticMethodID(this_class, name, sig);
+ if (ret == nullptr) {
+ std::string error_message = "Unable to find static method " + std::string(name) +
+ " within signature" + std::string(sig);
+ return arrow::Status::Invalid(error_message);
+ }
+ return ret;
+}
+
+std::shared_ptr<arrow::Schema> SchemaFromColumnNames(
+ const std::shared_ptr<arrow::Schema>& input,
+ const std::vector<std::string>& column_names) {
+ std::vector<std::shared_ptr<arrow::Field>> columns;
+ for (const auto& name : column_names) {
+ columns.push_back(input->GetFieldByName(name));
+ }
+ return std::make_shared<arrow::Schema>(columns);
+}
+
+arrow::Result<std::shared_ptr<arrow::dataset::FileFormat>> GetFileFormat(jint id) {
+ switch (id) {
+ case 0:
+ return std::make_shared<arrow::dataset::ParquetFileFormat>();
+ default:
+ std::string error_message = "illegal file format id: " + std::to_string(id);
+ return arrow::Status::Invalid(error_message);
+ }
+}
+
+std::string JStringToCString(JNIEnv* env, jstring string) {
+ if (string == nullptr) {
+ return std::string();
+ }
+ jboolean copied;
+ const char* chars = env->GetStringUTFChars(string, &copied);
Review comment:
since copied is unused just pass a nullptr (according to JNI docs it seems like this is allowed. /*copied=*/nullptr
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] emkornfield closed pull request #7030: ARROW-7808: [Java][Dataset] Implement Dataset Java API by JNI to C++
Posted by GitBox <gi...@apache.org>.
emkornfield closed pull request #7030:
URL: https://github.com/apache/arrow/pull/7030
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org