You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by GitBox <gi...@apache.org> on 2022/02/24 19:13:04 UTC
[GitHub] [arrow] lidavidm opened a new pull request #12509: ARROW-15781: [Python] Release GIL in ensure_complete_metadata
lidavidm opened a new pull request #12509:
URL: https://github.com/apache/arrow/pull/12509
Closes #12501.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] ericl commented on pull request #12509: ARROW-15781: [Python] Release GIL in ensure_complete_metadata
Posted by GitBox <gi...@apache.org>.
ericl commented on pull request #12509:
URL: https://github.com/apache/arrow/pull/12509#issuecomment-1050248549
I ran into some trouble building pyarrow, but the above test case looks great :+1: Looking forwards to trying it out in the next release.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] lidavidm edited a comment on pull request #12509: ARROW-15781: [Python] Release GIL in ensure_complete_metadata
Posted by GitBox <gi...@apache.org>.
lidavidm edited a comment on pull request #12509:
URL: https://github.com/apache/arrow/pull/12509#issuecomment-1050175988
@ericl would you be able to build this and ensure it fixes your case too?
I tested with the following:
```diff
diff --git a/cpp/src/arrow/dataset/file_parquet.cc b/cpp/src/arrow/dataset/file_parquet.cc
index cdee1f006..52de0ea11 100644
--- a/cpp/src/arrow/dataset/file_parquet.cc
+++ b/cpp/src/arrow/dataset/file_parquet.cc
@@ -17,6 +17,8 @@
#include "arrow/dataset/file_parquet.h"
+#include <chrono>
+#include <thread>
#include <memory>
#include <mutex>
#include <unordered_map>
@@ -561,6 +563,7 @@ Status ParquetFileFragment::EnsureCompleteMetadata(parquet::arrow::FileReader* r
return EnsureCompleteMetadata(reader.get());
}
+ std::this_thread::sleep_for(std::chrono::seconds(1));
std::shared_ptr<Schema> schema;
RETURN_NOT_OK(reader->GetSchema(&schema));
if (physical_schema_ && !physical_schema_->Equals(*schema)) {
```
and
```python
from concurrent.futures import ThreadPoolExecutor
import time
import pyarrow as pa
import pyarrow.parquet as pq
dataset = pq.ParquetDataset([
"/home/lidavidm/Documents/ursa-labs-taxi-data/2009/03/data.parquet",
"/home/lidavidm/Documents/ursa-labs-taxi-data/2009/04/data.parquet",
], use_legacy_dataset=False)
print(dataset.fragments)
pool = ThreadPoolExecutor()
start = time.monotonic()
for x in pool.map(lambda x: x.metadata, dataset.fragments):
print(x)
end = time.monotonic()
print(end - start)
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] jorisvandenbossche closed pull request #12509: ARROW-15781: [Python] Release GIL in ensure_complete_metadata
Posted by GitBox <gi...@apache.org>.
jorisvandenbossche closed pull request #12509:
URL: https://github.com/apache/arrow/pull/12509
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] github-actions[bot] commented on pull request #12509: ARROW-15781: [Python] Release GIL in ensure_complete_metadata
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on pull request #12509:
URL: https://github.com/apache/arrow/pull/12509#issuecomment-1050174949
https://issues.apache.org/jira/browse/ARROW-15781
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] ursabot commented on pull request #12509: ARROW-15781: [Python] Release GIL in ensure_complete_metadata
Posted by GitBox <gi...@apache.org>.
ursabot commented on pull request #12509:
URL: https://github.com/apache/arrow/pull/12509#issuecomment-1050946191
Benchmark runs are scheduled for baseline = 89cc6b3248fec31464f1b893bf4a7925ccccba29 and contender = f135b986db44358c1b41d7552d0e77009aa8c1c7. f135b986db44358c1b41d7552d0e77009aa8c1c7 is a master commit associated with this PR. Results will be available as each benchmark for each run completes.
Conbench compare runs links:
[Scheduled] [ec2-t3-xlarge-us-east-2](https://conbench.ursa.dev/compare/runs/cd8c7a4f5b354fbc8ab26ae7d5949172...aea46201de69499daec3910592f35d2f/)
[Scheduled] [test-mac-arm](https://conbench.ursa.dev/compare/runs/0805a430c74d4ab9adf01f5e620f6d03...24dba55113124b6392941f082c50b52d/)
[Scheduled] [ursa-i9-9960x](https://conbench.ursa.dev/compare/runs/5603e92d7897489d9273d4187d5e7280...081399effe43433bbb28ae7efb77a35b/)
[Scheduled] [ursa-thinkcentre-m75q](https://conbench.ursa.dev/compare/runs/9b3e5efd949e4b9aadc7cbde64ff83d6...00cf8edfbbc04db49e89fa09bd57ee3d/)
Supported benchmarks:
ec2-t3-xlarge-us-east-2: Supported benchmark langs: Python. Runs only benchmarks with cloud = True
test-mac-arm: Supported benchmark langs: C++, Python, R
ursa-i9-9960x: Supported benchmark langs: Python, R, JavaScript
ursa-thinkcentre-m75q: Supported benchmark langs: C++, Java
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] lidavidm commented on pull request #12509: ARROW-15781: [Python] Release GIL in ensure_complete_metadata
Posted by GitBox <gi...@apache.org>.
lidavidm commented on pull request #12509:
URL: https://github.com/apache/arrow/pull/12509#issuecomment-1050274094
Got it. We do have [nightly packages](https://arrow.apache.org/docs/dev/python/install.html#installing-nightly-packages) for conda/pip so you can test a little sooner than that.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] ursabot edited a comment on pull request #12509: ARROW-15781: [Python] Release GIL in ensure_complete_metadata
Posted by GitBox <gi...@apache.org>.
ursabot edited a comment on pull request #12509:
URL: https://github.com/apache/arrow/pull/12509#issuecomment-1050946191
Benchmark runs are scheduled for baseline = 89cc6b3248fec31464f1b893bf4a7925ccccba29 and contender = f135b986db44358c1b41d7552d0e77009aa8c1c7. f135b986db44358c1b41d7552d0e77009aa8c1c7 is a master commit associated with this PR. Results will be available as each benchmark for each run completes.
Conbench compare runs links:
[Finished :arrow_down:0.0% :arrow_up:0.0%] [ec2-t3-xlarge-us-east-2](https://conbench.ursa.dev/compare/runs/cd8c7a4f5b354fbc8ab26ae7d5949172...aea46201de69499daec3910592f35d2f/)
[Scheduled] [test-mac-arm](https://conbench.ursa.dev/compare/runs/0805a430c74d4ab9adf01f5e620f6d03...24dba55113124b6392941f082c50b52d/)
[Scheduled] [ursa-i9-9960x](https://conbench.ursa.dev/compare/runs/5603e92d7897489d9273d4187d5e7280...081399effe43433bbb28ae7efb77a35b/)
[Scheduled] [ursa-thinkcentre-m75q](https://conbench.ursa.dev/compare/runs/9b3e5efd949e4b9aadc7cbde64ff83d6...00cf8edfbbc04db49e89fa09bd57ee3d/)
Supported benchmarks:
ec2-t3-xlarge-us-east-2: Supported benchmark langs: Python. Runs only benchmarks with cloud = True
test-mac-arm: Supported benchmark langs: C++, Python, R
ursa-i9-9960x: Supported benchmark langs: Python, R, JavaScript
ursa-thinkcentre-m75q: Supported benchmark langs: C++, Java
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] ursabot edited a comment on pull request #12509: ARROW-15781: [Python] Release GIL in ensure_complete_metadata
Posted by GitBox <gi...@apache.org>.
ursabot edited a comment on pull request #12509:
URL: https://github.com/apache/arrow/pull/12509#issuecomment-1050946191
Benchmark runs are scheduled for baseline = 89cc6b3248fec31464f1b893bf4a7925ccccba29 and contender = f135b986db44358c1b41d7552d0e77009aa8c1c7. f135b986db44358c1b41d7552d0e77009aa8c1c7 is a master commit associated with this PR. Results will be available as each benchmark for each run completes.
Conbench compare runs links:
[Finished :arrow_down:0.0% :arrow_up:0.0%] [ec2-t3-xlarge-us-east-2](https://conbench.ursa.dev/compare/runs/cd8c7a4f5b354fbc8ab26ae7d5949172...aea46201de69499daec3910592f35d2f/)
[Finished :arrow_down:0.38% :arrow_up:0.0%] [test-mac-arm](https://conbench.ursa.dev/compare/runs/0805a430c74d4ab9adf01f5e620f6d03...24dba55113124b6392941f082c50b52d/)
[Scheduled] [ursa-i9-9960x](https://conbench.ursa.dev/compare/runs/5603e92d7897489d9273d4187d5e7280...081399effe43433bbb28ae7efb77a35b/)
[Finished :arrow_down:0.09% :arrow_up:0.09%] [ursa-thinkcentre-m75q](https://conbench.ursa.dev/compare/runs/9b3e5efd949e4b9aadc7cbde64ff83d6...00cf8edfbbc04db49e89fa09bd57ee3d/)
Supported benchmarks:
ec2-t3-xlarge-us-east-2: Supported benchmark langs: Python. Runs only benchmarks with cloud = True
test-mac-arm: Supported benchmark langs: C++, Python, R
ursa-i9-9960x: Supported benchmark langs: Python, R, JavaScript
ursa-thinkcentre-m75q: Supported benchmark langs: C++, Java
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] lidavidm commented on pull request #12509: ARROW-15781: [Python] Release GIL in ensure_complete_metadata
Posted by GitBox <gi...@apache.org>.
lidavidm commented on pull request #12509:
URL: https://github.com/apache/arrow/pull/12509#issuecomment-1050175988
@ericl would you be able to build this and ensure it fixes your case too?
I tested with the following:
```diff
diff --git a/cpp/src/arrow/dataset/file_parquet.cc b/cpp/src/arrow/dataset/file_parquet.cc
index cdee1f006..52de0ea11 100644
--- a/cpp/src/arrow/dataset/file_parquet.cc
+++ b/cpp/src/arrow/dataset/file_parquet.cc
@@ -17,6 +17,8 @@
#include "arrow/dataset/file_parquet.h"
+#include <chrono>
+#include <thread>
#include <memory>
#include <mutex>
#include <unordered_map>
@@ -561,6 +563,7 @@ Status ParquetFileFragment::EnsureCompleteMetadata(parquet::arrow::FileReader* r
return EnsureCompleteMetadata(reader.get());
}
+ std::this_thread::sleep_for(std::chrono::seconds(1));
std::shared_ptr<Schema> schema;
RETURN_NOT_OK(reader->GetSchema(&schema));
if (physical_schema_ && !physical_schema_->Equals(*schema)) {
```
and
```
from concurrent.futures import ThreadPoolExecutor
import time
import pyarrow as pa
import pyarrow.parquet as pq
dataset = pq.ParquetDataset([
"/home/lidavidm/Documents/ursa-labs-taxi-data/2009/03/data.parquet",
"/home/lidavidm/Documents/ursa-labs-taxi-data/2009/04/data.parquet",
], use_legacy_dataset=False)
print(dataset.fragments)
pool = ThreadPoolExecutor()
start = time.monotonic()
for x in pool.map(lambda x: x.metadata, dataset.fragments):
print(x)
end = time.monotonic()
print(end - start)
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] ursabot edited a comment on pull request #12509: ARROW-15781: [Python] Release GIL in ensure_complete_metadata
Posted by GitBox <gi...@apache.org>.
ursabot edited a comment on pull request #12509:
URL: https://github.com/apache/arrow/pull/12509#issuecomment-1050946191
Benchmark runs are scheduled for baseline = 89cc6b3248fec31464f1b893bf4a7925ccccba29 and contender = f135b986db44358c1b41d7552d0e77009aa8c1c7. f135b986db44358c1b41d7552d0e77009aa8c1c7 is a master commit associated with this PR. Results will be available as each benchmark for each run completes.
Conbench compare runs links:
[Finished :arrow_down:0.0% :arrow_up:0.0%] [ec2-t3-xlarge-us-east-2](https://conbench.ursa.dev/compare/runs/cd8c7a4f5b354fbc8ab26ae7d5949172...aea46201de69499daec3910592f35d2f/)
[Finished :arrow_down:0.38% :arrow_up:0.0%] [test-mac-arm](https://conbench.ursa.dev/compare/runs/0805a430c74d4ab9adf01f5e620f6d03...24dba55113124b6392941f082c50b52d/)
[Finished :arrow_down:0.0% :arrow_up:0.0%] [ursa-i9-9960x](https://conbench.ursa.dev/compare/runs/5603e92d7897489d9273d4187d5e7280...081399effe43433bbb28ae7efb77a35b/)
[Finished :arrow_down:0.09% :arrow_up:0.09%] [ursa-thinkcentre-m75q](https://conbench.ursa.dev/compare/runs/9b3e5efd949e4b9aadc7cbde64ff83d6...00cf8edfbbc04db49e89fa09bd57ee3d/)
Supported benchmarks:
ec2-t3-xlarge-us-east-2: Supported benchmark langs: Python. Runs only benchmarks with cloud = True
test-mac-arm: Supported benchmark langs: C++, Python, R
ursa-i9-9960x: Supported benchmark langs: Python, R, JavaScript
ursa-thinkcentre-m75q: Supported benchmark langs: C++, Java
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org