You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@orc.apache.org by "cxzl25 (via GitHub)" <gi...@apache.org> on 2024/01/03 13:26:46 UTC
[PR] ORC-1567: Remove orc-tools restriction on orc suffix name [orc]
cxzl25 opened a new pull request, #1722:
URL: https://github.com/apache/orc/pull/1722
### What changes were proposed in this pull request?
### Why are the changes needed?
The `count` and `sizes` commands provided by `orc-tools` now require that the file must have an orc suffix, but the orc files generated by Hive and Spark do not necessarily have an orc suffix, which makes it inconvenient to use.
### How was this patch tested?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@orc.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] ORC-1567: Support `-ignoreExtension` option at `sizes` and `count` commands of orc-tools [orc]
Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun commented on PR #1722:
URL: https://github.com/apache/orc/pull/1722#issuecomment-1878109273
Please update the PR description, @cxzl25 . It looks outdated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@orc.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] ORC-1567: Add the `-ignoreExtension` configuration to the `sizes` and `count` commands of orc-tools [orc]
Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun commented on code in PR #1722:
URL: https://github.com/apache/orc/pull/1722#discussion_r1442480774
##########
java/tools/src/java/org/apache/orc/tools/RowCount.java:
##########
@@ -54,7 +69,22 @@ public static void main(Configuration conf, String[] args) throws IOException {
System.exit(bad == 0 ? 0 : 1);
}
- public static void main(String[] args) throws IOException {
+ public static void main(String[] args) throws Exception {
main(new Configuration(), args);
}
+
+ private static Options createOptions() {
+ Options result = new Options();
+
+ result.addOption(Option.builder()
Review Comment:
Please add a short opt here.
https://github.com/apache/orc/blob/68f4a8bb970cb370faba6b907ae93164e3243042/java/tools/src/java/org/apache/orc/tools/FileDump.java#L786
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@orc.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] ORC-1567: Remove orc-tools restriction on orc suffix name [orc]
Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun commented on PR #1722:
URL: https://github.com/apache/orc/pull/1722#issuecomment-1877813372
For this PR, could you try to add a new additional configuration to ignore the extension while keeping the existing default behavior, @cxzl25 ? For the additional feature, we can accept that while avoiding any breaking change.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@orc.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] ORC-1567: Add the `-ignoreExtension` configuration to the `sizes` and `count` commands of orc-tools [orc]
Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun commented on code in PR #1722:
URL: https://github.com/apache/orc/pull/1722#discussion_r1442481106
##########
java/tools/src/java/org/apache/orc/tools/RowCount.java:
##########
@@ -54,7 +69,22 @@ public static void main(Configuration conf, String[] args) throws IOException {
System.exit(bad == 0 ? 0 : 1);
}
- public static void main(String[] args) throws IOException {
+ public static void main(String[] args) throws Exception {
main(new Configuration(), args);
}
+
+ private static Options createOptions() {
+ Options result = new Options();
+
+ result.addOption(Option.builder()
+ .longOpt("ignoreExtension")
Review Comment:
Like `FileDump.java`, please use 4-space indentation for this pattern.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@orc.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] ORC-1567: Support `-ignoreExtension` option at `sizes` and `count` commands of orc-tools [orc]
Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun commented on PR #1722:
URL: https://github.com/apache/orc/pull/1722#issuecomment-1878100454
I set the milestone, `1.9.3`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@orc.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] ORC-1567: Remove orc-tools restriction on orc suffix name [orc]
Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun commented on PR #1722:
URL: https://github.com/apache/orc/pull/1722#issuecomment-1877809293
To @cxzl25 , Apache Spark has three ORC code path.
- `sql` module's `OrcFileFormat`:
- https://github.com/apache/spark/blob/master/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/orc/OrcFileFormat.scala
- `hive` module's `OrcFileFormat`:
- https://github.com/apache/spark/blob/master/sql/hive/src/main/scala/org/apache/spark/sql/hive/orc/OrcFileFormat.scala
- `hive` module's `HiveFileFormat`:
- https://github.com/apache/spark/blob/master/sql/hive/src/main/scala/org/apache/spark/sql/hive/execution/HiveFileFormat.scala
It seems that you are using `HiveFileFormat`. It follows Hive's behavior whose file extension is determined by `hive.output.file.extension`. Please try the following from the beginning.
```
$ bin/spark-sql -c hive.output.file.extension=orc
...
spark-sql (default)> create table t stored as orc as select id from range(1);
Time taken: 1.706 seconds
$ ls -al spark-warehouse/t
total 40
drwxr-xr-x 8 dongjoon staff 256 Jan 4 13:50 .
drwxr-xr-x 3 dongjoon staff 96 Jan 4 13:50 ..
-rw-r--r-- 1 dongjoon staff 8 Jan 4 13:50 ._SUCCESS.crc
-rw-r--r-- 1 dongjoon staff 12 Jan 4 13:50 .part-00000-492978c5-6833-4dd3-a417-eba7a49b9d44-c000.snappy.orc.crc
-rw-r--r-- 1 dongjoon staff 12 Jan 4 13:50 .part-00009-492978c5-6833-4dd3-a417-eba7a49b9d44-c000.snappy.orc.crc
-rw-r--r-- 1 dongjoon staff 0 Jan 4 13:50 _SUCCESS
-rw-r--r-- 1 dongjoon staff 116 Jan 4 13:50 part-00000-492978c5-6833-4dd3-a417-eba7a49b9d44-c000.snappy.orc
-rw-r--r-- 1 dongjoon staff 237 Jan 4 13:50 part-00009-492978c5-6833-4dd3-a417-eba7a49b9d44-c000.snappy.orc
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@orc.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] ORC-1567: Support `-ignoreExtension` option at `sizes` and `count` commands of orc-tools [orc]
Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun commented on PR #1722:
URL: https://github.com/apache/orc/pull/1722#issuecomment-1878218315
Merged to main for Apache ORC 2.0.0. Thank you, @cxzl25 and @deshanxiao .
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@orc.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] ORC-1567: Support `-ignoreExtension` option at `sizes` and `count` commands of orc-tools [orc]
Posted by "cxzl25 (via GitHub)" <gi...@apache.org>.
cxzl25 commented on PR #1722:
URL: https://github.com/apache/orc/pull/1722#issuecomment-1878111470
> Please update the PR description
Thanks. Updated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@orc.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] ORC-1567: Support `-ignoreExtension` option at `sizes` and `count` commands of orc-tools [orc]
Posted by "cxzl25 (via GitHub)" <gi...@apache.org>.
cxzl25 commented on PR #1722:
URL: https://github.com/apache/orc/pull/1722#issuecomment-1878286387
Thank you for your help and review! @dongjoon-hyun @deshanxiao
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@orc.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] ORC-1567: Remove orc-tools restriction on orc suffix name [orc]
Posted by "cxzl25 (via GitHub)" <gi...@apache.org>.
cxzl25 commented on PR #1722:
URL: https://github.com/apache/orc/pull/1722#issuecomment-1878077115
> It seems that you are using `HiveFileFormat`. It follows Hive's behavior whose file extension is determined by `hive.output.file.extension`
Thanks for the reminder, because spark will convert the ctas statement, so I corrected the command slightly.(`.orc`)
```bash
./bin/spark-sql -c spark.hive.output.file.extension=.orc -c spark.sql.hive.convertMetastoreCtas=false
```
```sql
create table tmp_orc stored as orcfile as select id from range(1);
```
```bash
ls -al spark-warehouse/tmp_orc
```
```
-rwxr-xr-x@ 1 csy staff 8 1 5 11:28 .part-00000-d2fb8ac6-e34b-4b1b-9a08-6c3580e4afc6-c000.orc.crc
-rwxr-xr-x@ 1 csy staff 12 1 5 11:28 .part-00011-d2fb8ac6-e34b-4b1b-9a08-6c3580e4afc6-c000.orc.crc
-rwxr-xr-x@ 1 csy staff 0 1 5 11:28 part-00000-d2fb8ac6-e34b-4b1b-9a08-6c3580e4afc6-c000.orc
-rwxr-xr-x@ 1 csy staff 202 1 5 11:28 part-00011-d2fb8ac6-e34b-4b1b-9a08-6c3580e4afc6-c000.orc
```
> could you try to add a new additional configuration to ignore the extension while keeping the existing default behavior
Thanks, let me do it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@orc.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] ORC-1567: Support `-ignoreExtension` option at `sizes` and `count` commands of orc-tools [orc]
Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun closed pull request #1722: ORC-1567: Support `-ignoreExtension` option at `sizes` and `count` commands of orc-tools
URL: https://github.com/apache/orc/pull/1722
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@orc.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] ORC-1567: Remove orc-tools restriction on orc suffix name [orc]
Posted by "cxzl25 (via GitHub)" <gi...@apache.org>.
cxzl25 commented on PR #1722:
URL: https://github.com/apache/orc/pull/1722#issuecomment-1877310082
If we don't use orc datasource, using the implementation of hive will generate a file name without orc suffix.
```bash
./bin/spark-sql
```
```sql
set spark.sql.hive.convertMetastoreCtas=false;
create table tmp_orc stored as orcfile as select id from range(1);
```
```bash
ls -al spark-warehouse/tmp_orc
total 24
drwxr-xr-x@ 6 csy staff 192 1 4 23:34 .
drwxr-xr-x@ 4 csy staff 128 1 4 23:34 ..
-rwxr-xr-x@ 1 csy staff 8 1 4 23:34 .part-00000-5888ab4a-8773-4376-a320-0cd6f4df5889-c000.crc
-rwxr-xr-x@ 1 csy staff 12 1 4 23:34 .part-00011-5888ab4a-8773-4376-a320-0cd6f4df5889-c000.crc
-rwxr-xr-x@ 1 csy staff 0 1 4 23:34 part-00000-5888ab4a-8773-4376-a320-0cd6f4df5889-c000
-rwxr-xr-x@ 1 csy staff 202 1 4 23:34 part-00011-5888ab4a-8773-4376-a320-0cd6f4df5889-c000
```
Can we do this? If the input is only one file, ignore the orc suffix?
This behavior is similar to `FileDump`. `FileDump` does not require the file to have an orc suffix.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@orc.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] ORC-1567: Remove orc-tools restriction on orc suffix name [orc]
Posted by "deshanxiao (via GitHub)" <gi...@apache.org>.
deshanxiao commented on PR #1722:
URL: https://github.com/apache/orc/pull/1722#issuecomment-1876259651
+1 for @dongjoon-hyun's concern. In fact, I am thinking that using the file suffix does not seem accurate too. Maybe we can consider the magic number "ORC" starting with orc to judge whether it is a ORC file. This will not have any performance impact as it is just the orc tool.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@orc.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org