You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@orc.apache.org by "HuwCampbell (via GitHub)" <gi...@apache.org> on 2023/02/17 04:03:58 UTC
[GitHub] [orc] HuwCampbell opened a new pull request, #1413: Improve Specification documents
HuwCampbell opened a new pull request, #1413:
URL: https://github.com/apache/orc/pull/1413
The order of the data dictionary and length streams in dictionary columns is backwards compared to what all current writers emit; also, the readers gracefully handle absent streams (they're actually optional, as observed in the example files).
The documentation for nanosecond encoding was unclear and had broken sentences.
### What changes were proposed in this pull request?
Improvement of documentation in specification documents.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@orc.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Improve Specification documents [orc]
Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun closed pull request #1413: Improve Specification documents
URL: https://github.com/apache/orc/pull/1413
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@orc.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [orc] wgtmac commented on pull request #1413: Improve Specification documents
Posted by "wgtmac (via GitHub)" <gi...@apache.org>.
wgtmac commented on PR #1413:
URL: https://github.com/apache/orc/pull/1413#issuecomment-1434306386
On the C++ side, `DICTIONARY_DATA` is flushed before `LENGTH` stream. https://github.com/apache/orc/blob/main/c%2B%2B/src/ColumnWriter.cc#L1143
Actually the specs should not rule the order of the streams. In some implementations, streams of columns are even sorted to mitigate small I/Os.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@orc.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [orc] HuwCampbell commented on pull request #1413: Improve Specification documents
Posted by "HuwCampbell (via GitHub)" <gi...@apache.org>.
HuwCampbell commented on PR #1413:
URL: https://github.com/apache/orc/pull/1413#issuecomment-1439735580
That should be fine. I'll raise a Jira ticket first.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@orc.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [orc] HuwCampbell commented on pull request #1413: Improve Specification documents
Posted by "HuwCampbell (via GitHub)" <gi...@apache.org>.
HuwCampbell commented on PR #1413:
URL: https://github.com/apache/orc/pull/1413#issuecomment-1434326584
One can observe the streams with the orc tools.
I'll get around to a Jira in the coming days (it's Friday evening, so maybe Monday).
```
> orc-metadata --verbose examples/TestOrcFile.test1.orc
{
...
"stripes": [
...
"encodings": [
...
{ "column": 14, "encoding": "dictionary rle2", "count": 2 },
...
],
"streams": [
...
{ "id": 37, "column": 14, "kind": "data", "offset": 683, "length": 6 },
{ "id": 38, "column": 14, "kind": "length", "offset": 689, "length": 6 },
{ "id": 39, "column": 14, "kind": "dictionary", "offset": 695, "length": 10 },
...
],
...
}
]
}
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@orc.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [orc] dongjoon-hyun commented on pull request #1413: Improve Specification documents
Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun commented on PR #1413:
URL: https://github.com/apache/orc/pull/1413#issuecomment-1434365363
Given the current status, I'm also +1 for @wgtmac and @guiyanakuang 's comments. Maybe, shall we add an official note to the document?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@orc.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [orc] wgtmac commented on pull request #1413: Improve Specification documents
Posted by "wgtmac (via GitHub)" <gi...@apache.org>.
wgtmac commented on PR #1413:
URL: https://github.com/apache/orc/pull/1413#issuecomment-1434337993
BTW, do you have an interest to add https://github.com/HuwCampbell/orc-haskell to here: https://github.com/apache/orc/blob/main/proto/orc_proto.proto#L370 ? @HuwCampbell
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@orc.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Improve Specification documents [orc]
Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun commented on PR #1413:
URL: https://github.com/apache/orc/pull/1413#issuecomment-1837288776
Gentle ping, @HuwCampbell .
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@orc.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [orc] guiyanakuang commented on pull request #1413: Improve Specification documents
Posted by "guiyanakuang (via GitHub)" <gi...@apache.org>.
guiyanakuang commented on PR #1413:
URL: https://github.com/apache/orc/pull/1413#issuecomment-1434345294
> Actually the specs should not rule the order of the streams. In some implementations, streams of columns are even sorted to mitigate small I/Os.
On the Java side, writers sort by Kind. But the reader doesn't care about the order. I agree with you.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@orc.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Improve Specification documents [orc]
Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun commented on PR #1413:
URL: https://github.com/apache/orc/pull/1413#issuecomment-1884148251
The `specification` is spun off into a new `orc-format` repo. From now, we need to use `orc-format` as the ground-truth and maintains the files in `orc` repo by syncing before each releases.
- https://github.com/apache/orc-format/tree/main/specification
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@orc.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [orc] guiyanakuang commented on pull request #1413: Improve Specification documents
Posted by "guiyanakuang (via GitHub)" <gi...@apache.org>.
guiyanakuang commented on PR #1413:
URL: https://github.com/apache/orc/pull/1413#issuecomment-1434297093
https://github.com/apache/orc/blob/d403e11b7eab126259fb0683c0eb5c6131fd347b/java/core/src/java/org/apache/orc/impl/StreamName.java#L62-L78
https://github.com/apache/orc/blob/d403e11b7eab126259fb0683c0eb5c6131fd347b/proto/orc_proto.proto#L124-L129
> The order of the data dictionary and length streams in dictionary columns is backwards compared to what all current writers emit;
@deshanxiao , I think @HuwCampbell is right in that the order in which the streams are flushed to the file is actually based on this `compareTo ` method. `LENGTH` is written before `DICTIONARY_DATA` if it is in the same column.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@orc.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [orc] deshanxiao commented on pull request #1413: Improve Specification documents
Posted by "deshanxiao (via GitHub)" <gi...@apache.org>.
deshanxiao commented on PR #1413:
URL: https://github.com/apache/orc/pull/1413#issuecomment-1434153494
Hi @HuwCampbell
In [ flushDictionary ](https://github.com/apache/orc/blob/d403e11b7eab126259fb0683c0eb5c6131fd347b/java/core/src/java/org/apache/orc/impl/writer/StringBaseTreeWriter.java#L155)
``` java
context.writeBytes(stringOutput);
lengthOutput.write(context.getLength());
dumpOrder[context.getOriginalPosition()] = currentId++;
```
I saw that the `StringBaseTreeWriter` write `DICTIONARY_DATA` first and then write the `LENGTH`, right?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@orc.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [orc] deshanxiao commented on pull request #1413: Improve Specification documents
Posted by "deshanxiao (via GitHub)" <gi...@apache.org>.
deshanxiao commented on PR #1413:
URL: https://github.com/apache/orc/pull/1413#issuecomment-1439598045
Thank you @dongjoon-hyun @guiyanakuang @wgtmac
So, the order of these DICTIONARY doesn't really matter, right?
@HuwCampbell could you please update the document to add a note about it?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@orc.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org