You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@lucene.apache.org by "dungba88 (via GitHub)" <gi...@apache.org> on 2023/10/18 08:56:04 UTC
[PR] Consolidate the FSTStore and BytesStore in FST (#12543) [lucene]
dungba88 opened a new pull request, #12691:
URL: https://github.com/apache/lucene/pull/12691
### Description
This change will make the transition to off-heap FST writing easier. The idea is to only build the FST after the end of the process (when calling `compile()`).
- Consolidate the FSTStore and BytesStore in FST. The two are similar, except that FSTStore has an `init()` method, which is not needed for BytesStore. Thus I extracted the common methods to FSTReader (maybe a bad name). FST no longer needs to have if-else conditional logics to choose between the two.
- Remove all write methods out of FST so that it will be completely read-only. These methods are only used by FSTCompiler when constructing FST.
- Decouple NodeHash and FST by extract the functionality into FSTTraversal (again, could be bad name). The FST will only be created after completion.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
Re: [PR] Make FST fully read-only and streamline FST constructor [lucene]
Posted by "dungba88 (via GitHub)" <gi...@apache.org>.
dungba88 closed pull request #12691: Make FST fully read-only and streamline FST constructor
URL: https://github.com/apache/lucene/pull/12691
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
Re: [PR] Make FST fully read-only and streamline FST constructor [lucene]
Posted by "dungba88 (via GitHub)" <gi...@apache.org>.
dungba88 commented on PR #12691:
URL: https://github.com/apache/lucene/pull/12691#issuecomment-1793570275
Yeah, this PR was originally opened for another purpose: consolidate the FSTStore and BytesStore, and that was already done.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
Re: [PR] Consolidate the FSTStore and BytesStore in FST (#12543) [lucene]
Posted by "dungba88 (via GitHub)" <gi...@apache.org>.
dungba88 commented on code in PR #12691:
URL: https://github.com/apache/lucene/pull/12691#discussion_r1364843168
##########
lucene/core/src/java/org/apache/lucene/util/fst/FST.java:
##########
@@ -552,14 +527,11 @@ public void save(DataOutput metaOut, DataOutput out) throws IOException {
}
metaOut.writeByte(t);
metaOut.writeVLong(startNode);
- if (bytes != null) {
- long numBytes = bytes.getPosition();
- metaOut.writeVLong(numBytes);
- bytes.writeTo(out);
- } else {
- assert fstStore != null;
- fstStore.writeTo(out);
+ // nocommit why does the FSTStore case does not need the numBytes?
Review Comment:
Found it, it is actually an issue: https://github.com/apache/lucene/issues/12697
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
Re: [PR] Consolidate the FSTStore and BytesStore in FST (#12543) [lucene]
Posted by "dungba88 (via GitHub)" <gi...@apache.org>.
dungba88 commented on code in PR #12691:
URL: https://github.com/apache/lucene/pull/12691#discussion_r1365567273
##########
lucene/core/src/java/org/apache/lucene/util/fst/FST.java:
##########
@@ -552,14 +527,11 @@ public void save(DataOutput metaOut, DataOutput out) throws IOException {
}
metaOut.writeByte(t);
metaOut.writeVLong(startNode);
- if (bytes != null) {
- long numBytes = bytes.getPosition();
- metaOut.writeVLong(numBytes);
- bytes.writeTo(out);
- } else {
- assert fstStore != null;
- fstStore.writeTo(out);
+ // nocommit why does the FSTStore case does not need the numBytes?
Review Comment:
Yeah, but it seems to be a rare case. User needs to use save a FSTStore-backed FST with a different DataOutput for metadata. I assume most people will use the same DataOutput for metadata and main FST body, which would not trigger the error.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
Re: [PR] Make FST fully read-only and streamline FST constructor [lucene]
Posted by "mikemccand (via GitHub)" <gi...@apache.org>.
mikemccand commented on PR #12691:
URL: https://github.com/apache/lucene/pull/12691#issuecomment-1793486552
We closed this PR in favor of #12758?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
Re: [PR] Consolidate the FSTStore and BytesStore in FST (#12543) [lucene]
Posted by "mikemccand (via GitHub)" <gi...@apache.org>.
mikemccand commented on code in PR #12691:
URL: https://github.com/apache/lucene/pull/12691#discussion_r1365488019
##########
lucene/core/src/java/org/apache/lucene/util/fst/FST.java:
##########
@@ -552,14 +527,11 @@ public void save(DataOutput metaOut, DataOutput out) throws IOException {
}
metaOut.writeByte(t);
metaOut.writeVLong(startNode);
- if (bytes != null) {
- long numBytes = bytes.getPosition();
- metaOut.writeVLong(numBytes);
- bytes.writeTo(out);
- } else {
- assert fstStore != null;
- fstStore.writeTo(out);
+ // nocommit why does the FSTStore case does not need the numBytes?
Review Comment:
Egads, this looks like a serious bug? What would a user need to do to tickle this?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
Re: [PR] Consolidate the FSTStore and BytesStore in FST (#12543) [lucene]
Posted by "dungba88 (via GitHub)" <gi...@apache.org>.
dungba88 commented on code in PR #12691:
URL: https://github.com/apache/lucene/pull/12691#discussion_r1364843168
##########
lucene/core/src/java/org/apache/lucene/util/fst/FST.java:
##########
@@ -552,14 +527,11 @@ public void save(DataOutput metaOut, DataOutput out) throws IOException {
}
metaOut.writeByte(t);
metaOut.writeVLong(startNode);
- if (bytes != null) {
- long numBytes = bytes.getPosition();
- metaOut.writeVLong(numBytes);
- bytes.writeTo(out);
- } else {
- assert fstStore != null;
- fstStore.writeTo(out);
+ // nocommit why does the FSTStore case does not need the numBytes?
Review Comment:
Found it, it is actually an issue: https://github.com/apache/lucene/issues/12697. Fix is here: https://github.com/apache/lucene/pull/12698
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
Re: [PR] Consolidate the FSTStore and BytesStore in FST (#12543) [lucene]
Posted by "dungba88 (via GitHub)" <gi...@apache.org>.
dungba88 commented on code in PR #12691:
URL: https://github.com/apache/lucene/pull/12691#discussion_r1365569612
##########
lucene/core/src/java/org/apache/lucene/util/fst/FST.java:
##########
@@ -552,14 +527,11 @@ public void save(DataOutput metaOut, DataOutput out) throws IOException {
}
metaOut.writeByte(t);
metaOut.writeVLong(startNode);
- if (bytes != null) {
- long numBytes = bytes.getPosition();
- metaOut.writeVLong(numBytes);
- bytes.writeTo(out);
- } else {
- assert fstStore != null;
- fstStore.writeTo(out);
+ // nocommit why does the FSTStore case does not need the numBytes?
Review Comment:
Anyhow https://github.com/apache/lucene/pull/12698 should fix this, and we wouldn't need to handle 2 separate cases.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
Re: [PR] Consolidate the FSTStore and BytesStore in FST (#12543) [lucene]
Posted by "dungba88 (via GitHub)" <gi...@apache.org>.
dungba88 commented on code in PR #12691:
URL: https://github.com/apache/lucene/pull/12691#discussion_r1363511529
##########
lucene/core/src/java/org/apache/lucene/util/fst/FST.java:
##########
@@ -552,14 +527,11 @@ public void save(DataOutput metaOut, DataOutput out) throws IOException {
}
metaOut.writeByte(t);
metaOut.writeVLong(startNode);
- if (bytes != null) {
- long numBytes = bytes.getPosition();
- metaOut.writeVLong(numBytes);
- bytes.writeTo(out);
- } else {
- assert fstStore != null;
- fstStore.writeTo(out);
+ // nocommit why does the FSTStore case does not need the numBytes?
Review Comment:
I wasn't sure about this, need to debug on why the numBytes is not needed when writing with FSTStore
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org