You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@arrow.apache.org by "lidavidm (via GitHub)" <gi...@apache.org> on 2023/09/13 18:33:31 UTC
[GitHub] [arrow] lidavidm opened a new issue, #37702: [Java] Add validation functionality
lidavidm opened a new issue, #37702:
URL: https://github.com/apache/arrow/issues/37702
### Describe the enhancement requested
Other implementations can validate that the data in memory meets all the requirements of the Arrow specification (for example: offsets are nondecreasing for strings/lists; data is UTF-8 for strings). This would also be useful in Java (especially since vectors are mutable and it's a known shortcoming that mutators do not necessarily preserve type invariants).
### Component(s)
Java
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@arrow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] jduo commented on issue #37702: [Java] Add validation functionality
Posted by "jduo (via GitHub)" <gi...@apache.org>.
jduo commented on issue #37702:
URL: https://github.com/apache/arrow/issues/37702#issuecomment-1736376096
@lidavidm , I see that there is ValueVectorUtility#validate() and ValueVectorUtility#Validate().
Is the ask to add instance methods to the Vector classes to validate()? Or is it to add validity checks that are implemented in C++ but not Java (the UTF-8 check and offset check mentioned above, and others)?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] jduo commented on issue #37702: [Java] Add validation functionality
Posted by "jduo (via GitHub)" <gi...@apache.org>.
jduo commented on issue #37702:
URL: https://github.com/apache/arrow/issues/37702#issuecomment-1736348642
take
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] jduo commented on issue #37702: [Java] Add validation functionality
Posted by "jduo (via GitHub)" <gi...@apache.org>.
jduo commented on issue #37702:
URL: https://github.com/apache/arrow/issues/37702#issuecomment-1738130022
It seems the definition of validate() and validateFull() differ between C++ and Java.
In C++, validate() is an O(k) operation where k is the number of descendants and validateFull() is an O(k*n) operation where n is the length of each entry (in a string or list vector).
Whereas in Java, ValueVectorUtility#validate() is an O(1) operation that does not do any checks that iterate over the vector's fields and ValueVectorUtility#validateFull() checks offsets for string/binary/lists vectors and it drills down into container vectors. Note that Java's validateFull() currently does not do checks such as ensuring Decimal data is valid or ensuring string data is UTF-8
It seems like Java's validateFull() is somewhat similar to C++'s validate(), and Java's validate() is more like validateShallow().
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] lidavidm closed issue #37702: [Java] Add validation functionality
Posted by "lidavidm (via GitHub)" <gi...@apache.org>.
lidavidm closed issue #37702: [Java] Add validation functionality
URL: https://github.com/apache/arrow/issues/37702
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] lidavidm commented on issue #37702: [Java] Add validation functionality
Posted by "lidavidm (via GitHub)" <gi...@apache.org>.
lidavidm commented on issue #37702:
URL: https://github.com/apache/arrow/issues/37702#issuecomment-1736378640
Honestly, I never realized that existed. Thanks for finding those. I think we should do both: add instance methods, and then generally make sure that Java validates the same things C++ does.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] lidavidm commented on issue #37702: [Java] Add validation functionality
Posted by "lidavidm (via GitHub)" <gi...@apache.org>.
lidavidm commented on issue #37702:
URL: https://github.com/apache/arrow/issues/37702#issuecomment-1738203201
If validateFull is already checking offsets, I think it's appropriate to check string data as well? That said if the only difference is validating decimal/utf8 data then perhaps we can punt on that. (Does it validate union type codes/dense union offsets?)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org