You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@fury.apache.org by "ZhiQiang-Tiger (via GitHub)" <gi...@apache.org> on 2025/06/03 05:39:02 UTC
[PR] add log downgrade configuration [fury]
ZhiQiang-Tiger opened a new pull request, #2284:
URL: https://github.com/apache/fury/pull/2284
<!--
**Thanks for contributing to Fory.**
**If this is your first time opening a PR on fory, you can refer to [CONTRIBUTING.md](https://github.com/apache/fory/blob/main/CONTRIBUTING.md).**
Contribution Checklist
- The **Apache Fory (incubating)** community has restrictions on the naming of pr titles. You can also find instructions in [CONTRIBUTING.md](https://github.com/apache/fory/blob/main/CONTRIBUTING.md).
- Fory has a strong focus on performance. If the PR you submit will have an impact on performance, please benchmark it first and provide the benchmark result here.
-->
## What does this PR do?
<!-- Describe the purpose of this PR. -->
Users can choose the level of error messages to be printed.
## Related issues
<!--
Is there any related issue? Please attach here.
- #xxxx0
- #xxxx1
- #xxxx2
-->
Deserialize big object using Fury frequently, triggering a Full GC when logging failed object #2259
## Does this PR introduce any user-facing change?
<!--
If any user-facing interface changes, please [open an issue](https://github.com/apache/fory/issues/new/choose) describing the need to do so and update the document if necessary.
-->
The user can select the log mode by adding the configuration ExceptionLogMode via FuryBuilder. When log sampling, set the sampling step by logSampleStep.
- [ ] add enum class ExceptionLogMode.
- [ ] add sample step param.
## Benchmark
<!--
When the PR has an impact on performance (if you don't know whether the PR will have an impact on performance, you can submit the PR first, and if it will have impact on performance, the code reviewer will explain it), be sure to attach a benchmark data here.
-->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@fury.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@fury.apache.org
For additional commands, e-mail: commits-help@fury.apache.org
Re: [PR] add log downgrade configuration [fury]
Posted by "LiangliangSui (via GitHub)" <gi...@apache.org>.
LiangliangSui commented on code in PR #2284:
URL: https://github.com/apache/fury/pull/2284#discussion_r2122806990
##########
java/fury-core/src/main/java/org/apache/fury/config/ExceptionLogMode.java:
##########
@@ -0,0 +1,7 @@
+package org.apache.fury.config;
Review Comment:
The newly added files need to have a license header inserted at the top.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@fury.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@fury.apache.org
For additional commands, e-mail: commits-help@fury.apache.org
Re: [PR] feat(java): add log downgrade configuration [fury]
Posted by "ZhiQiang-Tiger (via GitHub)" <gi...@apache.org>.
ZhiQiang-Tiger commented on code in PR #2284:
URL: https://github.com/apache/fury/pull/2284#discussion_r2122999140
##########
java/fury-core/src/main/java/org/apache/fury/Fury.java:
##########
@@ -1679,6 +1681,13 @@ public boolean compressLong() {
return config.compressLong();
}
+ public ExceptionLogMode getExceptionLogMode() {
+ return config.getExceptionLogMode();
+ }
+ public int getLogSampleStep() {
Review Comment:
> And I think we don't need to duplicate fields here, we could just pass value directly from Config
ok. At the time it was to regulate
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@fury.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@fury.apache.org
For additional commands, e-mail: commits-help@fury.apache.org
Re: [PR] feat(java): add log downgrade configuration [fury]
Posted by "chaokunyang (via GitHub)" <gi...@apache.org>.
chaokunyang commented on code in PR #2284:
URL: https://github.com/apache/fury/pull/2284#discussion_r2122963815
##########
java/fury-core/src/main/java/org/apache/fury/Fury.java:
##########
@@ -1679,6 +1681,13 @@ public boolean compressLong() {
return config.compressLong();
}
+ public ExceptionLogMode getExceptionLogMode() {
+ return config.getExceptionLogMode();
+ }
+ public int getLogSampleStep() {
Review Comment:
And I think we don't need to duplicate fields here, we could just pass value directly from Config
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@fury.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@fury.apache.org
For additional commands, e-mail: commits-help@fury.apache.org
Re: [PR] add log downgrade configuration [fury]
Posted by "ZhiQiang-Tiger (via GitHub)" <gi...@apache.org>.
ZhiQiang-Tiger commented on code in PR #2284:
URL: https://github.com/apache/fury/pull/2284#discussion_r2122814350
##########
java/fury-core/src/main/java/org/apache/fury/config/ExceptionLogMode.java:
##########
@@ -0,0 +1,7 @@
+package org.apache.fury.config;
Review Comment:
> The newly added files need to have a license header inserted at the top.
okay, I will fix it later.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@fury.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@fury.apache.org
For additional commands, e-mail: commits-help@fury.apache.org
Re: [PR] add log downgrade configuration [fury]
Posted by "chaokunyang (via GitHub)" <gi...@apache.org>.
chaokunyang commented on code in PR #2284:
URL: https://github.com/apache/fury/pull/2284#discussion_r2122758916
##########
java/fury-core/src/main/java/org/apache/fury/Fury.java:
##########
@@ -19,47 +19,18 @@
package org.apache.fury;
-import java.io.ByteArrayOutputStream;
-import java.io.IOException;
-import java.io.OutputStream;
-import java.nio.ByteOrder;
-import java.util.ArrayList;
-import java.util.Arrays;
-import java.util.HashMap;
-import java.util.Iterator;
-import java.util.function.Consumer;
-import java.util.function.Function;
-import javax.annotation.concurrent.NotThreadSafe;
import org.apache.fury.builder.JITContext;
import org.apache.fury.collection.IdentityMap;
-import org.apache.fury.config.CompatibleMode;
-import org.apache.fury.config.Config;
-import org.apache.fury.config.FuryBuilder;
-import org.apache.fury.config.Language;
-import org.apache.fury.config.LongEncoding;
+import org.apache.fury.config.*;
Review Comment:
Fory don't use `*` for imports, please execute `mvn spotless:apply & mvn checkstyle:check`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@fury.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@fury.apache.org
For additional commands, e-mail: commits-help@fury.apache.org
Re: [PR] add log downgrade configuration [fury]
Posted by "chaokunyang (via GitHub)" <gi...@apache.org>.
chaokunyang commented on code in PR #2284:
URL: https://github.com/apache/fury/pull/2284#discussion_r2122781395
##########
java/fury-core/src/main/java/org/apache/fury/config/FuryBuilder.java:
##########
@@ -142,6 +145,20 @@ public FuryBuilder serializeEnumByName(boolean serializeEnumByName) {
return this;
}
+ /** exception log level
+ * choose the log level, print the error message.
+ * */
+ public FuryBuilder exceptionLogMode(ExceptionLogMode exceptionLogMode) {
Review Comment:
```suggestion
// -1 indicates all sample,s 0 indcates no samples, a positive int indicates max samples
public FuryBuilder withDeserializationExceptionMaxRecordSamples(int maxSamples) {
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@fury.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@fury.apache.org
For additional commands, e-mail: commits-help@fury.apache.org
Re: [PR] feat(java): add log downgrade configuration [fury]
Posted by "ZhiQiang-Tiger (via GitHub)" <gi...@apache.org>.
ZhiQiang-Tiger closed pull request #2284: feat(java): add log downgrade configuration
URL: https://github.com/apache/fury/pull/2284
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@fury.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@fury.apache.org
For additional commands, e-mail: commits-help@fury.apache.org
Re: [PR] add log downgrade configuration [fury]
Posted by "ZhiQiang-Tiger (via GitHub)" <gi...@apache.org>.
ZhiQiang-Tiger commented on code in PR #2284:
URL: https://github.com/apache/fury/pull/2284#discussion_r2122803670
##########
java/fury-core/src/main/java/org/apache/fury/util/ExceptionUtils.java:
##########
@@ -57,15 +59,36 @@ public static StackOverflowError trySetStackOverflowErrorMessage(
public static RuntimeException handleReadFailed(Fury fury, Throwable t) {
if (fury.getRefResolver() instanceof MapRefResolver) {
- ObjectArray readObjects = ((MapRefResolver) fury.getRefResolver()).getReadObjects();
- // carry with read objects for better trouble shooting.
- List<Object> objects = Arrays.asList(readObjects.objects).subList(0, readObjects.size);
- throw new DeserializationException(objects, t);
+ List<Object> exceptionObjects = getExceptionObjects(fury);
+ throw new DeserializationException(exceptionObjects, t);
} else {
Platform.throwException(t);
throw new IllegalStateException("unreachable");
}
}
+ public static List<Object> getExceptionObjects(Fury fury) {
+ ObjectArray readObjects = ((MapRefResolver) fury.getRefResolver()).getReadObjects();
+ // carry with read objects for better trouble shooting.
+ List<Object> objects = Arrays.asList(readObjects.objects).subList(0, readObjects.size);
+ switch (fury.getExceptionLogMode()) {
+ case NONE_PRINT:
+ return new ArrayList<>();
+ case SAMPLE_PRINT:
Review Comment:
> how about limiting max samples instead of configuring sample step? Using step will print little things for small object, and still print lots of things for big object
I think that the user will select a suitable log mode by judging the situation of the object. In other words,our design is scalable and we can employ multiple logging strategies to solve different situation.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@fury.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@fury.apache.org
For additional commands, e-mail: commits-help@fury.apache.org
Re: [PR] feat(java): add log downgrade configuration [fury]
Posted by "chaokunyang (via GitHub)" <gi...@apache.org>.
chaokunyang commented on code in PR #2284:
URL: https://github.com/apache/fury/pull/2284#discussion_r2122964933
##########
java/fury-core/src/main/java/org/apache/fury/util/ExceptionUtils.java:
##########
@@ -57,15 +59,35 @@ public static StackOverflowError trySetStackOverflowErrorMessage(
public static RuntimeException handleReadFailed(Fury fury, Throwable t) {
if (fury.getRefResolver() instanceof MapRefResolver) {
- ObjectArray readObjects = ((MapRefResolver) fury.getRefResolver()).getReadObjects();
- // carry with read objects for better trouble shooting.
- List<Object> objects = Arrays.asList(readObjects.objects).subList(0, readObjects.size);
- throw new DeserializationException(objects, t);
+ List<Object> exceptionObjects = getExceptionObjects(fury);
+ throw new DeserializationException(exceptionObjects, t);
} else {
Platform.throwException(t);
throw new IllegalStateException("unreachable");
}
}
+ public static List<Object> getExceptionObjects(Fury fury) {
+ ObjectArray readObjects = ((MapRefResolver) fury.getRefResolver()).getReadObjects();
+ // carry with read objects for better trouble shooting.
+ List<Object> objects = Arrays.asList(readObjects.objects).subList(0, readObjects.size);
+ switch (fury.getExceptionLogMode()) {
Review Comment:
```suggestion
switch (fury.getConfig().getExceptionLogMode()) {
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@fury.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@fury.apache.org
For additional commands, e-mail: commits-help@fury.apache.org
Re: [PR] feat(java): add log downgrade configuration [fury]
Posted by "chaokunyang (via GitHub)" <gi...@apache.org>.
chaokunyang commented on code in PR #2284:
URL: https://github.com/apache/fury/pull/2284#discussion_r2122959672
##########
java/fury-core/src/main/java/org/apache/fury/Fury.java:
##########
@@ -1679,6 +1681,13 @@ public boolean compressLong() {
return config.compressLong();
}
+ public ExceptionLogMode getExceptionLogMode() {
+ return config.getExceptionLogMode();
+ }
+ public int getLogSampleStep() {
Review Comment:
Please format code by `mvn spotless:apply`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@fury.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@fury.apache.org
For additional commands, e-mail: commits-help@fury.apache.org
Re: [PR] add log downgrade configuration [fury]
Posted by "chaokunyang (via GitHub)" <gi...@apache.org>.
chaokunyang commented on PR #2284:
URL: https://github.com/apache/fury/pull/2284#issuecomment-2933716782
Please check branch from latest main branch, we renamed fury to fory in #2263
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@fury.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@fury.apache.org
For additional commands, e-mail: commits-help@fury.apache.org
Re: [PR] add log downgrade configuration [fury]
Posted by "chaokunyang (via GitHub)" <gi...@apache.org>.
chaokunyang commented on code in PR #2284:
URL: https://github.com/apache/fury/pull/2284#discussion_r2122781395
##########
java/fury-core/src/main/java/org/apache/fury/config/FuryBuilder.java:
##########
@@ -142,6 +145,20 @@ public FuryBuilder serializeEnumByName(boolean serializeEnumByName) {
return this;
}
+ /** exception log level
+ * choose the log level, print the error message.
+ * */
+ public FuryBuilder exceptionLogMode(ExceptionLogMode exceptionLogMode) {
Review Comment:
```suggestion
// -1 indicates all sample,s 0 indcates no samples, a positive int indicates max samples
public FuryBuilder withDeserializationExceptionMsgMaxSamples(int maxSamples) {
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@fury.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@fury.apache.org
For additional commands, e-mail: commits-help@fury.apache.org
Re: [PR] add log downgrade configuration [fury]
Posted by "chaokunyang (via GitHub)" <gi...@apache.org>.
chaokunyang commented on code in PR #2284:
URL: https://github.com/apache/fury/pull/2284#discussion_r2122850500
##########
java/fury-core/src/main/java/org/apache/fury/util/ExceptionUtils.java:
##########
@@ -57,15 +59,36 @@ public static StackOverflowError trySetStackOverflowErrorMessage(
public static RuntimeException handleReadFailed(Fury fury, Throwable t) {
if (fury.getRefResolver() instanceof MapRefResolver) {
- ObjectArray readObjects = ((MapRefResolver) fury.getRefResolver()).getReadObjects();
- // carry with read objects for better trouble shooting.
- List<Object> objects = Arrays.asList(readObjects.objects).subList(0, readObjects.size);
- throw new DeserializationException(objects, t);
+ List<Object> exceptionObjects = getExceptionObjects(fury);
+ throw new DeserializationException(exceptionObjects, t);
} else {
Platform.throwException(t);
throw new IllegalStateException("unreachable");
}
}
+ public static List<Object> getExceptionObjects(Fury fury) {
+ ObjectArray readObjects = ((MapRefResolver) fury.getRefResolver()).getReadObjects();
+ // carry with read objects for better trouble shooting.
+ List<Object> objects = Arrays.asList(readObjects.objects).subList(0, readObjects.size);
+ switch (fury.getExceptionLogMode()) {
+ case NONE_PRINT:
+ return new ArrayList<>();
+ case SAMPLE_PRINT:
Review Comment:
Users can know how big of theri object graph. Iif we use step, the issue won't be resolved in some cases. Or we could increase a new strategy which limit max samples, and let that mode as the default mode
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@fury.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@fury.apache.org
For additional commands, e-mail: commits-help@fury.apache.org
Re: [PR] add log downgrade configuration [fury]
Posted by "chaokunyang (via GitHub)" <gi...@apache.org>.
chaokunyang commented on code in PR #2284:
URL: https://github.com/apache/fury/pull/2284#discussion_r2122773852
##########
java/fury-core/src/main/java/org/apache/fury/util/ExceptionUtils.java:
##########
@@ -57,15 +59,36 @@ public static StackOverflowError trySetStackOverflowErrorMessage(
public static RuntimeException handleReadFailed(Fury fury, Throwable t) {
if (fury.getRefResolver() instanceof MapRefResolver) {
- ObjectArray readObjects = ((MapRefResolver) fury.getRefResolver()).getReadObjects();
- // carry with read objects for better trouble shooting.
- List<Object> objects = Arrays.asList(readObjects.objects).subList(0, readObjects.size);
- throw new DeserializationException(objects, t);
+ List<Object> exceptionObjects = getExceptionObjects(fury);
+ throw new DeserializationException(exceptionObjects, t);
} else {
Platform.throwException(t);
throw new IllegalStateException("unreachable");
}
}
+ public static List<Object> getExceptionObjects(Fury fury) {
+ ObjectArray readObjects = ((MapRefResolver) fury.getRefResolver()).getReadObjects();
+ // carry with read objects for better trouble shooting.
+ List<Object> objects = Arrays.asList(readObjects.objects).subList(0, readObjects.size);
+ switch (fury.getExceptionLogMode()) {
+ case NONE_PRINT:
+ return new ArrayList<>();
+ case SAMPLE_PRINT:
Review Comment:
how about limiting max samples instead of configuring sample step? Using step will print little things for small object, and still print lots of things for big object
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@fury.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@fury.apache.org
For additional commands, e-mail: commits-help@fury.apache.org
Re: [PR] add log downgrade configuration [fury]
Posted by "chaokunyang (via GitHub)" <gi...@apache.org>.
chaokunyang commented on code in PR #2284:
URL: https://github.com/apache/fury/pull/2284#discussion_r2122766374
##########
java/fury-core/src/test/java/org/apache/fury/LogDowngradeTest.java:
##########
@@ -0,0 +1,48 @@
+package org.apache.fury;
+
+
+import org.apache.fury.config.ExceptionLogMode;
+import org.apache.fury.config.Language;
+import org.testng.annotations.Test;
+
+import java.util.Base64;
+
+
+public class LogDowngradeTest extends FuryTestBase{
+ @Test
+ public void testLog(){
+ Fury fury = Fury.builder()
+ .exceptionLogMode(ExceptionLogMode.SAMPLE_PRINT, 2)
Review Comment:
```suggestion
.withExceptionLogMode(ExceptionLogMode.SAMPLE_PRINT, 2)
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@fury.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@fury.apache.org
For additional commands, e-mail: commits-help@fury.apache.org
Re: [PR] add log downgrade configuration [fury]
Posted by "chaokunyang (via GitHub)" <gi...@apache.org>.
chaokunyang commented on code in PR #2284:
URL: https://github.com/apache/fury/pull/2284#discussion_r2122756611
##########
java/fury-core/pom.xml:
##########
@@ -51,7 +51,16 @@
<maven.compiler.target>8</maven.compiler.target>
<fury.java.rootdir>${basedir}/..</fury.java.rootdir>
</properties>
-
+ <distributionManagement>
Review Comment:
Please remove this
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@fury.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@fury.apache.org
For additional commands, e-mail: commits-help@fury.apache.org