You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@inlong.apache.org by GitBox <gi...@apache.org> on 2022/01/28 05:30:43 UTC
[GitHub] [incubator-inlong] KevinWen007 opened a new pull request #2387: [INLONG-2346][Feature][InLong-Sort] Support avro and canal formats for sort sink
KevinWen007 opened a new pull request #2387:
URL: https://github.com/apache/incubator-inlong/pull/2387
### Title Name: [INLONG-2346][Feature][InLong-Sort] Support avro and canal formats for sort sink
Fixes #2346
### Motivation
*Explain here the context, and why you're making that change. What is the problem you're trying to solve.*
### Modifications
*Describe the modifications you've done.*
### Verifying this change
- [ ] Make sure that the change passes the CI checks.
*(Please pick either of the following options)*
This change is a trivial rework / code cleanup without any test coverage.
*(or)*
This change is already covered by existing tests, such as *(please describe tests)*.
*(or)*
This change added tests and can be verified as follows:
*(example:)*
- *Added integration tests for end-to-end deployment with large payloads (10MB)*
- *Extended integration test for recovery after broker failure*
### Documentation
- Does this pull request introduce a new feature? (yes / no)
- If yes, how is the feature documented? (not applicable / docs / JavaDocs / not documented)
- If a feature is not applicable for documentation, explain why?
- If a feature is not documented yet in this PR, please create a followup issue for adding the documentation
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@inlong.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-inlong] KevinWen007 commented on a change in pull request #2387: [INLONG-2346][Feature][InLong-Sort] Support avro and canal formats for sort sink
Posted by GitBox <gi...@apache.org>.
KevinWen007 commented on a change in pull request #2387:
URL: https://github.com/apache/incubator-inlong/pull/2387#discussion_r807732040
##########
File path: inlong-sort/sort-formats/format-base/src/main/java/org/apache/inlong/sort/formats/base/TableFormatUtils.java
##########
@@ -331,7 +343,7 @@ public static LogicalType deriveLogicalType(FormatInfo formatInfo) {
} else if (formatInfo instanceof DateFormatInfo) {
return new DateType();
} else if (formatInfo instanceof TimestampFormatInfo) {
- return new TimestampType();
+ return new TimestampType(DEFAULT_PRECISION_FOR_TIMESTAMP);
Review comment:
Precision is not supported in current TimestampFormatInfo, and modification above is only to support avro format. Precision is not used elsewhere except avro.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@inlong.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-inlong] ywh19911024 commented on a change in pull request #2387: [INLONG-2346][Feature][InLong-Sort] Support avro and canal formats for sort sink
Posted by GitBox <gi...@apache.org>.
ywh19911024 commented on a change in pull request #2387:
URL: https://github.com/apache/incubator-inlong/pull/2387#discussion_r807686964
##########
File path: inlong-sort/pom.xml
##########
@@ -63,6 +63,7 @@
<pulsar.version>2.8.1</pulsar.version>
<kafka.version>2.4.1</kafka.version>
<lzo.core.version>1.0.6</lzo.core.version>
+ <avro.version>1.10.0</avro.version>
Review comment:
这个可以删除了，没看到用的地方
##########
File path: inlong-sort/sort-formats/format-base/src/main/java/org/apache/inlong/sort/formats/base/TableFormatUtils.java
##########
@@ -331,7 +343,7 @@ public static LogicalType deriveLogicalType(FormatInfo formatInfo) {
} else if (formatInfo instanceof DateFormatInfo) {
return new DateType();
} else if (formatInfo instanceof TimestampFormatInfo) {
- return new TimestampType();
+ return new TimestampType(DEFAULT_PRECISION_FOR_TIMESTAMP);
Review comment:
这里判断下呢，大于等于3的统一设置成2，小于的按照formatInfo中的来
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@inlong.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-inlong] dockerzhang merged pull request #2387: [INLONG-2346][Feature][InLong-Sort] Support avro and canal formats for sort sink
Posted by GitBox <gi...@apache.org>.
dockerzhang merged pull request #2387:
URL: https://github.com/apache/incubator-inlong/pull/2387
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@inlong.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-inlong] codecov-commenter commented on pull request #2387: [INLONG-2346][Feature][InLong-Sort] Support avro and canal formats for sort sink
Posted by GitBox <gi...@apache.org>.
codecov-commenter commented on pull request #2387:
URL: https://github.com/apache/incubator-inlong/pull/2387#issuecomment-1035058132
# [Codecov](https://codecov.io/gh/apache/incubator-inlong/pull/2387?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#2387](https://codecov.io/gh/apache/incubator-inlong/pull/2387?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (9109dfb) into [master](https://codecov.io/gh/apache/incubator-inlong/commit/e6de80d36d8de0244755fec49ad80d100d07d1d6?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (e6de80d) will **increase** coverage by `0.14%`.
> The diff coverage is `n/a`.
[](https://codecov.io/gh/apache/incubator-inlong/pull/2387?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
```diff
@@ Coverage Diff @@
## master #2387 +/- ##
============================================
+ Coverage 12.53% 12.67% +0.14%
- Complexity 1218 1253 +35
============================================
Files 420 426 +6
Lines 36255 36325 +70
Branches 5674 5679 +5
============================================
+ Hits 4545 4605 +60
- Misses 30901 30916 +15
+ Partials 809 804 -5
```
| [Impacted Files](https://codecov.io/gh/apache/incubator-inlong/pull/2387?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
|---|---|---|
| [...nlong/tubemq/corebase/policies/FlowCtrlResult.java](https://codecov.io/gh/apache/incubator-inlong/pull/2387/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aW5sb25nLXR1YmVtcS90dWJlbXEtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvaW5sb25nL3R1YmVtcS9jb3JlYmFzZS9wb2xpY2llcy9GbG93Q3RybFJlc3VsdC5qYXZh) | `0.00% <0.00%> (-60.00%)` | :arrow_down: |
| [.../inlong/tubemq/corebase/policies/FlowCtrlItem.java](https://codecov.io/gh/apache/incubator-inlong/pull/2387/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aW5sb25nLXR1YmVtcS90dWJlbXEtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvaW5sb25nL3R1YmVtcS9jb3JlYmFzZS9wb2xpY2llcy9GbG93Q3RybEl0ZW0uamF2YQ==) | `36.66% <0.00%> (-2.23%)` | :arrow_down: |
| [.../tubemq/corebase/policies/FlowCtrlRuleHandler.java](https://codecov.io/gh/apache/incubator-inlong/pull/2387/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aW5sb25nLXR1YmVtcS90dWJlbXEtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvaW5sb25nL3R1YmVtcS9jb3JlYmFzZS9wb2xpY2llcy9GbG93Q3RybFJ1bGVIYW5kbGVyLmphdmE=) | `32.96% <0.00%> (-1.11%)` | :arrow_down: |
| [...q/client/consumer/SimpleClientBalanceConsumer.java](https://codecov.io/gh/apache/incubator-inlong/pull/2387/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aW5sb25nLXR1YmVtcS90dWJlbXEtY2xpZW50L3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9pbmxvbmcvdHViZW1xL2NsaWVudC9jb25zdW1lci9TaW1wbGVDbGllbnRCYWxhbmNlQ29uc3VtZXIuamF2YQ==) | `0.00% <0.00%> (ø)` | |
| [...inlong/tubemq/corebase/metric/impl/BaseMetric.java](https://codecov.io/gh/apache/incubator-inlong/pull/2387/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aW5sb25nLXR1YmVtcS90dWJlbXEtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvaW5sb25nL3R1YmVtcS9jb3JlYmFzZS9tZXRyaWMvaW1wbC9CYXNlTWV0cmljLmphdmE=) | `100.00% <0.00%> (ø)` | |
| [...long/tubemq/corebase/metric/impl/LongMinGauge.java](https://codecov.io/gh/apache/incubator-inlong/pull/2387/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aW5sb25nLXR1YmVtcS90dWJlbXEtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvaW5sb25nL3R1YmVtcS9jb3JlYmFzZS9tZXRyaWMvaW1wbC9Mb25nTWluR2F1Z2UuamF2YQ==) | `71.42% <0.00%> (ø)` | |
| [.../inlong/tubemq/corebase/metric/impl/SinceTime.java](https://codecov.io/gh/apache/incubator-inlong/pull/2387/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aW5sb25nLXR1YmVtcS90dWJlbXEtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvaW5sb25nL3R1YmVtcS9jb3JlYmFzZS9tZXRyaWMvaW1wbC9TaW5jZVRpbWUuamF2YQ==) | `100.00% <0.00%> (ø)` | |
| [.../tubemq/corebase/metric/impl/LongStatsCounter.java](https://codecov.io/gh/apache/incubator-inlong/pull/2387/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aW5sb25nLXR1YmVtcS90dWJlbXEtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvaW5sb25nL3R1YmVtcS9jb3JlYmFzZS9tZXRyaWMvaW1wbC9Mb25nU3RhdHNDb3VudGVyLmphdmE=) | `100.00% <0.00%> (ø)` | |
| [...tubemq/corebase/metric/impl/LongOnlineCounter.java](https://codecov.io/gh/apache/incubator-inlong/pull/2387/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aW5sb25nLXR1YmVtcS90dWJlbXEtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvaW5sb25nL3R1YmVtcS9jb3JlYmFzZS9tZXRyaWMvaW1wbC9Mb25nT25saW5lQ291bnRlci5qYXZh) | `100.00% <0.00%> (ø)` | |
| [...long/tubemq/corebase/metric/impl/LongMaxGauge.java](https://codecov.io/gh/apache/incubator-inlong/pull/2387/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aW5sb25nLXR1YmVtcS90dWJlbXEtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvaW5sb25nL3R1YmVtcS9jb3JlYmFzZS9tZXRyaWMvaW1wbC9Mb25nTWF4R2F1Z2UuamF2YQ==) | `71.42% <0.00%> (ø)` | |
| ... and [3 more](https://codecov.io/gh/apache/incubator-inlong/pull/2387/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/incubator-inlong/pull/2387?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/incubator-inlong/pull/2387?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [e6de80d...9109dfb](https://codecov.io/gh/apache/incubator-inlong/pull/2387?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@inlong.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org