You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@storm.apache.org by "Morrigan Jones (JIRA)" <ji...@apache.org> on 2016/05/20 14:59:12 UTC
[jira] [Updated] (STORM-1854) Trident transactional spouts are
broken in 1.0.x
[ https://issues.apache.org/jira/browse/STORM-1854?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Morrigan Jones updated STORM-1854:
----------------------------------
Description:
In the process of upgrading our Storm code from 0.10.0 to 1.0.0, I've run into an issue with TransactionalTridentKafkaSpout. When running one of our topologies I'm getting the following exception:
{code}
Caused by: java.lang.ClassCastException: java.util.ArrayList cannot be cast to java.lang.Integer
at org.apache.storm.trident.spout.PartitionedTridentSpoutExecutor$Coordinator.initializeTransaction(PartitionedTridentSpoutExecutor.java:55) ~[storm-core-1.0.0.jar:1.0.0]
at org.apache.storm.trident.spout.PartitionedTridentSpoutExecutor$Coordinator.initializeTransaction(PartitionedTridentSpoutExecutor.java:43) ~[storm-core-1.0.0.jar:1.0.0]
at org.apache.storm.trident.spout.TridentSpoutCoordinator.execute(TridentSpoutCoordinator.java:70) ~[storm-core-1.0.0.jar:1.0.0]
at org.apache.storm.topology.BasicBoltExecutor.execute(BasicBoltExecutor.java:50) ~[storm-core-1.0.0.jar:1.0.0]
{code}
The issue appears to be caused by a change in PartitionedTridentSpoutExecutor between the two versions, specifically this method:
1.0.0 - https://github.com/apache/storm/blob/v1.0.0/storm-core/src/jvm/org/apache/storm/trident/spout/PartitionedTridentSpoutExecutor.java#L51
{code}
public Integer initializeTransaction(long txid, Integer prevMetadata, Integer currMetadata) {
if(currMetadata!=null) {
return currMetadata;
} else {
return _coordinator.getPartitionsForBatch();
}
}
{code}
0.10.0 - https://github.com/apache/storm/blob/v0.10.0/storm-core/src/jvm/storm/trident/spout/PartitionedTridentSpoutExecutor.java#L51
{code}
public Object initializeTransaction(long txid, Object prevMetadata, Object currMetadata) {
if(currMetadata!=null) {
return currMetadata;
} else {
return _coordinator.getPartitionsForBatch();
}
}
{code}
This was introduced by: https://github.com/apache/storm/commit/9e4c3df17ffbc737210e606d3d8a9cdae8f86634
TransactionalTridentKafkaSpout uses List<GlobalPartitionInformation> for its metadata. Generally, transactional spouts should have metadata that is more complex than just an Integer. OpaquePartitionedTridentSpoutExecutor uses Object for its metadata and correctly handles the metadata used by OpaqueTridentKafkaSpout (List<GlobalPartitionInformation>).
It looks like reverting the metadata type for transactional spouts in PartitionedTridentSpoutExecutor should work, but I haven't tried this yet.
was:
In the process of upgrading our Storm code from 0.10.0 to 1.0.0, I've run into an issue with TransactionalTridentKafkaSpout. When running one of our topologies I'm getting the following exception:
Caused by: java.lang.ClassCastException: java.util.ArrayList cannot be cast to java.lang.Integer
at org.apache.storm.trident.spout.PartitionedTridentSpoutExecutor$Coordinator.initializeTransaction(PartitionedTridentSpoutExecutor.java:55) ~[storm-core-1.0.0.jar:1.0.0]
at org.apache.storm.trident.spout.PartitionedTridentSpoutExecutor$Coordinator.initializeTransaction(PartitionedTridentSpoutExecutor.java:43) ~[storm-core-1.0.0.jar:1.0.0]
at org.apache.storm.trident.spout.TridentSpoutCoordinator.execute(TridentSpoutCoordinator.java:70) ~[storm-core-1.0.0.jar:1.0.0]
at org.apache.storm.topology.BasicBoltExecutor.execute(BasicBoltExecutor.java:50) ~[storm-core-1.0.0.jar:1.0.0]
The issue appears to be caused by a change in PartitionedTridentSpoutExecutor between the two versions, specifically this method:
1.0.0 - https://github.com/apache/storm/blob/v1.0.0/storm-core/src/jvm/org/apache/storm/trident/spout/PartitionedTridentSpoutExecutor.java#L51
public Integer initializeTransaction(long txid, Integer prevMetadata, Integer currMetadata) {
if(currMetadata!=null) {
return currMetadata;
} else {
return _coordinator.getPartitionsForBatch();
}
}
0.10.0 - https://github.com/apache/storm/blob/v0.10.0/storm-core/src/jvm/storm/trident/spout/PartitionedTridentSpoutExecutor.java#L51
public Object initializeTransaction(long txid, Object prevMetadata, Object currMetadata) {
if(currMetadata!=null) {
return currMetadata;
} else {
return _coordinator.getPartitionsForBatch();
}
}
This was introduced by: https://github.com/apache/storm/commit/9e4c3df17ffbc737210e606d3d8a9cdae8f86634
TransactionalTridentKafkaSpout uses List<GlobalPartitionInformation> for its metadata. Generally, transactional spouts should have metadata that is more complex than just an Integer. OpaquePartitionedTridentSpoutExecutor uses Object for its metadata and correctly handles the metadata used by OpaqueTridentKafkaSpout (List<GlobalPartitionInformation>).
It looks like reverting the metadata type for transactional spouts in PartitionedTridentSpoutExecutor should work, but I haven't tried this yet.
> Trident transactional spouts are broken in 1.0.x
> ------------------------------------------------
>
> Key: STORM-1854
> URL: https://issues.apache.org/jira/browse/STORM-1854
> Project: Apache Storm
> Issue Type: Bug
> Components: trident
> Affects Versions: 1.0.0, 1.0.1
> Reporter: Morrigan Jones
>
> In the process of upgrading our Storm code from 0.10.0 to 1.0.0, I've run into an issue with TransactionalTridentKafkaSpout. When running one of our topologies I'm getting the following exception:
> {code}
> Caused by: java.lang.ClassCastException: java.util.ArrayList cannot be cast to java.lang.Integer
> at org.apache.storm.trident.spout.PartitionedTridentSpoutExecutor$Coordinator.initializeTransaction(PartitionedTridentSpoutExecutor.java:55) ~[storm-core-1.0.0.jar:1.0.0]
> at org.apache.storm.trident.spout.PartitionedTridentSpoutExecutor$Coordinator.initializeTransaction(PartitionedTridentSpoutExecutor.java:43) ~[storm-core-1.0.0.jar:1.0.0]
> at org.apache.storm.trident.spout.TridentSpoutCoordinator.execute(TridentSpoutCoordinator.java:70) ~[storm-core-1.0.0.jar:1.0.0]
> at org.apache.storm.topology.BasicBoltExecutor.execute(BasicBoltExecutor.java:50) ~[storm-core-1.0.0.jar:1.0.0]
> {code}
> The issue appears to be caused by a change in PartitionedTridentSpoutExecutor between the two versions, specifically this method:
> 1.0.0 - https://github.com/apache/storm/blob/v1.0.0/storm-core/src/jvm/org/apache/storm/trident/spout/PartitionedTridentSpoutExecutor.java#L51
> {code}
> public Integer initializeTransaction(long txid, Integer prevMetadata, Integer currMetadata) {
> if(currMetadata!=null) {
> return currMetadata;
> } else {
> return _coordinator.getPartitionsForBatch();
> }
> }
> {code}
> 0.10.0 - https://github.com/apache/storm/blob/v0.10.0/storm-core/src/jvm/storm/trident/spout/PartitionedTridentSpoutExecutor.java#L51
> {code}
> public Object initializeTransaction(long txid, Object prevMetadata, Object currMetadata) {
> if(currMetadata!=null) {
> return currMetadata;
> } else {
> return _coordinator.getPartitionsForBatch();
> }
> }
> {code}
> This was introduced by: https://github.com/apache/storm/commit/9e4c3df17ffbc737210e606d3d8a9cdae8f86634
> TransactionalTridentKafkaSpout uses List<GlobalPartitionInformation> for its metadata. Generally, transactional spouts should have metadata that is more complex than just an Integer. OpaquePartitionedTridentSpoutExecutor uses Object for its metadata and correctly handles the metadata used by OpaqueTridentKafkaSpout (List<GlobalPartitionInformation>).
> It looks like reverting the metadata type for transactional spouts in PartitionedTridentSpoutExecutor should work, but I haven't tried this yet.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)