You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@storm.apache.org by rban1 <gi...@git.apache.org> on 2017/03/03 03:37:04 UTC
[GitHub] storm pull request #1985: STORM-2371: New eventhub implementation
GitHub user rban1 opened a pull request:
https://github.com/apache/storm/pull/1985
STORM-2371: New eventhub implementation
Made the changes with the latest comments
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/rban1/storm eventhub1
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/storm/pull/1985.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #1985
----
commit e3545aa692673bd379d88d5009c2b8662a9817ec
Author: Ranjan Banerjee <ra...@microsoft.com>
Date: 2017-03-03T03:28:49Z
STORM-2371: New eventhub implementation
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] storm pull request #1985: STORM-2371: New eventhub implementation
Posted by SreeramGarlapati <gi...@git.apache.org>.
Github user SreeramGarlapati commented on a diff in the pull request:
https://github.com/apache/storm/pull/1985#discussion_r104280244
--- Diff: external/storm-eventhubs/src/main/java/org/apache/storm/eventhubs/spout/StringEventDataScheme.java ---
@@ -46,19 +39,11 @@
private static final long serialVersionUID = 1L;
@Override
- public List<Object> deserialize(Message message) {
+ public List<Object> deserialize(EventData eventData) {
final List<Object> fieldContents = new ArrayList<Object>();
-
- for (Section section : message.getPayload()) {
- if (section instanceof Data) {
- Data data = (Data) section;
- fieldContents.add(new String(data.getValue().getArray()));
- } else if (section instanceof AmqpValue) {
- AmqpValue amqpValue = (AmqpValue) section;
- fieldContents.add(amqpValue.getValue().toString());
- }
- }
-
+ String messageData = "";
+ messageData = new String (eventData.getBody(),eventData.getBodyOffset(),eventData.getBodyLength(),Charset.defaultCharset());
--- End diff --
Add null check for getbody()
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] storm pull request #1985: STORM-2371: New eventhub implementation
Posted by SreeramGarlapati <gi...@git.apache.org>.
Github user SreeramGarlapati commented on a diff in the pull request:
https://github.com/apache/storm/pull/1985#discussion_r104280360
--- Diff: external/storm-eventhubs/src/main/java/org/apache/storm/eventhubs/spout/EventDataScheme.java ---
@@ -46,25 +42,11 @@
private static final long serialVersionUID = 1L;
@Override
- public List<Object> deserialize(Message message) {
+ public List<Object> deserialize(EventData eventData) {
final List<Object> fieldContents = new ArrayList<Object>();
-
- Map metaDataMap = new HashMap();
String messageData = "";
-
- for (Section section : message.getPayload()) {
- if (section instanceof Data) {
- Data data = (Data) section;
- messageData = new String(data.getValue().getArray());
- } else if (section instanceof AmqpValue) {
- AmqpValue amqpValue = (AmqpValue) section;
- messageData = amqpValue.getValue().toString();
- } else if (section instanceof ApplicationProperties) {
- final ApplicationProperties applicationProperties = (ApplicationProperties) section;
- metaDataMap = applicationProperties.getValue();
- }
- }
-
+ messageData = new String (eventData.getBody(),eventData.getBodyOffset(),eventData.getBodyLength(),Charset.defaultCharset());
--- End diff --
The old code handles the case when AmqpValue is species as payload - if you need that support here - you will need to wait until 0.12.0
Refer to this pr for details : https://github.com/Azure/azure-event-hubs-java/pull/66
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] storm pull request #1985: STORM-2371: New eventhub implementation
Posted by rban1 <gi...@git.apache.org>.
Github user rban1 commented on a diff in the pull request:
https://github.com/apache/storm/pull/1985#discussion_r104231241
--- Diff: external/storm-eventhubs/src/main/java/org/apache/storm/eventhubs/spout/EventHubReceiverImpl.java ---
@@ -65,77 +60,80 @@ public EventHubReceiverImpl(EventHubSpoutConfig config, String partitionId) {
}
@Override
- public void open(IEventHubFilter filter) throws EventHubException {
- logger.info("creating eventhub receiver: partitionId=" + partitionId +
- ", filterString=" + filter.getFilterString());
+ public void open(String offset) throws EventHubException {
+ logger.info("creating eventhub receiver: partitionId=" + partitionId +
+ ", offset=" + offset);
long start = System.currentTimeMillis();
- receiver = new ResilientEventHubReceiver(connectionString, entityName,
- partitionId, consumerGroupName, defaultCredits, filter);
- receiver.initialize();
-
+ try {
+ ehClient = EventHubClient.createFromConnectionStringSync(connectionString);
+ receiver = ehClient.createEpochReceiverSync(
+ consumerGroupName,
+ partitionId,
+ offset,
+ false,
+ 1);
+ }catch (Exception e){
+ logger.info("Exception in creating EventhubClient"+e.toString());
+ }
long end = System.currentTimeMillis();
logger.info("created eventhub receiver, time taken(ms): " + (end-start));
}
@Override
- public void close() {
+ public void close(){
if(receiver != null) {
- receiver.close();
+ try {
+ receiver.close().whenComplete((voidargs,error)->{
+ try{
+ if(error!=null){
+ logger.error("Exception during receiver close phase"+error.toString());
+ }
+ ehClient.closeSync();
+ }catch (Exception e){
+ logger.error("Exception during ehclient close phase"+e.toString());
+ }
+ }).get();
+ }catch (InterruptedException e){
+ logger.error("Exception occured during close phase"+e.toString());
+ }catch (ExecutionException e){
+ logger.error("Exception occured during close phase"+e.toString());
+ }
logger.info("closed eventhub receiver: partitionId=" + partitionId );
receiver = null;
+ ehClient = null;
}
}
+
@Override
public boolean isOpen() {
return (receiver != null);
}
@Override
- public EventData receive(long timeoutInMilliseconds) {
+ public EventDataWrap receive() {
long start = System.currentTimeMillis();
- Message message = receiver.receive(timeoutInMilliseconds);
+ Iterable<EventData> receivedEvents=null;
+ /*Get one message at a time for backward compatibility behaviour*/
+ try {
+ receivedEvents = receiver.receiveSync(1);
+ }catch (Exception e){
+ logger.error("Exception occured during receive"+e.toString());
+ }
long end = System.currentTimeMillis();
long millis = (end - start);
receiveApiLatencyMean.update(millis);
receiveApiCallCount.incr();
-
- if (message == null) {
- //Temporary workaround for AMQP/EH bug of failing to receive messages
- /*if(timeoutInMilliseconds > 100 && millis < timeoutInMilliseconds/2) {
- throw new RuntimeException(
- "Restart EventHubSpout due to failure of receiving messages in "
- + millis + " millisecond");
- }*/
+ if (receivedEvents == null) {
--- End diff --
check for length also
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] storm pull request #1985: STORM-2371: New eventhub implementation
Posted by SreeramGarlapati <gi...@git.apache.org>.
Github user SreeramGarlapati commented on a diff in the pull request:
https://github.com/apache/storm/pull/1985#discussion_r104280323
--- Diff: external/storm-eventhubs/src/main/java/org/apache/storm/eventhubs/spout/EventDataScheme.java ---
@@ -46,25 +42,11 @@
private static final long serialVersionUID = 1L;
@Override
- public List<Object> deserialize(Message message) {
+ public List<Object> deserialize(EventData eventData) {
final List<Object> fieldContents = new ArrayList<Object>();
-
- Map metaDataMap = new HashMap();
String messageData = "";
-
- for (Section section : message.getPayload()) {
- if (section instanceof Data) {
- Data data = (Data) section;
- messageData = new String(data.getValue().getArray());
- } else if (section instanceof AmqpValue) {
- AmqpValue amqpValue = (AmqpValue) section;
- messageData = amqpValue.getValue().toString();
- } else if (section instanceof ApplicationProperties) {
- final ApplicationProperties applicationProperties = (ApplicationProperties) section;
- metaDataMap = applicationProperties.getValue();
- }
- }
-
+ messageData = new String (eventData.getBody(),eventData.getBodyOffset(),eventData.getBodyLength(),Charset.defaultCharset());
--- End diff --
Null check getBody
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] storm pull request #1985: STORM-2371: New eventhub implementation
Posted by rban1 <gi...@git.apache.org>.
Github user rban1 commented on a diff in the pull request:
https://github.com/apache/storm/pull/1985#discussion_r104227737
--- Diff: external/storm-eventhubs/pom.xml ---
@@ -95,6 +95,11 @@
<version>${eventhubs.client.version}</version>
</dependency>
<dependency>
+ <groupId>com.microsoft.azure</groupId>
+ <artifactId>azure-eventhubs</artifactId>
+ <version>0.11.0</version>
--- End diff --
make it a property
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] storm pull request #1985: STORM-2371: New eventhub implementation
Posted by SreeramGarlapati <gi...@git.apache.org>.
Github user SreeramGarlapati commented on a diff in the pull request:
https://github.com/apache/storm/pull/1985#discussion_r104280523
--- Diff: external/storm-eventhubs/src/main/java/org/apache/storm/eventhubs/spout/EventHubReceiverImpl.java ---
@@ -65,77 +60,80 @@ public EventHubReceiverImpl(EventHubSpoutConfig config, String partitionId) {
}
@Override
- public void open(IEventHubFilter filter) throws EventHubException {
- logger.info("creating eventhub receiver: partitionId=" + partitionId +
- ", filterString=" + filter.getFilterString());
+ public void open(String offset) throws EventHubException {
+ logger.info("creating eventhub receiver: partitionId=" + partitionId +
+ ", offset=" + offset);
long start = System.currentTimeMillis();
- receiver = new ResilientEventHubReceiver(connectionString, entityName,
- partitionId, consumerGroupName, defaultCredits, filter);
- receiver.initialize();
-
+ try {
+ ehClient = EventHubClient.createFromConnectionStringSync(connectionString);
+ receiver = ehClient.createEpochReceiverSync(
+ consumerGroupName,
+ partitionId,
+ offset,
+ false,
+ 1);
--- End diff --
Please refrain from using catch all exceptions and move to catching more specific exceptions...
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] storm issue #1985: STORM-2371: New eventhub implementation
Posted by srdo <gi...@git.apache.org>.
Github user srdo commented on the issue:
https://github.com/apache/storm/pull/1985
I think you should push these changes back to your previous PR here https://github.com/apache/storm/pull/1951, otherwise it's hard to know which comments you're talking about. It also makes it harder for reviewers to know whether an issue has already been discussed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] storm pull request #1985: STORM-2371: New eventhub implementation
Posted by rban1 <gi...@git.apache.org>.
Github user rban1 commented on a diff in the pull request:
https://github.com/apache/storm/pull/1985#discussion_r104561334
--- Diff: external/storm-eventhubs/src/main/java/org/apache/storm/eventhubs/spout/EventHubReceiverImpl.java ---
@@ -65,77 +60,80 @@ public EventHubReceiverImpl(EventHubSpoutConfig config, String partitionId) {
}
@Override
- public void open(IEventHubFilter filter) throws EventHubException {
- logger.info("creating eventhub receiver: partitionId=" + partitionId +
- ", filterString=" + filter.getFilterString());
+ public void open(String offset) throws EventHubException {
+ logger.info("creating eventhub receiver: partitionId=" + partitionId +
+ ", offset=" + offset);
long start = System.currentTimeMillis();
- receiver = new ResilientEventHubReceiver(connectionString, entityName,
- partitionId, consumerGroupName, defaultCredits, filter);
- receiver.initialize();
-
+ try {
+ ehClient = EventHubClient.createFromConnectionStringSync(connectionString);
+ receiver = ehClient.createEpochReceiverSync(
+ consumerGroupName,
+ partitionId,
+ offset,
+ false,
+ 1);
--- End diff --
Will do
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] storm pull request #1985: STORM-2371: New eventhub implementation
Posted by rban1 <gi...@git.apache.org>.
Github user rban1 commented on a diff in the pull request:
https://github.com/apache/storm/pull/1985#discussion_r104561111
--- Diff: external/storm-eventhubs/src/main/java/org/apache/storm/eventhubs/spout/StringEventDataScheme.java ---
@@ -46,19 +39,11 @@
private static final long serialVersionUID = 1L;
@Override
- public List<Object> deserialize(Message message) {
+ public List<Object> deserialize(EventData eventData) {
final List<Object> fieldContents = new ArrayList<Object>();
-
- for (Section section : message.getPayload()) {
- if (section instanceof Data) {
- Data data = (Data) section;
- fieldContents.add(new String(data.getValue().getArray()));
- } else if (section instanceof AmqpValue) {
- AmqpValue amqpValue = (AmqpValue) section;
- fieldContents.add(amqpValue.getValue().toString());
- }
- }
-
+ String messageData = "";
+ messageData = new String (eventData.getBody(),eventData.getBodyOffset(),eventData.getBodyLength(),Charset.defaultCharset());
--- End diff --
will do
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] storm pull request #1985: STORM-2371: New eventhub implementation
Posted by rban1 <gi...@git.apache.org>.
Github user rban1 commented on a diff in the pull request:
https://github.com/apache/storm/pull/1985#discussion_r104228014
--- Diff: external/storm-eventhubs/src/main/java/org/apache/storm/eventhubs/bolt/EventHubBolt.java ---
@@ -18,14 +18,16 @@
package org.apache.storm.eventhubs.bolt;
import java.util.Map;
+import java.util.concurrent.ExecutionException;
--- End diff --
order the imports
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] storm pull request #1985: STORM-2371: New eventhub implementation
Posted by rban1 <gi...@git.apache.org>.
Github user rban1 commented on a diff in the pull request:
https://github.com/apache/storm/pull/1985#discussion_r104561293
--- Diff: external/storm-eventhubs/src/main/java/org/apache/storm/eventhubs/spout/EventDataScheme.java ---
@@ -46,25 +42,11 @@
private static final long serialVersionUID = 1L;
@Override
- public List<Object> deserialize(Message message) {
+ public List<Object> deserialize(EventData eventData) {
final List<Object> fieldContents = new ArrayList<Object>();
-
- Map metaDataMap = new HashMap();
String messageData = "";
-
- for (Section section : message.getPayload()) {
- if (section instanceof Data) {
- Data data = (Data) section;
- messageData = new String(data.getValue().getArray());
- } else if (section instanceof AmqpValue) {
- AmqpValue amqpValue = (AmqpValue) section;
- messageData = amqpValue.getValue().toString();
- } else if (section instanceof ApplicationProperties) {
- final ApplicationProperties applicationProperties = (ApplicationProperties) section;
- metaDataMap = applicationProperties.getValue();
- }
- }
-
+ messageData = new String (eventData.getBody(),eventData.getBodyOffset(),eventData.getBodyLength(),Charset.defaultCharset());
--- End diff --
Thanks. Is there any changes for now that you see in the serialization code that needs to be incorporated? Otherwise I will keep an eye for the 0.12.0 version and then incorporate AmqpValue also as a payload
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---