You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@bahir.apache.org by sbcd90 <gi...@git.apache.org> on 2017/03/27 04:56:58 UTC
[GitHub] bahir pull request #38: [BAHIR-97] Akka as a streaming source for SQL Stream...
GitHub user sbcd90 opened a pull request:
https://github.com/apache/bahir/pull/38
[BAHIR-97] Akka as a streaming source for SQL Streaming.
This PR is created to propose the addition of Akka compatible streaming source for Spark SQL Streaming.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/sbcd90/bahir sql-streaming-akka
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/bahir/pull/38.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #38
----
commit 565e35dbd0c5b748ab120d228595e0cdff173401
Author: Subhobrata Dey <sb...@gmail.com>
Date: 2017-03-27T04:30:30Z
[BAHIR-97] Akka as a streaming source for SQL Streaming.
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] bahir pull request #38: [BAHIR-97] Akka as a streaming source for SQL Stream...
Posted by ckadner <gi...@git.apache.org>.
Github user ckadner commented on a diff in the pull request:
https://github.com/apache/bahir/pull/38#discussion_r109263376
--- Diff: sql-streaming-akka/pom.xml ---
@@ -0,0 +1,120 @@
+<?xml version="1.0" encoding="UTF-8" ?>
+<!--
+ ~ Licensed to the Apache Software Foundation (ASF) under one or more
+ ~ contributor license agreements. See the NOTICE file distributed with
+ ~ this work for additional information regarding copyright ownership.
+ ~ The ASF licenses this file to You under the Apache License, Version 2.0
+ ~ (the "License"); you may not use this file except in compliance with
+ ~ the License. You may obtain a copy of the License at
+ ~
+ ~ http://www.apache.org/licenses/LICENSE-2.0
+ ~
+ ~ Unless required by applicable law or agreed to in writing, software
+ ~ distributed under the License is distributed on an "AS IS" BASIS,
+ ~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ ~ See the License for the specific language governing permissions and
+ ~ limitations under the License.
+ -->
+
+<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
+ <modelVersion>4.0.0</modelVersion>
+ <parent>
+ <groupId>org.apache.bahir</groupId>
+ <artifactId>bahir-parent_2.11</artifactId>
+ <version>2.2.0-SNAPSHOT</version>
+ <relativePath>../pom.xml</relativePath>
+ </parent>
+
+ <groupId>org.apache.bahir</groupId>
+ <artifactId>spark-sql-streaming-akka_2.11</artifactId>
+ <properties>
+ <sbt.project.name>sql-streaming-akka</sbt.project.name>
+ </properties>
+ <packaging>jar</packaging>
+ <name>Apache Bahir - Spark SQL Streaming Akka</name>
+ <url>http://bahir.apache.org</url>
+
+ <dependencies>
+ <dependency>
+ <groupId>org.apache.spark</groupId>
+ <artifactId>spark-tags_${scala.binary.version}</artifactId>
+ </dependency>
+ <dependency>
+ <groupId>org.apache.spark</groupId>
+ <artifactId>spark-sql_${scala.binary.version}</artifactId>
+ <version>${spark.version}</version>
+ </dependency>
+ <dependency>
+ <groupId>org.apache.spark</groupId>
+ <artifactId>spark-sql_${scala.binary.version}</artifactId>
+ <version>${spark.version}</version>
+ <type>test-jar</type>
+ <scope>test</scope>
+ </dependency>
+ <dependency>
+ <groupId>org.apache.spark</groupId>
+ <artifactId>spark-core_${scala.binary.version}</artifactId>
+ <version>${spark.version}</version>
+ <type>test-jar</type>
+ <scope>test</scope>
+ </dependency>
+ <dependency>
+ <groupId>${akka.group}</groupId>
+ <artifactId>akka-actor_${scala.binary.version}</artifactId>
+ <version>${akka.version}</version>
+ </dependency>
+ <dependency>
+ <groupId>${akka.group}</groupId>
+ <artifactId>akka-remote_${scala.binary.version}</artifactId>
+ <version>${akka.version}</version>
+ </dependency>
+ <dependency>
+ <groupId>${akka.group}</groupId>
+ <artifactId>akka-slf4j_${scala.binary.version}</artifactId>
+ <version>${akka.version}</version>
+ </dependency>
+ <dependency>
+ <groupId>org.rocksdb</groupId>
+ <artifactId>rocksdbjni</artifactId>
+ <version>5.1.2</version>
+ </dependency>
+ </dependencies>
+
+ <build>
+ <outputDirectory>target/scala-${scala.binary.version}/classes</outputDirectory>
+ <testOutputDirectory>target/scala-${scala.binary.version}/test-classes</testOutputDirectory>
+
+ <plugins>
+ <plugin>
+ <groupId>org.apache.maven.plugins</groupId>
+ <artifactId>maven-source-plugin</artifactId>
+ </plugin>
+
+ <!-- Assemble a jar with test dependencies for Python tests -->
+ <plugin>
+ <groupId>org.apache.maven.plugins</groupId>
+ <artifactId>maven-assembly-plugin</artifactId>
+ <executions>
+ <execution>
+ <id>test-jar-with-dependencies</id>
+ <phase>package</phase>
+ <goals>
+ <goal>single</goal>
+ </goals>
+ <configuration>
+ <!-- Make sure the file path is same as the sbt build -->
+ <finalName>spark-streaming-akka-test-${project.version}</finalName>
+ <outputDirectory>${project.build.directory}/scala-${scala.binary.version}</outputDirectory>
+ <appendAssemblyId>false</appendAssemblyId>
+ <!-- Don't publish it since it's only for Python tests -->
+ <attach>false</attach>
+ <descriptors>
+ <descriptor>src/main/assembly/assembly.xml</descriptor>
+ </descriptors>
+ </configuration>
+ </execution>
+ </executions>
+ </plugin>
+ </plugins>
+ </build>
+</project>
--- End diff --
add new line
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] bahir issue #38: [BAHIR-97] Akka as a streaming source for SQL Streaming.
Posted by sbcd90 <gi...@git.apache.org>.
Github user sbcd90 commented on the issue:
https://github.com/apache/bahir/pull/38
retest this please
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] bahir pull request #38: [BAHIR-97] Akka as a streaming source for SQL Stream...
Posted by ckadner <gi...@git.apache.org>.
Github user ckadner commented on a diff in the pull request:
https://github.com/apache/bahir/pull/38#discussion_r109263328
--- Diff: sql-streaming-akka/src/main/assembly/assembly.xml ---
@@ -0,0 +1,44 @@
+<!--
+ ~ Licensed to the Apache Software Foundation (ASF) under one or more
+ ~ contributor license agreements. See the NOTICE file distributed with
+ ~ this work for additional information regarding copyright ownership.
+ ~ The ASF licenses this file to You under the Apache License, Version 2.0
+ ~ (the "License"); you may not use this file except in compliance with
+ ~ the License. You may obtain a copy of the License at
+ ~
+ ~ http://www.apache.org/licenses/LICENSE-2.0
+ ~
+ ~ Unless required by applicable law or agreed to in writing, software
+ ~ distributed under the License is distributed on an "AS IS" BASIS,
+ ~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ ~ See the License for the specific language governing permissions and
+ ~ limitations under the License.
+ -->
+<assembly>
+ <id>test-jar-with-dependencies</id>
+ <formats>
+ <format>jar</format>
+ </formats>
+ <includeBaseDirectory>false</includeBaseDirectory>
+
+ <fileSets>
+ <fileSet>
+ <directory>${project.build.directory}/scala-${scala.binary.version}/test-classes</directory>
+ <outputDirectory></outputDirectory>
+ </fileSet>
+ </fileSets>
+
+ <dependencySets>
+ <dependencySet>
+ <useTransitiveDependencies>true</useTransitiveDependencies>
+ <scope>test</scope>
+ <unpack>true</unpack>
+ <excludes>
+ <exclude>org.apache.hadoop:*:jar</exclude>
+ <exclude>org.apache.zookeeper:*:jar</exclude>
+ <exclude>org.apache.avro:*:jar</exclude>
+ </excludes>
+ </dependencySet>
+ </dependencySets>
+
+</assembly>
--- End diff --
add new line
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] bahir issue #38: [BAHIR-97] Akka as a streaming source for SQL Streaming.
Posted by sbcd90 <gi...@git.apache.org>.
Github user sbcd90 commented on the issue:
https://github.com/apache/bahir/pull/38
Hi @ckadner , I have provided a README & examples as well..
Also, I have added a newline at the end of all the files..
plz have a look..
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] bahir issue #38: [BAHIR-97] Akka as a streaming source for SQL Streaming.
Posted by ApacheBahir <gi...@git.apache.org>.
Github user ApacheBahir commented on the issue:
https://github.com/apache/bahir/pull/38
Can one of the admins verify this patch?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] bahir issue #38: [BAHIR-97] Akka as a streaming source for SQL Streaming.
Posted by sbcd90 <gi...@git.apache.org>.
Github user sbcd90 commented on the issue:
https://github.com/apache/bahir/pull/38
Hi @lresende , I have removed the usage of the deprecated api s
& also opened a jira issue named [BAHIR-103](https://issues.apache.org/jira/browse/BAHIR-103)
for refactoring the utils classes into bahir-common.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] bahir issue #38: [BAHIR-97] Akka as a streaming source for SQL Streaming.
Posted by ApacheBahir <gi...@git.apache.org>.
Github user ApacheBahir commented on the issue:
https://github.com/apache/bahir/pull/38
Refer to this link for build results (access rights to CI server needed):
http://169.45.79.58:8080/job/bahir_spark_pr_builder/42/
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] bahir issue #38: [BAHIR-97] Akka as a streaming source for SQL Streaming.
Posted by ApacheBahir <gi...@git.apache.org>.
Github user ApacheBahir commented on the issue:
https://github.com/apache/bahir/pull/38
Build successful
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] bahir issue #38: [BAHIR-97] Akka as a streaming source for SQL Streaming.
Posted by ApacheBahir <gi...@git.apache.org>.
Github user ApacheBahir commented on the issue:
https://github.com/apache/bahir/pull/38
Build successful
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] bahir issue #38: [BAHIR-97] Akka as a streaming source for SQL Streaming.
Posted by ckadner <gi...@git.apache.org>.
Github user ckadner commented on the issue:
https://github.com/apache/bahir/pull/38
http://169.45.79.58:8080/job/Apache%20Bahir%20-%20Pull%20Request%20Builder/35/console
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] bahir issue #38: [BAHIR-97] Akka as a streaming source for SQL Streaming.
Posted by sbcd90 <gi...@git.apache.org>.
Github user sbcd90 commented on the issue:
https://github.com/apache/bahir/pull/38
retest this please
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] bahir pull request #38: [BAHIR-97] Akka as a streaming source for SQL Stream...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/bahir/pull/38
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] bahir issue #38: [BAHIR-97] Akka as a streaming source for SQL Streaming.
Posted by lresende <gi...@git.apache.org>.
Github user lresende commented on the issue:
https://github.com/apache/bahir/pull/38
Please test this Jenkins
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] bahir pull request #38: [BAHIR-97] Akka as a streaming source for SQL Stream...
Posted by lresende <gi...@git.apache.org>.
Github user lresende commented on a diff in the pull request:
https://github.com/apache/bahir/pull/38#discussion_r110021854
--- Diff: sql-streaming-akka/src/test/scala/org/apache/bahir/sql/streaming/akka/AkkaStreamSourceSuite.scala ---
@@ -0,0 +1,188 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.bahir.sql.streaming.akka
+
+import java.io.File
+
+import scala.collection.JavaConverters._
+import scala.collection.mutable
+
+import org.scalatest.BeforeAndAfter
+
+import org.apache.spark.{SharedSparkContext, SparkFunSuite}
+import org.apache.spark.sql.{DataFrame, SQLContext}
+import org.apache.spark.sql.execution.streaming.FileStreamSource.Timestamp
+import org.apache.spark.sql.execution.streaming.LongOffset
+
+import org.apache.bahir.utils.BahirUtils
+
+class AkkaStreamSourceSuite extends SparkFunSuite with SharedSparkContext with BeforeAndAfter {
+
+ protected var akkaTestUtils: AkkaTestUtils = _
+ protected val tempDir: File =
+ new File(System.getProperty("java.io.tmpdir") + "/spark-akka-persistence")
+
+ akkaTestUtils = new AkkaTestUtils
+ akkaTestUtils.setup()
+
+ before {
+ tempDir.mkdirs()
+ }
+
+ after {
+ Persistence.close()
+ BahirUtils.recursiveDeleteDir(tempDir)
+ }
+
+ protected val tmpDir: String = tempDir.getAbsolutePath
+
+ protected def createStreamingDataframe(dir: String = tmpDir): (SQLContext, DataFrame) = {
+
+ val sqlContext: SQLContext = new SQLContext(sc)
+
--- End diff --
Would you please change to use Sparksession instead of directly instantiating a SQLContext.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] bahir issue #38: [BAHIR-97] Akka as a streaming source for SQL Streaming.
Posted by ApacheBahir <gi...@git.apache.org>.
Github user ApacheBahir commented on the issue:
https://github.com/apache/bahir/pull/38
Build successful
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] bahir issue #38: [BAHIR-97] Akka as a streaming source for SQL Streaming.
Posted by lresende <gi...@git.apache.org>.
Github user lresende commented on the issue:
https://github.com/apache/bahir/pull/38
Once we remove the deprecated api usages, we should be ready to merge unless someone has other comments.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] bahir issue #38: [BAHIR-97] Akka as a streaming source for SQL Streaming.
Posted by ApacheBahir <gi...@git.apache.org>.
Github user ApacheBahir commented on the issue:
https://github.com/apache/bahir/pull/38
Build successful
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] bahir issue #38: [BAHIR-97] Akka as a streaming source for SQL Streaming.
Posted by lresende <gi...@git.apache.org>.
Github user lresende commented on the issue:
https://github.com/apache/bahir/pull/38
retest this please
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] bahir issue #38: [BAHIR-97] Akka as a streaming source for SQL Streaming.
Posted by ApacheBahir <gi...@git.apache.org>.
Github user ApacheBahir commented on the issue:
https://github.com/apache/bahir/pull/38
Build successful
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] bahir issue #38: [BAHIR-97] Akka as a streaming source for SQL Streaming.
Posted by sbcd90 <gi...@git.apache.org>.
Github user sbcd90 commented on the issue:
https://github.com/apache/bahir/pull/38
Hi @ckadner , the test cases are skipped..any reason for that?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] bahir issue #38: [BAHIR-97] Akka as a streaming source for SQL Streaming.
Posted by ApacheBahir <gi...@git.apache.org>.
Github user ApacheBahir commented on the issue:
https://github.com/apache/bahir/pull/38
Refer to this link for build results (access rights to CI server needed):
http://169.45.79.58:8080/job/bahir_spark_pr_builder/48/
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] bahir issue #38: [BAHIR-97] Akka as a streaming source for SQL Streaming.
Posted by sbcd90 <gi...@git.apache.org>.
Github user sbcd90 commented on the issue:
https://github.com/apache/bahir/pull/38
retest this please
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] bahir issue #38: [BAHIR-97] Akka as a streaming source for SQL Streaming.
Posted by ApacheBahir <gi...@git.apache.org>.
Github user ApacheBahir commented on the issue:
https://github.com/apache/bahir/pull/38
Build failed, see build log for details
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] bahir issue #38: [BAHIR-97] Akka as a streaming source for SQL Streaming.
Posted by ApacheBahir <gi...@git.apache.org>.
Github user ApacheBahir commented on the issue:
https://github.com/apache/bahir/pull/38
Refer to this link for build results (access rights to CI server needed):
http://169.45.79.58:8080/job/bahir_spark_pr_builder/50/
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] bahir pull request #38: [BAHIR-97] Akka as a streaming source for SQL Stream...
Posted by lresende <gi...@git.apache.org>.
Github user lresende commented on a diff in the pull request:
https://github.com/apache/bahir/pull/38#discussion_r110021902
--- Diff: sql-streaming-akka/src/test/scala/org/apache/bahir/sql/streaming/akka/AkkaStreamSourceSuite.scala ---
@@ -0,0 +1,188 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.bahir.sql.streaming.akka
+
+import java.io.File
+
+import scala.collection.JavaConverters._
+import scala.collection.mutable
+
+import org.scalatest.BeforeAndAfter
+
+import org.apache.spark.{SharedSparkContext, SparkFunSuite}
+import org.apache.spark.sql.{DataFrame, SQLContext}
+import org.apache.spark.sql.execution.streaming.FileStreamSource.Timestamp
+import org.apache.spark.sql.execution.streaming.LongOffset
+
+import org.apache.bahir.utils.BahirUtils
+
+class AkkaStreamSourceSuite extends SparkFunSuite with SharedSparkContext with BeforeAndAfter {
+
+ protected var akkaTestUtils: AkkaTestUtils = _
+ protected val tempDir: File =
+ new File(System.getProperty("java.io.tmpdir") + "/spark-akka-persistence")
+
+ akkaTestUtils = new AkkaTestUtils
+ akkaTestUtils.setup()
+
+ before {
+ tempDir.mkdirs()
+ }
+
+ after {
+ Persistence.close()
+ BahirUtils.recursiveDeleteDir(tempDir)
+ }
+
+ protected val tmpDir: String = tempDir.getAbsolutePath
+
+ protected def createStreamingDataframe(dir: String = tmpDir): (SQLContext, DataFrame) = {
+
+ val sqlContext: SQLContext = new SQLContext(sc)
+
+ sqlContext.setConf("spark.sql.streaming.checkpointLocation", dir + "/checkpoint")
+
+ val dataFrame: DataFrame =
+ sqlContext.readStream.format("org.apache.bahir.sql.streaming.akka.AkkaStreamSourceProvider")
+ .option("urlOfPublisher", akkaTestUtils.getFeederActorUri())
+ .option("persistenceDirPath", dir + "/persistence").load()
+ (sqlContext, dataFrame)
+ }
+}
+
+class BasicAkkaSourceSuite extends AkkaStreamSourceSuite {
+
+ private def writeStreamResults(sqlContext: SQLContext, dataFrame: DataFrame,
+ waitDuration: Long): Boolean = {
+ import sqlContext.implicits._
+ dataFrame.as[(String, Timestamp)].writeStream.format("parquet")
+ .start(s"$tmpDir/parquet/t.parquet").awaitTermination(waitDuration)
+ }
+
+ private def readBackSreamingResults(sqlContext: SQLContext): mutable.Buffer[String] = {
+ import sqlContext.implicits._
+ val asList =
+ sqlContext.read.schema(AkkaStreamConstants.SCHEMA_DEFAULT)
+ .parquet(s"$tmpDir/parquet/t.parquet").as[(String, Timestamp)].map(_._1)
+ .collectAsList().asScala
+ asList
+ }
+
+ test("basic usage") {
+ val message = "Akka is a reactive framework"
+
+ akkaTestUtils.setMessage(message)
+ akkaTestUtils.setCountOfMessages(1)
+
+ val (sqlContext: SQLContext, dataFrame: DataFrame) = createStreamingDataframe()
+
+ writeStreamResults(sqlContext, dataFrame, 10000)
+
+ val resultBuffer: mutable.Buffer[String] = readBackSreamingResults(sqlContext)
+
+ assert(resultBuffer.size === 1)
+ assert(resultBuffer.head === message)
+ }
+
+ test("Send and receive 100 messages.") {
+ val message = "Akka is a reactive framework"
+
+ akkaTestUtils.setMessage(message)
+ akkaTestUtils.setCountOfMessages(100)
+
+ val (sqlContext: SQLContext, dataFrame: DataFrame) = createStreamingDataframe()
+
+ writeStreamResults(sqlContext, dataFrame, 10000)
+
+ val resultBuffer: mutable.Buffer[String] = readBackSreamingResults(sqlContext)
+
+ assert(resultBuffer.size === 100)
+ assert(resultBuffer.head === message)
+ }
+
+ test("params not provided") {
+ val persistenceDirPath = tempDir.getAbsolutePath + "/persistence"
+
+ val provider = new AkkaStreamSourceProvider
+ val sqlContext: SQLContext = new SQLContext(sc)
+
--- End diff --
Would you please change to use Sparksession instead of directly instantiating a SQLContext.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] bahir issue #38: [BAHIR-97] Akka as a streaming source for SQL Streaming.
Posted by ckadner <gi...@git.apache.org>.
Github user ckadner commented on the issue:
https://github.com/apache/bahir/pull/38
retest this please
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] bahir issue #38: [BAHIR-97] Akka as a streaming source for SQL Streaming.
Posted by ckadner <gi...@git.apache.org>.
Github user ckadner commented on the issue:
https://github.com/apache/bahir/pull/38
@sbcd90 -- can you change your test suite to chose the `akka.remote.netty.tcp.port` dynamically?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] bahir issue #38: [BAHIR-97] Akka as a streaming source for SQL Streaming.
Posted by ApacheBahir <gi...@git.apache.org>.
Github user ApacheBahir commented on the issue:
https://github.com/apache/bahir/pull/38
Refer to this link for build results (access rights to CI server needed):
http://169.45.79.58:8080/job/Apache%20Bahir%20-%20Pull%20Request%20Builder/36/
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] bahir issue #38: [BAHIR-97] Akka as a streaming source for SQL Streaming.
Posted by ckadner <gi...@git.apache.org>.
Github user ckadner commented on the issue:
https://github.com/apache/bahir/pull/38
@sbcd90 -- thanks for you continuous updates and sorry for the piecemeal review from my end ... I started with your test cases since we were still in the process of fixing our Jenkins build setup. But your test cases are great now :-)
Perhaps more important, my first request to you should have been to add a README and examples so users can start using your connector without having to read through too much code ... i.e. please further follow the precedence set by `sql-streaming-mqtt`.
**Thank you!**
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] bahir issue #38: [BAHIR-97] Akka as a streaming source for SQL Streaming.
Posted by sbcd90 <gi...@git.apache.org>.
Github user sbcd90 commented on the issue:
https://github.com/apache/bahir/pull/38
Hi @ckadner , I haven't seen any updates from you..plz let me know if anything else is required from my side...
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] bahir issue #38: [BAHIR-97] Akka as a streaming source for SQL Streaming.
Posted by ApacheBahir <gi...@git.apache.org>.
Github user ApacheBahir commented on the issue:
https://github.com/apache/bahir/pull/38
Build failed, see build log for details
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] bahir issue #38: [BAHIR-97] Akka as a streaming source for SQL Streaming.
Posted by ckadner <gi...@git.apache.org>.
Github user ckadner commented on the issue:
https://github.com/apache/bahir/pull/38
retest this please
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] bahir issue #38: [BAHIR-97] Akka as a streaming source for SQL Streaming.
Posted by sbcd90 <gi...@git.apache.org>.
Github user sbcd90 commented on the issue:
https://github.com/apache/bahir/pull/38
Hi @ckadner , I have made the changes you mentioned to remove the remaining verbose. Please have a look.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] bahir issue #38: [BAHIR-97] Akka as a streaming source for SQL Streaming.
Posted by ckadner <gi...@git.apache.org>.
Github user ckadner commented on the issue:
https://github.com/apache/bahir/pull/38
Note, our Jenkins build server does not currently run Scalatests ...
> 17:20:55 No tests were executed.
```
17:20:54 [INFO]
17:20:54 [INFO] --- maven-surefire-plugin:2.19.1:test (default-test) @ spark-sql-streaming-akka_2.11 ---
17:20:54
17:20:54 -------------------------------------------------------
17:20:54 T E S T S
17:20:54 -------------------------------------------------------
17:20:54 OpenJDK 64-Bit Server VM warning: ignoring option MaxPermSize=512m; support was removed in 8.0
17:20:54
17:20:54 Results :
17:20:54
17:20:54 Tests run: 0, Failures: 0, Errors: 0, Skipped: 0
17:20:54
17:20:54 [INFO]
17:20:54 [INFO] --- maven-surefire-plugin:2.19.1:test (test) @ spark-sql-streaming-akka_2.11 ---
17:20:54 [INFO] Skipping execution of surefire because it has already been run for this configuration
17:20:54 [INFO]
17:20:54 [INFO] --- scalatest-maven-plugin:1.0:test (test) @ spark-sql-streaming-akka_2.11 ---
17:20:54 OpenJDK 64-Bit Server VM warning: ignoring option MaxPermSize=512m; support was removed in 8.0
17:20:55 Discovery starting.
17:20:55 Discovery completed in 36 milliseconds.
17:20:55 Run starting. Expected test count is: 0
17:20:55 DiscoverySuite:
17:20:55 Run completed in 100 milliseconds.
17:20:55 Total number of tests run: 0
17:20:55 Suites: completed 1, aborted 0
17:20:55 Tests: succeeded 0, failed 0, canceled 0, ignored 0, pending 0
17:20:55 No tests were executed.
17:20:55 [INFO]
```
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] bahir issue #38: [BAHIR-97] Akka as a streaming source for SQL Streaming.
Posted by ApacheBahir <gi...@git.apache.org>.
Github user ApacheBahir commented on the issue:
https://github.com/apache/bahir/pull/38
Refer to this link for build results (access rights to CI server needed):
http://169.45.79.58:8080/job/Apache%20Bahir%20-%20Pull%20Request%20Builder/35/
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] bahir issue #38: [BAHIR-97] Akka as a streaming source for SQL Streaming.
Posted by ktoso <gi...@git.apache.org>.
Github user ktoso commented on the issue:
https://github.com/apache/bahir/pull/38
Hi there, interesting patch :)
May I interest you with Akka Streams though? I think it would be the natural fit for the Akka side here. You could implement what you made Actors here as a GraphStage ( http://doc.akka.io/docs/akka/2.4.17/scala/stream/stream-customize.html ), and our blog series about how to write custom connectors ( http://blog.akka.io/integrations/2016/09/05/flow-control-at-the-akka-stream-boundary ).
Hope this helps! Would be super exciting to have a connection to Spark SQL Streaming directly exposed as Akka Source/Sink :-)
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] bahir pull request #38: [BAHIR-97] Akka as a streaming source for SQL Stream...
Posted by lresende <gi...@git.apache.org>.
Github user lresende commented on a diff in the pull request:
https://github.com/apache/bahir/pull/38#discussion_r110022955
--- Diff: sql-streaming-akka/src/main/scala/org/apache/bahir/utils/BahirUtils.scala ---
@@ -0,0 +1,47 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.bahir.utils
+
+import java.io.{File, IOException}
+import java.nio.file.{Files, FileVisitResult, Path, SimpleFileVisitor}
+import java.nio.file.attribute.BasicFileAttributes
+
+object BahirUtils extends Logging {
+
--- End diff --
We should not duplicate these two classes on each extension. Could you create a jira for refactoring these into a bahir-common project that is shared with the other extensions.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] bahir issue #38: [BAHIR-97] Akka as a streaming source for SQL Streaming.
Posted by ApacheBahir <gi...@git.apache.org>.
Github user ApacheBahir commented on the issue:
https://github.com/apache/bahir/pull/38
Build successful
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] bahir issue #38: [BAHIR-97] Akka as a streaming source for SQL Streaming.
Posted by sbcd90 <gi...@git.apache.org>.
Github user sbcd90 commented on the issue:
https://github.com/apache/bahir/pull/38
retest this please
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] bahir issue #38: [BAHIR-97] Akka as a streaming source for SQL Streaming.
Posted by ckadner <gi...@git.apache.org>.
Github user ckadner commented on the issue:
https://github.com/apache/bahir/pull/38
@sbcd90 -- Thanks, much better! To remove the remaining noise, I believe you would have to add a dependency to `akka-slf4j` in `sql-streaming-akka/pom.xml` and configure Akka to use the `akka.event.slf4j.Slf4jLogger` ...
```XML
<dependency>
<groupId>${akka.group}</groupId>
<artifactId>akka-slf4j_${scala.binary.version}</artifactId>
<version>${akka.version}</version>
</dependency>
```
```Scala
akka.loggers.0 = "akka.event.slf4j.Slf4jLogger"
akka.log-dead-letters-during-shutdown = "off"
```
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] bahir issue #38: [BAHIR-97] Akka as a streaming source for SQL Streaming.
Posted by ckadner <gi...@git.apache.org>.
Github user ckadner commented on the issue:
https://github.com/apache/bahir/pull/38
@sbcd90 -- your test cases ran fine, just with a lot of "noise".
http://169.45.79.58:8080/job/bahir_spark_pr_builder/38/consoleFull
```
00:27:23 [INFO] --- scalatest-maven-plugin:1.0:test (test) @ spark-sql-streaming-akka_2.11 ---
...
00:27:24 Discovery starting.
00:27:24 log4j:ERROR Could not read configuration file from URL [file:src/test/resources/log4j.properties].
00:27:24 java.io.FileNotFoundException: src/test/resources/log4j.properties (No such file or directory)
...
00:27:24 log4j:ERROR Ignoring configuration file [file:src/test/resources/log4j.properties].
...
00:27:24 Discovery completed in 698 milliseconds.
00:27:24 Run starting. Expected test count is: 5
00:27:24 StressTestAkkaSource:
00:27:24 Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties
...
00:28:23 Run completed in 59 seconds, 534 milliseconds.
00:28:23 Total number of tests run: 5
00:28:23 Suites: completed 4, aborted 0
00:28:23 Tests: succeeded 5, failed 0, canceled 0, ignored 0, pending 0
00:28:23 All tests passed.
```
Could you add a `log4.j.properties` file in the test source folder to reduce the log verbosity?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] bahir issue #38: [BAHIR-97] Akka as a streaming source for SQL Streaming.
Posted by sbcd90 <gi...@git.apache.org>.
Github user sbcd90 commented on the issue:
https://github.com/apache/bahir/pull/38
Hi @ckadner , I have made `akka.remote.netty.tcp.port` dynamic.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] bahir issue #38: [BAHIR-97] Akka as a streaming source for SQL Streaming.
Posted by ApacheBahir <gi...@git.apache.org>.
Github user ApacheBahir commented on the issue:
https://github.com/apache/bahir/pull/38
Refer to this link for build results (access rights to CI server needed):
http://169.45.79.58:8080/job/bahir_spark_pr_builder/53/
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] bahir issue #38: [BAHIR-97] Akka as a streaming source for SQL Streaming.
Posted by ckadner <gi...@git.apache.org>.
Github user ckadner commented on the issue:
https://github.com/apache/bahir/pull/38
ok to test
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] bahir issue #38: [BAHIR-97] Akka as a streaming source for SQL Streaming.
Posted by ApacheBahir <gi...@git.apache.org>.
Github user ApacheBahir commented on the issue:
https://github.com/apache/bahir/pull/38
Refer to this link for build results (access rights to CI server needed):
http://169.45.79.58:8080/job/Apache%20Bahir%20-%20Pull%20Request%20Builder/34/
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] bahir pull request #38: [BAHIR-97] Akka as a streaming source for SQL Stream...
Posted by ckadner <gi...@git.apache.org>.
Github user ckadner commented on a diff in the pull request:
https://github.com/apache/bahir/pull/38#discussion_r109263538
--- Diff: sql-streaming-akka/src/test/resources/feeder_actor.conf ---
@@ -0,0 +1,34 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+akka {
+ loglevel = "INFO"
+ actor {
+ provider = "akka.remote.RemoteActorRefProvider"
+ }
+ remote {
+ enabled-transports = ["akka.remote.netty.tcp"]
+ netty.tcp {
+ hostname = "127.0.0.1"
+ port = 0
+ }
+ log-sent-messages = on
+ log-received-messages = on
+ }
+ loggers.0 = "akka.event.slf4j.Slf4jLogger"
+ log-dead-letters-during-shutdown = "off"
+}
--- End diff --
add new line
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] bahir issue #38: [BAHIR-97] Akka as a streaming source for SQL Streaming.
Posted by sbcd90 <gi...@git.apache.org>.
Github user sbcd90 commented on the issue:
https://github.com/apache/bahir/pull/38
Hello @ckadner ,
Are any actions need to be taken from my side? Like provide Java tests(Junits) instead of scala?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] bahir issue #38: [BAHIR-97] Akka as a streaming source for SQL Streaming.
Posted by sbcd90 <gi...@git.apache.org>.
Github user sbcd90 commented on the issue:
https://github.com/apache/bahir/pull/38
retest this please
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] bahir issue #38: [BAHIR-97] Akka as a streaming source for SQL Streaming.
Posted by ApacheBahir <gi...@git.apache.org>.
Github user ApacheBahir commented on the issue:
https://github.com/apache/bahir/pull/38
Refer to this link for build results (access rights to CI server needed):
http://169.45.79.58:8080/job/bahir_spark_pr_builder/37/
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] bahir issue #38: [BAHIR-97] Akka as a streaming source for SQL Streaming.
Posted by ApacheBahir <gi...@git.apache.org>.
Github user ApacheBahir commented on the issue:
https://github.com/apache/bahir/pull/38
Build successful
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] bahir issue #38: [BAHIR-97] Akka as a streaming source for SQL Streaming.
Posted by ApacheBahir <gi...@git.apache.org>.
Github user ApacheBahir commented on the issue:
https://github.com/apache/bahir/pull/38
Refer to this link for build results (access rights to CI server needed):
http://169.45.79.58:8080/job/bahir_spark_pr_builder/38/
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] bahir issue #38: [BAHIR-97] Akka as a streaming source for SQL Streaming.
Posted by ApacheBahir <gi...@git.apache.org>.
Github user ApacheBahir commented on the issue:
https://github.com/apache/bahir/pull/38
Refer to this link for build results (access rights to CI server needed):
http://169.45.79.58:8080/job/bahir_spark_pr_builder/39/
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] bahir pull request #38: [BAHIR-97] Akka as a streaming source for SQL Stream...
Posted by lresende <gi...@git.apache.org>.
Github user lresende commented on a diff in the pull request:
https://github.com/apache/bahir/pull/38#discussion_r110023018
--- Diff: sql-streaming-akka/src/main/scala/org/apache/bahir/utils/Logging.scala ---
@@ -0,0 +1,24 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.bahir.utils
+
+import org.slf4j.LoggerFactory
+
+trait Logging {
+ final val log = LoggerFactory.getLogger(this.getClass.getName.stripSuffix("$"))
+}
--- End diff --
Similar comment as file above.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] bahir issue #38: [BAHIR-97] Akka as a streaming source for SQL Streaming.
Posted by ckadner <gi...@git.apache.org>.
Github user ckadner commented on the issue:
https://github.com/apache/bahir/pull/38
@sbcd90 -- Scalatests should be sufficient. We need to fix our Jenkins integration test setup. Not an action item for you :-)
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] bahir issue #38: [BAHIR-97] Akka as a streaming source for SQL Streaming.
Posted by ckadner <gi...@git.apache.org>.
Github user ckadner commented on the issue:
https://github.com/apache/bahir/pull/38
Thanks @lresende and @sbcd90 - LGTM
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] bahir issue #38: [BAHIR-97] Akka as a streaming source for SQL Streaming.
Posted by ApacheBahir <gi...@git.apache.org>.
Github user ApacheBahir commented on the issue:
https://github.com/apache/bahir/pull/38
Build successful
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] bahir issue #38: [BAHIR-97] Akka as a streaming source for SQL Streaming.
Posted by sbcd90 <gi...@git.apache.org>.
Github user sbcd90 commented on the issue:
https://github.com/apache/bahir/pull/38
Hi @ckadner , Thanks for your reply. I have added `log4j.properties` in the test sources folder.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---