You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by PangZhi <gi...@git.apache.org> on 2017/04/21 00:59:37 UTC

[GitHub] flink pull request #3748: [FLINK-6225] [Cassandra Connector] add CassandraTa...

GitHub user PangZhi opened a pull request:

    https://github.com/apache/flink/pull/3748

    [FLINK-6225] [Cassandra Connector] add CassandraTableSink

    This PR is to address https://issues.apache.org/jira/browse/FLINK-6225
    
    - add Row type support for CassandraSink
    - add StreamTableSink implementation for Cassandra


You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/PangZhi/flink FLINK-6225_add_cassandra_table_sink

Alternatively you can review and apply these changes as the patch at:

    https://github.com/apache/flink/pull/3748.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #3748
    
----
commit 8bd2816a5d5999575bd98b5d3096af5f36f6f173
Author: Jing Fan <ji...@uber.com>
Date:   2017-04-21T00:54:36Z

    [FLINK-6225] [Cassandra Connector] add CassandraTableSink

----


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request #3748: [FLINK-6225] [Cassandra Connector] add CassandraTa...

Posted by zentol <gi...@git.apache.org>.
Github user zentol commented on a diff in the pull request:

    https://github.com/apache/flink/pull/3748#discussion_r112616773
  
    --- Diff: flink-connectors/flink-connector-cassandra/src/main/java/org/apache/flink/streaming/connectors/cassandra/CassandraTableSink.java ---
    @@ -0,0 +1,101 @@
    +/*
    + * Licensed to the Apache Software Foundation (ASF) under one or more
    + * contributor license agreements.  See the NOTICE file distributed with
    + * this work for additional information regarding copyright ownership.
    + * The ASF licenses this file to You under the Apache License, Version 2.0
    + * (the "License"); you may not use this file except in compliance with
    + * the License.  You may obtain a copy of the License at
    + *
    + *    http://www.apache.org/licenses/LICENSE-2.0
    + *
    + * Unless required by applicable law or agreed to in writing, software
    + * distributed under the License is distributed on an "AS IS" BASIS,
    + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    + * See the License for the specific language governing permissions and
    + * limitations under the License.
    + */
    +
    +package org.apache.flink.streaming.connectors.cassandra;
    +
    +import com.datastax.driver.core.Cluster;
    +import org.apache.flink.api.common.typeinfo.TypeInformation;
    +import org.apache.flink.api.java.typeutils.RowTypeInfo;
    +import org.apache.flink.streaming.api.datastream.DataStream;
    +import org.apache.flink.table.sinks.StreamTableSink;
    +import org.apache.flink.table.sinks.TableSink;
    +import org.apache.flink.types.Row;
    +import org.apache.flink.util.Preconditions;
    +
    +import java.net.InetSocketAddress;
    +import java.util.Collection;
    +import java.util.List;
    +import java.util.Properties;
    +
    +/**
    + * A cassandra  {@link StreamTableSink}.
    + *
    + */
    +class CassandraTableSink implements StreamTableSink<Row> {
    +	private final List<InetSocketAddress> hostAddrs;
    +	private final String cql;
    +	private final String[] fieldNames;
    +	private final TypeInformation[] fieldTypes;
    +	private final Properties properties;
    +
    +	public CassandraTableSink(List<InetSocketAddress> hostAddrs, String cql, String[] fieldNames, TypeInformation[] fieldTypes, Properties properties) {
    --- End diff --
    
    why are the fieldNames/Types both in the constructor and in the configure() method? What the purpose of the Properties parameter?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request #3748: [FLINK-6225] [Cassandra Connector] add CassandraTa...

Posted by haohui <gi...@git.apache.org>.
Github user haohui commented on a diff in the pull request:

    https://github.com/apache/flink/pull/3748#discussion_r116092436
  
    --- Diff: flink-connectors/flink-connector-cassandra/src/test/java/org/apache/flink/streaming/connectors/cassandra/CassandraConnectorITCase.java ---
    @@ -438,6 +462,45 @@ public void cancel() {
     	}
     
     	@Test
    +	public void testCassandraTableSink() throws Exception {
    +		StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();
    +		env.setParallelism(1);
    +
    +		DataStreamSource<Row> source = env.fromCollection(rowCollection);
    +		CassandraTableSink cassandraTableSink = new CassandraTableSink(new ClusterBuilder() {
    +			@Override
    +			protected Cluster buildCluster(Cluster.Builder builder) {
    +				return builder.addContactPointsWithPorts(new InetSocketAddress(HOST, PORT)).build();
    +			}
    +		}, INSERT_DATA_QUERY, new Properties());
    +		CassandraTableSink newCassandrTableSink = cassandraTableSink.configure(FIELD_NAMES, FIELD_TYPES);
    +
    +		newCassandrTableSink.emitDataStream(source);
    +
    +		env.execute();
    +		ResultSet rs = session.execute(SELECT_DATA_QUERY);
    +		Assert.assertEquals(20, rs.all().size());
    +	}
    +
    +	@Test
    +	public void testCassandraTableSinkE2E() throws Exception {
    +		StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();
    +		env.setParallelism(4);
    +		StreamTableEnvironment tEnv = StreamTableEnvironment.getTableEnvironment(env);
    --- End diff --
    
    Agree. I think you're right. @PangZhi can you please make the suggested changes?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request #3748: [FLINK-6225] [Cassandra Connector] add CassandraTa...

Posted by haohui <gi...@git.apache.org>.
Github user haohui commented on a diff in the pull request:

    https://github.com/apache/flink/pull/3748#discussion_r115935549
  
    --- Diff: flink-connectors/flink-connector-cassandra/src/test/java/org/apache/flink/streaming/connectors/cassandra/CassandraConnectorITCase.java ---
    @@ -438,6 +462,45 @@ public void cancel() {
     	}
     
     	@Test
    +	public void testCassandraTableSink() throws Exception {
    +		StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();
    +		env.setParallelism(1);
    +
    +		DataStreamSource<Row> source = env.fromCollection(rowCollection);
    +		CassandraTableSink cassandraTableSink = new CassandraTableSink(new ClusterBuilder() {
    +			@Override
    +			protected Cluster buildCluster(Cluster.Builder builder) {
    +				return builder.addContactPointsWithPorts(new InetSocketAddress(HOST, PORT)).build();
    +			}
    +		}, INSERT_DATA_QUERY, new Properties());
    +		CassandraTableSink newCassandrTableSink = cassandraTableSink.configure(FIELD_NAMES, FIELD_TYPES);
    +
    +		newCassandrTableSink.emitDataStream(source);
    +
    +		env.execute();
    +		ResultSet rs = session.execute(SELECT_DATA_QUERY);
    +		Assert.assertEquals(20, rs.all().size());
    +	}
    +
    +	@Test
    +	public void testCassandraTableSinkE2E() throws Exception {
    +		StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();
    +		env.setParallelism(4);
    +		StreamTableEnvironment tEnv = StreamTableEnvironment.getTableEnvironment(env);
    --- End diff --
    
    @zentol here is the unit test that implicitly requires the scale dependency.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request #3748: [FLINK-6225] [Cassandra Connector] add CassandraTa...

Posted by zentol <gi...@git.apache.org>.
Github user zentol commented on a diff in the pull request:

    https://github.com/apache/flink/pull/3748#discussion_r116833487
  
    --- Diff: flink-connectors/flink-connector-cassandra/src/test/java/org/apache/flink/streaming/connectors/cassandra/CassandraConnectorITCase.java ---
    @@ -392,6 +425,45 @@ public void testCassandraPojoAtLeastOnceSink() throws Exception {
     	}
     
     	@Test
    +	public void testCassandraTableSink() throws Exception {
    +		StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();
    +		env.setParallelism(1);
    +
    +		DataStreamSource<Row> source = env.fromCollection(rowCollection);
    +		CassandraTableSink cassandraTableSink = new CassandraTableSink(new ClusterBuilder() {
    +			@Override
    +			protected Cluster buildCluster(Cluster.Builder builder) {
    +				return builder.addContactPointsWithPorts(new InetSocketAddress(HOST, PORT)).build();
    --- End diff --
    
    Why aren't you using static `builder` here?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request #3748: [FLINK-6225] [Cassandra Connector] add CassandraTa...

Posted by zentol <gi...@git.apache.org>.
Github user zentol commented on a diff in the pull request:

    https://github.com/apache/flink/pull/3748#discussion_r113541703
  
    --- Diff: flink-connectors/flink-connector-cassandra/src/test/java/org/apache/flink/streaming/connectors/cassandra/CassandraConnectorITCase.java ---
    @@ -108,10 +109,16 @@ protected Cluster buildCluster(Cluster.Builder builder) {
     	private static final String SELECT_DATA_QUERY = "SELECT * FROM flink.test;";
     
     	private static final ArrayList<Tuple3<String, Integer, Integer>> collection = new ArrayList<>(20);
    +	private static final ArrayList<org.apache.flink.types.Row> rowCollection = new ArrayList<>(20);
    --- End diff --
    
    I see, In that case i would suggest to import the Flink for type and use the qualified name for the Datastax row.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request #3748: [FLINK-6225] [Cassandra Connector] add CassandraTa...

Posted by fhueske <gi...@git.apache.org>.
Github user fhueske commented on a diff in the pull request:

    https://github.com/apache/flink/pull/3748#discussion_r113745320
  
    --- Diff: flink-connectors/flink-connector-cassandra/pom.xml ---
    @@ -176,5 +176,10 @@ under the License.
     				</exclusion>
     			</exclusions>
     		</dependency>
    +		<dependency>
    +			<groupId>org.apache.flink</groupId>
    +			<artifactId>flink-table_2.10</artifactId>
    +			<version>1.3-SNAPSHOT</version>
    --- End diff --
    
    The version should not be hard-coded. Use `<version>${project.version}</version>` instead.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request #3748: [FLINK-6225] [Cassandra Connector] add CassandraTa...

Posted by zentol <gi...@git.apache.org>.
Github user zentol commented on a diff in the pull request:

    https://github.com/apache/flink/pull/3748#discussion_r113408171
  
    --- Diff: flink-connectors/flink-connector-cassandra/pom.xml ---
    @@ -176,5 +176,10 @@ under the License.
     				</exclusion>
     			</exclusions>
     		</dependency>
    +		<dependency>
    +			<groupId>org.apache.flink</groupId>
    +			<artifactId>flink-table_2.10</artifactId>
    +			<version>1.3-SNAPSHOT</version>
    --- End diff --
    
    should be set to provided.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink issue #3748: [FLINK-6225] [Cassandra Connector] add CassandraTableSink

Posted by zentol <gi...@git.apache.org>.
Github user zentol commented on the issue:

    https://github.com/apache/flink/pull/3748
  
    A few minor things left, but we're in pretty good shape now.


---

[GitHub] flink issue #3748: [FLINK-6225] [Cassandra Connector] add CassandraTableSink

Posted by PangZhi <gi...@git.apache.org>.
Github user PangZhi commented on the issue:

    https://github.com/apache/flink/pull/3748
  
    @zentol thx, fixed.


---

[GitHub] flink pull request #3748: [FLINK-6225] [Cassandra Connector] add CassandraTa...

Posted by zentol <gi...@git.apache.org>.
Github user zentol commented on a diff in the pull request:

    https://github.com/apache/flink/pull/3748#discussion_r144854453
  
    --- Diff: flink-connectors/flink-connector-cassandra/src/test/java/org/apache/flink/streaming/connectors/cassandra/CassandraConnectorITCase.java ---
    @@ -465,10 +530,10 @@ public void testCassandraScalaTupleAtLeastSink() throws Exception {
     		sink.close();
     
     		ResultSet rs = session.execute(injectTableName(SELECT_DATA_QUERY));
    -		List<Row> rows = rs.all();
    +		List<com.datastax.driver.core.Row > rows = rs.all();
    --- End diff --
    
    the added space should be removed


---

[GitHub] flink pull request #3748: [FLINK-6225] [Cassandra Connector] add CassandraTa...

Posted by zentol <gi...@git.apache.org>.
Github user zentol commented on a diff in the pull request:

    https://github.com/apache/flink/pull/3748#discussion_r116832836
  
    --- Diff: flink-connectors/flink-connector-cassandra/src/test/java/org/apache/flink/streaming/connectors/cassandra/CassandraConnectorITCase.java ---
    @@ -392,6 +425,45 @@ public void testCassandraPojoAtLeastOnceSink() throws Exception {
     	}
     
     	@Test
    +	public void testCassandraTableSink() throws Exception {
    +		StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();
    +		env.setParallelism(1);
    +
    +		DataStreamSource<Row> source = env.fromCollection(rowCollection);
    +		CassandraTableSink cassandraTableSink = new CassandraTableSink(new ClusterBuilder() {
    +			@Override
    +			protected Cluster buildCluster(Cluster.Builder builder) {
    +				return builder.addContactPointsWithPorts(new InetSocketAddress(HOST, PORT)).build();
    +			}
    +		}, injectTableName(INSERT_DATA_QUERY), new Properties());
    +		CassandraTableSink newCassandrTableSink = cassandraTableSink.configure(FIELD_NAMES, FIELD_TYPES);
    +
    +		newCassandrTableSink.emitDataStream(source);
    +
    +		env.execute();
    +		ResultSet rs = session.execute(injectTableName(SELECT_DATA_QUERY));
    +		Assert.assertEquals(20, rs.all().size());
    +	}
    +
    +	@Test
    +	public void testCassandraTableSinkE2E() throws Exception {
    --- End diff --
    
    What's the difference between this test and `TestCassandraTableSink`?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink issue #3748: [FLINK-6225] [Cassandra Connector] add CassandraTableSink

Posted by PangZhi <gi...@git.apache.org>.
Github user PangZhi commented on the issue:

    https://github.com/apache/flink/pull/3748
  
    @zentol @fhueske Can you take another look.


---

[GitHub] flink pull request #3748: [FLINK-6225] [Cassandra Connector] add CassandraTa...

Posted by fhueske <gi...@git.apache.org>.
Github user fhueske commented on a diff in the pull request:

    https://github.com/apache/flink/pull/3748#discussion_r113747908
  
    --- Diff: flink-connectors/flink-connector-cassandra/src/main/java/org/apache/flink/streaming/connectors/cassandra/CassandraSink.java ---
    @@ -190,10 +192,13 @@ private CassandraSink(SingleOutputStreamOperator<IN> sink) {
     	 * @param <IN>  input type
     	 * @return CassandraSinkBuilder, to further configure the sink
     	 */
    -	public static <IN, T extends Tuple> CassandraSinkBuilder<IN> addSink(DataStream<IN> input) {
    +	public static <IN, T extends Tuple, R extends Row> CassandraSinkBuilder<IN> addSink(DataStream<IN> input) {
    --- End diff --
    
    do we need `T` and `R`? Couldn't these be removed if we cast to `DataStream<Tuple>` and `DataStream<Row>`?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request #3748: [FLINK-6225] [Cassandra Connector] add CassandraTa...

Posted by zentol <gi...@git.apache.org>.
Github user zentol commented on a diff in the pull request:

    https://github.com/apache/flink/pull/3748#discussion_r113781674
  
    --- Diff: flink-connectors/flink-connector-cassandra/src/main/java/org/apache/flink/streaming/connectors/cassandra/CassandraSink.java ---
    @@ -190,10 +192,13 @@ private CassandraSink(SingleOutputStreamOperator<IN> sink) {
     	 * @param <IN>  input type
     	 * @return CassandraSinkBuilder, to further configure the sink
     	 */
    -	public static <IN, T extends Tuple> CassandraSinkBuilder<IN> addSink(DataStream<IN> input) {
    +	public static <IN, T extends Tuple, R extends Row> CassandraSinkBuilder<IN> addSink(DataStream<IN> input) {
    --- End diff --
    
    Yes we can probably remove them.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request #3748: [FLINK-6225] [Cassandra Connector] add CassandraTa...

Posted by zentol <gi...@git.apache.org>.
Github user zentol commented on a diff in the pull request:

    https://github.com/apache/flink/pull/3748#discussion_r115944431
  
    --- Diff: flink-connectors/flink-connector-cassandra/src/test/java/org/apache/flink/streaming/connectors/cassandra/CassandraConnectorITCase.java ---
    @@ -438,6 +462,45 @@ public void cancel() {
     	}
     
     	@Test
    +	public void testCassandraTableSink() throws Exception {
    +		StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();
    +		env.setParallelism(1);
    +
    +		DataStreamSource<Row> source = env.fromCollection(rowCollection);
    +		CassandraTableSink cassandraTableSink = new CassandraTableSink(new ClusterBuilder() {
    +			@Override
    +			protected Cluster buildCluster(Cluster.Builder builder) {
    +				return builder.addContactPointsWithPorts(new InetSocketAddress(HOST, PORT)).build();
    +			}
    +		}, INSERT_DATA_QUERY, new Properties());
    +		CassandraTableSink newCassandrTableSink = cassandraTableSink.configure(FIELD_NAMES, FIELD_TYPES);
    +
    +		newCassandrTableSink.emitDataStream(source);
    +
    +		env.execute();
    +		ResultSet rs = session.execute(SELECT_DATA_QUERY);
    +		Assert.assertEquals(20, rs.all().size());
    +	}
    +
    +	@Test
    +	public void testCassandraTableSinkE2E() throws Exception {
    +		StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();
    +		env.setParallelism(4);
    +		StreamTableEnvironment tEnv = StreamTableEnvironment.getTableEnvironment(env);
    --- End diff --
    
    OK, this does in fact require the t dependency on `flink-streaming-scala`. The `flink-scala` dependency isn't needed though. 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request #3748: [FLINK-6225] [Cassandra Connector] add CassandraTa...

Posted by zentol <gi...@git.apache.org>.
Github user zentol commented on a diff in the pull request:

    https://github.com/apache/flink/pull/3748#discussion_r143965663
  
    --- Diff: flink-connectors/flink-connector-cassandra/pom.xml ---
    @@ -187,5 +187,17 @@ under the License.
     				</exclusion>
     			</exclusions>
     		</dependency>
    +		<dependency>
    +			<groupId>org.apache.flink</groupId>
    +			<artifactId>flink-table_2.10</artifactId>
    --- End diff --
    
    the scala version should not be hard-coded. see the flink-test-utils dependency on how to do it.


---

[GitHub] flink pull request #3748: [FLINK-6225] [Cassandra Connector] add CassandraTa...

Posted by zentol <gi...@git.apache.org>.
Github user zentol commented on a diff in the pull request:

    https://github.com/apache/flink/pull/3748#discussion_r113410371
  
    --- Diff: flink-connectors/flink-connector-cassandra/src/test/java/org/apache/flink/streaming/connectors/cassandra/CassandraConnectorITCase.java ---
    @@ -108,10 +109,16 @@ protected Cluster buildCluster(Cluster.Builder builder) {
     	private static final String SELECT_DATA_QUERY = "SELECT * FROM flink.test;";
     
     	private static final ArrayList<Tuple3<String, Integer, Integer>> collection = new ArrayList<>(20);
    +	private static final ArrayList<org.apache.flink.types.Row> rowCollection = new ArrayList<>(20);
    +
    +	private static final String[] FIELD_NAMES = new String[] {"id", "counter", "batch_id"};
    +	private static final TypeInformation[] FIELD_TYPES = new TypeInformation[] {BasicTypeInfo.STRING_TYPE_INFO,
    --- End diff --
    
    Move `STRING_TYPE_INFO` to the next line.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request #3748: [FLINK-6225] [Cassandra Connector] add CassandraTa...

Posted by zentol <gi...@git.apache.org>.
Github user zentol commented on a diff in the pull request:

    https://github.com/apache/flink/pull/3748#discussion_r112616241
  
    --- Diff: flink-connectors/flink-connector-cassandra/src/main/java/org/apache/flink/streaming/connectors/cassandra/CassandraTableSink.java ---
    @@ -0,0 +1,101 @@
    +/*
    + * Licensed to the Apache Software Foundation (ASF) under one or more
    + * contributor license agreements.  See the NOTICE file distributed with
    + * this work for additional information regarding copyright ownership.
    + * The ASF licenses this file to You under the Apache License, Version 2.0
    + * (the "License"); you may not use this file except in compliance with
    + * the License.  You may obtain a copy of the License at
    + *
    + *    http://www.apache.org/licenses/LICENSE-2.0
    + *
    + * Unless required by applicable law or agreed to in writing, software
    + * distributed under the License is distributed on an "AS IS" BASIS,
    + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    + * See the License for the specific language governing permissions and
    + * limitations under the License.
    + */
    +
    +package org.apache.flink.streaming.connectors.cassandra;
    +
    +import com.datastax.driver.core.Cluster;
    +import org.apache.flink.api.common.typeinfo.TypeInformation;
    +import org.apache.flink.api.java.typeutils.RowTypeInfo;
    +import org.apache.flink.streaming.api.datastream.DataStream;
    +import org.apache.flink.table.sinks.StreamTableSink;
    +import org.apache.flink.table.sinks.TableSink;
    +import org.apache.flink.types.Row;
    +import org.apache.flink.util.Preconditions;
    +
    +import java.net.InetSocketAddress;
    +import java.util.Collection;
    +import java.util.List;
    +import java.util.Properties;
    +
    +/**
    + * A cassandra  {@link StreamTableSink}.
    + *
    + */
    +class CassandraTableSink implements StreamTableSink<Row> {
    +	private final List<InetSocketAddress> hostAddrs;
    +	private final String cql;
    +	private final String[] fieldNames;
    +	private final TypeInformation[] fieldTypes;
    +	private final Properties properties;
    +
    +	public CassandraTableSink(List<InetSocketAddress> hostAddrs, String cql, String[] fieldNames, TypeInformation[] fieldTypes, Properties properties) {
    +		this.hostAddrs = Preconditions.checkNotNull(hostAddrs, "hostAddrs");
    +		this.cql = Preconditions.checkNotNull(cql, "cql");
    +		this.fieldNames = Preconditions.checkNotNull(fieldNames, "fieldNames");
    +		this.fieldTypes = Preconditions.checkNotNull(fieldTypes, "fieldTypes");
    +		this.properties = Preconditions.checkNotNull(properties, "properties");
    +	}
    +
    +	@Override
    +	public TypeInformation<Row> getOutputType() {
    +		return new RowTypeInfo(fieldTypes);
    +	}
    +
    +	@Override
    +	public String[] getFieldNames() {
    +		return this.fieldNames;
    +	}
    +
    +	@Override
    +	public TypeInformation<?>[] getFieldTypes() {
    +		return this.fieldTypes;
    +	}
    +
    +	@Override
    +	public TableSink<Row> configure(String[] fieldNames, TypeInformation<?>[] fieldTypes) {
    +		fieldNames = Preconditions.checkNotNull(fieldNames, "fieldNames");
    +		fieldTypes = Preconditions.checkNotNull(fieldTypes, "fieldTypes");
    +		Preconditions.checkArgument(fieldNames.length == fieldTypes.length,
    +			"Number of provided field names and types does not match.");
    +		return new CassandraTableSink(this.hostAddrs, this.cql, fieldNames, fieldTypes, this.properties);
    +	}
    +
    +	@Override
    +	public void emitDataStream(DataStream<Row> dataStream) {
    +		try {
    +			CassandraSink.addSink(dataStream)
    +				.setClusterBuilder(new CassandraClusterBuilder(this.hostAddrs))
    --- End diff --
    
    It must be possible for the user to set his own cluster builder for the plethora of options it supports.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request #3748: [FLINK-6225] [Cassandra Connector] add CassandraTa...

Posted by zentol <gi...@git.apache.org>.
Github user zentol commented on a diff in the pull request:

    https://github.com/apache/flink/pull/3748#discussion_r144854466
  
    --- Diff: flink-connectors/flink-connector-cassandra/src/test/java/org/apache/flink/streaming/connectors/cassandra/CassandraConnectorITCase.java ---
    @@ -465,10 +530,10 @@ public void testCassandraScalaTupleAtLeastSink() throws Exception {
     		sink.close();
     
     		ResultSet rs = session.execute(injectTableName(SELECT_DATA_QUERY));
    -		List<Row> rows = rs.all();
    +		List<com.datastax.driver.core.Row > rows = rs.all();
     		Assert.assertEquals(scalaTupleCollection.size(), rows.size());
     
    -		for (Row row : rows) {
    +		for (com.datastax.driver.core.Row  row : rows) {
    --- End diff --
    
    the added space should be removed


---

[GitHub] flink issue #3748: [FLINK-6225] [Cassandra Connector] add CassandraTableSink

Posted by fhueske <gi...@git.apache.org>.
Github user fhueske commented on the issue:

    https://github.com/apache/flink/pull/3748
  
    Thanks for the update and reminder @PangZhi.
    I'll have a look tomorrow. Maybe we can include this in the upcoming 1.4.0 release.
    
    Thanks, Fabian


---

[GitHub] flink pull request #3748: [FLINK-6225] [Cassandra Connector] add CassandraTa...

Posted by zentol <gi...@git.apache.org>.
Github user zentol commented on a diff in the pull request:

    https://github.com/apache/flink/pull/3748#discussion_r116833798
  
    --- Diff: flink-connectors/flink-connector-cassandra/src/test/java/org/apache/flink/streaming/connectors/cassandra/CassandraConnectorITCase.java ---
    @@ -39,8 +40,12 @@
     import org.apache.flink.configuration.Configuration;
     import org.apache.flink.core.io.InputSplit;
     import org.apache.flink.runtime.testutils.CommonTestUtils;
    +import org.apache.flink.streaming.api.datastream.DataStreamSource;
    +import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
     import org.apache.flink.streaming.runtime.operators.WriteAheadSinkTestBase;
    +import org.apache.flink.table.api.java.StreamTableEnvironment;
     
    +import org.apache.flink.types.Row;
    --- End diff --
    
    This line should be moved up by 1 line.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request #3748: [FLINK-6225] [Cassandra Connector] add CassandraTa...

Posted by zentol <gi...@git.apache.org>.
Github user zentol commented on a diff in the pull request:

    https://github.com/apache/flink/pull/3748#discussion_r113410447
  
    --- Diff: flink-connectors/flink-connector-cassandra/src/test/java/org/apache/flink/streaming/connectors/cassandra/CassandraConnectorITCase.java ---
    @@ -108,10 +109,16 @@ protected Cluster buildCluster(Cluster.Builder builder) {
     	private static final String SELECT_DATA_QUERY = "SELECT * FROM flink.test;";
     
     	private static final ArrayList<Tuple3<String, Integer, Integer>> collection = new ArrayList<>(20);
    +	private static final ArrayList<org.apache.flink.types.Row> rowCollection = new ArrayList<>(20);
    +
    +	private static final String[] FIELD_NAMES = new String[] {"id", "counter", "batch_id"};
    --- End diff --
    
    remove the space after `[]`.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink issue #3748: [FLINK-6225] [Cassandra Connector] add CassandraTableSink

Posted by fhueske <gi...@git.apache.org>.
Github user fhueske commented on the issue:

    https://github.com/apache/flink/pull/3748
  
    Thanks for the update @PangZhi. 
    Btw. it is recommended to post a short comment when you update a PR. Pushing an update does not trigger a notification.
    
    The PR looks good. Will merge this.


---

[GitHub] flink pull request #3748: [FLINK-6225] [Cassandra Connector] add CassandraTa...

Posted by zentol <gi...@git.apache.org>.
Github user zentol commented on a diff in the pull request:

    https://github.com/apache/flink/pull/3748#discussion_r112615718
  
    --- Diff: flink-connectors/flink-connector-cassandra/src/test/java/org/apache/flink/streaming/connectors/cassandra/CassandraConnectorITCase.java ---
    @@ -402,6 +410,20 @@ public void testCassandraTupleAtLeastOnceSink() throws Exception {
     	}
     
     	@Test
    +	public void testCassandraRowAtLeastOnceSink() throws Exception {
    +		StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();
    +		env.setParallelism(1);
    +
    +		DataStreamSource<org.apache.flink.types.Row> source = env.fromCollection(rowCollection);
    +		source.addSink(new CassandraRowSink(INSERT_DATA_QUERY, builder));
    --- End diff --
    
    This should use the CassandraRowSinkBuilder to test the row type detection. I know other tests don't do that either, but the should actually.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request #3748: [FLINK-6225] [Cassandra Connector] add CassandraTa...

Posted by PangZhi <gi...@git.apache.org>.
Github user PangZhi commented on a diff in the pull request:

    https://github.com/apache/flink/pull/3748#discussion_r143098834
  
    --- Diff: flink-connectors/flink-connector-cassandra/src/test/java/org/apache/flink/streaming/connectors/cassandra/CassandraConnectorITCase.java ---
    @@ -392,6 +425,45 @@ public void testCassandraPojoAtLeastOnceSink() throws Exception {
     	}
     
     	@Test
    +	public void testCassandraTableSink() throws Exception {
    +		StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();
    +		env.setParallelism(1);
    +
    +		DataStreamSource<Row> source = env.fromCollection(rowCollection);
    +		CassandraTableSink cassandraTableSink = new CassandraTableSink(new ClusterBuilder() {
    +			@Override
    +			protected Cluster buildCluster(Cluster.Builder builder) {
    +				return builder.addContactPointsWithPorts(new InetSocketAddress(HOST, PORT)).build();
    +			}
    +		}, injectTableName(INSERT_DATA_QUERY), new Properties());
    +		CassandraTableSink newCassandrTableSink = cassandraTableSink.configure(FIELD_NAMES, FIELD_TYPES);
    +
    +		newCassandrTableSink.emitDataStream(source);
    +
    +		env.execute();
    +		ResultSet rs = session.execute(injectTableName(SELECT_DATA_QUERY));
    +		Assert.assertEquals(20, rs.all().size());
    +	}
    +
    +	@Test
    +	public void testCassandraTableSinkE2E() throws Exception {
    --- End diff --
    
    this is added as another reviewer think better to add e2e test using sql api.


---

[GitHub] flink issue #3748: [FLINK-6225] [Cassandra Connector] add CassandraTableSink

Posted by zentol <gi...@git.apache.org>.
Github user zentol commented on the issue:

    https://github.com/apache/flink/pull/3748
  
    Apart from the above this looks ok to me.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink issue #3748: [FLINK-6225] [Cassandra Connector] add CassandraTableSink

Posted by zentol <gi...@git.apache.org>.
Github user zentol commented on the issue:

    https://github.com/apache/flink/pull/3748
  
    When importing guava you have to explicitly add an exclusion to https://github.com/apache/flink/blob/master/tools/maven/suppressions.xml.


---

[GitHub] flink issue #3748: [FLINK-6225] [Cassandra Connector] add CassandraTableSink

Posted by PangZhi <gi...@git.apache.org>.
Github user PangZhi commented on the issue:

    https://github.com/apache/flink/pull/3748
  
    @zentol Do you mind taking another look.


---

[GitHub] flink pull request #3748: [FLINK-6225] [Cassandra Connector] add CassandraTa...

Posted by zentol <gi...@git.apache.org>.
Github user zentol commented on a diff in the pull request:

    https://github.com/apache/flink/pull/3748#discussion_r143966527
  
    --- Diff: flink-connectors/flink-connector-cassandra/src/main/java/org/apache/flink/streaming/connectors/cassandra/CassandraRowSink.java ---
    @@ -0,0 +1,59 @@
    +/*
    + * Licensed to the Apache Software Foundation (ASF) under one or more
    + * contributor license agreements.  See the NOTICE file distributed with
    + * this work for additional information regarding copyright ownership.
    + * The ASF licenses this file to You under the Apache License, Version 2.0
    + * (the "License"); you may not use this file except in compliance with
    + * the License.  You may obtain a copy of the License at
    + *
    + *    http://www.apache.org/licenses/LICENSE-2.0
    + *
    + * Unless required by applicable law or agreed to in writing, software
    + * distributed under the License is distributed on an "AS IS" BASIS,
    + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    + * See the License for the specific language governing permissions and
    + * limitations under the License.
    + */
    +
    +package org.apache.flink.streaming.connectors.cassandra;
    +
    +import org.apache.flink.configuration.Configuration;
    +import org.apache.flink.types.Row;
    +
    +import com.datastax.driver.core.PreparedStatement;
    +import com.datastax.driver.core.ResultSet;
    +import com.google.common.util.concurrent.ListenableFuture;
    +
    +/**
    + * Flink Sink to save data into a Cassandra cluster.
    + *
    + */
    +public class CassandraRowSink extends CassandraSinkBase<Row, ResultSet> {
    --- End diff --
    
    you could extend AbstractCassandraTupleSink which would cover open(), send(), the constructor and fields.


---

[GitHub] flink pull request #3748: [FLINK-6225] [Cassandra Connector] add CassandraTa...

Posted by zentol <gi...@git.apache.org>.
Github user zentol commented on a diff in the pull request:

    https://github.com/apache/flink/pull/3748#discussion_r113410291
  
    --- Diff: flink-connectors/flink-connector-cassandra/src/test/java/org/apache/flink/streaming/connectors/cassandra/CassandraConnectorITCase.java ---
    @@ -108,10 +109,16 @@ protected Cluster buildCluster(Cluster.Builder builder) {
     	private static final String SELECT_DATA_QUERY = "SELECT * FROM flink.test;";
     
     	private static final ArrayList<Tuple3<String, Integer, Integer>> collection = new ArrayList<>(20);
    +	private static final ArrayList<org.apache.flink.types.Row> rowCollection = new ArrayList<>(20);
    +
    +	private static final String[] FIELD_NAMES = new String[] {"id", "counter", "batch_id"};
    +	private static final TypeInformation[] FIELD_TYPES = new TypeInformation[] {BasicTypeInfo.STRING_TYPE_INFO,
    +	BasicTypeInfo.INT_TYPE_INFO, BasicTypeInfo.INT_TYPE_INFO};
     
     	static {
     		for (int i = 0; i < 20; i++) {
     			collection.add(new Tuple3<>(UUID.randomUUID().toString(), i, 0));
    +			rowCollection.add(org.apache.flink.types.Row.of(UUID.randomUUID().toString(), i, 0));
    --- End diff --
    
    same as above regarding the import


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request #3748: [FLINK-6225] [Cassandra Connector] add CassandraTa...

Posted by zentol <gi...@git.apache.org>.
Github user zentol commented on a diff in the pull request:

    https://github.com/apache/flink/pull/3748#discussion_r113409179
  
    --- Diff: flink-connectors/flink-connector-cassandra/src/main/java/org/apache/flink/streaming/connectors/cassandra/CassandraTableSink.java ---
    @@ -0,0 +1,83 @@
    +/*
    + * Licensed to the Apache Software Foundation (ASF) under one or more
    + * contributor license agreements.  See the NOTICE file distributed with
    + * this work for additional information regarding copyright ownership.
    + * The ASF licenses this file to You under the Apache License, Version 2.0
    + * (the "License"); you may not use this file except in compliance with
    + * the License.  You may obtain a copy of the License at
    + *
    + *    http://www.apache.org/licenses/LICENSE-2.0
    + *
    + * Unless required by applicable law or agreed to in writing, software
    + * distributed under the License is distributed on an "AS IS" BASIS,
    + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    + * See the License for the specific language governing permissions and
    + * limitations under the License.
    + */
    +
    +package org.apache.flink.streaming.connectors.cassandra;
    +
    +import org.apache.flink.api.common.typeinfo.TypeInformation;
    +import org.apache.flink.api.java.typeutils.RowTypeInfo;
    +import org.apache.flink.streaming.api.datastream.DataStream;
    +import org.apache.flink.table.sinks.StreamTableSink;
    +import org.apache.flink.table.sinks.TableSink;
    +import org.apache.flink.types.Row;
    +import org.apache.flink.util.Preconditions;
    +
    +import java.util.Properties;
    +
    +/**
    + * A cassandra  {@link StreamTableSink}.
    + *
    + */
    +class CassandraTableSink implements StreamTableSink<Row> {
    +	private final ClusterBuilder builder;
    +	private final String cql;
    +	private String[] fieldNames;
    +	private TypeInformation[] fieldTypes;
    +	private final Properties properties;
    +
    +	public CassandraTableSink(ClusterBuilder builder, String cql, Properties properties) {
    +		this.builder = Preconditions.checkNotNull(builder, "builder");
    --- End diff --
    
    Please include a slightly longer exception message, i.e "ClusterBuilder must not be null." The same applies to other calls to `Preconditions#checkNotNull()`.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request #3748: [FLINK-6225] [Cassandra Connector] add CassandraTa...

Posted by zentol <gi...@git.apache.org>.
Github user zentol commented on a diff in the pull request:

    https://github.com/apache/flink/pull/3748#discussion_r143965738
  
    --- Diff: flink-connectors/flink-connector-cassandra/pom.xml ---
    @@ -187,5 +187,17 @@ under the License.
     				</exclusion>
     			</exclusions>
     		</dependency>
    +		<dependency>
    +			<groupId>org.apache.flink</groupId>
    +			<artifactId>flink-table_2.10</artifactId>
    +			<version>${project.version}</version>
    +			<scope>provided</scope>
    +		</dependency>
    +		<dependency>
    +			<groupId>org.apache.flink</groupId>
    +			<artifactId>flink-streaming-scala_2.10</artifactId>
    --- End diff --
    
    scala version


---

[GitHub] flink issue #3748: [FLINK-6225] [Cassandra Connector] add CassandraTableSink

Posted by zentol <gi...@git.apache.org>.
Github user zentol commented on the issue:

    https://github.com/apache/flink/pull/3748
  
    ```
    [ERROR] src/main/java/org/apache/flink/streaming/connectors/cassandra/CassandraRowSink.java:[20,8] (imports) UnusedImports: Unused import: org.apache.flink.configuration.Configuration.
    [ERROR] src/main/java/org/apache/flink/streaming/connectors/cassandra/CassandraRowSink.java:[23,8] (imports) UnusedImports: Unused import: com.datastax.driver.core.PreparedStatement.
    [ERROR] src/main/java/org/apache/flink/streaming/connectors/cassandra/CassandraRowSink.java:[24,8] (imports) UnusedImports: Unused import: com.datastax.driver.core.ResultSet.
    [ERROR] src/main/java/org/apache/flink/streaming/connectors/cassandra/CassandraRowSink.java:[25,8] (imports) UnusedImports: Unused import: com.google.common.util.concurrent.ListenableFuture.
    ```


---

[GitHub] flink issue #3748: [FLINK-6225] [Cassandra Connector] add CassandraTableSink

Posted by zentol <gi...@git.apache.org>.
Github user zentol commented on the issue:

    https://github.com/apache/flink/pull/3748
  
    You will have to rebase this PR and adjust the tests. I recently modified the cassandra tests to no longer run any actual jobs.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request #3748: [FLINK-6225] [Cassandra Connector] add CassandraTa...

Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:

    https://github.com/apache/flink/pull/3748


---

[GitHub] flink pull request #3748: [FLINK-6225] [Cassandra Connector] add CassandraTa...

Posted by zentol <gi...@git.apache.org>.
Github user zentol commented on a diff in the pull request:

    https://github.com/apache/flink/pull/3748#discussion_r113409930
  
    --- Diff: flink-connectors/flink-connector-cassandra/src/main/java/org/apache/flink/streaming/connectors/cassandra/CassandraTableSink.java ---
    @@ -0,0 +1,83 @@
    +/*
    + * Licensed to the Apache Software Foundation (ASF) under one or more
    + * contributor license agreements.  See the NOTICE file distributed with
    + * this work for additional information regarding copyright ownership.
    + * The ASF licenses this file to You under the Apache License, Version 2.0
    + * (the "License"); you may not use this file except in compliance with
    + * the License.  You may obtain a copy of the License at
    + *
    + *    http://www.apache.org/licenses/LICENSE-2.0
    + *
    + * Unless required by applicable law or agreed to in writing, software
    + * distributed under the License is distributed on an "AS IS" BASIS,
    + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    + * See the License for the specific language governing permissions and
    + * limitations under the License.
    + */
    +
    +package org.apache.flink.streaming.connectors.cassandra;
    +
    +import org.apache.flink.api.common.typeinfo.TypeInformation;
    +import org.apache.flink.api.java.typeutils.RowTypeInfo;
    +import org.apache.flink.streaming.api.datastream.DataStream;
    +import org.apache.flink.table.sinks.StreamTableSink;
    +import org.apache.flink.table.sinks.TableSink;
    +import org.apache.flink.types.Row;
    +import org.apache.flink.util.Preconditions;
    +
    +import java.util.Properties;
    +
    +/**
    + * A cassandra  {@link StreamTableSink}.
    + *
    --- End diff --
    
    remove this line.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request #3748: [FLINK-6225] [Cassandra Connector] add CassandraTa...

Posted by zentol <gi...@git.apache.org>.
Github user zentol commented on a diff in the pull request:

    https://github.com/apache/flink/pull/3748#discussion_r113410687
  
    --- Diff: flink-connectors/flink-connector-cassandra/src/test/java/org/apache/flink/streaming/connectors/cassandra/CassandraConnectorITCase.java ---
    @@ -438,6 +461,27 @@ public void cancel() {
     	}
     
     	@Test
    +	public void testCassandraTableSink() throws Exception {
    +		StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();
    +		env.setParallelism(1);
    +
    +		DataStreamSource<org.apache.flink.types.Row> source = env.fromCollection(rowCollection);
    +		CassandraTableSink cassandraTableSink = new CassandraTableSink(new ClusterBuilder() {
    +			@Override
    +			protected Cluster buildCluster(Cluster.Builder builder) {
    +				return builder.addContactPointsWithPorts(new InetSocketAddress(HOST, PORT)).build();
    +			}
    +		}, INSERT_DATA_QUERY, new Properties());
    +		cassandraTableSink.configure(FIELD_NAMES, FIELD_TYPES);
    --- End diff --
    
    Shouldn't we assign the returned value to `cassandraTableSink`?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink issue #3748: [FLINK-6225] [Cassandra Connector] add CassandraTableSink

Posted by PangZhi <gi...@git.apache.org>.
Github user PangZhi commented on the issue:

    https://github.com/apache/flink/pull/3748
  
    @zentol and do you know why import com.google.common.util.concurrent.ListenableFuture becomes illegal import? The checkstyle error is blocking CI. However this class is brought in from base class.


---

[GitHub] flink pull request #3748: [FLINK-6225] [Cassandra Connector] add CassandraTa...

Posted by fhueske <gi...@git.apache.org>.
Github user fhueske commented on a diff in the pull request:

    https://github.com/apache/flink/pull/3748#discussion_r147442488
  
    --- Diff: flink-connectors/flink-connector-cassandra/src/main/java/org/apache/flink/streaming/connectors/cassandra/CassandraSink.java ---
    @@ -205,12 +207,16 @@ private CassandraSink(SingleOutputStreamOperator<IN> sink) {
     	 * @param <IN>  input type
     	 * @return CassandraSinkBuilder, to further configure the sink
     	 */
    -	public static <IN, T extends Tuple> CassandraSinkBuilder<IN> addSink(DataStream<IN> input) {
    +	public static <IN> CassandraSinkBuilder<IN> addSink(DataStream<IN> input) {
     		TypeInformation<IN> typeInfo = input.getType();
     		if (typeInfo instanceof TupleTypeInfo) {
    -			DataStream<T> tupleInput = (DataStream<T>) input;
    +			DataStream<Tuple> tupleInput = (DataStream<Tuple>) input;
     			return (CassandraSinkBuilder<IN>) new CassandraTupleSinkBuilder<>(tupleInput, tupleInput.getType(), tupleInput.getType().createSerializer(tupleInput.getExecutionEnvironment().getConfig()));
     		}
    +		if (typeInfo instanceof RowTypeInfo) {
    --- End diff --
    
    Can check here for a concrete class with `Row.class.equals(typeInfo.getTypeClass())` to also support `GenericType<Row>` as well.


---

[GitHub] flink pull request #3748: [FLINK-6225] [Cassandra Connector] add CassandraTa...

Posted by PangZhi <gi...@git.apache.org>.
Github user PangZhi commented on a diff in the pull request:

    https://github.com/apache/flink/pull/3748#discussion_r113505564
  
    --- Diff: flink-connectors/flink-connector-cassandra/src/test/java/org/apache/flink/streaming/connectors/cassandra/CassandraConnectorITCase.java ---
    @@ -108,10 +109,16 @@ protected Cluster buildCluster(Cluster.Builder builder) {
     	private static final String SELECT_DATA_QUERY = "SELECT * FROM flink.test;";
     
     	private static final ArrayList<Tuple3<String, Integer, Integer>> collection = new ArrayList<>(20);
    +	private static final ArrayList<org.apache.flink.types.Row> rowCollection = new ArrayList<>(20);
    --- End diff --
    
    @zentol There is another Row class used here. So need to use qualified name.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink issue #3748: [FLINK-6225] [Cassandra Connector] add CassandraTableSink

Posted by haohui <gi...@git.apache.org>.
Github user haohui commented on the issue:

    https://github.com/apache/flink/pull/3748
  
    @zentol can you please take another look? Thanks.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink issue #3748: [FLINK-6225] [Cassandra Connector] add CassandraTableSink

Posted by PangZhi <gi...@git.apache.org>.
Github user PangZhi commented on the issue:

    https://github.com/apache/flink/pull/3748
  
    @zentol Hey zentol, when do you think we will be able to merge in.


---

[GitHub] flink pull request #3748: [FLINK-6225] [Cassandra Connector] add CassandraTa...

Posted by fhueske <gi...@git.apache.org>.
Github user fhueske commented on a diff in the pull request:

    https://github.com/apache/flink/pull/3748#discussion_r147456818
  
    --- Diff: flink-connectors/flink-connector-cassandra/src/main/java/org/apache/flink/streaming/connectors/cassandra/CassandraSink.java ---
    @@ -375,6 +381,34 @@ protected void sanityCheck() {
     	}
     
     	/**
    +	 * Builder for a {@link CassandraRowSink}.
    +	 */
    +	public static class CassandraRowSinkBuilder extends CassandraSinkBuilder<Row> {
    +		public CassandraRowSinkBuilder(DataStream<Row> input, TypeInformation<Row> typeInfo, TypeSerializer<Row> serializer) {
    +			super(input, typeInfo, serializer);
    +		}
    +
    +		@Override
    +		protected void sanityCheck() {
    +			super.sanityCheck();
    +			if (query == null || query.length() == 0) {
    +				throw new IllegalArgumentException("Query must not be null or empty.");
    +			}
    +		}
    +
    +		@Override
    +		protected CassandraSink<Row> createSink() throws Exception {
    +			return new CassandraSink<>(input.addSink(new CassandraRowSink(query, builder)).name("Cassandra Sink"));
    +
    +		}
    +
    +		@Override
    +		protected CassandraSink<Row> createWriteAheadSink() throws Exception {
    +			throw new IllegalArgumentException("Exactly-once guarantees can only be provided for tuple types.");
    --- End diff --
    
    I had a look at `CassandraTupleWriteAheadSink`. It would be straightforward to copy and adapt it for `Row`.


---

[GitHub] flink pull request #3748: [FLINK-6225] [Cassandra Connector] add CassandraTa...

Posted by zentol <gi...@git.apache.org>.
Github user zentol commented on a diff in the pull request:

    https://github.com/apache/flink/pull/3748#discussion_r116834041
  
    --- Diff: flink-connectors/flink-connector-cassandra/src/main/java/org/apache/flink/streaming/connectors/cassandra/CassandraRowSink.java ---
    @@ -0,0 +1,59 @@
    +/*
    + * Licensed to the Apache Software Foundation (ASF) under one or more
    + * contributor license agreements.  See the NOTICE file distributed with
    + * this work for additional information regarding copyright ownership.
    + * The ASF licenses this file to You under the Apache License, Version 2.0
    + * (the "License"); you may not use this file except in compliance with
    + * the License.  You may obtain a copy of the License at
    + *
    + *    http://www.apache.org/licenses/LICENSE-2.0
    + *
    + * Unless required by applicable law or agreed to in writing, software
    + * distributed under the License is distributed on an "AS IS" BASIS,
    + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    + * See the License for the specific language governing permissions and
    + * limitations under the License.
    + */
    +
    +package org.apache.flink.streaming.connectors.cassandra;
    +
    +import com.datastax.driver.core.PreparedStatement;
    +import com.datastax.driver.core.ResultSet;
    +import com.google.common.util.concurrent.ListenableFuture;
    +import org.apache.flink.configuration.Configuration;
    +import org.apache.flink.types.Row;
    +
    +/**
    + * Flink Sink to save data into a Cassandra cluster.
    + *
    + * @param <IN> Type of the elements emitted by this sink, it must extend {@link Row}
    + */
    +public class CassandraRowSink<IN extends Row> extends CassandraSinkBase<IN, ResultSet> {
    --- End diff --
    
    Could we not use `Row` directly here like in the builder?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request #3748: [FLINK-6225] [Cassandra Connector] add CassandraTa...

Posted by fhueske <gi...@git.apache.org>.
Github user fhueske commented on a diff in the pull request:

    https://github.com/apache/flink/pull/3748#discussion_r147517530
  
    --- Diff: flink-connectors/flink-connector-cassandra/src/test/java/org/apache/flink/streaming/connectors/cassandra/CassandraConnectorITCase.java ---
    @@ -392,6 +425,45 @@ public void testCassandraPojoAtLeastOnceSink() throws Exception {
     	}
     
     	@Test
    +	public void testCassandraTableSink() throws Exception {
    +		StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();
    +		env.setParallelism(1);
    +
    +		DataStreamSource<Row> source = env.fromCollection(rowCollection);
    +		CassandraTableSink cassandraTableSink = new CassandraTableSink(new ClusterBuilder() {
    +			@Override
    +			protected Cluster buildCluster(Cluster.Builder builder) {
    +				return builder.addContactPointsWithPorts(new InetSocketAddress(HOST, PORT)).build();
    +			}
    +		}, injectTableName(INSERT_DATA_QUERY), new Properties());
    +		CassandraTableSink newCassandrTableSink = cassandraTableSink.configure(FIELD_NAMES, FIELD_TYPES);
    +
    +		newCassandrTableSink.emitDataStream(source);
    +
    +		env.execute();
    +		ResultSet rs = session.execute(injectTableName(SELECT_DATA_QUERY));
    +		Assert.assertEquals(20, rs.all().size());
    +	}
    +
    +	@Test
    +	public void testCassandraTableSinkE2E() throws Exception {
    --- End diff --
    
    I think `testCassandraTableSink` can be removed in favor of this test.


---

[GitHub] flink pull request #3748: [FLINK-6225] [Cassandra Connector] add CassandraTa...

Posted by haohui <gi...@git.apache.org>.
Github user haohui commented on a diff in the pull request:

    https://github.com/apache/flink/pull/3748#discussion_r115927398
  
    --- Diff: flink-connectors/flink-connector-cassandra/pom.xml ---
    @@ -176,5 +176,23 @@ under the License.
     				</exclusion>
     			</exclusions>
     		</dependency>
    +		<dependency>
    +			<groupId>org.apache.flink</groupId>
    +			<artifactId>flink-table_2.10</artifactId>
    +			<version>${project.version}</version>
    +			<scope>provided</scope>
    +		</dependency>
    +		<dependency>
    +			<groupId>org.apache.flink</groupId>
    +			<artifactId>flink-scala_2.10</artifactId>
    --- End diff --
    
    You're correct.
    
    The problem is that in order to actually use the connector in a meaningful way users do need to depend on the packages, as `ExecutionEnvironment` and `StreamExecutionEnviornment` are exposed in the APIs:
    
    ```
    $ javap flink-libraries/flink-table/target/classes/org/apache/flink/table/api/TableEnvironment.class
    ```
    
    ```
      public static org.apache.flink.table.api.scala.BatchTableEnvironment getTableEnvironment(org.apache.flink.api.scala.ExecutionEnvironment);
      public static org.apache.flink.table.api.java.BatchTableEnvironment getTableEnvironment(org.apache.flink.api.java.ExecutionEnvironment, org.apache.flink.table.api.TableConfig);
      public static org.apache.flink.table.api.java.BatchTableEnvironment getTableEnvironment(org.apache.flink.api.java.ExecutionEnvironment);
    ```
    
    This is captured by the integration test in the patch. However, I do think these two dependency should be `test` instead of `provided`. At the very least the dependency does not get leaked to the users.
    
    @PangZhi can you please make the changes?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request #3748: [FLINK-6225] [Cassandra Connector] add CassandraTa...

Posted by zentol <gi...@git.apache.org>.
Github user zentol commented on a diff in the pull request:

    https://github.com/apache/flink/pull/3748#discussion_r113410046
  
    --- Diff: flink-connectors/flink-connector-cassandra/src/test/java/org/apache/flink/streaming/connectors/cassandra/CassandraConnectorITCase.java ---
    @@ -86,11 +83,15 @@
     
     	private static EmbeddedCassandraService cassandra;
     
    +	private static String HOST = "127.0.0.1";
    +
    +	private static int PORT = 9042;
    --- End diff --
    
    should be final


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request #3748: [FLINK-6225] [Cassandra Connector] add CassandraTa...

Posted by fhueske <gi...@git.apache.org>.
Github user fhueske commented on a diff in the pull request:

    https://github.com/apache/flink/pull/3748#discussion_r147517745
  
    --- Diff: flink-connectors/flink-connector-cassandra/src/main/java/org/apache/flink/streaming/connectors/cassandra/CassandraTableSink.java ---
    @@ -0,0 +1,82 @@
    +/*
    + * Licensed to the Apache Software Foundation (ASF) under one or more
    + * contributor license agreements.  See the NOTICE file distributed with
    + * this work for additional information regarding copyright ownership.
    + * The ASF licenses this file to You under the Apache License, Version 2.0
    + * (the "License"); you may not use this file except in compliance with
    + * the License.  You may obtain a copy of the License at
    + *
    + *    http://www.apache.org/licenses/LICENSE-2.0
    + *
    + * Unless required by applicable law or agreed to in writing, software
    + * distributed under the License is distributed on an "AS IS" BASIS,
    + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    + * See the License for the specific language governing permissions and
    + * limitations under the License.
    + */
    +
    +package org.apache.flink.streaming.connectors.cassandra;
    +
    +import org.apache.flink.api.common.typeinfo.TypeInformation;
    +import org.apache.flink.api.java.typeutils.RowTypeInfo;
    +import org.apache.flink.streaming.api.datastream.DataStream;
    +import org.apache.flink.table.sinks.AppendStreamTableSink;
    +import org.apache.flink.types.Row;
    +import org.apache.flink.util.Preconditions;
    +
    +import java.util.Properties;
    +
    +/**
    + * A cassandra  {@link AppendStreamTableSink}.
    --- End diff --
    
    `cassandra` -> `Cassandra`


---

[GitHub] flink pull request #3748: [FLINK-6225] [Cassandra Connector] add CassandraTa...

Posted by zentol <gi...@git.apache.org>.
Github user zentol commented on a diff in the pull request:

    https://github.com/apache/flink/pull/3748#discussion_r115911445
  
    --- Diff: flink-connectors/flink-connector-cassandra/pom.xml ---
    @@ -176,5 +176,23 @@ under the License.
     				</exclusion>
     			</exclusions>
     		</dependency>
    +		<dependency>
    +			<groupId>org.apache.flink</groupId>
    +			<artifactId>flink-table_2.10</artifactId>
    +			<version>${project.version}</version>
    +			<scope>provided</scope>
    +		</dependency>
    +		<dependency>
    +			<groupId>org.apache.flink</groupId>
    +			<artifactId>flink-scala_2.10</artifactId>
    --- End diff --
    
    I still don't understand this :/
    
    If neither flink-table nor we actually depend on flink-scala, nor flink-scala-streaming then we don't need the dependency. If the user wants to work against the scala api he has to specify this dependency himself.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink issue #3748: [FLINK-6225] [Cassandra Connector] add CassandraTableSink

Posted by PangZhi <gi...@git.apache.org>.
Github user PangZhi commented on the issue:

    https://github.com/apache/flink/pull/3748
  
    @fhueske Hi zentol hasn't replied can you help to take a look. thx



---

[GitHub] flink pull request #3748: [FLINK-6225] [Cassandra Connector] add CassandraTa...

Posted by zentol <gi...@git.apache.org>.
Github user zentol commented on a diff in the pull request:

    https://github.com/apache/flink/pull/3748#discussion_r115934327
  
    --- Diff: flink-connectors/flink-connector-cassandra/pom.xml ---
    @@ -176,5 +176,23 @@ under the License.
     				</exclusion>
     			</exclusions>
     		</dependency>
    +		<dependency>
    +			<groupId>org.apache.flink</groupId>
    +			<artifactId>flink-table_2.10</artifactId>
    +			<version>${project.version}</version>
    +			<scope>provided</scope>
    +		</dependency>
    +		<dependency>
    +			<groupId>org.apache.flink</groupId>
    +			<artifactId>flink-scala_2.10</artifactId>
    --- End diff --
    
    Please point me to the test that actually uses the scala dependendies, because i can't find any.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request #3748: [FLINK-6225] [Cassandra Connector] add CassandraTa...

Posted by zentol <gi...@git.apache.org>.
Github user zentol commented on a diff in the pull request:

    https://github.com/apache/flink/pull/3748#discussion_r115859400
  
    --- Diff: flink-connectors/flink-connector-cassandra/pom.xml ---
    @@ -176,5 +176,23 @@ under the License.
     				</exclusion>
     			</exclusions>
     		</dependency>
    +		<dependency>
    +			<groupId>org.apache.flink</groupId>
    +			<artifactId>flink-table_2.10</artifactId>
    +			<version>${project.version}</version>
    +			<scope>provided</scope>
    +		</dependency>
    +		<dependency>
    +			<groupId>org.apache.flink</groupId>
    +			<artifactId>flink-scala_2.10</artifactId>
    --- End diff --
    
    Do we actually depend on the scala dependencies?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink issue #3748: [FLINK-6225] [Cassandra Connector] add CassandraTableSink

Posted by PangZhi <gi...@git.apache.org>.
Github user PangZhi commented on the issue:

    https://github.com/apache/flink/pull/3748
  
    @zentol can you take another look?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request #3748: [FLINK-6225] [Cassandra Connector] add CassandraTa...

Posted by zentol <gi...@git.apache.org>.
Github user zentol commented on a diff in the pull request:

    https://github.com/apache/flink/pull/3748#discussion_r113410244
  
    --- Diff: flink-connectors/flink-connector-cassandra/src/test/java/org/apache/flink/streaming/connectors/cassandra/CassandraConnectorITCase.java ---
    @@ -108,10 +109,16 @@ protected Cluster buildCluster(Cluster.Builder builder) {
     	private static final String SELECT_DATA_QUERY = "SELECT * FROM flink.test;";
     
     	private static final ArrayList<Tuple3<String, Integer, Integer>> collection = new ArrayList<>(20);
    +	private static final ArrayList<org.apache.flink.types.Row> rowCollection = new ArrayList<>(20);
    --- End diff --
    
    Please add an import for `Row` instead of using the qualified name.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request #3748: [FLINK-6225] [Cassandra Connector] add CassandraTa...

Posted by fhueske <gi...@git.apache.org>.
Github user fhueske commented on a diff in the pull request:

    https://github.com/apache/flink/pull/3748#discussion_r147507230
  
    --- Diff: flink-connectors/flink-connector-cassandra/src/main/java/org/apache/flink/streaming/connectors/cassandra/CassandraTableSink.java ---
    @@ -0,0 +1,82 @@
    +/*
    + * Licensed to the Apache Software Foundation (ASF) under one or more
    + * contributor license agreements.  See the NOTICE file distributed with
    + * this work for additional information regarding copyright ownership.
    + * The ASF licenses this file to You under the Apache License, Version 2.0
    + * (the "License"); you may not use this file except in compliance with
    + * the License.  You may obtain a copy of the License at
    + *
    + *    http://www.apache.org/licenses/LICENSE-2.0
    + *
    + * Unless required by applicable law or agreed to in writing, software
    + * distributed under the License is distributed on an "AS IS" BASIS,
    + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    + * See the License for the specific language governing permissions and
    + * limitations under the License.
    + */
    +
    +package org.apache.flink.streaming.connectors.cassandra;
    +
    +import org.apache.flink.api.common.typeinfo.TypeInformation;
    +import org.apache.flink.api.java.typeutils.RowTypeInfo;
    +import org.apache.flink.streaming.api.datastream.DataStream;
    +import org.apache.flink.table.sinks.AppendStreamTableSink;
    +import org.apache.flink.types.Row;
    +import org.apache.flink.util.Preconditions;
    +
    +import java.util.Properties;
    +
    +/**
    + * A cassandra  {@link AppendStreamTableSink}.
    + */
    +public class CassandraTableSink implements AppendStreamTableSink<Row> {
    --- End diff --
    
    rename to `CassandraAppendTableSink`


---

[GitHub] flink pull request #3748: [FLINK-6225] [Cassandra Connector] add CassandraTa...

Posted by zentol <gi...@git.apache.org>.
Github user zentol commented on a diff in the pull request:

    https://github.com/apache/flink/pull/3748#discussion_r113410023
  
    --- Diff: flink-connectors/flink-connector-cassandra/src/test/java/org/apache/flink/streaming/connectors/cassandra/CassandraConnectorITCase.java ---
    @@ -86,11 +83,15 @@
     
     	private static EmbeddedCassandraService cassandra;
     
    +	private static String HOST = "127.0.0.1";
    --- End diff --
    
    should be final


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request #3748: [FLINK-6225] [Cassandra Connector] add CassandraTa...

Posted by fhueske <gi...@git.apache.org>.
Github user fhueske commented on a diff in the pull request:

    https://github.com/apache/flink/pull/3748#discussion_r113776064
  
    --- Diff: flink-connectors/flink-connector-cassandra/src/main/java/org/apache/flink/streaming/connectors/cassandra/CassandraSink.java ---
    @@ -335,6 +340,30 @@ protected void sanityCheck() {
     		}
     	}
     
    +	public static class CassandraRowSinkBuilder<IN extends Row> extends CassandraSinkBuilder<IN> {
    +		public CassandraRowSinkBuilder(DataStream<IN> input, TypeInformation<IN> typeInfo, TypeSerializer<IN> serializer) {
    +			super(input, typeInfo, serializer);
    +		}
    +
    +		@Override
    +		protected void sanityCheck() {
    +			super.sanityCheck();
    +			if (query == null || query.length() == 0) {
    +				throw new IllegalArgumentException("Query must not be null or empty.");
    +			}
    +		}
    +
    +		@Override
    +		public CassandraSink<IN> build() throws Exception {
    +			sanityCheck();
    +			if (isWriteAheadLogEnabled) {
    +				throw new UnsupportedOperationException();
    --- End diff --
    
    Please add an error message here.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request #3748: [FLINK-6225] [Cassandra Connector] add CassandraTa...

Posted by fhueske <gi...@git.apache.org>.
Github user fhueske commented on a diff in the pull request:

    https://github.com/apache/flink/pull/3748#discussion_r113778001
  
    --- Diff: flink-connectors/flink-connector-cassandra/src/main/java/org/apache/flink/streaming/connectors/cassandra/CassandraTableSink.java ---
    @@ -0,0 +1,83 @@
    +/*
    + * Licensed to the Apache Software Foundation (ASF) under one or more
    + * contributor license agreements.  See the NOTICE file distributed with
    + * this work for additional information regarding copyright ownership.
    + * The ASF licenses this file to You under the Apache License, Version 2.0
    + * (the "License"); you may not use this file except in compliance with
    + * the License.  You may obtain a copy of the License at
    + *
    + *    http://www.apache.org/licenses/LICENSE-2.0
    + *
    + * Unless required by applicable law or agreed to in writing, software
    + * distributed under the License is distributed on an "AS IS" BASIS,
    + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    + * See the License for the specific language governing permissions and
    + * limitations under the License.
    + */
    +
    +package org.apache.flink.streaming.connectors.cassandra;
    +
    +import org.apache.flink.api.common.typeinfo.TypeInformation;
    +import org.apache.flink.api.java.typeutils.RowTypeInfo;
    +import org.apache.flink.streaming.api.datastream.DataStream;
    +import org.apache.flink.table.sinks.StreamTableSink;
    +import org.apache.flink.table.sinks.TableSink;
    +import org.apache.flink.types.Row;
    +import org.apache.flink.util.Preconditions;
    +
    +import java.util.Properties;
    +
    +/**
    + * A cassandra  {@link StreamTableSink}.
    + *
    + */
    +class CassandraTableSink implements StreamTableSink<Row> {
    +	private final ClusterBuilder builder;
    +	private final String cql;
    +	private String[] fieldNames;
    +	private TypeInformation[] fieldTypes;
    +	private final Properties properties;
    +
    +	public CassandraTableSink(ClusterBuilder builder, String cql, Properties properties) {
    +		this.builder = Preconditions.checkNotNull(builder, "builder");
    +		this.cql = Preconditions.checkNotNull(cql, "cql");
    +		this.properties = Preconditions.checkNotNull(properties, "properties");
    +	}
    +
    +	@Override
    +	public TypeInformation<Row> getOutputType() {
    +		return new RowTypeInfo(fieldTypes);
    +	}
    +
    +	@Override
    +	public String[] getFieldNames() {
    +		return this.fieldNames;
    +	}
    +
    +	@Override
    +	public TypeInformation<?>[] getFieldTypes() {
    +		return this.fieldTypes;
    +	}
    +
    +	@Override
    +	public TableSink<Row> configure(String[] fieldNames, TypeInformation<?>[] fieldTypes) {
    +		CassandraTableSink cassandraTableSink = new CassandraTableSink(this.builder, this.cql, this.properties);
    +		cassandraTableSink.fieldNames = Preconditions.checkNotNull(fieldNames, "fieldNames");
    --- End diff --
    
    Better error messages


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request #3748: [FLINK-6225] [Cassandra Connector] add CassandraTa...

Posted by haohui <gi...@git.apache.org>.
Github user haohui commented on a diff in the pull request:

    https://github.com/apache/flink/pull/3748#discussion_r115861002
  
    --- Diff: flink-connectors/flink-connector-cassandra/pom.xml ---
    @@ -176,5 +176,23 @@ under the License.
     				</exclusion>
     			</exclusions>
     		</dependency>
    +		<dependency>
    +			<groupId>org.apache.flink</groupId>
    +			<artifactId>flink-table_2.10</artifactId>
    +			<version>${project.version}</version>
    +			<scope>provided</scope>
    +		</dependency>
    +		<dependency>
    +			<groupId>org.apache.flink</groupId>
    +			<artifactId>flink-scala_2.10</artifactId>
    --- End diff --
    
    This is interesting as now `flink-table_2.10` no longer pulls in `flink-streaming-scala_2.10` as a transitive dependency. Therefore the change needs to introduce the dependency in order to compile anything against the `flink-streaming-scala` module (e.g., `StreamExecutionEnviornment`).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request #3748: [FLINK-6225] [Cassandra Connector] add CassandraTa...

Posted by zentol <gi...@git.apache.org>.
Github user zentol commented on a diff in the pull request:

    https://github.com/apache/flink/pull/3748#discussion_r113410392
  
    --- Diff: flink-connectors/flink-connector-cassandra/src/test/java/org/apache/flink/streaming/connectors/cassandra/CassandraConnectorITCase.java ---
    @@ -108,10 +109,16 @@ protected Cluster buildCluster(Cluster.Builder builder) {
     	private static final String SELECT_DATA_QUERY = "SELECT * FROM flink.test;";
     
     	private static final ArrayList<Tuple3<String, Integer, Integer>> collection = new ArrayList<>(20);
    +	private static final ArrayList<org.apache.flink.types.Row> rowCollection = new ArrayList<>(20);
    +
    +	private static final String[] FIELD_NAMES = new String[] {"id", "counter", "batch_id"};
    +	private static final TypeInformation[] FIELD_TYPES = new TypeInformation[] {BasicTypeInfo.STRING_TYPE_INFO,
    +	BasicTypeInfo.INT_TYPE_INFO, BasicTypeInfo.INT_TYPE_INFO};
    --- End diff --
    
    missing indentation.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink issue #3748: [FLINK-6225] [Cassandra Connector] add CassandraTableSink

Posted by haohui <gi...@git.apache.org>.
Github user haohui commented on the issue:

    https://github.com/apache/flink/pull/3748
  
    Given we have dropped the support for Java 7, it might make sense to just rewrite the class to use the completablefuture in java 8.


---

[GitHub] flink pull request #3748: [FLINK-6225] [Cassandra Connector] add CassandraTa...

Posted by fhueske <gi...@git.apache.org>.
Github user fhueske commented on a diff in the pull request:

    https://github.com/apache/flink/pull/3748#discussion_r113779310
  
    --- Diff: flink-connectors/flink-connector-cassandra/src/test/java/org/apache/flink/streaming/connectors/cassandra/CassandraConnectorITCase.java ---
    @@ -438,6 +461,27 @@ public void cancel() {
     	}
     
     	@Test
    +	public void testCassandraTableSink() throws Exception {
    +		StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();
    +		env.setParallelism(1);
    +
    +		DataStreamSource<org.apache.flink.types.Row> source = env.fromCollection(rowCollection);
    +		CassandraTableSink cassandraTableSink = new CassandraTableSink(new ClusterBuilder() {
    +			@Override
    +			protected Cluster buildCluster(Cluster.Builder builder) {
    +				return builder.addContactPointsWithPorts(new InetSocketAddress(HOST, PORT)).build();
    +			}
    +		}, INSERT_DATA_QUERY, new Properties());
    +		cassandraTableSink.configure(FIELD_NAMES, FIELD_TYPES);
    --- End diff --
    
    Yes. It would also be good to test the TableSink in an actual Table API program. Most of the methods like `configure()` and `emitDataStream()` are internally called by the API.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---