You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@beam.apache.org by Adam Scott <ad...@gmail.com> on 2023/02/03 20:59:05 UTC
[Question] [CassandraIO]Using a query
HI All,
does anyone have an example of using CassandraIO to query a table?
The following mentions "Alternatively, one may use
CassandraIO.<Person>readAll()
.withCoder(SerializableCoder.of(Person.class)) to query a subset of the
Cassandra database by creating a PCollection of CassandraIO.Read<Person>
each with their own query or RingRange."
https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/io/cassandra/CassandraIO.html
Hoping there would be an example of this.
TIA,
Adam
Re: [Question] [CassandraIO]Using a query
Posted by Vincent Marquez <vi...@gmail.com>.
I've not used the BigTable writer, but It should be just creating a
function that takes a MyClass and converts it to an
Iterable<bigtable.v2.Mutation>,
then calling that from a DoFn you create and outputting the resulting
Iterable to the OutputReceiver. Then your pipeline would be something like
pipeline.apply(CassandraIO.read().withTable("MyClass").withBlahblah)
.apply(new CustomDoFnToConvert())
.apply(BigTable.write())
*~Vincent*
On Tue, Feb 7, 2023 at 11:11 AM Adam Scott <ad...@gmail.com> wrote:
> Thank you Vincent! Thanks for the quick response!
>
> The goal is to read from a Cassandra query and write to a Bigtable table.
>
> I've tried combining this technique, below, with the test examples you
> pointed out. But I am afraid my efforts were wholly naive.
>
> https://github.com/GoogleCloudPlatform/DataflowTemplates/blob/main/v1/src/main/java/com/google/cloud/teleport/bigtable/CassandraToBigtable.java
>
> It seems I would want, if I were dreaming, instead of a
> BeamRowToBigtableFn (
> https://github.com/GoogleCloudPlatform/DataflowTemplates/blob/cc799861a458eef329aa5a15189a8045109e43e0/v1/src/main/java/com/google/cloud/teleport/bigtable/CassandraToBigtable.java#L220)
> , a Beam<myClasss>toBigtableFn.
>
> Any hints on how to add a write/output to Bigtable?
>
> TIA,
> Adam
>
>
>
> On Fri, Feb 3, 2023 at 1:25 PM Vincent Marquez <vi...@gmail.com>
> wrote:
>
>> There are some examples in the test code that should be easy enough to
>> follow.
>>
>> Here is an example of just querying the entire table:
>>
>>
>> https://github.com/apache/beam/blob/master/sdks/java/io/cassandra/src/test/java/org/apache/beam/sdk/io/cassandra/CassandraIOTest.java#L460
>>
>> Here's an example of using readAll to only pull certain keys:
>>
>>
>> https://github.com/apache/beam/blob/master/sdks/java/io/cassandra/src/test/java/org/apache/beam/sdk/io/cassandra/CassandraIOTest.java#L381
>>
>>
>>
>>
>> *~Vincent*
>>
>>
>> On Fri, Feb 3, 2023 at 12:59 PM Adam Scott <ad...@gmail.com>
>> wrote:
>>
>>> HI All,
>>>
>>> does anyone have an example of using CassandraIO to query a table?
>>>
>>> The following mentions "Alternatively, one may use CassandraIO.<Person>readAll()
>>> .withCoder(SerializableCoder.of(Person.class)) to query a subset of the
>>> Cassandra database by creating a PCollection of CassandraIO.Read<Person>
>>> each with their own query or RingRange."
>>>
>>> https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/io/cassandra/CassandraIO.html
>>>
>>> Hoping there would be an example of this.
>>>
>>> TIA,
>>> Adam
>>>
>>>
>>>
>>>
Re: [Question] [CassandraIO]Using a query
Posted by Adam Scott <ad...@gmail.com>.
Thank you Vincent! Thanks for the quick response!
The goal is to read from a Cassandra query and write to a Bigtable table.
I've tried combining this technique, below, with the test examples you
pointed out. But I am afraid my efforts were wholly naive.
https://github.com/GoogleCloudPlatform/DataflowTemplates/blob/main/v1/src/main/java/com/google/cloud/teleport/bigtable/CassandraToBigtable.java
It seems I would want, if I were dreaming, instead of a BeamRowToBigtableFn
(
https://github.com/GoogleCloudPlatform/DataflowTemplates/blob/cc799861a458eef329aa5a15189a8045109e43e0/v1/src/main/java/com/google/cloud/teleport/bigtable/CassandraToBigtable.java#L220)
, a Beam<myClasss>toBigtableFn.
Any hints on how to add a write/output to Bigtable?
TIA,
Adam
On Fri, Feb 3, 2023 at 1:25 PM Vincent Marquez <vi...@gmail.com>
wrote:
> There are some examples in the test code that should be easy enough to
> follow.
>
> Here is an example of just querying the entire table:
>
>
> https://github.com/apache/beam/blob/master/sdks/java/io/cassandra/src/test/java/org/apache/beam/sdk/io/cassandra/CassandraIOTest.java#L460
>
> Here's an example of using readAll to only pull certain keys:
>
>
> https://github.com/apache/beam/blob/master/sdks/java/io/cassandra/src/test/java/org/apache/beam/sdk/io/cassandra/CassandraIOTest.java#L381
>
>
>
>
> *~Vincent*
>
>
> On Fri, Feb 3, 2023 at 12:59 PM Adam Scott <ad...@gmail.com> wrote:
>
>> HI All,
>>
>> does anyone have an example of using CassandraIO to query a table?
>>
>> The following mentions "Alternatively, one may use CassandraIO.<Person>readAll()
>> .withCoder(SerializableCoder.of(Person.class)) to query a subset of the
>> Cassandra database by creating a PCollection of CassandraIO.Read<Person>
>> each with their own query or RingRange."
>>
>> https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/io/cassandra/CassandraIO.html
>>
>> Hoping there would be an example of this.
>>
>> TIA,
>> Adam
>>
>>
>>
>>
Re: [Question] [CassandraIO]Using a query
Posted by Vincent Marquez <vi...@gmail.com>.
There are some examples in the test code that should be easy enough to
follow.
Here is an example of just querying the entire table:
https://github.com/apache/beam/blob/master/sdks/java/io/cassandra/src/test/java/org/apache/beam/sdk/io/cassandra/CassandraIOTest.java#L460
Here's an example of using readAll to only pull certain keys:
https://github.com/apache/beam/blob/master/sdks/java/io/cassandra/src/test/java/org/apache/beam/sdk/io/cassandra/CassandraIOTest.java#L381
*~Vincent*
On Fri, Feb 3, 2023 at 12:59 PM Adam Scott <ad...@gmail.com> wrote:
> HI All,
>
> does anyone have an example of using CassandraIO to query a table?
>
> The following mentions "Alternatively, one may use CassandraIO.<Person>readAll()
> .withCoder(SerializableCoder.of(Person.class)) to query a subset of the
> Cassandra database by creating a PCollection of CassandraIO.Read<Person>
> each with their own query or RingRange."
>
> https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/io/cassandra/CassandraIO.html
>
> Hoping there would be an example of this.
>
> TIA,
> Adam
>
>
>
>