You are viewing a plain text version of this content. The canonical link for it is here.

Posted to user@beam.apache.org by Neville Li <ne...@gmail.com> on 2020/01/08 18:03:29 UTC

Scio 0.8.0 released

Hi all,

We just released Scio 0.8.0. This is based on the most recent Beam 2.17.0
release and includes a lot of new features & bug fixes over the past 10
months.

Cheers,
Neville

https://github.com/spotify/scio/releases/tag/v0.8.0

*"Amato Animo Animato Animagus"*
Breaking changes & deprecations

   - See v0.8.0 Migration Guide
   <https://spotify.github.io/scio/migrations/v0.8.0-Migration-Guide.html> for
   detailed instructions
   - Remove @experimental from transform #2537
   <https://github.com/spotify/scio/pull/2537>
   - Deprecate scio-elasticsearch2 and scio-cassandra2 #2414
   <https://github.com/spotify/scio/pull/2414> #2421
   <https://github.com/spotify/scio/pull/2421>
   - Deprecate hashFilter #2442 <https://github.com/spotify/scio/pull/2442>
   - Deprecate legacy components in scio-extras #2533
   <https://github.com/spotify/scio/pull/2533>

Features

   - Bump Beam to 2.17.0 #2577 <https://github.com/spotify/scio/pull/2577>
   - Add sharded Sparkey support. #2336
   <https://github.com/spotify/scio/pull/2336>
   - Rework side input cache #2363
   <https://github.com/spotify/scio/pull/2363>
   - Cleanup Side Inputs API, introduce Singleton Set SideInputs #2424
   <https://github.com/spotify/scio/pull/2424>
   - Add schema support for GenericRecord #2514
   <https://github.com/spotify/scio/pull/2514>

Bug fixes & improvements

   - Add file:linenum only to outer transform #2405
   <https://github.com/spotify/scio/pull/2405>
   - Fix join transform names #2444
   <https://github.com/spotify/scio/pull/2444>
   - Remove Coder context bound for partitionByKey #2451
   <https://github.com/spotify/scio/pull/2451>
   - Rename that method argument in join functions to rhs #2466
   <https://github.com/spotify/scio/pull/2466>
   - Replace custom ClosureCleaner with chill's #2423Use chill's
   Externalizer to serialize predicates in SCollectionMatchers #2410
   <https://github.com/spotify/scio/pull/2410>
   - Add errmsg when beamOpts == null in JobTest, fix #2430
   <https://github.com/spotify/scio/issues/2430> #2545
   <https://github.com/spotify/scio/pull/2545>
   - Add bigQuerySelect() method with default flattenResults value #2500
   <https://github.com/spotify/scio/pull/2500>
   - Better consistency around BigQuery API #2412Fail early on malformed
   BigQuery spec #2345 <https://github.com/spotify/scio/pull/2345>
   - Rewrite typedBigQueryStorage #2434
   <https://github.com/spotify/scio/pull/2434>
   - Add DML query support to bigquery client #2418
   <https://github.com/spotify/scio/pull/2418>
   - Treat Avro array as java List in BigQuery read, fix #2068
   <https://github.com/spotify/scio/issues/2068> #2415
   <https://github.com/spotify/scio/pull/2415>
   - Fix NPE in scio-bigtable's ChannelPoolCreator when credentials aren't
   set #2317Fix bigtable scollection ops return type #2486
   <https://github.com/spotify/scio/pull/2486>
   - Refactor PubsubIO for more typesafety #2457
   <https://github.com/spotify/scio/pull/2457>
   - Avoid Mutation coder fallback for Spanner #2478
   <https://github.com/spotify/scio/pull/2478>
   - Fix Parquet sink suffix #2367Improve iterable equality #2483
   <https://github.com/spotify/scio/pull/2483>
   - Improve back compat with Scio 0.7 #2401
   <https://github.com/spotify/scio/pull/2401>
   - Improve coder gen by checking companion implicits #2522
   <https://github.com/spotify/scio/pull/2522>
   - Make recursive coders serializable #2404
   <https://github.com/spotify/scio/pull/2404>
   - Remove kryo coder override in intermediate steps #2422Fix fallback
   warning when implicit is in scope #2511
   <https://github.com/spotify/scio/pull/2511>
   - Improve the schema compatibility error message #2366
   <https://github.com/spotify/scio/pull/2366>
   - Remove schema fallback #2489
   <https://github.com/spotify/scio/pull/2489>
   - Add Schemas support for more types #2364
   <https://github.com/spotify/scio/pull/2364>
   - Assert FileStorage.isDone in MaterializeTap #2518
   <https://github.com/spotify/scio/pull/2518>
   - Add support for cleaning up TF models on shutdown #2549
   <https://github.com/spotify/scio/pull/2549>
   - Rework TensorFlow predict ops #2343
   <https://github.com/spotify/scio/pull/2343>
   - Remove unused/deprecated TensorFlow graph DoFn #2339
   <https://github.com/spotify/scio/pull/2339>
   - Mark some APIs in scio-extras as experimental #2517
   <https://github.com/spotify/scio/issues/2517> #2572
   <https://github.com/spotify/scio/pull/2572>

Re: Scio 0.8.0 released

Posted by Kenneth Knowles <kl...@google.com>.

So fast! Excellent.

On Wed, Jan 8, 2020 at 11:28 AM Robert Bradshaw <ro...@google.com> wrote:

> Nice!
>
> On Wed, Jan 8, 2020 at 10:03 AM Neville Li <ne...@gmail.com> wrote:
>
>> Hi all,
>>
>> We just released Scio 0.8.0. This is based on the most recent Beam 2.17.0
>> release and includes a lot of new features & bug fixes over the past 10
>> months.
>>
>> Cheers,
>> Neville
>>
>> https://github.com/spotify/scio/releases/tag/v0.8.0
>>
>> *"Amato Animo Animato Animagus"*
>> Breaking changes & deprecations
>>
>>    - See v0.8.0 Migration Guide
>>    <https://spotify.github.io/scio/migrations/v0.8.0-Migration-Guide.html> for
>>    detailed instructions
>>    - Remove @experimental from transform #2537
>>    <https://github.com/spotify/scio/pull/2537>
>>    - Deprecate scio-elasticsearch2 and scio-cassandra2 #2414
>>    <https://github.com/spotify/scio/pull/2414> #2421
>>    <https://github.com/spotify/scio/pull/2421>
>>    - Deprecate hashFilter #2442
>>    <https://github.com/spotify/scio/pull/2442>
>>    - Deprecate legacy components in scio-extras #2533
>>    <https://github.com/spotify/scio/pull/2533>
>>
>> Features
>>
>>    - Bump Beam to 2.17.0 #2577
>>    <https://github.com/spotify/scio/pull/2577>
>>    - Add sharded Sparkey support. #2336
>>    <https://github.com/spotify/scio/pull/2336>
>>    - Rework side input cache #2363
>>    <https://github.com/spotify/scio/pull/2363>
>>    - Cleanup Side Inputs API, introduce Singleton Set SideInputs #2424
>>    <https://github.com/spotify/scio/pull/2424>
>>    - Add schema support for GenericRecord #2514
>>    <https://github.com/spotify/scio/pull/2514>
>>
>> Bug fixes & improvements
>>
>>    - Add file:linenum only to outer transform #2405
>>    <https://github.com/spotify/scio/pull/2405>
>>    - Fix join transform names #2444
>>    <https://github.com/spotify/scio/pull/2444>
>>    - Remove Coder context bound for partitionByKey #2451
>>    <https://github.com/spotify/scio/pull/2451>
>>    - Rename that method argument in join functions to rhs #2466
>>    <https://github.com/spotify/scio/pull/2466>
>>    - Replace custom ClosureCleaner with chill's #2423Use chill's
>>    Externalizer to serialize predicates in SCollectionMatchers #2410
>>    <https://github.com/spotify/scio/pull/2410>
>>    - Add errmsg when beamOpts == null in JobTest, fix #2430
>>    <https://github.com/spotify/scio/issues/2430> #2545
>>    <https://github.com/spotify/scio/pull/2545>
>>    - Add bigQuerySelect() method with default flattenResults value #2500
>>    <https://github.com/spotify/scio/pull/2500>
>>    - Better consistency around BigQuery API #2412Fail early on malformed
>>    BigQuery spec #2345 <https://github.com/spotify/scio/pull/2345>
>>    - Rewrite typedBigQueryStorage #2434
>>    <https://github.com/spotify/scio/pull/2434>
>>    - Add DML query support to bigquery client #2418
>>    <https://github.com/spotify/scio/pull/2418>
>>    - Treat Avro array as java List in BigQuery read, fix #2068
>>    <https://github.com/spotify/scio/issues/2068> #2415
>>    <https://github.com/spotify/scio/pull/2415>
>>    - Fix NPE in scio-bigtable's ChannelPoolCreator when credentials
>>    aren't set #2317Fix bigtable scollection ops return type #2486
>>    <https://github.com/spotify/scio/pull/2486>
>>    - Refactor PubsubIO for more typesafety #2457
>>    <https://github.com/spotify/scio/pull/2457>
>>    - Avoid Mutation coder fallback for Spanner #2478
>>    <https://github.com/spotify/scio/pull/2478>
>>    - Fix Parquet sink suffix #2367Improve iterable equality #2483
>>    <https://github.com/spotify/scio/pull/2483>
>>    - Improve back compat with Scio 0.7 #2401
>>    <https://github.com/spotify/scio/pull/2401>
>>    - Improve coder gen by checking companion implicits #2522
>>    <https://github.com/spotify/scio/pull/2522>
>>    - Make recursive coders serializable #2404
>>    <https://github.com/spotify/scio/pull/2404>
>>    - Remove kryo coder override in intermediate steps #2422Fix fallback
>>    warning when implicit is in scope #2511
>>    <https://github.com/spotify/scio/pull/2511>
>>    - Improve the schema compatibility error message #2366
>>    <https://github.com/spotify/scio/pull/2366>
>>    - Remove schema fallback #2489
>>    <https://github.com/spotify/scio/pull/2489>
>>    - Add Schemas support for more types #2364
>>    <https://github.com/spotify/scio/pull/2364>
>>    - Assert FileStorage.isDone in MaterializeTap #2518
>>    <https://github.com/spotify/scio/pull/2518>
>>    - Add support for cleaning up TF models on shutdown #2549
>>    <https://github.com/spotify/scio/pull/2549>
>>    - Rework TensorFlow predict ops #2343
>>    <https://github.com/spotify/scio/pull/2343>
>>    - Remove unused/deprecated TensorFlow graph DoFn #2339
>>    <https://github.com/spotify/scio/pull/2339>
>>    - Mark some APIs in scio-extras as experimental #2517
>>    <https://github.com/spotify/scio/issues/2517> #2572
>>    <https://github.com/spotify/scio/pull/2572>
>>
>>
>>
>

Re: Scio 0.8.0 released

Posted by Robert Bradshaw <ro...@google.com>.

Nice!

On Wed, Jan 8, 2020 at 10:03 AM Neville Li <ne...@gmail.com> wrote:

> Hi all,
>
> We just released Scio 0.8.0. This is based on the most recent Beam 2.17.0
> release and includes a lot of new features & bug fixes over the past 10
> months.
>
> Cheers,
> Neville
>
> https://github.com/spotify/scio/releases/tag/v0.8.0
>
> *"Amato Animo Animato Animagus"*
> Breaking changes & deprecations
>
>    - See v0.8.0 Migration Guide
>    <https://spotify.github.io/scio/migrations/v0.8.0-Migration-Guide.html> for
>    detailed instructions
>    - Remove @experimental from transform #2537
>    <https://github.com/spotify/scio/pull/2537>
>    - Deprecate scio-elasticsearch2 and scio-cassandra2 #2414
>    <https://github.com/spotify/scio/pull/2414> #2421
>    <https://github.com/spotify/scio/pull/2421>
>    - Deprecate hashFilter #2442
>    <https://github.com/spotify/scio/pull/2442>
>    - Deprecate legacy components in scio-extras #2533
>    <https://github.com/spotify/scio/pull/2533>
>
> Features
>
>    - Bump Beam to 2.17.0 #2577 <https://github.com/spotify/scio/pull/2577>
>    - Add sharded Sparkey support. #2336
>    <https://github.com/spotify/scio/pull/2336>
>    - Rework side input cache #2363
>    <https://github.com/spotify/scio/pull/2363>
>    - Cleanup Side Inputs API, introduce Singleton Set SideInputs #2424
>    <https://github.com/spotify/scio/pull/2424>
>    - Add schema support for GenericRecord #2514
>    <https://github.com/spotify/scio/pull/2514>
>
> Bug fixes & improvements
>
>    - Add file:linenum only to outer transform #2405
>    <https://github.com/spotify/scio/pull/2405>
>    - Fix join transform names #2444
>    <https://github.com/spotify/scio/pull/2444>
>    - Remove Coder context bound for partitionByKey #2451
>    <https://github.com/spotify/scio/pull/2451>
>    - Rename that method argument in join functions to rhs #2466
>    <https://github.com/spotify/scio/pull/2466>
>    - Replace custom ClosureCleaner with chill's #2423Use chill's
>    Externalizer to serialize predicates in SCollectionMatchers #2410
>    <https://github.com/spotify/scio/pull/2410>
>    - Add errmsg when beamOpts == null in JobTest, fix #2430
>    <https://github.com/spotify/scio/issues/2430> #2545
>    <https://github.com/spotify/scio/pull/2545>
>    - Add bigQuerySelect() method with default flattenResults value #2500
>    <https://github.com/spotify/scio/pull/2500>
>    - Better consistency around BigQuery API #2412Fail early on malformed
>    BigQuery spec #2345 <https://github.com/spotify/scio/pull/2345>
>    - Rewrite typedBigQueryStorage #2434
>    <https://github.com/spotify/scio/pull/2434>
>    - Add DML query support to bigquery client #2418
>    <https://github.com/spotify/scio/pull/2418>
>    - Treat Avro array as java List in BigQuery read, fix #2068
>    <https://github.com/spotify/scio/issues/2068> #2415
>    <https://github.com/spotify/scio/pull/2415>
>    - Fix NPE in scio-bigtable's ChannelPoolCreator when credentials
>    aren't set #2317Fix bigtable scollection ops return type #2486
>    <https://github.com/spotify/scio/pull/2486>
>    - Refactor PubsubIO for more typesafety #2457
>    <https://github.com/spotify/scio/pull/2457>
>    - Avoid Mutation coder fallback for Spanner #2478
>    <https://github.com/spotify/scio/pull/2478>
>    - Fix Parquet sink suffix #2367Improve iterable equality #2483
>    <https://github.com/spotify/scio/pull/2483>
>    - Improve back compat with Scio 0.7 #2401
>    <https://github.com/spotify/scio/pull/2401>
>    - Improve coder gen by checking companion implicits #2522
>    <https://github.com/spotify/scio/pull/2522>
>    - Make recursive coders serializable #2404
>    <https://github.com/spotify/scio/pull/2404>
>    - Remove kryo coder override in intermediate steps #2422Fix fallback
>    warning when implicit is in scope #2511
>    <https://github.com/spotify/scio/pull/2511>
>    - Improve the schema compatibility error message #2366
>    <https://github.com/spotify/scio/pull/2366>
>    - Remove schema fallback #2489
>    <https://github.com/spotify/scio/pull/2489>
>    - Add Schemas support for more types #2364
>    <https://github.com/spotify/scio/pull/2364>
>    - Assert FileStorage.isDone in MaterializeTap #2518
>    <https://github.com/spotify/scio/pull/2518>
>    - Add support for cleaning up TF models on shutdown #2549
>    <https://github.com/spotify/scio/pull/2549>
>    - Rework TensorFlow predict ops #2343
>    <https://github.com/spotify/scio/pull/2343>
>    - Remove unused/deprecated TensorFlow graph DoFn #2339
>    <https://github.com/spotify/scio/pull/2339>
>    - Mark some APIs in scio-extras as experimental #2517
>    <https://github.com/spotify/scio/issues/2517> #2572
>    <https://github.com/spotify/scio/pull/2572>
>
>
>