You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@beam.apache.org by Neville Li <ne...@gmail.com> on 2020/08/05 18:45:58 UTC
Scio 0.9.3 released
Hi all,
We just released Scio 0.9.3. This bumps Beam SDK to 2.23.0 and includes a
lot of improvements & bug fixes.
Cheers,
Neville
https://github.com/spotify/scio/releases/tag/v0.9.3
*"Petrificus Totalus"*
There are no breaking changes in this release, but some were introduced
with v0.9.0:
See v0.9.0 Migration Guide
<https://spotify.github.io/scio/migrations/v0.9.0-Migration-Guide.html> for
detailed instructions.
Improvements
- Allow user-supplied filename prefix for smb writes/reads (#3215
<https://github.com/spotify/scio/pull/3215>)
- Refactor SortedBucketTransform into a BoundedSource + reuse merge
logic (#3097 <https://github.com/spotify/scio/pull/3097>)
- Add keyGroupFilter optimization to scio-smb (#3160
<https://github.com/spotify/scio/pull/3160>)
- Add error message to BaseAsyncLookupDoFn preconditions check (#3176
<https://github.com/spotify/scio/pull/3176>)
- Add Elasticsearch 5,6,7 add/update alias on multiple indices ops (#3134
<https://github.com/spotify/scio/pull/3134>)
- Add initial update alias op to ES7(#2920
<https://github.com/spotify/scio/pull/2920>)
- Add ScioContext#applyTransform (#3146
<https://github.com/spotify/scio/pull/3146>)
- Allow SCollection#transform name override (#3142
<https://github.com/spotify/scio/pull/3142>)
- Allow setting default name through SCollection#applyTransform (#3144
<https://github.com/spotify/scio/pull/3144>)
- Update 0.9 migration doc and add Bigquery Type read schema
documentation(#3148 <https://github.com/spotify/scio/pull/3148>)
Bug Fixes
- AvroBucketMetadata should validate keyPath (fix #3038
<https://github.com/spotify/scio/issues/3038>) (#3140
<https://github.com/spotify/scio/pull/3140>)
- Allow union types in non leaf field for key (#3187
<https://github.com/spotify/scio/pull/3187>)
- Fix issue with union type as non-leaf field of smb key (#3193
<https://github.com/spotify/scio/pull/3193>)
- Fix ContextAndArgs#typed overloading issue (#3199
<https://github.com/spotify/scio/pull/3199>)
- Fix error propagation on Scala Future onSuccess callback (#3178
<https://github.com/spotify/scio/pull/3178>)
- Fix ByteBuffer should be readOnly (#3220
<https://github.com/spotify/scio/pull/3220>)
- Fix compiler warnings (#3183
<https://github.com/spotify/scio/pull/3183>)
- Fix JdbcShardedReadOptions.fetchSize description (#3209
<https://github.com/spotify/scio/pull/3209>)
- Fix FAQ typo (#3194 <https://github.com/spotify/scio/pull/3194>)
- Fix scalafix error in SortMergeBucketScioContextSyntax (#3158
<https://github.com/spotify/scio/pull/3158>)
- Add scalafix ExplicitReturnType and ProcedureSyntax rules (#3179
<https://github.com/spotify/scio/pull/3179>)
- Cleanup a few more unused and unchecked params (#3223
<https://github.com/spotify/scio/pull/3223>)
- Use GcpOptions#getWorkerZone instead of deprecated GcpOptions#getZone (
#3224 <https://github.com/spotify/scio/pull/3224>)
- Use raw coder in SCollection#applyKvTransform (#3171
<https://github.com/spotify/scio/pull/3171>)
- Add raw beam coder materializer (#3164
<https://github.com/spotify/scio/pull/3164>)
- Avoid circular dep between SCollection and PCollectionWrapper (#3163
<https://github.com/spotify/scio/pull/3163>)
- Remove unused param of internal partitionFn (#3166
<https://github.com/spotify/scio/pull/3166>)
- Remove unused CoderRegistry (#3165
<https://github.com/spotify/scio/pull/3165>)
- Remove defunct scio-bench (#3150
<https://github.com/spotify/scio/pull/3150>)
- Reuse applyTransform (#3162 <https://github.com/spotify/scio/pull/3162>
)
- Make multijoin.py python3
- Use TextIO#withCompression (#3145
<https://github.com/spotify/scio/pull/3145>)
Dependency Updates
- Update Beam SDK to 2.23.0 (#3197
<https://github.com/spotify/scio/pull/3197>)
- Update dependencies to be inline with 2.23.0 (#3225
<https://github.com/spotify/scio/pull/3225>)
- Update to scala 2.12.12 (#3157
<https://github.com/spotify/scio/pull/3157>)
- Update auto-value to 1.7.4 (#3147
<https://github.com/spotify/scio/pull/3147>)
- Update breeze to 1.1 (#3211 <https://github.com/spotify/scio/pull/3211>
)
- Update cassandra-all to 3.11.7 (#3186
<https://github.com/spotify/scio/pull/3186>)
- Update cassandra-driver-core to 3.10.0 (#3195
<https://github.com/spotify/scio/pull/3195>)
- Update commons-lang3 to 3.11 (#3161
<https://github.com/spotify/scio/pull/3161>)
- Update commons-text to 1.9 (#3185
<https://github.com/spotify/scio/pull/3185>)
- Update contributing guidelines with current tools (#3149
<https://github.com/spotify/scio/pull/3149>)
- Update elasticsearch-rest-client, ... to 7.8.1 (#3192
<https://github.com/spotify/scio/pull/3192>)
- Update elasticsearch, ... to 6.8.11 (#3188
<https://github.com/spotify/scio/pull/3188>)
- Update jackson-module-scala to 2.10.5 (#3169
<https://github.com/spotify/scio/pull/3169>)
- Update jna to 5.6.0 (#3156 <https://github.com/spotify/scio/pull/3156>)
- Update magnolify to 0.2.2 (#3154
<https://github.com/spotify/scio/pull/3154>)
- Update mysql-connector-java to 8.0.21 (#3153
<https://github.com/spotify/scio/pull/3153>)
- Update pprint to 0.6.0 (#3203
<https://github.com/spotify/scio/pull/3203>)
- Update protobuf version to 3.11.4 (#3200
<https://github.com/spotify/scio/pull/3200>)
- Update sbt-scalafix to 0.9.18 (#3138
<https://github.com/spotify/scio/pull/3138>)
- Update sbt-sonatype to 3.9.4 (#3136
<https://github.com/spotify/scio/pull/3136>)
- Update scalafmt-core to 2.6.2 (#3139
<https://github.com/spotify/scio/pull/3139>)
- Update scalafmt-core to 2.6.3 (#3152
<https://github.com/spotify/scio/pull/3152>)
- Update scalafmt-core to 2.6.4 (#3167
<https://github.com/spotify/scio/pull/3167>)
- Update sparkey to 3.1.0 (#3204
<https://github.com/spotify/scio/pull/3204>)
- Fix conflicting gcsio dependency (#3180
<https://github.com/spotify/scio/pull/3180>)
Re: Scio 0.9.3 released
Posted by Marcin Kuthan <ma...@gmail.com>.
Just upgraded to Scio 0.9.2 / Beam 2.22 and the stack is not up-to-date
again ;)
Thanks for the update (Scio and Beam teams)!
On Thu, 6 Aug 2020 at 02:09, Robert Bradshaw <ro...@google.com> wrote:
> Thanks for the update!
>
> On Wed, Aug 5, 2020 at 11:46 AM Neville Li <ne...@gmail.com> wrote:
> >
> > Hi all,
> >
> > We just released Scio 0.9.3. This bumps Beam SDK to 2.23.0 and includes
> a lot of improvements & bug fixes.
> >
> > Cheers,
> > Neville
> >
> > https://github.com/spotify/scio/releases/tag/v0.9.3
> >
> > "Petrificus Totalus"
> >
> > There are no breaking changes in this release, but some were introduced
> with v0.9.0:
> >
> > See v0.9.0 Migration Guide for detailed instructions.
> >
> > Improvements
> >
> > Allow user-supplied filename prefix for smb writes/reads (#3215)
> > Refactor SortedBucketTransform into a BoundedSource + reuse merge logic
> (#3097)
> > Add keyGroupFilter optimization to scio-smb (#3160)
> > Add error message to BaseAsyncLookupDoFn preconditions check (#3176)
> > Add Elasticsearch 5,6,7 add/update alias on multiple indices ops (#3134)
> > Add initial update alias op to ES7(#2920)
> > Add ScioContext#applyTransform (#3146)
> > Allow SCollection#transform name override (#3142)
> > Allow setting default name through SCollection#applyTransform (#3144)
> > Update 0.9 migration doc and add Bigquery Type read schema
> documentation(#3148)
> >
> > Bug Fixes
> >
> > AvroBucketMetadata should validate keyPath (fix #3038) (#3140)
> > Allow union types in non leaf field for key (#3187)
> > Fix issue with union type as non-leaf field of smb key (#3193)
> > Fix ContextAndArgs#typed overloading issue (#3199)
> > Fix error propagation on Scala Future onSuccess callback (#3178)
> > Fix ByteBuffer should be readOnly (#3220)
> > Fix compiler warnings (#3183)
> > Fix JdbcShardedReadOptions.fetchSize description (#3209)
> > Fix FAQ typo (#3194)
> > Fix scalafix error in SortMergeBucketScioContextSyntax (#3158)
> > Add scalafix ExplicitReturnType and ProcedureSyntax rules (#3179)
> > Cleanup a few more unused and unchecked params (#3223)
> > Use GcpOptions#getWorkerZone instead of deprecated GcpOptions#getZone
> (#3224)
> > Use raw coder in SCollection#applyKvTransform (#3171)
> > Add raw beam coder materializer (#3164)
> > Avoid circular dep between SCollection and PCollectionWrapper (#3163)
> > Remove unused param of internal partitionFn (#3166)
> > Remove unused CoderRegistry (#3165)
> > Remove defunct scio-bench (#3150)
> > Reuse applyTransform (#3162)
> > Make multijoin.py python3
> > Use TextIO#withCompression (#3145)
> >
> > Dependency Updates
> >
> > Update Beam SDK to 2.23.0 (#3197)
> > Update dependencies to be inline with 2.23.0 (#3225)
> > Update to scala 2.12.12 (#3157)
> > Update auto-value to 1.7.4 (#3147)
> > Update breeze to 1.1 (#3211)
> > Update cassandra-all to 3.11.7 (#3186)
> > Update cassandra-driver-core to 3.10.0 (#3195)
> > Update commons-lang3 to 3.11 (#3161)
> > Update commons-text to 1.9 (#3185)
> > Update contributing guidelines with current tools (#3149)
> > Update elasticsearch-rest-client, ... to 7.8.1 (#3192)
> > Update elasticsearch, ... to 6.8.11 (#3188)
> > Update jackson-module-scala to 2.10.5 (#3169)
> > Update jna to 5.6.0 (#3156)
> > Update magnolify to 0.2.2 (#3154)
> > Update mysql-connector-java to 8.0.21 (#3153)
> > Update pprint to 0.6.0 (#3203)
> > Update protobuf version to 3.11.4 (#3200)
> > Update sbt-scalafix to 0.9.18 (#3138)
> > Update sbt-sonatype to 3.9.4 (#3136)
> > Update scalafmt-core to 2.6.2 (#3139)
> > Update scalafmt-core to 2.6.3 (#3152)
> > Update scalafmt-core to 2.6.4 (#3167)
> > Update sparkey to 3.1.0 (#3204)
> > Fix conflicting gcsio dependency (#3180)
>
Re: Scio 0.9.3 released
Posted by Robert Bradshaw <ro...@google.com>.
Thanks for the update!
On Wed, Aug 5, 2020 at 11:46 AM Neville Li <ne...@gmail.com> wrote:
>
> Hi all,
>
> We just released Scio 0.9.3. This bumps Beam SDK to 2.23.0 and includes a lot of improvements & bug fixes.
>
> Cheers,
> Neville
>
> https://github.com/spotify/scio/releases/tag/v0.9.3
>
> "Petrificus Totalus"
>
> There are no breaking changes in this release, but some were introduced with v0.9.0:
>
> See v0.9.0 Migration Guide for detailed instructions.
>
> Improvements
>
> Allow user-supplied filename prefix for smb writes/reads (#3215)
> Refactor SortedBucketTransform into a BoundedSource + reuse merge logic (#3097)
> Add keyGroupFilter optimization to scio-smb (#3160)
> Add error message to BaseAsyncLookupDoFn preconditions check (#3176)
> Add Elasticsearch 5,6,7 add/update alias on multiple indices ops (#3134)
> Add initial update alias op to ES7(#2920)
> Add ScioContext#applyTransform (#3146)
> Allow SCollection#transform name override (#3142)
> Allow setting default name through SCollection#applyTransform (#3144)
> Update 0.9 migration doc and add Bigquery Type read schema documentation(#3148)
>
> Bug Fixes
>
> AvroBucketMetadata should validate keyPath (fix #3038) (#3140)
> Allow union types in non leaf field for key (#3187)
> Fix issue with union type as non-leaf field of smb key (#3193)
> Fix ContextAndArgs#typed overloading issue (#3199)
> Fix error propagation on Scala Future onSuccess callback (#3178)
> Fix ByteBuffer should be readOnly (#3220)
> Fix compiler warnings (#3183)
> Fix JdbcShardedReadOptions.fetchSize description (#3209)
> Fix FAQ typo (#3194)
> Fix scalafix error in SortMergeBucketScioContextSyntax (#3158)
> Add scalafix ExplicitReturnType and ProcedureSyntax rules (#3179)
> Cleanup a few more unused and unchecked params (#3223)
> Use GcpOptions#getWorkerZone instead of deprecated GcpOptions#getZone (#3224)
> Use raw coder in SCollection#applyKvTransform (#3171)
> Add raw beam coder materializer (#3164)
> Avoid circular dep between SCollection and PCollectionWrapper (#3163)
> Remove unused param of internal partitionFn (#3166)
> Remove unused CoderRegistry (#3165)
> Remove defunct scio-bench (#3150)
> Reuse applyTransform (#3162)
> Make multijoin.py python3
> Use TextIO#withCompression (#3145)
>
> Dependency Updates
>
> Update Beam SDK to 2.23.0 (#3197)
> Update dependencies to be inline with 2.23.0 (#3225)
> Update to scala 2.12.12 (#3157)
> Update auto-value to 1.7.4 (#3147)
> Update breeze to 1.1 (#3211)
> Update cassandra-all to 3.11.7 (#3186)
> Update cassandra-driver-core to 3.10.0 (#3195)
> Update commons-lang3 to 3.11 (#3161)
> Update commons-text to 1.9 (#3185)
> Update contributing guidelines with current tools (#3149)
> Update elasticsearch-rest-client, ... to 7.8.1 (#3192)
> Update elasticsearch, ... to 6.8.11 (#3188)
> Update jackson-module-scala to 2.10.5 (#3169)
> Update jna to 5.6.0 (#3156)
> Update magnolify to 0.2.2 (#3154)
> Update mysql-connector-java to 8.0.21 (#3153)
> Update pprint to 0.6.0 (#3203)
> Update protobuf version to 3.11.4 (#3200)
> Update sbt-scalafix to 0.9.18 (#3138)
> Update sbt-sonatype to 3.9.4 (#3136)
> Update scalafmt-core to 2.6.2 (#3139)
> Update scalafmt-core to 2.6.3 (#3152)
> Update scalafmt-core to 2.6.4 (#3167)
> Update sparkey to 3.1.0 (#3204)
> Fix conflicting gcsio dependency (#3180)