You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@pekko.apache.org by "pjfanning (via GitHub)" <gi...@apache.org> on 2023/06/10 18:20:35 UTC
[GitHub] [incubator-pekko-connectors] pjfanning opened a new pull request, #167: [DRAFT] add scala3 support for s3
pjfanning opened a new pull request, #167:
URL: https://github.com/apache/incubator-pekko-connectors/pull/167
relates to #166
* have grabbed some code from futiles lib to remove the dependency (which does not yet support scala 3)
* this will need to acknowledged in our license - futiles is Apache licensed
* Scala 3 is stricter about private access and it looks like we'll need to make some pekko-http classes public.
* `Content-Length`
* `Content-Type`
* A couple more compile problems also
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@pekko.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@pekko.apache.org
For additional commands, e-mail: notifications-help@pekko.apache.org
[GitHub] [incubator-pekko-connectors] pjfanning commented on a diff in pull request #167: add scala3 support for s3
Posted by "pjfanning (via GitHub)" <gi...@apache.org>.
pjfanning commented on code in PR #167:
URL: https://github.com/apache/incubator-pekko-connectors/pull/167#discussion_r1249754728
##########
s3/src/main/scala/org/apache/pekko/stream/connectors/s3/impl/S3Stream.scala:
##########
@@ -1177,11 +1177,15 @@ import scala.util.{ Failure, Success, Try }
import conf.multipartUploadSettings.retrySettings._
- SplitAfterSize(chunkSize, chunkBufferSize)(atLeastOneByteString)
- .via(getChunkBuffer(chunkSize, chunkBufferSize, maxRetries)) // creates the chunks
- .mergeSubstreamsWithParallelism(parallelism)
+ val source1: SubFlow[Chunk, NotUsed, Flow[ByteString, ByteString, NotUsed]#Repr, Sink[ByteString, NotUsed]] =
+ SplitAfterSize(chunkSize, chunkBufferSize)(atLeastOneByteString)
+ .via(getChunkBuffer(chunkSize, chunkBufferSize, maxRetries)) // creates the chunks
+
+ val source2 = source1.mergeSubstreamsWithParallelism(parallelism)
Review Comment:
code no longer compiles when I start removing the types on these params
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@pekko.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@pekko.apache.org
For additional commands, e-mail: notifications-help@pekko.apache.org
[GitHub] [incubator-pekko-connectors] mdedetrich commented on a diff in pull request #167: add scala3 support for s3
Posted by "mdedetrich (via GitHub)" <gi...@apache.org>.
mdedetrich commented on code in PR #167:
URL: https://github.com/apache/incubator-pekko-connectors/pull/167#discussion_r1249748399
##########
s3/src/main/scala/org/apache/pekko/stream/connectors/s3/impl/S3Stream.scala:
##########
@@ -1177,11 +1177,15 @@ import scala.util.{ Failure, Success, Try }
import conf.multipartUploadSettings.retrySettings._
- SplitAfterSize(chunkSize, chunkBufferSize)(atLeastOneByteString)
- .via(getChunkBuffer(chunkSize, chunkBufferSize, maxRetries)) // creates the chunks
- .mergeSubstreamsWithParallelism(parallelism)
+ val source1: SubFlow[Chunk, NotUsed, Flow[ByteString, ByteString, NotUsed]#Repr, Sink[ByteString, NotUsed]] =
+ SplitAfterSize(chunkSize, chunkBufferSize)(atLeastOneByteString)
+ .via(getChunkBuffer(chunkSize, chunkBufferSize, maxRetries)) // creates the chunks
+
+ val source2 = source1.mergeSubstreamsWithParallelism(parallelism)
Review Comment:
The `source2` here seems to be a leftover from refactoring/figuring out types locally? You should be able to just go
```scala
source1.mergeSubstreamsWithParallelism(parallelism)
.filter(_.size > 0)
.via(atLeastOne)
.zip(requestInfoOrUploadState(s3Location, contentType, s3Headers, initialUploadState))
// etc etc
```
##########
s3/src/main/scala/org/apache/pekko/stream/connectors/s3/javadsl/S3.scala:
##########
@@ -485,8 +486,11 @@ object S3 {
* @return A [[pekko.stream.javadsl.Source]] containing the objects data as a [[pekko.util.ByteString]] along with a materialized value containing the
* [[pekko.stream.connectors.s3.ObjectMetadata]]
*/
- def getObject(bucket: String, key: String): Source[ByteString, CompletionStage[ObjectMetadata]] =
- new Source(S3Stream.getObject(S3Location(bucket, key), None, None, S3Headers.empty).toCompletionStage())
+ def getObject(bucket: String, key: String): Source[ByteString, CompletionStage[ObjectMetadata]] = {
+ val objectSource: SourceToCompletionStage[ByteString, ObjectMetadata] =
Review Comment:
I think this type can be simplified but as before will have to check out locally (also applies for other future instances).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@pekko.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@pekko.apache.org
For additional commands, e-mail: notifications-help@pekko.apache.org
[GitHub] [incubator-pekko-connectors] mdedetrich commented on a diff in pull request #167: add scala3 support for s3
Posted by "mdedetrich (via GitHub)" <gi...@apache.org>.
mdedetrich commented on code in PR #167:
URL: https://github.com/apache/incubator-pekko-connectors/pull/167#discussion_r1249747354
##########
s3/src/main/scala/org/apache/pekko/stream/connectors/s3/impl/S3Stream.scala:
##########
@@ -1177,11 +1177,15 @@ import scala.util.{ Failure, Success, Try }
import conf.multipartUploadSettings.retrySettings._
- SplitAfterSize(chunkSize, chunkBufferSize)(atLeastOneByteString)
- .via(getChunkBuffer(chunkSize, chunkBufferSize, maxRetries)) // creates the chunks
- .mergeSubstreamsWithParallelism(parallelism)
+ val source1: SubFlow[Chunk, NotUsed, Flow[ByteString, ByteString, NotUsed]#Repr, Sink[ByteString, NotUsed]] =
Review Comment:
I presume that this type annotation was auto generated from Intellij and if so it likely can be simplified a bit more. Ill check out this PR and see if I can improve it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@pekko.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@pekko.apache.org
For additional commands, e-mail: notifications-help@pekko.apache.org
[GitHub] [incubator-pekko-connectors] mdedetrich commented on pull request #167: [DRAFT] add scala3 support for s3
Posted by "mdedetrich (via GitHub)" <gi...@apache.org>.
mdedetrich commented on PR #167:
URL: https://github.com/apache/incubator-pekko-connectors/pull/167#issuecomment-1585764282
Since this is only used for testing I am fine with this approach
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@pekko.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@pekko.apache.org
For additional commands, e-mail: notifications-help@pekko.apache.org
[GitHub] [incubator-pekko-connectors] pjfanning commented on pull request #167: add scala3 support for s3
Posted by "pjfanning (via GitHub)" <gi...@apache.org>.
pjfanning commented on PR #167:
URL: https://github.com/apache/incubator-pekko-connectors/pull/167#issuecomment-1616660367
> I'll review this when I get back home and have laptop
@mdedetrich would you have time to review this?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@pekko.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@pekko.apache.org
For additional commands, e-mail: notifications-help@pekko.apache.org
[GitHub] [incubator-pekko-connectors] mdedetrich commented on pull request #167: add scala3 support for s3
Posted by "mdedetrich (via GitHub)" <gi...@apache.org>.
mdedetrich commented on PR #167:
URL: https://github.com/apache/incubator-pekko-connectors/pull/167#issuecomment-1675808632
@pjfanning Is this ready or is it still in draft status?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@pekko.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@pekko.apache.org
For additional commands, e-mail: notifications-help@pekko.apache.org
[GitHub] [incubator-pekko-connectors] mdedetrich commented on pull request #167: add scala3 support for s3
Posted by "mdedetrich (via GitHub)" <gi...@apache.org>.
mdedetrich commented on PR #167:
URL: https://github.com/apache/incubator-pekko-connectors/pull/167#issuecomment-1676831707
> The code works but the type annotations to help the Scala 3 compiler are not ideal.
Ill checkout hte code locally to see what is going on but either way I don't think this is a deal breaker
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@pekko.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@pekko.apache.org
For additional commands, e-mail: notifications-help@pekko.apache.org
[GitHub] [incubator-pekko-connectors] pjfanning commented on pull request #167: add scala3 support for s3
Posted by "pjfanning (via GitHub)" <gi...@apache.org>.
pjfanning commented on PR #167:
URL: https://github.com/apache/incubator-pekko-connectors/pull/167#issuecomment-1675811014
> @pjfanning Is this ready or is it still in draft status?
The code works but the type annotations to help the Scala 3 compiler are not ideal.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@pekko.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@pekko.apache.org
For additional commands, e-mail: notifications-help@pekko.apache.org
[GitHub] [incubator-pekko-connectors] mdedetrich commented on a diff in pull request #167: add scala3 support for s3
Posted by "mdedetrich (via GitHub)" <gi...@apache.org>.
mdedetrich commented on code in PR #167:
URL: https://github.com/apache/incubator-pekko-connectors/pull/167#discussion_r1249756816
##########
s3/src/main/scala/org/apache/pekko/stream/connectors/s3/impl/S3Stream.scala:
##########
@@ -1177,11 +1177,15 @@ import scala.util.{ Failure, Success, Try }
import conf.multipartUploadSettings.retrySettings._
- SplitAfterSize(chunkSize, chunkBufferSize)(atLeastOneByteString)
- .via(getChunkBuffer(chunkSize, chunkBufferSize, maxRetries)) // creates the chunks
- .mergeSubstreamsWithParallelism(parallelism)
+ val source1: SubFlow[Chunk, NotUsed, Flow[ByteString, ByteString, NotUsed]#Repr, Sink[ByteString, NotUsed]] =
+ SplitAfterSize(chunkSize, chunkBufferSize)(atLeastOneByteString)
+ .via(getChunkBuffer(chunkSize, chunkBufferSize, maxRetries)) // creates the chunks
+
+ val source2 = source1.mergeSubstreamsWithParallelism(parallelism)
Review Comment:
I can see that for `source1`, its `source2` that seems redundant (`source2` has no type annotation)?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@pekko.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@pekko.apache.org
For additional commands, e-mail: notifications-help@pekko.apache.org
[GitHub] [incubator-pekko-connectors] mdedetrich commented on a diff in pull request #167: add scala3 support for s3
Posted by "mdedetrich (via GitHub)" <gi...@apache.org>.
mdedetrich commented on code in PR #167:
URL: https://github.com/apache/incubator-pekko-connectors/pull/167#discussion_r1249756816
##########
s3/src/main/scala/org/apache/pekko/stream/connectors/s3/impl/S3Stream.scala:
##########
@@ -1177,11 +1177,15 @@ import scala.util.{ Failure, Success, Try }
import conf.multipartUploadSettings.retrySettings._
- SplitAfterSize(chunkSize, chunkBufferSize)(atLeastOneByteString)
- .via(getChunkBuffer(chunkSize, chunkBufferSize, maxRetries)) // creates the chunks
- .mergeSubstreamsWithParallelism(parallelism)
+ val source1: SubFlow[Chunk, NotUsed, Flow[ByteString, ByteString, NotUsed]#Repr, Sink[ByteString, NotUsed]] =
+ SplitAfterSize(chunkSize, chunkBufferSize)(atLeastOneByteString)
+ .via(getChunkBuffer(chunkSize, chunkBufferSize, maxRetries)) // creates the chunks
+
+ val source2 = source1.mergeSubstreamsWithParallelism(parallelism)
Review Comment:
I can see that for `source1`, its `source2` that seems redundant (`source2` has no type annotation).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@pekko.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@pekko.apache.org
For additional commands, e-mail: notifications-help@pekko.apache.org
[GitHub] [incubator-pekko-connectors] pjfanning merged pull request #167: add scala3 support for s3
Posted by "pjfanning (via GitHub)" <gi...@apache.org>.
pjfanning merged PR #167:
URL: https://github.com/apache/incubator-pekko-connectors/pull/167
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@pekko.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@pekko.apache.org
For additional commands, e-mail: notifications-help@pekko.apache.org
[GitHub] [incubator-pekko-connectors] mdedetrich commented on pull request #167: add scala3 support for s3
Posted by "mdedetrich (via GitHub)" <gi...@apache.org>.
mdedetrich commented on PR #167:
URL: https://github.com/apache/incubator-pekko-connectors/pull/167#issuecomment-1608404609
I'll review this when I get back home and have laptop
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@pekko.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@pekko.apache.org
For additional commands, e-mail: notifications-help@pekko.apache.org
[GitHub] [incubator-pekko-connectors] pjfanning commented on a diff in pull request #167: add scala3 support for s3
Posted by "pjfanning (via GitHub)" <gi...@apache.org>.
pjfanning commented on code in PR #167:
URL: https://github.com/apache/incubator-pekko-connectors/pull/167#discussion_r1249752826
##########
s3/src/main/scala/org/apache/pekko/stream/connectors/s3/impl/S3Stream.scala:
##########
@@ -1177,11 +1177,15 @@ import scala.util.{ Failure, Success, Try }
import conf.multipartUploadSettings.retrySettings._
- SplitAfterSize(chunkSize, chunkBufferSize)(atLeastOneByteString)
- .via(getChunkBuffer(chunkSize, chunkBufferSize, maxRetries)) // creates the chunks
- .mergeSubstreamsWithParallelism(parallelism)
+ val source1: SubFlow[Chunk, NotUsed, Flow[ByteString, ByteString, NotUsed]#Repr, Sink[ByteString, NotUsed]] =
+ SplitAfterSize(chunkSize, chunkBufferSize)(atLeastOneByteString)
+ .via(getChunkBuffer(chunkSize, chunkBufferSize, maxRetries)) // creates the chunks
+
+ val source2 = source1.mergeSubstreamsWithParallelism(parallelism)
Review Comment:
the original code wouldn't compile in scala 3 and splitting the code up has helped to get it to compile
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@pekko.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@pekko.apache.org
For additional commands, e-mail: notifications-help@pekko.apache.org
[GitHub] [incubator-pekko-connectors] mdedetrich commented on a diff in pull request #167: add scala3 support for s3
Posted by "mdedetrich (via GitHub)" <gi...@apache.org>.
mdedetrich commented on code in PR #167:
URL: https://github.com/apache/incubator-pekko-connectors/pull/167#discussion_r1249747354
##########
s3/src/main/scala/org/apache/pekko/stream/connectors/s3/impl/S3Stream.scala:
##########
@@ -1177,11 +1177,15 @@ import scala.util.{ Failure, Success, Try }
import conf.multipartUploadSettings.retrySettings._
- SplitAfterSize(chunkSize, chunkBufferSize)(atLeastOneByteString)
- .via(getChunkBuffer(chunkSize, chunkBufferSize, maxRetries)) // creates the chunks
- .mergeSubstreamsWithParallelism(parallelism)
+ val source1: SubFlow[Chunk, NotUsed, Flow[ByteString, ByteString, NotUsed]#Repr, Sink[ByteString, NotUsed]] =
Review Comment:
I presume that this type annotation was auto generated from Intellij and if so it likely can be simplified a bit more.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@pekko.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@pekko.apache.org
For additional commands, e-mail: notifications-help@pekko.apache.org
[GitHub] [incubator-pekko-connectors] pjfanning commented on pull request #167: add scala3 support for s3
Posted by "pjfanning (via GitHub)" <gi...@apache.org>.
pjfanning commented on PR #167:
URL: https://github.com/apache/incubator-pekko-connectors/pull/167#issuecomment-1616809420
> There seems to be a case of a leftover refactoring (see comments). Also to me it seems like the `legal/GoogleCommonLicense.txt` is unrelated to this PR???
that change is part of this PR - up until now, the only LICENSE extras were for google-common - so the build was using the root LICENSE for the google-common jar - now we have s3 stuff in the root LICENSE, so I need to create a special license for google-common that omits the s3 stuff
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@pekko.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@pekko.apache.org
For additional commands, e-mail: notifications-help@pekko.apache.org