[POC - 3] New commit interface #1042
Draft
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
guizmaii
force-pushed
the
comsumer_commit_records_cleaned
branch
from
0fb7fcb
to
a2f446e
Compare
guizmaii
commented
Sep 9, 2023
| .flatMap(offsetBatch => | ||
| commitTransactionWithOffsets( | ||
| offsetBatch, | ||
| consumerGroupMetadata = None // TODO Jules |
There was a problem hiding this comment.
TODO @guizmaii: How to pass this information here?
guizmaii
commented
Sep 9, 2023
| @@ -219,23 +214,16 @@ private[consumer] final class Runloop private ( | |||
| if (streams.isEmpty) ZIO.succeed(fulfillResult) | |||
| else { | |||
| for { | |||
| consumerGroupMetadata <- getConsumerGroupMetadataIfAny | |||
| /*consumerGroupMetadata*/ _ <- getConsumerGroupMetadataIfAny // TODO Jules: Use: how? | |||
There was a problem hiding this comment.
TODO @guizmaii: Find how to pass this information to the transactional producer
guizmaii
force-pushed
the
comsumer_commit_records_cleaned
branch
2 times, most recently
from
da3ed50
to
b7bc8d7
Compare
guizmaii
commented
Sep 9, 2023
| } | ||
| .transduce(Consumer.offsetBatches) |
There was a problem hiding this comment.
TODO @guizmaii: What was .tranduce doing?
guizmaii
force-pushed
the
comsumer_commit_records_cleaned
branch
from
b7bc8d7
to
b1ade41
Compare
guizmaii
commented
Sep 9, 2023
Comment on lines
+11
to
+34
| @inline | ||
| private[zio] def topicPartition(record: ConsumerRecord[_, _]): TopicPartition = | ||
| new TopicPartition(record.topic(), record.partition()) | ||
|
|
||
| private[zio] def deserializeWith[R, K, V]( | ||
| keyDeserializer: Deserializer[R, K], | ||
| valueDeserializer: Deserializer[R, V] | ||
| )(record: ConsumerRecord[Array[Byte], Array[Byte]]): RIO[R, ConsumerRecord[K, V]] = | ||
| for { | ||
| key <- keyDeserializer.deserialize(record.topic(), record.headers(), record.key()) | ||
| value <- valueDeserializer.deserialize(record.topic(), record.headers(), record.value()) | ||
| } yield new ConsumerRecord[K, V]( | ||
| record.topic(), | ||
| record.partition(), | ||
| record.offset(), | ||
| record.timestamp(), | ||
| record.timestampType(), | ||
| record.serializedKeySize(), | ||
| record.serializedValueSize(), | ||
| key, | ||
| value, | ||
| record.headers(), | ||
| record.leaderEpoch() | ||
| ) |
guizmaii
changed the base branch from
comsumer_commit_records_awaitable
to
master
guizmaii
force-pushed
the
comsumer_commit_records_cleaned
branch
from
1ae11f0
to
ad7fb69
Compare
guizmaii
force-pushed
the
comsumer_commit_records_cleaned
branch
from
fc9b7e9
to
2b88dce
Compare
guizmaii
force-pushed
the
comsumer_commit_records_cleaned
branch
from
2b88dce
to
a60db8e
Compare
guizmaii
commented
Sep 10, 2023
| @@ -30,11 +31,11 @@ private[consumer] final class RunloopAccess private ( | |||
| diagnostics: Diagnostics | |||
| ) { | |||
|
|
|||
| private def withRunloopZIO[E]( | |||
There was a problem hiding this comment.
TODO @guizmaii: Put back previous signature
svroonland
reviewed
Sep 23, 2023
| @@ -37,6 +38,14 @@ trait Consumer { | |||
| timeout: Duration = Duration.Infinity | |||
| ): Task[Map[TopicPartition, Long]] | |||
|
|
|||
| def commit(record: ConsumerRecord[_, _]): Task[Unit] | |||
There was a problem hiding this comment.
This is 6 extra methods. I propose we commit only Offset like objects, not entire ConsumerRecords. Semantically that is better anyway, to 'commit an offset' instead of 'commit a record', the latter suggesting you can ack single records.
It would be nice to have a supertype for the Offset and the OffsetBatch, maybe CommittableOffset. Alpakka has Committable for the same concept.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
No description provided.