Since committing offsets waits until the outstanding message queue is empty, if acknowledging all of them takes anywhere close to the commit offset timeout (5 sec by default), the commit process will fail. Let’s throw a hypothesis for what might be happening here: It makes sense, it wants to know what records have been successfully produced to commit their offsets, or whatever reference it is used (e.g., GTID, binlog position). One key point here is that the commit process waits until the outstanding messages queue is empty to proceed. Simple, right? Well, maybe not, but hopefully clear enough. Finally, coming from a different thread, every (by default, 60 seconds) the WorkerSourceTask gets its commitOffsets method invoked, which triggers the process that results in the errors seeing the previous section.When it gets acknowledged by the Kafka cluster, a callback is involved, which will remove the acknowledged record from the pending queue. If you know how KafkaProducer instances work, they don’t send the record over the network when send returns. Every SourceRecord is then send using the KafkaProducer. Those SourceRecord are stored in a sort of outstanding messages queue, before being provided to a KafkaProducer instance.In every iteration of the loop, the WorkerSourceTask polls many SourceRecord instances provided by the source connector plugin (Debezium in this case).The execute method is, basically, a big while loop that runs indefinitely, unless the task state changes to stop or paused.
0 Comments
Leave a Reply. |