kafka多线程消费异常

kafka CommitFailedException rebalanced poll

问题：

org.apache.kafka.clients.consumer.CommitFailedException: Commit cannot be completed since the group has already rebalanced and assigned the partitions to another member. This means that the time between subsequent calls to poll() was longer than the configured max.poll.interval.ms, which typically implies that the poll loop is spending too much time message processing. You can address this either by increasing the session timeout or by reducing the maximum size of batches returned in poll() with max.poll.records.

原因：

kafka等待消费者返回消费消息成功与否的响应时间超时。

解决方案：

配置参数：session.timeout.ms：是消费者心跳响应的周期，若在这段时间内，没心跳响应，kafka的broker就会移除这个consumer然后rebalance。max.poll.records的大小应该置为一个合理的范围，最好可以在session.timeout.ms的范围内处理完所有的请求。

The timeout used to detect consumer failures when using Kafka’s group management facility. The consumer sends periodic heartbeats to indicate its liveness to the broker. If no heartbeats are received by the broker before the expiration of this session timeout, then the broker will remove this consumer from the group and initiate a rebalance. Note that the value must be in the allowable range as configured in the broker configuration by “group.min.session.timeout.ms” and “group.max.session.timeout.ms”.