Kafka Streams error: “PolicyViolationException: Topic replication factor must be 3”

I’m creating a Streams app to consume a Topic and do a count with results in a KTable, and I’ve got this error:

2020-05-03 15:23:26,373 [streamsapp1-e7955018-aca6-43ce-8967-abd6c6d238d9-StreamThread-1] ERROR org.apache.kafka.streams.processor.internals.InternalTopicManager [] - stream-thread [main] Unexpected error during topic creation for streamsapp1-KSTREAM-AGGREGATE-STATE-STORE-0000000001-changelog.
Error message was: org.apache.kafka.common.errors.PolicyViolationException: Topic replication factor must be 3
2020-05-03 15:23:26,374 [streamsapp1-e7955018-aca6-43ce-8967-abd6c6d238d9-StreamThread-1] ERROR org.apache.kafka.streams.processor.internals.StreamThread [] - stream-thread [streamsapp1-e7955018-aca6-43ce-8967-abd6c6d238d9-StreamThread-1] Encountered the following unexpected Kafka exception during processing, this usually indicate Streams internal errors:
org.apache.kafka.streams.errors.StreamsException: Could not create topic streamsapp1-KSTREAM-AGGREGATE-STATE-STORE-0000000001-changelog.

I think the clue here is that the app is trying to create an intermediate topic, streamsapp1-KSTREAM-AGGREGATE-STATE-STORE-0000000001-changelog, and is failing because the Replication Factor for the intermediate topic does not match the rf of the source topic. This is described in the Streams docs for option config options here.

Adding config property replication.factor=3 to match the rf of the source topic fixes the issue.

Leave a Reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.