Month: May 2020

Posted on

The Pragmatic Programmer: 1st edition vs 20th anniversary edition – what are the major changes?

I might come back and do a more thorough look at each of these differences later, but I wanted to get a rough idea of what were the main changes to each of the chapters in the recently released 20th Anniversary Edition of the classic book, The Pragmatic Programmer.

There’s probably plenty more other changes even in the sections that are common between both editions, but this is a high level overview of the major differences in each chapter, summarized as:

  • New sections added
  • Sections with major changes
  • Sections removed
Posted on

Kafka Streams error: “PolicyViolationException: Topic replication factor must be 3”

I’m creating a Streams app to consume a Topic and do a count with results in a KTable, and I’ve got this error:

2020-05-03 15:23:26,373 [streamsapp1-e7955018-aca6-43ce-8967-abd6c6d238d9-StreamThread-1] ERROR org.apache.kafka.streams.processor.internals.InternalTopicManager [] - stream-thread [main] Unexpected error during topic creation for streamsapp1-KSTREAM-AGGREGATE-STATE-STORE-0000000001-changelog.
Error message was: org.apache.kafka.common.errors.PolicyViolationException: Topic replication factor must be 3
2020-05-03 15:23:26,374 [streamsapp1-e7955018-aca6-43ce-8967-abd6c6d238d9-StreamThread-1] ERROR org.apache.kafka.streams.processor.internals.StreamThread [] - stream-thread [streamsapp1-e7955018-aca6-43ce-8967-abd6c6d238d9-StreamThread-1] Encountered the following unexpected Kafka exception during processing, this usually indicate Streams internal errors:
org.apache.kafka.streams.errors.StreamsException: Could not create topic streamsapp1-KSTREAM-AGGREGATE-STATE-STORE-0000000001-changelog.

I think the clue here is that the app is trying to create an intermediate topic, streamsapp1-KSTREAM-AGGREGATE-STATE-STORE-0000000001-changelog, and is failing because the Replication Factor for the intermediate topic does not match the rf of the source topic. This is described in the Streams docs for option config options here.

Adding config property replication.factor=3 to match the rf of the source topic fixes the issue.

Proudly powered by WordPress