Kafka Message Key Hashing

 In Kafka each event message contains an optional key and a value.

  • key == null : messages are distributed evenly across partitions in a topic (a round-robin strategy).
  • key != null : All messages that share same key will always be sent and stored in the same partition.

** A key can be anything to identify a message (a string, numeric value, binary value etc.).

Key Hashing is the process of determining the mapping of a key to a partition. A Kafka partitioner is a code logic that takes a record and determines to which partition to send it into.

** In the default Kafka partitioner, the keys are hashed using murmur2 algorithm (murmur hash).

targetPartition = Math.abs(Utils.murmur2(keyBytes) % (numPartitions - 1))

Example Flow:

  • Message 1: account_id = 12345 hash(12345) % 2 might result in 0 -> Partition 0
  • Message 2: account_id = 67890 hash(67890) % 2 might result in 1 -> Partition 1
  • Message 3: account_id = 12345 (same as Message 1)hash(12345) % 2 still results in 0 -> Partition 0 again, ensuring that all messages with account_id = 12345 are processed in the same order.

Benefits of Keyed Partitioning:

  • Message Ordering: By sending all messages with the same key to the same partition, Kafka ensures that they are processed in the order they were sent.
  • Data Locality: Grouping related messages into the same partition can also improve performance by ensuring that they are processed together.

Yorumlar

Yorum Gönder

Bu blogdaki popüler yayınlar

JUnit 5 Parameterized Tests

  JUnit 5 provides parameterized tests, which let you run a test multiple times with different arguments. This is especially useful when you want to test a function with various inputs without writing repetitive test cases. Basic Structure of a Parameterized Test To create a parameterized test, use the @ParameterizedTest annotation. You also need a source of parameters, which JUnit provides through different annotations like @ValueSource , @CsvSource , @CsvFileSource. ValueSource : Supplies a single array of primitive types or strings. @Service public class AccountValidationService { // Example rules: // - Account number must be 10 digits // - Must start with "ACC" public boolean validateAccountNumber (String accountNumber) { if (accountNumber == null ) { return false ; } if (!accountNumber.startsWith( "ACC" )) { return false ; } if (accountNumber.length() != 10 ) { ...
Devamı

Transactional Outbox Pattern

Resim
Dual Write Problem The dual write problem means it’s hard to keep different data sources working together when you write or save things. We need to make sure that all the writes are done correctly and the same in all places. If one write operation fails or only partially succeeds, it can lead to data integrity issues. Dual Write Problem Diagram The Transactional Outbox pattern helps solve the Dual Write Problem. It adds a middle step to keep writing actions separate. How the Pattern Works Let’s assume that we also need to send an event to the notification service when a customer is created. It doesn’t immediately tell another service. Instead, it puts the important details in a special table called an outbox table. Customer savedCustomer = customerRepository.save( new Customer (customerDto.username(), customerDto.email())); //customerProducerService.sendCustomerCreatedEvent(savedCustomer); OutboxEvent outboxEvent = new OutboxEvent (); outboxEvent.setEventPayloa...
Devamı