Message queuing best practices

Catalog

Message system role ？

The news is not lost ？

The news is not repeated ？

Message sequencing ？

---

This article talks about Message queuing functions . Let me talk about the best practices of message queuing .

Message system role ？

Message queuing system is mainly composed of three roles ： producer 、 Message queue 、 consumer .

producer ： Produce messages to the message queue .

Message queue ： Store messages produced by producers , Provide information to consumers for consumption .

consumer ： Consume messages stored in the message queue .

The news is not lost ？

To ensure that the message is not lost , We can first consider when messages will be lost .

Producer Perspective

1 The producer sent a message , After that, it doesn't matter , Whether the message queue writes messages or not has nothing to do with itself , Suppose that the message queue does not write messages . The message is lost .

2 Suppose in extreme cases , The producer's network failed , Cause message loss .

If the business requires, the message must not be lost , How do you guarantee that ？

There are synchronous and asynchronous ways to send messages .

Sync ： The producer sends the message to the message queue , Wait until the write confirmation of the message queue , The message is written successfully , Relatively low performance .

asynchronous ： The producer sends the message to the message queue , Go straight back to , If the message queue is written successfully or fails, the producer will be called back , Relatively high performance .

Producers send messages synchronously , Wait for the response from the message queue , If the message queue response fails , retry .

The producer failed to send a message. How many times is the retry threshold appropriate ？

This can be determined through their own business , There is no fixed number of times . Can be encapsulated in the producer client .

If the number of retries reaches the threshold or fails ？

If the number of retries reaches the threshold, it still fails , Can write DB, After that, it will be re sent in the form of scheduled tasks . Of course, the idempotency of messages needs to be considered in the process of execution .

Message queuing

The message queue does not persist the received message , Process to hang , At this point, the producer sends messages asynchronously . Lost message .

To ensure that the message is not lost , Cooperate with producers to use synchronous message sending mechanism . Ensure that once the producer's message is sent successfully , The news will not be lost .

Consumer perspective

Those business scenarios will lose messages ？

1 Consumer consumption to news , submitted offset, The consumer process was suddenly killed , Consumer failed to process message , Lost message .

2 Messages are valid in message queues , Because consumer spending news is too slow , Messages that cause expiration will be uniformly cleaned up , Lost message .

3 Message consumption failed , Enter the retry message queue ; retry 16 Every time it's consumption failure , Enter the dead letter message queue , Without human intervention , Lost message .

How to avoid the above loss of information ？

1 You have to wait for the consumer to successfully process the message , To submit offset site .

2 Accelerate the consumption speed of consumers , Try to consume messages before they expire .

3 For this retry message queue 、 Dead letter message queue , Let's do manual intervention .

The news is not repeated ？

From the producer's point of view ： Due to some abnormal conditions 、 Business retry 、 Code bug wait , Can cause duplicate messages to be sent . Therefore, the producer does not guarantee that the message is not repeated .

From a message queue perspective ： Message queue is just a channel for message transmission , It's a carrier of messages , Message duplication is a business level problem . Therefore, message queuing does not guarantee the non repetition of messages .

From the perspective of consumers ： The news that consumers are queuing up , Because neither producer nor message queue can guarantee that messages are not repeated , We need to ensure idempotency when messages are consumed repeatedly .

How do consumers consume messages from message queues ? 

Producer production message , Each message is written to the message queue , All have offset , This offset Represents the sequence number of the message .

Consumers also consume information according to  offset Sequential messages , After consuming the message, it will be submitted  offset.

How to ensure idempotency when consumers consume messages repeatedly ？

1 If there is only one copy of data in the database , You can read it first , If the data exists , Can be updated ; If the data doesn't exist , Direct write .

2 Add a unique constraint to the database , To ensure that data cannot be written multiple times . To avoid the generation of dirty data .

3 If you need to write Redis, That's OK , Anyway, every time set, Natural idempotence .

Message sequencing ？

How to understand the order of messages ？

From the producer's point of view , The production messages are sequential A->B->C.

From the perspective of message queuing , The stored messages are sequential A->B->C.

From the perspective of consumers , The consumption messages are in order A->B-C.

How to ensure the sequence of messages ？

producer ： Send in synchronization in the same thread .

Message queue ： Ensure that the messages to be sorted are routed to the same queue .

consumer ： Do not adjust the order of messages received , The message of a partition can be processed by only one thread .