nodejs link redis/mysql Connection pool

redis It's a single thread job , So whether the query task is sent by one link or multiple links ,redis Is a serial execution . And return to the client through the current link .nodejs Accept redis After the return of , Parallel or not , You have to wait until the main thread is idle to process the data returned by the server one by one .

Look again mysql~

mysql Not a single threaded service , Multiple query requests can be processed in parallel .

mysql A separate thread query is created for each link .redis The data is basically in memory ,mysql Will read a large number of disks I/O, Multithreading is faster .

however nodejs It's single threaded . But it calls I/O Instructions, etc. are done through another thread ,I/O After the instruction is completed, give the main thread a small task , Callback function .

nodejs The main thread is a , however IO There will be multiple threads .

nodejs Use multiple connections to connect mysql. Multiple connections require connection pooling , Having a connection pool avoids the consumption of creating and destroying each connection .

Sum up ：nodejs + mysql There's nothing wrong with using thread pools .nodejs + redis Just a single connection is enough .

So there is the connection management module ,egg-redis.

redis performance

The reason for the error ：redis client Business code and redis client Of I/O performance .

redis client It adopts the single link mode , The bottom layer adopts non blocking network I/O,

tuning :pipeline,script

Non dependent batch requests adopt pipeline.

redis High performance is reflected in the processing capacity of the server , But bottlenecks often appear on the client side , So enhance the client I/O Multi client is the high concurrency solution .

ioredis

Performance centered , A fully functional , Support Redis >= 2.6.12 and (Node.js >= 6).

npm install ioredis

basic usage

var Redis = require('ioredis');
var redis = new Redis();
 
redis.set('foo', 'bar');
redis.get('foo', function (err, result) {
    
  console.log(result);
});
 
// Or using a promise if the last argument isn't a function
redis.get('foo').then(function (result) {
    
  console.log(result);
});
 
// Arguments to commands are flattened, so the following are the same:
redis.sadd('set', 1, 3, 5, 7);
redis.sadd('set', [1, 3, 5, 7]);
 
// All arguments are passed directly to the redis server:
redis.set('key', 100, 'EX', 10);

Connect to Redis

new Redis()       // Connect to 127.0.0.1:6379
new Redis(6380)   // 127.0.0.1:6380
new Redis(6379, '192.168.1.1')        // 192.168.1.1:6379
new Redis('/tmp/redis.sock')
new Redis({
    
  port: 6379,          // Redis port
  host: '127.0.0.1',   // Redis host
  family: 4,           // 4 (IPv4) or 6 (IPv6)
  password: 'auth',
  db: 0
})

A special connection method ：

// Connect to 127.0.0.1:6380, db 4, using password "authpassword":
new Redis('redis://:[email protected]:6380/4')

pub/sub Release / subscribe

publish/subscribe

redis.on Monitor events

var Redis = require('ioredis');
var redis = new Redis();
var pub = new Redis();
redis.subscribe('news', 'music', function (err, count) {
    
  //  Now we subscribe to news and music Two channels 
  // `count`  Represents the serial number of our current subscription 
 
  pub.publish('news', 'Hello world!');
  pub.publish('music', 'Hello again!');
});
 
redis.on('message', function (channel, message) {
    
  //  Received a message  Hello world! from channel news
  //  Received a message  Hello again! from channel music
  console.log('Receive message %s from channel %s', message, channel);
});

//  Another event is called `messageBuffer`, and message It's the same 
//  return buffer, Alternative string 
redis.on('messageBuffer', function (channel, message) {
    
  // Both `channel` and `message` are buffers.
});

PSUBSCRIBE Release

redis.psubscribe('pattern', function (err, count) {
    });
redis.on('pmessage', function (pattern, channel, message) {
    });
redis.on('pmessageBuffer', function (pattern, channel, message) {
    });

When a client issues a subscription or PSUBSCRIBE when , The connection will be put into “ subscribe ” Pattern . here , Only the command to modify the subscription set is valid . When the subscription set is empty , The connection will return to normal mode .

If you send a normal command in subscription mode , Only a new connection will be opened .

Processing binary data ：

redis.set('foo', Buffer.from('bar'));

Get the data in the cache ：

redis.getBuffer('foo', function (err, result) {
    
  // result is a buffer.
});

The Conduit Pipelining

send out 5 More than orders , You can use the pipeline to put commands into a queue in memory , Then send to redis, This performance can be improved by 50-300.

These commands are in the memory queue of Li Amin , And cooperate with exec Method to execute .

var pipeline = redis.pipeline();
pipeline.set('foo', 'bar');
pipeline.del('cc');
pipeline.exec(function (err, results) {
    
  // `err`  Always null, and `results`  Array of responses 
  // corresponding to the sequence of queued commands.
  // Each response follows the format `[err, result]`.
});
 
//  You can even chain call commands 
redis.pipeline().set('foo', 'bar').del('cc').exec(function (err, results) {
    
});
 
// `exec`  Also return promise
var promise = redis.pipeline().set('foo', 'bar').get('foo').exec();
promise.then(function (result) {
    
  // result === [[null, 'OK'], [null, 'bar']]
});

Each chained command can also have a callback , When the command is answered, it will be called ：

redis.pipeline().set('foo', 'bar').get('foo', function (err, result) {
    
  // result === 'bar'
}).exec(function (err, result) {
    
  // result[1][1] === 'bar'
});

In addition to adding commands to the pipeline queue separately , You can also pass a set of commands and parameters to the constructor ：

redis.pipeline([
  ['set', 'foo', 'bar'],
  ['get', 'foo']
]).exec(function () {
     /* ... */ });

length Properties of ：

const length = redis.pipeline().set('foo', 'bar').get('foo').length;
// length === 2

Business ：

redis.multi().set('foo', 'bar').get('foo').exec(function (err, results) {
    
  // results === [[null, 'OK'], [null, 'bar']]
});

If there is a syntax error in the command chain of the transaction （ for example , Wrong number of parameters , Wrong command name, etc ）, No command will be executed , And return error ：

redis.multi().set('foo').set('foo', 'new value').exec(function (err, results) {
    
  // err:
  // { [ReplyError: EXECABORT Transaction discarded because of previous errors.]
  // name: 'ReplyError',
  // message: 'EXECABORT Transaction discarded because of previous errors.',
  // command: { name: 'exec', args: [] },
  // previousErrors:
  // [ { [ReplyError: ERR wrong number of arguments for 'set' command]
  // name: 'ReplyError',
  // message: 'ERR wrong number of arguments for \'set\' command',
  // command: [Object] } ] }
});

In terms of interfaces ,multi The difference from the pipeline is , When specifying a callback for each link command , The queued state will be passed to the callback instead of the result of the command ：

redis.multi({
     pipeline: false });
redis.set('foo', 'bar');
redis.get('foo');
redis.exec(function (err, result) {
    
  // result === [[null, 'OK'], [null, 'bar']]
});

redis.multi([
  ['set', 'foo', 'bar'],
  ['get', 'foo']
]).exec(function () {
     /* ... */ });

The pipeline supports inline transactions , This means that you can group a subset of commands in the pipeline into transactions ：
redis.pipeline().get('foo').multi().set('foo', 'bar').get('foo').exec().get('foo').exec();

Lua Script

ioredis Support all script commands , for example EVAL,EVALSHA and SCRIPT.

ioredis It's a public one defineCommand Method , Make scripting easier to use ：

var redis = new Redis();
 
// This will define a command echo:
redis.defineCommand('echo', {
    
  numberOfKeys: 2,
  lua: 'return {KEYS[1],KEYS[2],ARGV[1],ARGV[2]}'
});
 
// Now `echo` can be used just like any other ordinary command,
// and ioredis will try to use `EVALSHA` internally when possible for better performance.
redis.echo('k1', 'k2', 'a1', 'a2', function (err, result) {
    
  // result === ['k1', 'k2', 'a1', 'a2']
});
 
// `echoBuffer` is also defined automatically to return buffers instead of strings:
redis.echoBuffer('k1', 'k2', 'a1', 'a2', function (err, result) {
    
  // result[0] equals to Buffer.from('k1');
});
 
// And of course it works with pipeline:
redis.pipeline().set('foo', 'bar').echo('k1', 'k2', 'a1', 'a2').exec();

If the number of keys cannot be determined when defining the command , You can omit numberOfKeys attribute , And pass the number of keys as the first parameter when calling the command ：

redis.defineCommand('echoDynamicKeyNumber', {
    
  lua: 'return {KEYS[1],KEYS[2],ARGV[1],ARGV[2]}'
});
 
// Now you have to pass the number of keys as the first argument every time
// you invoke the `echoDynamicKeyNumber` command:
redis.echoDynamicKeyNumber(2, 'k1', 'k2', 'a1', 'a2', function (err, result) {
    
  // result === ['k1', 'k2', 'a1', 'a2']
});

Redis standard

cluster The constructor takes two arguments ,

• The first parameter is the list of cluster nodes connected to , There is no need to enumerate all cluster nodes , But if one node cannot access, the client will try the next node , And the client will automatically discover other nodes when connecting at least one node .

• The second parameter is the option

  • clusterRetryStrategy: When no node is started to be accessible ,clusterRetryStrategy Will call , Return a number ,ioredis Will try at the specified delay （ Milliseconds ） Then reconnect to the boot node from scratch , Otherwise, no boot node available error will be returned .

  //  The default value of the option 
  function (times) {
        
    var delay = Math.min(100 + times * 2, 2000);
    return delay;
  }
  

  • enableOfflineQueue： Class like enableOfflineQueue Options Redis.
  • enableReadyCheck: When enabled , Only in cluster info A command that reports that the cluster is ready to process commands is issued “ be ready ” event . otherwise , It will be “connect” Send out immediately after sending out .
  • scaleReads： Configure the location where the read query is sent .
  • maxRedirections： When a cluster related error is received （ for example MOVED, etc. ） when , The client redirects the command to another node . This option limits the maximum redirection allowed when sending commands . The default value is .ASK``CLUSTERDOWN``16
  • retryDelayOnFailover： If the target node is disconnected when sending the command ,ioredis Will retry after the specified delay . The default value is 100. You should make sure that retryDelayOnFailover * maxRedirections > cluster-node-timeout Ensure that no commands fail during failover .
  • retryDelayOnClusterDown： When the cluster shuts down , All orders will be rejected , Error for CLUSTERDOWN. If this option is numeric （ By default 100）, Then the client will at the specified time （ In Milliseconds ） Then resend the command
  • retryDelayOnTryAgain： If this option is a number （ By default 100）, Then the client will TRYAGAIN At the appointed time （ In Milliseconds ） Then resend the rejected command .
  • redisOptions：Redis The default options passed to the constructor when connecting to a node .
  • slotsRefreshTimeout： The number of milliseconds before the timeout occurred when refreshing slots from the cluster （ The default value is 1000）
  • slotsRefreshInterval： The number of milliseconds between each automatic slot refresh （ Default 5000）

  Publish and subscribe in cluster mode ：（ It is consistent with the publication and subscription in independent mode ）

  sub.on('message', function (channel, message) {
        
    console.log(channel, message);
  });
   
  sub.subscribe('news', function () {
        
    pub.publish('news', 'highlights');
  });