Logger Log specification

Request response log ：

• Parameters to be recorded ： Request method , Request routing , Request parameters . As a api What you need token Or other authorization Information , response time , Response state （http status).
• The format of log recording is customized

resolvent ：

• Request response log At present, we are more in front Nginx That floor to record , It's not difficult to customize one by yourself ,ctx.logger That piece should override the default format Of .

Logs can be encapsulated and parsed ：

• Directly introducing egg-logger after , The native log output does not meet the basic requirements , And the format is not uniform .

Full link configurable ：

request - header in , Different companies , The unique flag of the whole link that may be used is different .
Some companies use traceId、 And some use request-id, And so on , If you need to change the source code to complete , Whether the deployment of production is an obstacle ？

resolvent ：

• Full link tagging is configurable This belongs to tracelog Category , This is actually related to the internal structure of the enterprise , Need to be customized . We have an eagle eye system inside , And the corresponding plug-ins , If you are interested, you can follow up and promote this RFC

Elegant use egg-logger journal

• The format of the request log

Recorded parameters ： Request method , Request routing , Request parameters （query,body;param Collection is unlikely ？), As a api What you need token Or other authorization Information , response time , Response state （http status)

• Full link tagging is configurable

request - header in , Different companies , The unique flag of the whole link that may be used is different .
Some companies use traceId、 And some use request-id, And so on , If you need to change the source code to complete , Whether the deployment of production is an obstacle ？

• Solution ： Use koa-log4 To reproduce the formed log system
• Turn off the egg-logger

config.logger.level = 'NONE' and config.logger.consoleLevel = 'NONE'

• Native log output cfork Output a third-party library to stdout Of , This must be hard to control , But it will not be written to file Inside , It doesn't affect your analysis .

• Full link routing

• Try to modify the log format through middleware ：

Node Network programming

It has event driven , Non blocking , Single thread and other features .

Node Provides net,dgram,http,https this 4 A module , They are used to deal with TCP,UDP,HTTP,HTTPS, Suitable for server and client .

structure TCP application （ Transmission control protocol ）

TCP Transmission control protocol

OSI： Open system interconnection communication reference model . A conceptual model , Proposed by the international organization for standardization , A standard framework that attempts to interconnect computers worldwide as networks .

Seven layers ：

The physical layer （ Network physical hardware ）,

Data link layer （ Network specific link interface ）,

The network layer （IP）,

Transport layer （TCP/UDP）,

The session layer （ Communication connection / Maintain conversation ）,

The presentation layer （ encryption / Decryption, etc ）,

application layer （HTTP,SMTP,IMAP etc. ）.

TCP: Connection oriented protocol , The salient feature is that... Is required before transmission 3 Two handshakes form a conversation .

In the process of creating a session , The server and the client provide a socket respectively , These two sockets together form a connection . The connection between the server and the client is realized through sockets .

socket： Socket , A program that enables an application to read, write, and send and receive communications protocols and data .

TCP The creation process and link reduction process are performed by TCP/IP The protocol stack is created automatically .TCP The socket is writable and readable stream object .

TCP Service events ：

Server events , Connection event

TCP There are certain optimization strategies for small datagrams in the network ：Nagle Algorithm . If only one byte of content is sent at a time without optimization , The network will be filled with packets with only a few valid data , Waste network resources .

Nagle The algorithm aims at this situation , When the data in the buffer reaches a certain amount or a certain time, it will be sent out , So little data treasure will be Nagle Algorithm merging , To optimize the network . Although this optimization makes the network bandwidth used effectively , But the data may be delayed .

structure UDP service （ User packet protocol ）

A socket can be connected to more than one socket UDP Service communications .

Provide simple unreliable information transmission from plane to transaction , In the case of poor network, there is a serious problem of packet loss . No need to link , Low resource consumption , Fast and flexible processing .

structure HTTP service

Transport layer http,smtp

HTTP: Hypertext transfer protocol

structure websocket service

stay websocket Before , The most efficient way to communicate between web client and server is comet technology , Realization comet The technical detail is to use long polling or iframe flow , The principle of long polling is that the client sends a request to the server , The server only disconnects when it times out or there is corresponding data , The client reissued the request after receiving data or timeout , This request behavior has a long tail .

structure web application

With event driven and V8 High performance , Become a leader in the amount of services .

• It's on the front and back JS When , Across HTTP When communicating ：

No need to switch locales , Some knowledge will not be lost due to the switching of locale , Good context consistency .

data （ because JSON） It can be used directly across the front and back ends .

Business , You can choose the front end or the back end with light weight , Language is interlinked , The price is small .

• request The event occurred on a network connection

• Specific business needs

The judgment of the request method ,URL Path analysis of ,Cookie Parsing ,Basic authentication , Form data analysis , Any format file upload processing .

BASIC authentication ： One used to allow Web browser Or other client programs provide... On request user name and password A login authentication method of formal identity certificate

Introduction to higher-order functions ： May be infinitely complex , However, as long as the above function is returned as a parameter for the total result , Pass to CreateServer（） Just listen to the listener

• RESTful class web The method of request in the service is very important , Because it will determine the operation behavior of resources .

PUT Represents a new resource ,POST Indicates updating a resource ,GET Indicates viewing a resource ,DELETE Indicates deleting a resource .

• Path resolution
• cookie

The impact of performance ： Once the server has attached the settings to the client Cookie The intent of the , Unless Cookie Be overdue , Otherwise, the client will send these every request cookie To the server ,cookie Too much will lead to a larger header .

YSlow Performance rules ：

Reduce Cookie Size

Use different domain names for static components , Convert the domain name IP Need to carry out DNS Inquire about , One more domain name will be one more time DNS Inquire about

Reduce DNS Inquire about

• session

How to map the data in each client and server one by one ：

be based on cookie To map users and data

The server is on session, Will agree on a key value as session The password of , The server did not check the user request cookie Middle carry , Will generate a unique value that does not duplicate , And set the timeout .

The correspondence between browser and server-side data is realized by querying string

risk ： Because you want to send the address in the address bar to another person , Then he will be the same as you .cookie The scheme cannot take effect after changing the browser or computer , It's safer .

• session Centralized tools ：redis,memcached etc. ,Node Processes do not need to maintain data objects internally , Garbage collection and memory limitations , And the cache expiration strategy of the cache design is more reasonable .
• Reasons for using third-party caching

Node Maintain a long link with the cache service , Instead of frequent short connections , The delay caused by handshake only affects initialization .

The cache directly stores and accesses data in memory .

Caching services are usually associated with Node Processes run on the same machine or in the same computer room , Network speed is less affected .

session You need to get... Asynchronously .

• session Security

Sign the value with the private key ：

var sign=function(val,secret){
    
  return val+'.'+crypto.createHmac('sha256'.secret).update(val).digest('base64').replace(/\=+$/,'');
};

• Cached rules

add to expire or cache-control To the header

To configure ETags

Give Way ajax Caching

• Condition request

Conditional requests will be made to detect whether local resources are available , In ordinary GET The request message is accompanied by If-Modified-Since Field .

If the server does not have a new version , In response to a 304 The status code , The client uses the local version .

If the server has a new version , Send the new content to the client , Client abandons local version .

//  Check whether local resources are available 

var handle = function(req, res) {
    
  fs.stat(filename, function(err, stat) {
    
    var lastModified = stat.mtime.toUTCString();
    if (lastModified === req.headers['if-modified-since']) {
    
      res.writeHead(304, 'Not Modified');
      res.end();
    } else {
    
      fs.readFile(filename, function(err, file) {
    
        var lastModified = stat.mtime.toUTCString();
        res.setHeader('Last-Modified', lastModified);
        res.writeHead(200, 'OK');
        res.end();
      });
    }
  });
};

• Time stamp flaw

The time stamp of the file is changed, but the content is not necessarily changed .

The timestamp can only be accurate to seconds , Frequently updated content will not take effect .

HTTP/1.1 ETag To solve .

• Cache update mechanism

Every time I release , Follow... In the path web Version number of the application .

Every time I release , The path that follows the contents of the file Hash value .

• Cognitive return add account password

res.setHeader('WWW-Authenticate','Basic realm="Secure Area"');

• Judge whether there is content

var hasBody=function(req){
    
  return 'transfer-encoding' in req.headers || 'content-length' in req.headers;
}

//  Stream processing data
function(req, res) {
    
  if (hasBody(req)) {
    
    var buffers = [];
    req.on('data', function() {
    
      buffers.push(chunk);
    });
    req.on('end', function() {
    
      req.rawBody = Buffer.concat(buffers).toString();
      handle(req, res);
    });
  } else {
    
    handle(req, res);
  }
};

Judge content-type：

var mime=function(req){
  var str=req.headers['content-type']||'';
  return str.split(';')[0];
}

analysis xml file ：

var xml2js=require('xml2js');

var handle = function(req,res){
  if(mime(req)==='application/xml'){
    xml2js.parseString(req.rawBody,function(err,xml){
      if(err){
        res.writeHead(400);
        res.end('Invalid xml');
        return;
      }
      req.body=xml;
      todo(req,res);
    })
  }
}

• Because it's a file upload , So it's like a normal form ,json,xml In that way, receiving content first becomes unacceptable in the way of parsing .

When receiving data of unknown size ：

function (req,res){
  if(hasBody(req)){
    var done=function(){
      handle(req,res);
    }
    if(mime(req)==='application/json'){
      parseJSON(req,done);
    }else if(mime(req)==='application/xml'){
      parseXML(req,done);
    }else if(mime(req)==='multipart/form-data'){
      parseMulipart(req,done);
    }
  }else{
    handle(req,res);
  }
}

take req The stream object is directly handed over to the corresponding parsing method , There are parsing methods to process the uploaded content by yourself , Or accept content stored in memory , Or stream away .

• formiable

Parsing message based on stream understanding , Write the received file to the temporary folder of the system , And return the corresponding path

// formidable  Parsing message based on streaming processing 
var formidable = require('formidable');

var upload = function (req, res) {
  if (hasBody(req)) {
    if (MimeType(req) == 'mulipart/form-data') {
      var form = new formidable.IncomingForm();
      form.parse(req, function (err, fields, files) {
        req.body = fields;
        req.files = files;
        handle(req, res);
      });
    }
  } else {
    handle(req, res);
  }
}

• Memory limit status

Limit the size of uploaded content , Once the limit is exceeded, stop accepting data , And accordingly 400 Status code .

Through flow analysis , Direct data flow to disk ,Node Only keep small data such as file path .

//  Limit the size of the file 

var bytes = 1024;
var limit = function (req, res) {
  var received = 0;
  var len = req.headers['content-length'] ? parseInt(req.headers['content-length'], 10) : null;

  if (len && len > bytes) {
    res.writeHead(413);
    res.end();
    return;
  }

  //limit
  req.on('data', function (chunk) {
    received += chunk.length;
    if (received > bytes) {
      req.destory();
    }
  });
  handle(req, res);
}

For no content-length Request message for , At every data Just judge in the event ,but,json Document and XML The file may not complete parsing .

• CSRF

//  testing 
var validateCSRF = function (req, res) {
  var token = req.session._csrf || (req.session._csrf = generateRandom(24));
  var _csrf = req.body._csrf;
  if (token !== _csrf) {
    res.writeHead(403);
    res.end(' Blocking access ');
  } else {
    handle(req, res);
  }
}

// csrf add to 

var generateRandom = function (len) {
  return crypto.randomBytes(Math.ceil(len * 3 / 4)).toString('base64').slice(0, len);
}

• Route resolution

MVC The main idea of the model is to separate business logic according to responsibilities

controller

Model

View

Working mode ：

Route resolution , The behavior calls the relevant model for data operation , After the data operation, the base note and related data are used to render the page. , Output to client .