flannel principle And TUN Pattern

First ,TUN Pattern See the principle for details https://wonderful.blog.csdn.net/article/details/113105456 , It is usually used for two private networks to cross through the public network . In general, be familiar with TUN The characteristics of the device , Be routed to TUN Data sent by the equipment , Will be Open TUN Of socket Read , Intercept ;write To TUN The data of , Will be TUN Put it back on the protocol stack , Simulated TUN Package receiving operation of the machine where the equipment is located .

technological process

Assuming the current 2 platform node node1 by 11.238.116.75 node2 The physical IP by 11.238.116.73 And start flannel.flannel The process and simpletun The operating mode is very similar
main Function to find the execution Run()

  log.Info("Running backend.")
  wg.Add(1)
  go func() {
    
    bn.Run(ctx)
    wg.Done()
  }()

Run() Function in backend/udp/cproxy_amd64.go To realize .

func (n *network) Run(ctx context.Context) {
    
  defer func() {
    
    n.tun.Close()
    n.conn.Close()
    n.ctl.Close()
    n.ctl2.Close()
  }()

  // one for each goroutine below
  wg := sync.WaitGroup{
    }
  defer wg.Wait()

  wg.Add(1)
  go func() {
    
    //n.tun yes tun equipment 
    //n.conn yes listen Local address of , Used to receive packages from other physical machines overlay message 
    //n.ctl2 It's the control channel , Used to execute control surface messages 
    runCProxy(n.tun, n.conn, n.ctl2, n.tunNet.IP, n.MTU())
    wg.Done()
  }()

   ......
}

func runCProxy(tun *os.File, conn *net.UDPConn, ctl *os.File, tunIP ip.IP4, tunMTU int) {
    
  var log_errors int
  if log.V(1) {
    
    log_errors = 1
  }

  c, err := conn.File()
  if err != nil {
    
    log.Error("Converting UDPConn to File failed: ", err)
    return
  }
  defer c.Close()

  //C.run_proxy  The function is  backend/udp/proxy_amd64.c
  C.run_proxy(
    C.int(tun.Fd()),
    C.int(c.Fd()),
    C.int(ctl.Fd()),
    C.in_addr_t(tunIP.NetworkOrder()),
    C.size_t(tunMTU),
    C.int(log_errors),
  )
}

backend/udp/proxy_amd64.c

void run_proxy(int tun, int sock, int ctl, in_addr_t tun_ip, size_t tun_mtu, int log_errors) {
    
  char *buf;
  struct pollfd fds[PFD_CNT] = {
    
    {
    
      .fd = tun,
      .events = POLLIN
    },
    {
    
      .fd = sock,
      .events = POLLIN
    },
    {
    
      .fd = ctl,
      .events = POLLIN
    },
  };

  exit_flag = 0;
  tun_addr = tun_ip;
  log_enabled = log_errors;

  buf = (char *) malloc(tun_mtu);
  if( !buf ) {
    
    log_error("Failed to allocate %d byte buffer\n", tun_mtu);
    exit(1);
  }

  fcntl(tun, F_SETFL, O_NONBLOCK);

  while( !exit_flag ) {
    
    int nfds = poll(fds, PFD_CNT, -1), activity;
    if( nfds < 0 ) {
    
      if( errno == EINTR )
        continue;

      log_error("Poll failed: %s\n", strerror(errno));
      exit(1);
    }

    if( fds[PFD_CTL].revents & POLLIN )
      process_cmd(ctl);

    // If it is listen The port has readable information , No brain from listen Port read out , write in TUN, such TUN The device can simulate this message as a packet receiving action , At present Node Carry out packet receiving operation . from udp_to_tun Function processing .
    // If it is tun The device has a readable signal , Reference book Node The above container has data sent out , Then we need to judge the purpose ip Where is the address node On , Then send it to the node above flannel process . from tun_to_udp Function processing .
    if( fds[PFD_TUN].revents & POLLIN || fds[PFD_SOCK].revents & POLLIN )
      do {
    
        activity = 0;
        activity += tun_to_udp(tun, sock, buf, tun_mtu);
        activity += udp_to_tun(sock, tun, buf, tun_mtu);

        /* As long as tun or udp is readable bypass poll(). * We'll just occasionally get EAGAIN on an unreadable fd which * is cheaper than the poll() call, the rest of the time the * read/recvfrom call moves data which poll() never does for us. * * This is at the expense of the ctl socket, a counter could be * used to place an upper bound on how long we may neglect ctl. */
      } while( activity );
  }

  free(buf);
}

tun_to_udp function , Read tun The request , Judge its purpose ip, Purpose ip It must be something else node The network segment on the container （ Otherwise, it will not be routed to tun equipment ）. Then the main function of this function is according to the purpose ip The network segment , Find the corresponding node

static int tun_to_udp(int tun, int sock, char *buf, size_t buflen) {
    
  struct iphdr *iph;
  struct sockaddr_in *next_hop;

  ssize_t pktlen = tun_recv_packet(tun, buf, buflen);
  if( pktlen < 0 )
    return 0;
  
  iph = (struct iphdr *)buf;

  // Find a route ,next_hop  Is the purpose ip Where the container is node The physical address of 
  next_hop = find_route((in_addr_t) iph->daddr);
  if( !next_hop ) {
    
    send_net_unreachable(tun, buf);
    goto _active;
  }

  if( !decrement_ttl(iph) ) {
    
    /* TTL went to 0, discard. * TODO: send back ICMP Time Exceeded */
    goto _active;
  }

  // Leave the request intact , adopt UDP The package is sent to the corresponding node Of flannel process 
  sock_send_packet(sock, buf, pktlen, next_hop);
_active:
  return 1;
}

Where does the routing table come from ? Last one flannel principle And Subnet partition in , We mentioned that ,flannel The subnet segment assigned to this machine , Registered to $PREFIX/subnets/ Under the table of contents , that , other flannel The process only needs watch This directory will do .

func (esr *etcdSubnetRegistry) watchSubnets(ctx context.Context, since uint64) (Event, uint64, error) {
    
  key := path.Join(esr.etcdCfg.Prefix, "subnets")
  opts := &etcd.WatcherOptions{
    
    AfterIndex: since,
    Recursive:  true,
  }
  e, err := esr.client().Watcher(key, opts).Next(ctx)
  if err != nil {
    
    return Event{
    }, 0, err
  }

  evt, err := parseSubnetWatchResponse(e)
  return evt, e.Node.ModifiedIndex, err
}

The result of bag grabbing

There is now 2 platform node, Namely A 11.238.116.75 as well as B 11.238.116.73,A Container started in 182.48.56.2,B Container started in 182.48.17.2. stay A In the implementation of Ping 182.48.17.2.

First, in the A Of docker0 Upper grab bag , It can be seen that the message is original 3 Layer head

stay A Of flannel0 Upper grab bag , The source address changes to flannel0 The address of , This is because Docker Will be in node I added snat The rules

Chain POSTROUTING (policy ACCEPT 16250 packets, 1188K bytes)
 pkts bytes target     prot opt in     out     source               destination
  172 11132 MASQUERADE  all  --  *      !docker0  182.48.56.0/24       0.0.0.0/0

because It's from flannel0 get out , natural , The source address is replaced with flannel0 Of IP

stay A Of eth0 Upper grab bag , so ping The original message of the message , Encapsulated in a physical network (Node) Of ip, Sent to 182.48.17.2. Where this container is located Node Of flannel process . stay B Of eth0 Grab the bag , It must be the same message , There are no more screenshots here .
It should be noted that , This message is not actually the original ping message , original ping The message sip Has been changed to A Of flannel0 Of IP Address .(TODO The specific reasons need to be studied )

here ,overlay The message has arrived B Of eth0, And be B Of flannel Process monitoring , And read .B What to do , Is to load the message （ Which contains 3 Layer head Ping message ） Write his tun equipment .tun Equipment is flannel0.

stay B Of flannel0 Upper grab bag ,flannel0