flannel principle And Subnet partition

Docker

Usually , 2 Install... On each physical machine Docker,Docker Respectively in 2 Physical machine ,docker0 The bridge , Its IP The address belongs to the private network segment , For example, the physical machine address is 10.x.x.x,docker0 The address is usually 192.x.x.x. Anything that starts on a physical machine container, Its assigned eth0 The address is 192.x.x.x. however 2 On a separate physical machine container There is no communication between them , Because there is no... In the physical network 192.x.x.x Segment information , It cannot be routed , What's more, physical network as infrastructure , It is also impossible to dynamically add container network related routes .

flannel Problem solved , In fact, that is Deployed on multiple different physical machines (ECS) The container on , How to conduct network communication .

flannel

Subnet partition

flannel No matter which operation mode , Will be on the physical machine docker Modify the configuration information , for example flannel It will be modified after startup /usr/lib/systemd/system/docker.service Inject DOCKER_OPT_BIP="–bip=xx.xx.xx.xx/24" Make the container started on the current machine take this network segment as the address of the container . The specific network segment information is saved in etcd Medium , Each station node above flannel All monitored this ,flannel Read etcd analysis , Then determine your network segment , modify docker To configure .

ETCD Stored in advance flannel The network segment , for example ETCDCTL_API=2 etcdctl mk /atomic.io/network/config '{ "Network": "182.48.0.0/16" }' The configuration format is json Format , Need and flannel Parsing format match . This configuration tells all flannel process ,flannel Where node Containers applied for on ip, Must be within this network segment .

type Config struct {
  Network     ip.IP4Net
  SubnetMin   ip.IP4
  SubnetMax   ip.IP4
  SubnetLen   uint
  BackendType string          `json:"-"`
  Backend     json.RawMessage `json:",omitempty"`
}

flannel Read the network segment configuration and parsing process

  // Fetch the network config (i.e. what backend to use etc..).
  config, err := getConfig(ctx, sm)
  if err == errCanceled {
    
    wg.Wait()
    os.Exit(0)
  }

  func getConfig(ctx context.Context, sm subnet.Manager) (*subnet.Config, error) {
    
  // Retry every second until it succeeds
  for {
    
    config, err := sm.GetNetworkConfig(ctx)
    if err != nil {
    
      log.Errorf("Couldn't fetch network config: %s", err)
    } else if config == nil {
    
      log.Warningf("Couldn't find network config: %s", err)
    } else {
    
      log.Infof("Found network config - Backend type: %s", config.BackendType)
      return config, nil
    }
    select {
    
    case <-ctx.Done():
      return nil, errCanceled
    case <-time.After(1 * time.Second):
      fmt.Println("timed out")
    }
  }
}

# subnet/etcdv2/local_manager.go  Read etcd Network segment information in 
func (m *LocalManager) GetNetworkConfig(ctx context.Context) (*Config, error) {
    
  cfg, err := m.registry.getNetworkConfig(ctx)
  if err != nil {
    
    return nil, err
  }


  // Get  { "Network": "182.48.0.0/16" }, And then parse , Parsed, for example ,182.48.0.0/16 This segment , After the parsing  SubnetMin yes 182.48.1.0,SubnetMax yes 182.48.255.0
  return ParseConfig(cfg)
}

# subnet/etcdv2/registry.go
func (esr *etcdSubnetRegistry) getNetworkConfig(ctx context.Context) (string, error) {
    
  key := path.Join(esr.etcdCfg.Prefix, "config")
  resp, err := esr.client().Get(ctx, key, &etcd.GetOptions{
    Quorum: true})
  if err != nil {
    
    return "", err
  }

  // adopt etcd Get  { "Network": "182.48.0.0/16" }
  return resp.Node.Value, nil
}

flannel Read etcd After the data , Because there are multiple node The existence of , Each station node It needs to be done again according to 16 Bitmask , Determine your own subnet mask , such , Each station node The upper container starts ip There will be no conflict , stay ParseConfig in , There is already logic to determine 16 In case of bit mask, the subnet segment is 24 Mask , In other cases by n+2.24 Bit mask means value , At present node Make a container of ip yes 182.48.{y}.x,x by 0~254. Then there is a problem , It's every one of them node above {y} Definitely not the same configuration .

# main.go
bn, err := be.RegisterNetwork(ctx, &wg, config)
if err != nil {
  log.Errorf("Error registering network: %s", err)
  cancel()
  wg.Wait()
  os.Exit(1)
}

#  The path is related to the architecture 
func (be *UdpBackend) RegisterNetwork(ctx context.Context, wg *sync.WaitGroup, config *subnet.Config) (backend.Network, error) {

    l, err := be.sm.AcquireLease(ctx, &attrs)
    switch err {
    case nil:
    ......
    }
}

func (m *LocalManager) AcquireLease(ctx context.Context, attrs *LeaseAttrs) (*Lease, error) {
  config, err := m.GetNetworkConfig(ctx)
  if err != nil {
    return nil, err
  }

  for i := 0; i < raceRetries; i++ {
    l, err := m.tryAcquireLease(ctx, config, attrs.PublicIP, attrs)
    switch err {
    case nil:
      return l, nil
    case errTryAgain:
      continue
    default:
      return nil, err
    }
  }

  return nil, errors.New("Max retries reached trying to acquire a subnet")
}

tryAcquireLease Function , First from etcd Read the old configuration , Check whether the current node There is subnet configuration information , Usually if it's just flannel Restart and start successfully before , Then the configuration will be read at this time

flannel in , Deposit in etcd The path of the format is $PREFIX/subnets/,key Is a network segment ,value by node Of ip.

$ETCDCTL_API=2 etcdctl ls /coreos.com/network/subnets
/coreos.com/network/subnets/182.48.56.0-24
/coreos.com/network/subnets/182.48.17.0-24


$ETCDCTL_API=2 etcdctl get /coreos.com/network/subnets/182.48.56.0-24
{
    "PublicIP":"11.238.116.75"}

func (m *LocalManager) tryAcquireLease(ctx context.Context, config *Config, extIaddr ip.IP4, attrs *LeaseAttrs) (*Lease, error) {
    
  leases, _, err := m.registry.getSubnets(ctx)
  if err != nil {
    
    return nil, err
  }

  // Try to reuse a subnet if there's one that matches our IP
  // Go through all the acquired key, Judge it value Whether it's me node Of ip, Indicates that there is configuration information , Just use it 
  if l := findLeaseByIP(leases, extIaddr); l != nil {
    
    // Make sure the existing subnet is still within the configured network
    if isSubnetConfigCompat(config, l.Subnet) {
    
      log.Infof("Found lease (%v) for current IP (%v), reusing", l.Subnet, extIaddr)
    }
}

tryAcquireLease Function , If there is no current node Configuration of , Just create a configuration . Prevent multiple node Of flannel The method of creating the same network segment concurrently is also very simple , The parameters are used prevExist=false Prevent simultaneous write success ; After failure , The external loop creates the subnet again , Until success .

  if sn.Empty() {
    
    // no existing match, grab a new one
    sn, err = m.allocateSubnet(config, leases)
    if err != nil {
    
      return nil, err
    }
  }

  exp, err := m.registry.createSubnet(ctx, sn, attrs, subnetTTL)
  switch {
    
  case err == nil:
    log.Infof("Allocated lease (%v) to current node (%v) ", sn, extIaddr)
    return &Lease{
    
      Subnet:     sn,
      Attrs:      *attrs,
      Expiration: exp,
    }, nil
  case isErrEtcdNodeExist(err):
    return nil, errTryAgain
  default:
    return nil, err
  }

thus , Multiple physical machines (Node) Upper flannel process , from 182.48.0.0/16 In this configured network segment , Generated its own subnet , for example 182.48.1.0/24,182.48.2.0/24, And save the configuration in etcd above , To ensure that other nodes do not occupy . And use this subnet , Revised Docker Launch parameters , Yes Docker The container to be started is subject to the subnet .