Steven's Blog
0%

ONOS with Segment Routing

Posted on Edited on

This tutorial is to run Segment Routing application in ONOS on a 2x2 leaf-spine topology built with Mininet.

Experiment Environment

I have run this test successfully on Ubuntu 16.04 with ONOS 1.12 and 1.14 two versions.

Install And Run ONOS

Refer to ONOS Wiki.

Enable Command onos-netcfg

Source this file to use command onos-netcfg.

1
$ sourc $ONOS_ROOT/tools/dev/bash_profile

Or you can simply add the line in .bashrc.

Enable ONOS Applications

Enable necessary applications in ONOS shell.

1
onos> app activate drivers,openflow,netcfghostprovider,segmentrouting

Or you can simply add the following line in .bashrc.

1
export ONOS_APPS='drivers,openflow,netcfghostprovider,segmentrouting'

Install Mininet

Refer to Mininet Github page.

2x2 Leaf-Sping Topology

In this practice, we will run our segment routing application on 2x2 leaf-spine topology as the following picture.

Each leaf switches are attached with two hosts. The two subnets are 10.6.1.0/24 and 10.6.2.0/24.

ONOS Configuration

We are required to configure three network elements in ONOS - devices, ports and hosts.

Devices

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
"devices" : {
    "of:0000000000000191" : {
        "segmentrouting" : {
            "name" : "Spine-R1",
            "ipv4NodeSid" : 101,
            "ipv4Loopback" : "10.6.3.254",
            "routerMac" : "00:00:00:00:03:80",
            "isEdgeRouter" : false,
            "adjacencySids" : []
        },
        "basic" : {
            "driver" : "ofdpa-ovs"
        }
    }
}

To config the switches as segment routers the following settings are needed to do.

  • of:0000000000000191: Datapath ID (DPID) - ONOS uses this ID to configure corresponding switches.
  • segmentrouting: Segment routing application specific parameters
  • name: name of the switch
  • ipv4NodeSid: Node segment ID
  • ipv4Lookback: IP address of the loopback interface of the switch
  • routerMac: MAC address of the loopback interface - In this implementation, this MAC address is used as src/dst MAC address in Ethernet headers for communication to other switches, instead of using the interface-MAC addresses.
  • isEdgeRouter: The flag to determine the edge router
  • adjacencySids: Adjacency segment ID - This is not used in this practice.
  • basic: Basic parameters
  • driver: Driver of the switch

Ports

1
2
3
4
5
6
7
8
9
10
"ports" : {
    "of:0000000000000101/3" : {
        "interfaces" : [
            {
                "ips" : [ "10.6.1.254/24" ],
                "vlan-untagged" : 20
            }
        ]
    }
}

The following is needed to configure links between segment routers and hosts.

  • of:0000000000000101/3: Datapath Id and the port number
  • ips: IP of the host which is connected to the switch and it’s subnet
  • vlan-untagged: VLAN tag number

Hosts

1
2
3
4
5
6
7
8
"hosts" : {
    "00:00:00:00:00:01/-1" : {
        "basic": {
            "ips": ["10.6.1.1"],
            "locations": ["of:0000000000000101/3"]
        }
    }
}

The following is needed to configure hosts.

  • 00:00:00:00:00:01/-1: Host MAC address and VLAN tag number - In this case, -1 means no VLAN tag
  • ips: Host IP
  • locations: The datapath and port number to which the host attaches

The configuration file are shown below.

Configuration Fileview raw
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
{
  "devices" : {
    "of:0000000000000191" : {
      "segmentrouting" : {
        "name" : "Spine-R1",
        "ipv4NodeSid" : 101,
        "ipv4Loopback" : "10.6.3.254",
        "routerMac" : "00:00:00:00:03:80",
        "isEdgeRouter" : false,
        "adjacencySids" : []
      },
      "basic" : {
        "driver" : "ofdpa-ovs"
      }
    },
    "of:0000000000000192" : {
      "segmentrouting" : {
        "name" : "Spine-R2",
        "ipv4NodeSid" : 102,
        "ipv4Loopback" : "10.6.4.254",
        "routerMac" : "00:00:00:00:04:80",
        "isEdgeRouter" : false,
        "adjacencySids" : []
      },
      "basic" : {
        "driver" : "ofdpa-ovs"
      }
    },
    "of:0000000000000101" : {
      "segmentrouting" : {
        "name" : "Leaf-R1",
        "ipv4NodeSid" : 201,
        "ipv4Loopback" : "10.6.1.254",
        "routerMac" : "00:00:00:00:01:80",
        "isEdgeRouter" : true,
        "adjacencySids" : []
      },
      "basic" : {
        "driver" : "ofdpa-ovs"
      }
    },
    "of:0000000000000102" : {
      "segmentrouting" : {
        "name" : "Leaf-R2",
        "ipv4NodeSid" : 202,
        "ipv4Loopback" : "10.6.2.254",
        "routerMac" : "00:00:00:00:02:80",
        "isEdgeRouter" : true,
        "adjacencySids" : []
      },
      "basic" : {
        "driver" : "ofdpa-ovs"
      }
    }
  },
  "ports" : {
    "of:0000000000000101/3" : {
      "interfaces" : [{  
        "ips" : [ "10.6.1.254/24" ],
        "vlan-untagged" : 20
      }]
    },
    "of:0000000000000101/4" : {
      "interfaces" : [{  
        "ips" : [ "10.6.1.254/24" ],
        "vlan-untagged" : 20
      }]
    },
    "of:0000000000000102/3" : {
      "interfaces" : [{  
        "ips" : [ "10.6.2.254/24" ],
        "vlan-untagged" : 40
      }]
    },
    "of:0000000000000102/4" : {
      "interfaces" : [{  
        "ips" : [ "10.6.2.254/24" ],
        "vlan-untagged" : 40
      }]
    }
  },
  "hosts" : {
    "00:00:00:00:00:01/-1" : {
      "basic": {
        "ips": ["10.6.1.1"],
        "locations": ["of:0000000000000101/3"]
      }
    },
    "00:00:00:00:00:02/-1" : {
      "basic": {
        "ips": ["10.6.1.2"],
        "locations": ["of:0000000000000101/4"]
      }
    },
    "00:00:00:00:00:03/-1" : {
      "basic": {
        "ips": ["10.6.2.1"],
        "locations": ["of:0000000000000102/3"]
      }
    },
    "00:00:00:00:00:04/-1" : {
      "basic": {
        "ips": ["10.6.2.2"],
        "locations": ["of:0000000000000102/4"]
      }
    }
  }
}

Load the configuration.

1
$ onos-netcfg <onos_ip> <config_file>

Run Mininet

Here is the topology script to run in Mininet.

Topologyview raw
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
# 2-by-2 leaf-spine topology
from mininet.topo import Topo
from mininet.node import Host

class MyTopo(Topo):

    spineswitch = []
    leafswitch = []
    host = []

    def __init__(self):

        # initialize topology
        Topo.__init__(self)

        for i in range(1, 3):
            # add spine switches
            self.spineswitch.append(self.addSwitch("100"+str(i), dpid="000000000000019"+str(i)))

            # add leaf switches
            self.leafswitch.append(self.addSwitch("200"+str(i), dpid="000000000000010"+str(i)))

        # add hosts
        self.host.append(self.addHost("3001", cls=IpHost, mac="00:00:00:00:00:01", ip="10.6.1.1/24", gateway="10.6.1.254"))
        self.host.append(self.addHost("3002", cls=IpHost, mac="00:00:00:00:00:02", ip="10.6.1.2/24", gateway="10.6.1.254"))
        self.host.append(self.addHost("3003", cls=IpHost, mac="00:00:00:00:00:03", ip="10.6.2.1/24", gateway="10.6.2.254"))
        self.host.append(self.addHost("3004", cls=IpHost, mac="00:00:00:00:00:04", ip="10.6.2.2/24", gateway="10.6.2.254"))

        # add links
        for i in range(2):
            self.addLink(self.spineswitch[i], self.leafswitch[0])
            self.addLink(self.spineswitch[i], self.leafswitch[1])

        for i in range(2):
            self.addLink(self.leafswitch[i], self.host[i*2])
            self.addLink(self.leafswitch[i], self.host[i*2+1])

class IpHost(Host):
    def __init__(self, name, gateway, *args, **kwargs):
        super(IpHost, self).__init__(name,*args,**kwargs)
        self.gateway = gateway

    def config(self, **kwargs):
        Host.config(self,**kwargs)
        mtu = "ifconfig " + self.name + "-eth0 mtu 1490"
        self.cmd(mtu)
        self.cmd('ip route add default via %s' % self.gateway)

topos = {'mytopo': (lambda: MyTopo())}

Run Mininet.

1
$ sudo mn --switch=ovs,protocols=OpenFlow13 --custom=topo.py --topo=mytopo --controller=remote,127.0.0.1:6653

Mininet will create a 2x2 leaf-spine fabric with OpenFlow1.3 enabled Open vSwitches and connect to the ONOS controller running at localhost.

Test

We can run pingall at Mininet shell and verify the connectivity between all hosts.

Check OpenFlow Table

Check flow tables in switches.

1
$ ovs-ofctl -O OpenFlow13 dump-flows <bridge>

Check group tables in switches.

1
$ ovs-ofctl -O OpenFlow13 dump-groups <bridge>

See the Open vSwitch Manual for more details.

To run the commands shell Mininet shell, type the following.

1
mininet> 2001 ovs-ofctl -O OpenFlow13 dump-flows "2001"

Trace the flow tables and group tables and you can see switches push and pop MPLS labels.

Here are some points I got.

  • In this practice, switches did perform segment routing using MPLS labels to route.
  • For packets crossing subnets, the leaf switch would only push MPLS labels of the egress edge switch, which was ipv4NodeSid we set in device configuration.
  • As the packets arrived at spine switches, MPLS labels were popped. The reason was that for this 2x2 leaf-spine fabric, spine switches were called penultimate hop and would perform labels popping to reduce the load at egress switches.
  • Since labels were popped at penultimate switches, the egress switches only needed to do IP table lookup without MPLS lookup, solving the Egress LSR Double Lookup problem.

The following picture shows labels of each flow in this practice.

Update

After tracing the flows installed by Segment Routing application, configuration for hosts is not truly necessary. Segment Routing uses flow rules to match the subnet under switches, which is already included in port configuration, but not Ethernet or IP addresses.

Configuration with hosts omitted has been tested and packets could be correctly forwarded.