Programmable Internetworking & Communication Operating System Docs ... Click Spaces -> Space Directory to see docs for all releases ...
Page tree
Skip to end of metadata
Go to start of metadata

Introduction

EVPN Multihoming (MH) is a standards based replacement for MLAG to achieve standards based all-active server redundancy in datacenters and enterprise networks. Multihoming eliminates the need to have MLAG and unlike MLAG, does not require the group of devices used to create a multihoming group to have inter-connecting peer links.

EVPN multihoming is based on the concept of forming Ethernet Segments (ES). In the context of BGP EVPN, an Ethernet Segment is a set of links connecting a single end host or server with the EVPN VXLAN fabric. In the Figure below, Server1 is multihomed to the VXLAN fabric through links connecting it to VTEP1 and VTEP2, forming an Ethernet Segment. Similarly, Server2, Server3 and Server4 are connected to the fabric through links forming segment 2, segment 3 and segment 4 respectively.

              Figure 1. EVPN Multihoming


Key Benefits

  • Interconnecting switch links between TOR switches in a redundancy group are not required.
  • There can be more than two TOR switches in the redundancy group. A maximum of 8 devices can form one redundancy group as opposed to a maximum of two devices in an MLAG group.
  • Ease of management as it provides a single BGP-EVPN control plane.
  • Allows multi-vendor interoperability because of standards based implementation. 

 Brief Mechanism

As shown in Figure 1, if VTEP1 loses direct connection to Server1, it will signal to VTEP2, VTEP3 and VTEP4 that Server1 is no longer accessible and to change their next-hop group for Server1 if there are alternative routes to this ES. Since VTEP2 also has a route to this ES, VTEP3 and VTEP4 will change their next hop groups for Server1 MAC address to go through VTEP2 instead of VTEP1.

EVPN multihoming uses EVPN Type-1, Type-2 and Type-4 routes to discover these Ethernet Segments and deliver traffic to and from these segments. Each Ethernet Segment is identified by a unique ID called Ethernet Segment ID (ESI). ESI is unique across the entire EVPN VXLAN domain on all VTEPs. To configure ESI, we first need to configure the Ethernet Segment System MAC and a local ES-ID. The local ES system MAC and local ES ID are used to automatically generate the ESI. Different ES System MACs can be configured for different Ethernet Segments but the ES System MAC configured on interfaces connecting to the same server or host must be same. In the sections below we will briefly explore EVPN Type-1Auto Discovery and EVPN Type-4 Ethernet Segment Routes.

EVPN Type-1 Ethernet Auto Discovery (A-D) Route

EVPN Type-1 routes are used for fast convergence and aliasing for multihomed sites or end hosts. Fast convergence is important in networks with large number of hosts. When a link failure occurs, devices can signal to upstream PE devices or other VTEPs in the network to change their next-hop groups for the MAC addresses associated with that particular Ethernet Segment. Aliasing is used for load balancing on multiple outgoing links.


Route Distinguisher (RD) (8 octets)

Ethernet Segment Identifier (10 octets)

Ethernet Tag ID (4 octets)

MPLS Label (3 octets)

Table 1. EVPN Type-1 Route NLRI


Type-1 routes are only originated for multihomed devices, i.e. these routes are sent only when the Ethernet Segment ID is set to a non-zero value.

EVPN Type-4 Ethernet Segment Route

Type-4 routes are used for electing the Designated Forwarder (DF) in EVPN multihoming scenarios. The DF is responsible for sending broadcast, unknown unicast and multicast (BUM) traffic to multihomed hosts or servers. If a server is multihomed to two upstream devices in the EVPN fabric, only one would be elected as the DF for sending BUM traffic to the multihomed server on this ES.

The EVPN NLRI for Type-4 Ethernet Segment Route consists of the following four fields. For multihomed hosts, the Ethernet Segment Identifier (ESI) must be set to a non-zero value. A value of zero for the ESI indicates a single homed device.


RD (8 Octets)

Ethernet Segment Identifier (10 Octets)

IP Address Length (1 octet)

Originating Router's IP Address (4 or 16 octets)

  Table 2. EVPN Type-4 NLRI

 

EVPN multihoming works by utilizing EVPN Type-1, Type-2 and Type-4 routes. Three types of EVPN routes are used to fulfill three distinct EVPN multihoming requirements.


The PICOS EVPN multihoming can be summarized in the following three steps


Step1:

EVPN Type-2 routes are used for remote learning of end hosts. As shown in Figure 1 for example, Server1 is multihomed to VTEP1 and VTEP2. Both VTEP1 and VTEP2 will first locally learn Server1’s MAC/IP address and then advertise it to VTEP3 and VTEP4 using EVPN Type-2 routes (remote learning).

 

Step2:

With EVPN multihoming enabled on switch ports connecting Server1 to VTEP1 and VTEP2, both devices will configure the same Ethernet Subnet Identifier (ESI) on the two links and advertise this information to all the other VTEPs  using Type-4 routes. This will indicate to remote VTEPs that Server1 is multihomed to the EVPN fabric and is accessible through both VTEP1 and VTEP2.


Step3:

For fast convergence and aliasing, VTEP1 and VTEP2 will advertise Type-1 routes. This type of advertisement is sent in the event of a link failure. If the link connecting VTEP1 and Server1 goes down, VTEP1 will advertise Type-1 route indicating to all the other VTEPs that it has lost connection with Server1 (Ethernet Segment). If VTEP1 was elected the designated forward (DF) for this segment, VTEP2 will then assume the role of DF for Server1 whereas VTEP3 and VTEP4 will adjust their next-hop groups for Server1 accordingly.


Note

It is strongly suggested to remove any MLAG related configuration from the overlay devices (VTEPs) if you plan on deploying EVPN MH in your network. The two technologies are designed to address almost similar requirements of redundancy but MLAG is a layer2 technique whereas EVPN MH is deployed in the overlay and is a layer3 technique. The two technologies hence do not mix well and users are urged to stick to EVPN MH for its standards based implementation and interoperability benefits as opposed to MLAG especially in the overlay network design.

Designated Forwarder Election

For BGP EVPN, a Designated Forwarder (DF) is responsible for handling Broadcast, Unknown Unicast and Multicast (BUM) traffic for Ethernet Segments. As shown in Figure 1, Server1 constitutes an Ethernet Segment as it is multihomed to two VTEPs in the EVPN overlay. Only one of the two VTEPs has to be elected a DF to deliver BUM traffic to Server1 from remote VTEPs (VTEP3 and VTEP4). The administrator needs to configure EVPN multihoming preference for the Ethernet Segment. A higher preference value wins the election process and assumes the role of the DF. If the preference on both the VTEPs is same then the VTEP with the smallest IP wins the election. It’s important to note that the interface connecting the Ethernet Segment must be an Aggregate-Ethernet interface.

For every Ethernet Segment there must be only one DF.

The following command can be used to set the ES DF preference.

 

set interface aggregate-ethernet <interface> evpn mh es-df-pref <preference>

VTEP Uplink Status Tracking

Overlay VTEPs are capable of tracking the state of uplinks connecting the overlay leaf devices with underlay spine switches. This feature is automatically enabled when the user enables the EVPN MH. All the downstream links connecting the MH hosts are put in protocol down state when all the uplinks go down. In Figure 1, all the links connecting the underlay spine switches must go down before the downstream links are put in protocol down state. The links will not be put in protocol down state if there is only one link up and active connecting the overlay devices with the underlay.

EVPN MH Split Horizon

Split-horizon technique is used to stop BUM traffic from forming a loop in the EVPN VXLAN network that employs multihoming. Using split-horizon, BUM traffic from the same Ethernet Segment will be dropped by the peer device. As can been seen in Figure 2, Host1 is multihomed into the EVPN VXLAN fabric and connected to both VTEP1 and VTEP2 forming Ethernet Segment ES1.

In the example below, if Host1 sends some BUM traffic to DF and then it is forwarded to the spine switch and then relayed back to VTEP2, that traffic will be discarded at VTEP2 since VTEP2 has an interface configured for the same Ethernet Segment. Split horizon mechanism ensures BUM traffic can’t loop back to Host1 if Host1 originates this traffic.

Figure 2. EVPN MH Split-Horizon


Key points to note for EVPN MH Split Horizon and BUM traffic duplication:

  1. Only one DF is elected in one ESI (one LAG), so partial connection is not allowed for EVPN MH access connection.
  2. Non-DF node will drop the BUM traffic from other nodes, including peer of the same ES. BUM traffic from remote VTEPs is dropped by non-DF nodes to avoid duplication.


Note

Note: Single homed ports connecting a single device with the EVPN VXLAN fabric are not yet supported on VTEP devices. If a server is dual homed to two VTEPs and the connecting link to one of the VTEP fails, the server will still be considered a dual homed device even if there is only one active link connecting it to the fabric.


EVPN MH Basic Configuration

To configure EVPN multihoming on a LAG interface, the user needs to configure an Ethernet Segment identifier (ES-ID) and an Ethernet Segment System MAC. A type-3 ESI value is automatically generated by using these two parameters. The resulting 10-byte ESI value has the following format, where the XXs denote the 6-byte ES System MAC and the YYs denote the 3-byte local ES ID.


03 : XX : XX : XX : XX : XX : XX : YY : YY : YY


For each Ethernet segment, both the ES System MAC and the ES ID must be the same on all the switches forming the EVPN MH redundancy group. However, switches can be configured with different system MACs for different Ethernet segments or share the same system MAC for all Ethernet segments.


Switch Configuration and Topology

For this example configuration we have the following topology comprising of two spine switches, four leaf switches and four servers that are multihomed to this network. Two servers are multihomed to Leaf1 and Leaf2 switches whereas the other two servers are attached to Leaf3 and Leaf4 switches.


The four leaf switches form two MH redundancy groups for the four ES IDs. Two ES ID’s are configured under Leaf1 and Leaf2 whereas, another two ES ID’s are configured under Leaf3 and Leaf4. Each leaf switch has one physical interface added to a LAG interface to connect to the downstream server.

The two ES IDs are configured on each peer connected to the two downstream multihomed servers. On Leaf1, two physical ports connect to the multihomed servers; both the physical ports are first added to LAG interfaces. The LAG interface connected to Server1 is “ae2” whereas the interface connecting Server2 is “ae1”. For EVPN MH, it is a mandatory that the ports connecting the multihomed hosts must be LAG interfaces. Similarly, on all leaf switches there are two LAG interfaces and two ES IDs configured.


Figure 3. EVPN MH Example Topology


Note

Note: The physical ports connecting the multihomed servers or hosts must be added to a LAG interface and then configured for EVPN MH. Physical interfaces such as ge-1/1/1 cannot be directly used for MH configuration.

Leaf1 Configuration

Step 1. Configure LAG interfaces, set the LAG interface MTU, configure ES ID, configure the LAG interface VLAN membership and the LAG interface trunk mode. MTU configuration is optional, it can be configured when the MTU size needs to be of a particular size based on the customer use case environment. The LACP configuration is also optional.

admin@Leaf1# set interface aggregate-ethernet ae1 aggregated-ether-options lacp enable true
admin@Leaf1# set interface aggregate-ethernet ae2 aggregated-ether-options lacp enable true
admin@Leaf1# set interface aggregate-ethernet ae1 family ethernet-switching port-mode trunk
admin@Leaf1# set interface aggregate-ethernet ae1 family ethernet-switching vlan members 30
admin@Leaf1# set interface aggregate-ethernet ae1 family ethernet-switching vlan members 20
admin@Leaf1# set interface aggregate-ethernet ae2 family ethernet-switching port-mode trunk
admin@Leaf1# set interface aggregate-ethernet ae2 family ethernet-switching vlan members 30
admin@Leaf1# set interface aggregate-ethernet ae2 family ethernet-switching vlan members 20
admin@Leaf1# set interface aggregate-ethernet ae1 evpn mh es-id 4
admin@Leaf1# set interface aggregate-ethernet ae2 evpn mh es-id 2

Step2. Configure ES system MAC, it can be the same or different for different ethernet segments. Configure the physical interfaces and VLAN membership and the ports trunk mode. Also add physical interfaces to LAG interfaces.

admin@Leaf1# set interface aggregate-ethernet ae1 evpn mh es-sys-mac 00:22:22:22:22:22
admin@Leaf1# set interface aggregate-ethernet ae2 evpn mh es-sys-mac 00:22:22:22:22:22
admin@Leaf1# set interface gigabit-ethernet ge-1/1/7 family ethernet-switching native-vlan-id 20
admin@Leaf1# set interface gigabit-ethernet ge-1/1/7 family ethernet-switching port-mode trunk
admin@Leaf1# set interface gigabit-ethernet ge-1/1/7 family ethernet-switching vlan members 30
admin@Leaf1# set interface gigabit-ethernet ge-1/1/19 mtu 3000
admin@Leaf1# set interface gigabit-ethernet ge-1/1/19 family ethernet-switching native-vlan-id 4014
admin@Leaf1# set interface gigabit-ethernet ge-1/1/23 ether-options 802.3ad ae2
admin@Leaf1# set interface gigabit-ethernet ge-1/1/23 family ethernet-switching
admin@Leaf1# set interface gigabit-ethernet ge-1/1/24 disable true
admin@Leaf1# set interface gigabit-ethernet ge-1/1/27 mtu 3000
admin@Leaf1# set interface gigabit-ethernet ge-1/1/27 family ethernet-switching native-vlan-id 4012
admin@Leaf1# set interface gigabit-ethernet ge-1/1/37 ether-options 802.3ad ae1

Step3. Configure VRF and enable IP routing.

admin@Leaf1# set ip routing enable true
admin@Leaf1# set ip vrf vrf1

Step4. Configure loopback interfaces and assign IP addresses.

admin@Leaf1# set l3-interface loopback lo address 10.226.14.254 prefix-length 32
admin@Leaf1# set l3-interface loopback lo address 10.226.14.207 prefix-length 32
admin@Leaf1# set l3-interface loopback vrf1 address 17.17.17.17 prefix-length 32

Step5. Configure VLAN interfaces and assign IP addresses.

admin@Leaf1# set l3-interface vlan-interface vlan4014 address 10.226.15.65 prefix-length 30
admin@Leaf1# set l3-interface vlan-interface vlan4012 address 10.226.15.29 prefix-length 30
admin@Leaf1# set l3-interface vlan-interface vlan4094 address 10.226.13.26 prefix-length 30
admin@Leaf1# set l3-interface vlan-interface vlan99 vrf vrf1
admin@Leaf1# set l3-interface vlan-interface vlan20 vrf vrf1
admin@Leaf1# set l3-interface vlan-interface vlan20 address 192.168.0.3 prefix-length 24
admin@Leaf1# set l3-interface vlan-interface vlan20 address 2002:0:0:1::3 prefix-length 64
admin@Leaf1# set l3-interface vlan-interface vlan30 vrf vrf1
admin@Leaf1# set l3-interface vlan-interface vlan30 address 172.168.0.3 prefix-length 24
admin@Leaf1# set l3-interface vlan-interface vlan30 address 2003:0:0:1::3 prefix-length 64
admin@Leaf1# set l3-interface vlan-interface vlan20 router-mac 00:00:20:00:00:FE
admin@Leaf1# set l3-interface vlan-interface vlan30 router-mac 00:00:30:00:00:FE

Step6. Configure BGP and EVPN.

admin@Leaf1# set protocols bgp local-as 65001
admin@Leaf1# set protocols bgp router-id 10.226.14.207
admin@Leaf1# set protocols bgp neighbor 10.226.14.48 remote-as internal
admin@Leaf1# set protocols bgp neighbor 10.226.14.48 update-source 10.226.14.207
admin@Leaf1# set protocols bgp neighbor 10.226.14.48 evpn activate true
admin@Leaf1# set protocols bgp neighbor 10.226.14.208 remote-as internal
admin@Leaf1# set protocols bgp neighbor 10.226.14.208 update-source 10.226.14.207
admin@Leaf1# set protocols bgp neighbor 10.226.14.208 evpn activate true
admin@Leaf1# set protocols bgp neighbor 10.226.14.24 remote-as internal
admin@Leaf1# set protocols bgp neighbor 10.226.14.24 update-source 10.226.14.207
admin@Leaf1# set protocols bgp neighbor 10.226.14.24 evpn activate true
admin@Leaf1# set protocols bgp ipv4-unicast
admin@Leaf1# set protocols bgp evpn advertise-all-vni
admin@Leaf1# set protocols bgp evpn advertise ipv4-unicast
admin@Leaf1# set protocols bgp evpn advertise ipv6-unicast
admin@Leaf1# set protocols bgp evpn advertise-svi-ip
admin@Leaf1# set protocols bgp vrf vrf1 local-as 65001
admin@Leaf1# set protocols bgp vrf vrf1 router-id 17.17.17.17
admin@Leaf1# set protocols bgp vrf vrf1 evpn advertise ipv4-unicast
admin@Leaf1# set protocols bgp vrf vrf1 evpn advertise ipv6-unicast
admin@Leaf1# set protocols evpn mh
admin@Leaf1# set protocols lldp enable true

Step7. Configure OSPF.

admin@Leaf1# set protocols ospf router-id 17.17.17.17
admin@Leaf1# set protocols ospf network 10.226.15.64/30 area 0.0.0.0
admin@Leaf1# set protocols ospf network 10.226.15.28/30 area 0.0.0.0
admin@Leaf1# set protocols ospf network 10.226.14.254/32 area 0.0.0.0
admin@Leaf1# set protocols ospf network 10.226.14.207/32 area 0.0.0.0

Step8. Configure PIM

admin@Leaf1# set protocols pim rp 202.202.202.202 group 224.1.1.1/32
admin@Leaf1# set protocols pim rp 202.202.202.202 group 230.1.1.1/32
admin@Leaf1# set protocols pim interface lo
admin@Leaf1# set protocols pim interface vlan4014
admin@Leaf1# set protocols pim interface vlan4012

Step9. Configure hostname, inband mode and VLANs.

admin@Leaf1# set system hostname Leaf1
admin@Leaf1# set system inband enable true
admin@Leaf1# set vlans vlan-id 20 l3-interface vlan20
admin@Leaf1# set vlans vlan-id 30 l3-interface vlan30
admin@Leaf1# set vlans vlan-id 99 l3-interface vlan99
admin@Leaf1# set vlans vlan-id 4012 l3-interface vlan4012
admin@Leaf1# set vlans vlan-id 4014 l3-interface vlan4014
admin@Leaf1# set vlans vlan-id 4094 l3-interface vlan4094

Step10. Configure VXLANs.

admin@Leaf1# set vxlans source-interface lo address 10.226.14.254
admin@Leaf1# set vxlans vni 10030 decapsulation mode service-vlan-per-port
admin@Leaf1# set vxlans vni 10030 vlan 30
admin@Leaf1# set vxlans vni 10030 arp-nd-suppress disable false
admin@Leaf1# set vxlans vni 10020 decapsulation mode service-vlan-per-port
admin@Leaf1# set vxlans vni 10020 vlan 20
admin@Leaf1# set vxlans vni 10020 arp-nd-suppress disable false
admin@Leaf1# set vxlans vni 99 vlan 99
admin@Leaf1# set vxlans vrf vrf1 l3-vni 99
admin@Leaf1# set vxlans vni 10020 mcast-group 224.1.1.1
admin@Leaf1# set vxlans vni 10030 mcast-group 230.1.1.1
admin@Leaf1# commit

Leaf2 Configuration

admin@Leaf2# set interface aggregate-ethernet ae1 aggregated-ether-options lacp enable true
admin@Leaf2# set interface aggregate-ethernet ae2 aggregated-ether-options lacp enable true
admin@Leaf2# set interface aggregate-ethernet ae1 family ethernet-switching port-mode trunk
admin@Leaf2# set interface aggregate-ethernet ae1 family ethernet-switching vlan members 30
admin@Leaf2# set interface aggregate-ethernet ae1 family ethernet-switching vlan members 20
admin@Leaf2# set interface aggregate-ethernet ae1 evpn mh es-id 4
admin@Leaf2# set interface aggregate-ethernet ae1 evpn mh es-sys-mac 00:22:22:22:22:22
admin@Leaf2# set interface aggregate-ethernet ae2 family ethernet-switching port-mode "trunk"
admin@Leaf2# set interface aggregate-ethernet ae2 family ethernet-switching vlan members 30
admin@Leaf2# set interface aggregate-ethernet ae2 family ethernet-switching vlan members 20
admin@Leaf2# set interface aggregate-ethernet ae2 evpn mh es-id 2
admin@Leaf2# set interface aggregate-ethernet ae2 evpn mh es-sys-mac 00:22:22:22:22:22
admin@Leaf2# set interface gigabit-ethernet te-1/1/1 mtu 3000
admin@Leaf2# set interface gigabit-ethernet te-1/1/5 mtu 3000
admin@Leaf2# set interface gigabit-ethernet te-1/1/9 family ethernet-switching native-vlan-id 20
admin@Leaf2# set interface gigabit-ethernet te-1/1/9 family ethernet-switching port-mode trunk
admin@Leaf2# set interface gigabit-ethernet te-1/1/9 family ethernet-switching vlan members 30
admin@Leaf2# set interface gigabit-ethernet te-1/1/17 mtu 3000
admin@Leaf2# set interface gigabit-ethernet te-1/1/17 family ethernet-switching native-vlan-id 4014
admin@Leaf2# set interface gigabit-ethernet te-1/1/19 mtu 3000
admin@Leaf2# set interface gigabit-ethernet te-1/1/19 family ethernet-switching native-vlan-id 4013
admin@Leaf2# set interface gigabit-ethernet te-1/1/21 ether-options 802.3ad ae2
admin@Leaf2# set interface gigabit-ethernet te-1/1/23 ether-options 802.3ad ae1
admin@Leaf2# set interface ethernet-switching-options
admin@Leaf2# set ip routing enable true
admin@Leaf2# set ip vrf vrf1
admin@Leaf2# set l3-interface loopback lo address 10.226.14.253 prefix-length 32
admin@Leaf2# set l3-interface loopback vrf1 address 26.26.26.26 prefix-length 32
admin@Leaf2# set l3-interface vlan-interface vlan4013 address 10.226.15.25 prefix-length 30
admin@Leaf2# set l3-interface vlan-interface vlan4014 address 10.226.15.81 prefix-length 30
admin@Leaf2# set l3-interface vlan-interface vlan4094 address 10.226.13.25 prefix-length 30
admin@Leaf2# set l3-interface vlan-interface vlan99 vrf vrf1
admin@Leaf2# set l3-interface vlan-interface vlan20 vrf vrf1
admin@Leaf2# set l3-interface vlan-interface vlan20 address 192.168.0.3 prefix-length 24
admin@Leaf2# set l3-interface vlan-interface vlan20 address 2002:0:0:1::3 prefix-length 64
admin@Leaf2# set l3-interface vlan-interface vlan30 vrf vrf1
admin@Leaf2# set l3-interface vlan-interface vlan30 address 172.168.0.3 prefix-length 24
admin@Leaf2# set l3-interface vlan-interface vlan30 address 2003:0:0:1::3 prefix-length 64
admin@Leaf2# set l3-interface vlan-interface vlan20 router-mac 00:00:20:00:00:FE
admin@Leaf2# set l3-interface vlan-interface vlan30 router-mac 00:00:30:00:00:FE
admin@Leaf2# set protocols bgp local-as 65001
admin@Leaf2# set protocols bgp router-id 10.226.14.208
admin@Leaf2# set protocols bgp neighbor 10.226.14.48 remote-as internal
admin@Leaf2# set protocols bgp neighbor 10.226.14.48 update-source 10.226.14.208
admin@Leaf2# set protocols bgp neighbor 10.226.14.48 evpn activate true
admin@Leaf2# set protocols bgp neighbor 10.226.14.207 remote-as internal
admin@Leaf2# set protocols bgp neighbor 10.226.14.207 update-source 10.226.14.208
admin@Leaf2# set protocols bgp neighbor 10.226.14.207 evpn activate true
admin@Leaf2# set protocols bgp neighbor 10.226.14.24 remote-as internal
admin@Leaf2# set protocols bgp neighbor 10.226.14.24 update-source 10.226.14.208
admin@Leaf2# set protocols bgp neighbor 10.226.14.24 evpn activate true
admin@Leaf2# set protocols bgp ipv4-unicast
admin@Leaf2# set protocols bgp evpn advertise-all-vni
admin@Leaf2# set protocols bgp evpn advertise ipv4-unicast
admin@Leaf2# set protocols bgp evpn advertise ipv6-unicast
admin@Leaf2# set protocols bgp evpn advertise-svi-ip
admin@Leaf2# set protocols bgp vrf vrf1 local-as 65001
admin@Leaf2# set protocols bgp vrf vrf1 router-id 26.26.26.26
admin@Leaf2# set protocols bgp vrf vrf1 evpn advertise ipv4-unicast
admin@Leaf2# set protocols bgp vrf vrf1 evpn advertise ipv6-unicast
admin@Leaf2# set protocols evpn mh
admin@Leaf2# set protocols lldp enable true
admin@Leaf2# set protocols ospf router-id 110.110.110.110
admin@Leaf2# set protocols ospf network 10.226.15.24/30 area 0.0.0.0
admin@Leaf2# set protocols ospf network 10.226.15.80/30 area 0.0.0.0
admin@Leaf2# set protocols ospf network 10.226.14.208/32 area 0.0.0.0
admin@Leaf2# set protocols ospf network 10.226.14.253/32 area 0.0.0.0
admin@Leaf2# set protocols pim rp 202.202.202.202 group 224.1.1.1/32
admin@Leaf2# set protocols pim rp 202.202.202.202 group 230.1.1.1/32
admin@Leaf2# set protocols pim interface lo
admin@Leaf2# set protocols pim interface vlan4014
admin@Leaf2# set protocols pim interface vlan4013
admin@Leaf2# set vlans vlan-id 20 l3-interface vlan20
admin@Leaf2# set vlans vlan-id 30 l3-interface vlan30
admin@Leaf2# set vlans vlan-id 99 l3-interface vlan99
admin@Leaf2# set vlans vlan-id 4013 l3-interface vlan4013
admin@Leaf2# set vlans vlan-id 4014 l3-interface vlan4014
admin@Leaf2# set vlans vlan-id 4094 l3-interface vlan4094
admin@Leaf2# set vxlans source-interface lo address 10.226.14.253
admin@Leaf2# set vxlans vni 10030 decapsulation mode service-vlan-per-port
admin@Leaf2# set vxlans vni 10030 vlan 30
admin@Leaf2# set vxlans vni 10030 arp-nd-suppress disable false
admin@Leaf2# set vxlans vni 10020 decapsulation mode service-vlan-per-port
admin@Leaf2# set vxlans vni 10020 vlan 20
admin@Leaf2# set vxlans vni 10020 arp-nd-suppress disable false
admin@Leaf2# set vxlans vni 99 vlan 99
admin@Leaf2# set vxlans vrf vrf1 l3-vni 99
admin@Leaf2# set vxlans vni 10020 mcast-group 224.1.1.1
admin@Leaf2# set vxlans vni 10030 mcast-group 230.1.1.1
admin@Leaf2# commit

Leaf3 Configuration

admin@Leaf3# set interface aggregate-ethernet ae1 aggregated-ether-options lacp enable true
admin@Leaf3# set interface aggregate-ethernet ae2 aggregated-ether-options lacp enable true
admin@Leaf3# set interface aggregate-ethernet ae1 family ethernet-switching port-mode trunk
admin@Leaf3# set interface aggregate-ethernet ae1 family ethernet-switching vlan members 30
admin@Leaf3# set interface aggregate-ethernet ae1 family ethernet-switching vlan members 20
admin@Leaf3# set interface aggregate-ethernet ae1 evpn mh es-id 3
admin@Leaf3# set interface aggregate-ethernet ae1 evpn mh es-sys-mac 00:00:00:22:22:22
admin@Leaf3# set interface aggregate-ethernet ae2 family ethernet-switching port-mode trunk
admin@Leaf3# set interface aggregate-ethernet ae2 family ethernet-switching vlan members 30
admin@Leaf3# set interface aggregate-ethernet ae2 family ethernet-switching vlan members 20
admin@Leaf3# set interface aggregate-ethernet ae2 evpn mh es-id 1
admin@Leaf3# set interface aggregate-ethernet ae2 evpn mh es-sys-mac 00:00:00:22:22:22
admin@Leaf3# set interface gigabit-ethernet ge-1/1/3 mtu 3000
admin@Leaf3# set interface gigabit-ethernet ge-1/1/3 family ethernet-switching native-vlan-id 4016
admin@Leaf3# set interface gigabit-ethernet ge-1/1/5 mtu 3000
admin@Leaf3# set interface gigabit-ethernet ge-1/1/5 family ethernet-switching native-vlan-id 3012
admin@Leaf3# set interface gigabit-ethernet ge-1/1/19 family ethernet-switching native-vlan-id 20
admin@Leaf3# set interface gigabit-ethernet ge-1/1/19 family ethernet-switching port-mode trunk
admin@Leaf3# set interface gigabit-ethernet ge-1/1/19 family ethernet-switching vlan members 20
admin@Leaf3# set interface gigabit-ethernet ge-1/1/21 ether-options 802.3ad ae1
admin@Leaf3# set interface gigabit-ethernet ge-1/1/22 ether-options 802.3ad ae2
admin@Leaf3# set interface gigabit-ethernet ge-1/1/23 family ethernet-switching native-vlan-id 30
admin@Leaf3# set interface gigabit-ethernet ge-1/1/23 family ethernet-switching port-mode trunk
admin@Leaf3# set interface gigabit-ethernet ge-1/1/23 family ethernet-switching vlan members 20
admin@Leaf3# set interface gigabit-ethernet te-1/1/3 family ethernet-switching native-vlan-id 3011
admin@Leaf3# set interface ethernet-switching-options
admin@Leaf3# set ip routing enable true
admin@Leaf3# set ip vrf vrf1
admin@Leaf3# set l3-interface loopback lo address 10.226.14.24 prefix-length 32
admin@Leaf3# set l3-interface loopback lo address 10.226.14.201 prefix-length 32
admin@Leaf3# set l3-interface vlan-interface vlan3011 address 10.226.0.98 prefix-length 30
admin@Leaf3# set l3-interface vlan-interface vlan3012 address 10.226.0.94 prefix-length 30
admin@Leaf3# set l3-interface vlan-interface vlan99 vrf vrf1
admin@Leaf3# set l3-interface vlan-interface vlan20 vrf vrf1
admin@Leaf3# set l3-interface vlan-interface vlan20 address 192.168.0.3 prefix-length 24
admin@Leaf3# set l3-interface vlan-interface vlan20 address 2002:0:0:1::3 prefix-length 64
admin@Leaf3# set l3-interface vlan-interface vlan30 vrf vrf1
admin@Leaf3# set l3-interface vlan-interface vlan30 address 172.168.0.3 prefix-length 24
admin@Leaf3# set l3-interface vlan-interface vlan30 address 2003:0:0:1::3 prefix-length 64
admin@Leaf3# set l3-interface vlan-interface vlan20 router-mac 00:00:20:00:00:FE
admin@Leaf3# set l3-interface vlan-interface vlan30 router-mac 00:00:30:00:00:FE
admin@Leaf3# set protocols bgp local-as 65001
admin@Leaf3# set protocols bgp router-id 10.226.14.24
admin@Leaf3# set protocols bgp neighbor 10.226.14.207 remote-as internal
admin@Leaf3# set protocols bgp neighbor 10.226.14.207 timers delayopen 60
admin@Leaf3# set protocols bgp neighbor 10.226.14.207 update-source "10.226.14.24"
admin@Leaf3# set protocols bgp neighbor 10.226.14.207 evpn activate true
admin@Leaf3# set protocols bgp neighbor 10.226.14.208 remote-as "internal"
admin@Leaf3# set protocols bgp neighbor 10.226.14.208 timers delayopen 60
admin@Leaf3# set protocols bgp neighbor 10.226.14.208 update-source 10.226.14.24
admin@Leaf3# set protocols bgp neighbor 10.226.14.208 evpn activate true
admin@Leaf3# set protocols bgp neighbor 10.226.14.48 remote-as internal
admin@Leaf3# set protocols bgp neighbor 10.226.14.48 timers delayopen 60
admin@Leaf3# set protocols bgp neighbor 10.226.14.48 update-source 10.226.14.24
admin@Leaf3# set protocols bgp neighbor 10.226.14.48 evpn activate true
admin@Leaf3# set protocols bgp ipv4-unicast
admin@Leaf3# set protocols bgp evpn advertise-all-vni
admin@Leaf3# set protocols bgp evpn advertise ipv4-unicast
admin@Leaf3# set protocols bgp evpn advertise ipv6-unicast
admin@Leaf3# set protocols bgp evpn advertise-svi-ip
admin@Leaf3# set protocols lldp enable true
admin@Leaf3# set protocols ospf router-id 24.24.24.24
admin@Leaf3# set protocols ospf network 10.226.0.96/30 area 0.0.0.0
admin@Leaf3# set protocols ospf network 10.226.0.92/30 area 0.0.0.0
admin@Leaf3# set protocols ospf network 10.226.14.201/32 area 0.0.0.0
admin@Leaf3# set protocols ospf network 10.226.14.24/32 area 0.0.0.0
admin@Leaf3# set vlans vlan-id 20 l3-interface vlan20
admin@Leaf3# set vlans vlan-id 30 l3-interface vlan30
admin@Leaf3# set vlans vlan-id 99 l3-interface vlan99
admin@Leaf3# set vlans vlan-id 3011 l3-interface vlan3011
admin@Leaf3# set vlans vlan-id 3012 l3-interface vlan3012
admin@Leaf3# set vxlans source-interface lo address 10.226.14.201
admin@Leaf3# set vxlans vni 10020 decapsulation mode service-vlan-per-port
admin@Leaf3# set vxlans vni 10020 vlan 20
admin@Leaf3# set vxlans vni 10020 arp-nd-suppress disable false
admin@Leaf3# set vxlans vni 10030 decapsulation mode service-vlan-per-port
admin@Leaf3# set vxlans vni 10030 vlan 30
admin@Leaf3# set vxlans vni 10030 arp-nd-suppress disable false
admin@Leaf3# set vxlans vni 99 vlan 99
admin@Leaf3# set vxlans traceoptions flag all disable false
admin@Leaf3# set vxlans vrf vrf1 l3-vni 99
admin@Leaf3# set vxlans vni 10020 mcast-group 224.1.1.1
admin@Leaf3# set vxlans vni 10030 mcast-group 230.1.1.1
admin@Leaf3# commit

Leaf4 Configuration

admin@Leaf4# set interface aggregate-ethernet ae1 aggregated-ether-options lacp enable true
admin@Leaf4# set interface aggregate-ethernet ae2 aggregated-ether-options lacp enable true
admin@Leaf4# set interface aggregate-ethernet ae1 family ethernet-switching port-mode trunk
admin@Leaf4# set interface aggregate-ethernet ae1 family ethernet-switching vlan members 30
admin@Leaf4# set interface aggregate-ethernet ae1 family ethernet-switching vlan members 20
admin@Leaf4# set interface aggregate-ethernet ae1 evpn mh es-id 1
admin@Leaf4# set interface aggregate-ethernet ae1 evpn mh es-sys-mac 00:00:00:22:22:22
admin@Leaf4# set interface aggregate-ethernet ae2 mtu 3000
admin@Leaf4# set interface aggregate-ethernet ae2 family ethernet-switching port-mode trunk
admin@Leaf4# set interface aggregate-ethernet ae2 family ethernet-switching vlan members 30
admin@Leaf4# set interface aggregate-ethernet ae2 family ethernet-switching vlan members 20
admin@Leaf4# set interface aggregate-ethernet ae2 evpn mh es-id 3
admin@Leaf4# set interface aggregate-ethernet ae2 evpn mh es-sys-mac 00:00:00:22:22:22
admin@Leaf4# set interface gigabit-ethernet ge-1/1/1 disable true
admin@Leaf4# set interface gigabit-ethernet ge-1/1/3 mtu 3000
admin@Leaf4# set interface gigabit-ethernet ge-1/1/3 family ethernet-switching native-vlan-id 3011
admin@Leaf4# set interface gigabit-ethernet ge-1/1/5 mtu 3000
admin@Leaf4# set interface gigabit-ethernet ge-1/1/5 family ethernet-switching native-vlan-id 3012
admin@Leaf4# set interface gigabit-ethernet ge-1/1/19 family ethernet-switching native-vlan-id 20
admin@Leaf4# set interface gigabit-ethernet ge-1/1/19 family ethernet-switching port-mode trunk
admin@Leaf4# set interface gigabit-ethernet ge-1/1/19 family ethernet-switching vlan members 20
admin@Leaf4# set interface gigabit-ethernet ge-1/1/21 ether-options 802.3ad ae1
admin@Leaf4# set interface gigabit-ethernet ge-1/1/22 ether-options 802.3ad ae2
admin@Leaf4# set interface gigabit-ethernet ge-1/1/23 family ethernet-switching native-vlan-id 30
admin@Leaf4# set interface gigabit-ethernet ge-1/1/23 family ethernet-switching port-mode trunk
admin@Leaf4# set interface gigabit-ethernet ge-1/1/23 family ethernet-switching vlan members 20
admin@Leaf4# set interface gigabit-ethernet te-1/1/3 family ethernet-switching native-vlan-id 3011
admin@Leaf4# set ip routing enable true
admin@Leaf4# set ip vrf vrf1
admin@Leaf4# set l3-interface loopback lo address 10.226.14.202 prefix-length 32
admin@Leaf4# set l3-interface loopback lo address 10.226.14.48 prefix-length 32
admin@Leaf4# set l3-interface loopback vrf1 address 48.48.48.48 prefix-length 32
admin@Leaf4# set l3-interface vlan-interface vlan3011 address 10.226.15.70 prefix-length 30
admin@Leaf4# set l3-interface vlan-interface vlan3012 address 10.226.15.78 prefix-length 30
admin@Leaf4# set l3-interface vlan-interface vlan99 vrf vrf1
admin@Leaf4# set l3-interface vlan-interface vlan20 vrf vrf1
admin@Leaf4# set l3-interface vlan-interface vlan20 address 192.168.0.3 prefix-length 24
admin@Leaf4# set l3-interface vlan-interface vlan20 address 2002:0:0:1::3 prefix-length 64
admin@Leaf4# set l3-interface vlan-interface vlan30 vrf vrf1
admin@Leaf4# set l3-interface vlan-interface vlan30 address 172.168.0.3 prefix-length 24
admin@Leaf4# set l3-interface vlan-interface vlan30 address 2003:0:0:1::3 prefix-length 64
admin@Leaf4# set l3-interface vlan-interface vlan20 router-mac 00:00:20:00:00:FE
admin@Leaf4# set l3-interface vlan-interface vlan30 router-mac 00:00:30:00:00:FE
admin@Leaf4# set protocols bgp local-as 65001
admin@Leaf4# set protocols bgp router-id 10.226.14.48
admin@Leaf4# set protocols bgp neighbor 10.226.14.207 remote-as internal
admin@Leaf4# set protocols bgp neighbor 10.226.14.207 update-source 10.226.14.48
admin@Leaf4# set protocols bgp neighbor 10.226.14.207 evpn activate true
admin@Leaf4# set protocols bgp neighbor 10.226.14.208 remote-as internal
admin@Leaf4# set protocols bgp neighbor 10.226.14.208 update-source 10.226.14.48
admin@Leaf4# set protocols bgp neighbor 10.226.14.208 evpn activate true
admin@Leaf4# set protocols bgp neighbor 10.226.14.24 remote-as internal
admin@Leaf4# set protocols bgp neighbor 10.226.14.24 update-source 10.226.14.48
admin@Leaf4# set protocols bgp neighbor 10.226.14.24 evpn activate true
admin@Leaf4# set protocols bgp ipv4-unicast
admin@Leaf4# set protocols bgp evpn advertise-all-vni
admin@Leaf4# set protocols bgp evpn advertise ipv4-unicast
admin@Leaf4# set protocols bgp evpn advertise ipv6-unicast
admin@Leaf4# set protocols bgp evpn advertise-svi-ip
admin@Leaf4# set protocols bgp vrf vrf1 local-as 65001
admin@Leaf4# set protocols bgp vrf vrf1 router-id 48.48.48.48
admin@Leaf4# set protocols bgp vrf vrf1 evpn advertise ipv4-unicast
admin@Leaf4# set protocols bgp vrf vrf1 evpn advertise ipv6-unicast
admin@Leaf4# set protocols evpn mh startup-delay 60
admin@Leaf4# set protocols lldp enable true
admin@Leaf4# set protocols ospf router-id 48.48.48.48
admin@Leaf4# set protocols ospf network 10.226.14.202/32 area 0.0.0.0
admin@Leaf4# set protocols ospf network 10.226.14.48/32 area 0.0.0.0
admin@Leaf4# set protocols ospf network 10.226.15.68/30 area 0.0.0.0
admin@Leaf4# set protocols ospf network 10.226.15.76/30 area 0.0.0.0
admin@Leaf4# set vlans vlan-id 20 l3-interface vlan20
admin@Leaf4# set vlans vlan-id 30 l3-interface vlan30
admin@Leaf4# set vlans vlan-id 99 l3-interface vlan99
admin@Leaf4# set vlans vlan-id 3011 l3-interface vlan3011
admin@Leaf4# set vlans vlan-id 3012 l3-interface vlan3012
admin@Leaf4# set vxlans source-interface lo address 10.226.14.202
admin@Leaf4# set vxlans vni 10020 decapsulation mode service-vlan-per-port
admin@Leaf4# set vxlans vni 10020 vlan 20
admin@Leaf4# set vxlans vni 10020 arp-nd-suppress disable false
admin@Leaf4# set vxlans vni 10030 decapsulation mode service-vlan-per-port
admin@Leaf4# set vxlans vni 10030 vlan 30
admin@Leaf4# set vxlans vni 10030 arp-nd-suppress disable false
admin@Leaf4# set vxlans vni 99 vlan 99
admin@Leaf4# set vxlans vrf vrf1 l3-vni 99
admin@Leaf4# set vxlans vni 10020 mcast-group 224.1.1.1
admin@Leaf4# set vxlans vni 10030 mcast-group 230.1.1.1
admin@Leaf4# commit

Spine1 Configuration

admin@Spine1# set interface gigabit-ethernet te-1/1/11 mtu 2000
admin@Spine1# set interface gigabit-ethernet te-1/1/11 family ethernet-switching native-vlan-id 4011
admin@Spine1# set interface gigabit-ethernet te-1/1/11 speed 1000
admin@Spine1# set interface gigabit-ethernet te-1/1/21 mtu 2000
admin@Spine1# set interface gigabit-ethernet te-1/1/21 family ethernet-switching native-vlan-id 4092
admin@Spine1# set interface gigabit-ethernet te-1/1/21 speed 1000
admin@Spine1# set interface gigabit-ethernet te-1/1/35 mtu 2000
admin@Spine1# set interface gigabit-ethernet te-1/1/35 family ethernet-switching native-vlan-id 4013
admin@Spine1# set interface gigabit-ethernet te-1/1/39 disable true
admin@Spine1# set interface gigabit-ethernet te-1/1/41 mtu 2000
admin@Spine1# set interface gigabit-ethernet te-1/1/41 family ethernet-switching native-vlan-id 4014
admin@Spine1# set interface gigabit-ethernet te-1/1/44 mtu 2000
admin@Spine1# set interface gigabit-ethernet te-1/1/44 family ethernet-switching native-vlan-id 4016
admin@Spine1# set interface gigabit-ethernet te-1/1/45 family ethernet-switching native-vlan-id 4015
admin@Spine1# set ip routing enable true
admin@Spine1# set l3-interface loopback lo address 185.185.185.185 prefix-length 32
admin@Spine1# set l3-interface vlan-interface vlan4011 address 10.226.15.42 prefix-length 30
admin@Spine1# set l3-interface vlan-interface vlan4092 address 10.226.15.49 prefix-length 30
admin@Spine1# set l3-interface vlan-interface vlan4014 address 10.226.15.66 prefix-length 30
admin@Spine1# set l3-interface vlan-interface vlan4013 address 10.226.15.26 prefix-length 30
admin@Spine1# set l3-interface vlan-interface vlan4015 address 10.226.15.69 prefix-length 30
admin@Spine1# set l3-interface vlan-interface vlan4016 address 10.226.15.81 prefix-length 30
admin@Spine1# set protocols spanning-tree enable false
admin@Spine1# set protocols ospf router-id 185.185.185.185
admin@Spine1# set protocols ospf network 10.226.15.40/30 area 0.0.0.0
admin@Spine1# set protocols ospf network 10.226.15.48/30 area 0.0.0.0
admin@Spine1# set protocols ospf network 10.226.15.64/30 area 0.0.0.0
admin@Spine1# set protocols ospf network 10.226.15.24/30 area 0.0.0.0
admin@Spine1# set protocols ospf network 10.226.15.68/30 area 0.0.0.0
admin@Spine1# set protocols ospf network 10.226.15.80/30 area 0.0.0.0
admin@Spine1# set protocols ospf network 185.185.185.185/32 area 0.0.0.0
admin@Spine1# set protocols pim rp 202.202.202.202 group 224.1.1.1/32
admin@Spine1# set protocols pim rp 202.202.202.202 group 230.1.1.1/32
admin@Spine1# set protocols pim interface lo
admin@Spine1# set protocols pim interface vlan4011
admin@Spine1# set protocols pim interface vlan4092
admin@Spine1# set protocols pim interface vlan4014
admin@Spine1# set protocols pim interface vlan4013
admin@Spine1# set protocols pim interface vlan4015
admin@Spine1# set protocols pim interface vlan4016
admin@Spine1# set vlans vlan-id 4011 l3-interface vlan4011
admin@Spine1# set vlans vlan-id 4013 l3-interface vlan4013
admin@Spine1# set vlans vlan-id 4014 l3-interface vlan4014
admin@Spine1# set vlans vlan-id 4015 l3-interface vlan4015
admin@Spine1# set vlans vlan-id 4016 l3-interface vlan4016
admin@Spine1# set vlans vlan-id 4092 l3-interface vlan4092
admin@Spine1# commit

Spine2 Configuration

admin@Spine2# set interface gigabit-ethernet te-1/1/9 speed 1000
admin@Spine2# set interface gigabit-ethernet te-1/1/10 speed 1000
admin@Spine2# set interface gigabit-ethernet te-1/1/11 mtu 2000
admin@Spine2# set interface gigabit-ethernet te-1/1/11 family ethernet-switching native-vlan-id 4011
admin@Spine2# set interface gigabit-ethernet te-1/1/11 speed 1000
admin@Spine2# set interface gigabit-ethernet te-1/1/12 speed 1000
admin@Spine2# set interface gigabit-ethernet te-1/1/21 mtu 2000
admin@Spine2# set interface gigabit-ethernet te-1/1/21 family ethernet-switching native-vlan-id 4092
admin@Spine2# set interface gigabit-ethernet te-1/1/21 speed 1000
admin@Spine2# set interface gigabit-ethernet te-1/1/22 speed 1000
admin@Spine2# set interface gigabit-ethernet te-1/1/23 speed 1000
admin@Spine2# set interface gigabit-ethernet te-1/1/24 speed 1000
admin@Spine2# set interface gigabit-ethernet te-1/1/25 speed 1000
admin@Spine2# set interface gigabit-ethernet te-1/1/26 speed 1000
admin@Spine2# set interface gigabit-ethernet te-1/1/27 mtu 2000
admin@Spine2# set interface gigabit-ethernet te-1/1/27 family ethernet-switching native-vlan-id 4012
admin@Spine2# set interface gigabit-ethernet te-1/1/27 speed 1000
admin@Spine2# set interface gigabit-ethernet te-1/1/28 speed 1000
admin@Spine2# set interface gigabit-ethernet te-1/1/37 mtu 2000
admin@Spine2# set interface gigabit-ethernet te-1/1/37 family ethernet-switching native-vlan-id 4014
admin@Spine2# set interface gigabit-ethernet te-1/1/37 speed 1000
admin@Spine2# set interface gigabit-ethernet te-1/1/38 speed 1000
admin@Spine2# set interface gigabit-ethernet te-1/1/39 speed 1000
admin@Spine2# set interface gigabit-ethernet te-1/1/40 speed 1000
admin@Spine2# set interface gigabit-ethernet te-1/1/44 speed 1000
admin@Spine2# set interface gigabit-ethernet te-1/1/45 family ethernet-switching native-vlan-id 4015
admin@Spine2# set interface gigabit-ethernet te-1/1/45 family ethernet-switching native-vlan-id 4015
admin@Spine2# set interface gigabit-ethernet te-1/1/45 speed 1000
admin@Spine2# set interface gigabit-ethernet te-1/1/46 speed 1000
admin@Spine2# set interface gigabit-ethernet te-1/1/47 speed 1000
admin@Spine2# set interface gigabit-ethernet te-1/1/48 speed 1000
admin@Spine2# set ip routing enable true
admin@Spine2# set l3-interface loopback lo address 202.202.202.202 prefix-length 32
admin@Spine2# set l3-interface vlan-interface vlan4011 address 10.226.15.38 prefix-length 30
admin@Spine2# set l3-interface vlan-interface vlan4092 address 10.226.15.50 prefix-length 30
admin@Spine2# set l3-interface vlan-interface vlan4012 address 10.226.15.30 prefix-length 30
admin@Spine2# set l3-interface vlan-interface vlan4014 address 10.226.15.82 prefix-length 30
admin@Spine2# set l3-interface vlan-interface vlan4015 address 10.226.15.77 prefix-length 30
admin@Spine2# set protocols lldp enable true
admin@Spine2# set protocols spanning-tree enable false
admin@Spine2# set protocols ospf router-id 202.202.202.202
admin@Spine2# set protocols ospf network 10.226.15.36/30 area 0.0.0.0
admin@Spine2# set protocols ospf network 10.226.15.48/30 area 0.0.0.0
admin@Spine2# set protocols ospf network 10.226.15.28/30 area 0.0.0.0
admin@Spine2# set protocols ospf network 10.226.15.80/30 area 0.0.0.0
admin@Spine2# set protocols ospf network 10.226.15.76/30 area 0.0.0.0
admin@Spine2# set protocols ospf network 202.202.202.202/32 area 0.0.0.0
admin@Spine2# set protocols pim rp 202.202.202.202 group 224.1.1.1/32
admin@Spine2# set protocols pim rp 202.202.202.202 group 230.1.1.1/32
admin@Spine2# set protocols pim interface lo
admin@Spine2# set protocols pim interface vlan4011
admin@Spine2# set protocols pim interface vlan4092
admin@Spine2# set protocols pim interface vlan4012
admin@Spine2# set protocols pim interface vlan4014
admin@Spine2# set protocols pim interface vlan4015
admin@Spine2# set vlans vlan-id 4011 l3-interface vlan4011
admin@Spine2# set vlans vlan-id 4012 l3-interface vlan4012
admin@Spine2# set vlans vlan-id 4014 l3-interface vlan4014
admin@Spine2# set vlans vlan-id 4015 l3-interface vlan4015
admin@Spine2# set vlans vlan-id 4092 l3-interface vlan4092
admin@Spine2# commit

Verify Configuration

The above configuration uses the same default ES-DF preferences on all the three Leaf switches. The resulting effect is that the leaf switch with the smallest VTEP IP is selected as the DF. In the first EVPN MH redundancy group, Leaf2 is elected as the DF since it has the smallest VTEP IP in the group. Similarly in the second redundancy group, Leaf3 has the smallest VTEP IP and is elected the DF.

admin@Leaf1# run show evpn es 
Type: B bypass, L local, R remote, N non-DF
ESI                               Type   ES-IF           VTEPs
03:00:22:22:22:22:22:00:00:04     LRN   ae1           10.226.14.253
03:00:22:22:22:22:22:00:00:02     LRN   ae2           10.226.14.253

admin@Leaf2# run show evpn es 
Type: B bypass, L local, R remote, N non-DF
ESI                               Type   ES-IF            VTEPs
03:00:22:22:22:22:22:00:00:04     LR    ae1            10.226.14.254
03:00:22:22:22:22:22:00:00:02     LR    ae2            10.226.14.254

admin@Leaf3# run show evpn es
Type: B bypass, L local, R remote, N non-DF
ESI                                Type   ES-IF          VTEPs
03:00:00:00:22:22:22:00:00:03      LR    ae1           10.226.14.202
03:00:00:00:22:22:22:00:00:01      LR    ae2           10.226.14.202

admin@Leaf4# run show evpn es
Type: B bypass, L local, R remote, N non-DF
ESI                                Type   ES-IF           VTEPs
03:00:00:00:22:22:22:00:00:01      LRN    ae1          10.226.14.201
03:00:00:00:22:22:22:00:00:03      LRN    ae2          10.226.14.201


Now let’s change the ES-DF preference on Leaf1 to 65535 and see the output again. This time Leaf1 assumes the role of DF.

admin@Leaf1# set interface aggregate-ethernet ae1 evpn mh es-df-pref 65535
admin@Leaf1# set interface aggregate-ethernet ae2 evpn mh es-df-pref 65535
admin@Leaf1# commit 
Commit OK.
Save done

admin@Leaf1# run show evpn es 
Type: B bypass, L local, R remote, N non-DF
ESI                           Type   ES-IF              VTEPs
03:00:22:22:22:22:22:00:00:04     LR    ae1              10.226.14.253
03:00:22:22:22:22:22:00:00:02     LR    ae2              10.226.14.253

admin@Leaf2# run show evpn es 
Type: B bypass, L local, R remote, N non-DF 
ESI                           Type   ES-IF              VTEPs
03:00:22:22:22:22:22:00:00:04     LRN    ae1            10.226.14.254
03:00:22:22:22:22:22:00:00:02     LRN    ae2            10.226.14.254

To check the EVPN ES detail information, run the following show command. This command shows important information such as ESI, Type, interface, state, DF status and the VTEPs etc.

admin@Leaf3# run show evpn es detail 
ESI: 03:00:00:00:22:22:22:00:00:03
 Type: Local,Remote
 Interface: ae1
 State: up
 Bridge port: yes
 Ready for BGP: yes
 VNI Count: 2
 MAC Count: 0
 DF status: df 
 DF preference: 32767
 Nexthop group: 536870913
 VTEPs:
     10.226.14.202 df_alg: preference df_pref: 32767 nh: 268435459
ESI: 03:00:00:00:22:22:22:00:00:01
 Type: Local,Remote
 Interface: ae2
 State: up
 Bridge port: yes
 Ready for BGP: yes
 VNI Count: 2
 MAC Count: 0
 DF status: df 
 DF preference: 32767
 Nexthop group: 536870914
 VTEPs:
     10.226.14.202 df_alg: preference df_pref: 32767 nh: 268435459

Check the VXLAN address table with the following show command.

admin@Leaf3# run show vxlan address-table 
VNID           MAC address          Type       Interface          VTEP
-----------    -----------------    -------    ----------------    ---------------
99             04:f8:f8:20:67:7b    Dynamic                        10.226.14.254  
99             18:5a:58:37:55:e1    Dynamic                        10.226.14.253  
10020          00:1e:c9:bb:c0:3c    Dynamic       ge-1/1/19                          
10020          04:f8:f8:20:67:7b    Dynamic                        10.226.14.254  
10020          18:5a:58:37:55:e1    Dynamic                        10.226.14.253  
10030          04:f8:f8:20:67:7b    Dynamic                        10.226.14.254  
10030          18:5a:58:37:55:e1    Dynamic                        10.226.14.253  
10030          22:11:11:00:00:00    Dynamic       ae1                                
Entries in access port:  2
Entries in network port: 7

To display the number of MACs learned both locally and remotely in a VNI run the following show command.

admin@Leaf3# run show evpn mac vni 10030 
Number of MACs (local and remote) known for this VNI: 4
Flags: N=sync-neighs, I=local-inactive, P=peer-active, X=peer-proxy
MAC               Type   Flags  Intf/Remote ES/VTEP     VLAN  Seq #'s
22:11:11:00:00:00     local    P     ae1                      30    0/0
18:5a:58:37:55:e1     remote        10.226.14.253                    0/0
18:5a:58:37:64:61     local          vlan30                   30    0/0
04:f8:f8:20:67:7b     remote        10.226.14.254                    0/0

To get the tunnel information within a VNI, run the following show command.

admin@Leaf3# run show vxlan tunnel vni 10030 
Total number of tunnels: 2

VNI 10030, Encap:service-vlan-delete, Decap:service-vlan-per-port
  src addr:10.226.14.202, dst addr:10.226.14.253, state:UP
  traffic type:all
  Vtep type:EVPN 
  nexthops:10.226.15.69 10.226.15.77 
  output ports:ge-1/1/5 ge-1/1/3 

  src addr:10.226.14.202, dst addr:10.226.14.254, state:UP
  traffic type:all
  Vtep type:EVPN 
  nexthops:10.226.15.69 10.226.15.77 
  output ports:ge-1/1/5 ge-1/1/3 

To get more info on interfaces belonging to a particular VNI, run the following show command.

admin@Leaf3# run show vxlan vni 10030
Interface      ID             Type           Egress     Vlan ID
-----------    -----------    -----------    -------    --------
ae1            0x80000005    Access          100012     30      
ae2            0x80000006    Access          100013     30      
ge-1/1/23      0x80000007    Access         100014      30      
ge-1/1/5       0x80000008    Network(UC)    200258             
ge-1/1/5       0x8000000a    Network(MC)    100020             
ge-1/1/5       0x8000000b    Network(UC)    200258             
ge-1/1/5       0x8000000d    Network(MC)    100022             
ge-1/1/3       0x8000000f    Network(MC)    100024             
ge-1/1/3       0x80000010    Network(UC)    100017              

Show the VXLAN MAC address information on Leaf1. Port ge-1/1/7 is single homed host connected to a VXLAN access port in VNID 10030 with MAC address 22:00:00:00:00:00.

admin@Leaf1# run show vxlan address-table 
VNID           MAC address          Type       Interface          VTEP
-----------    -----------------    -------    ----------------    ---------------
99             18:5a:58:37:55:e1    Dynamic                        10.226.14.253  
10020          18:5a:58:37:55:e1    Dynamic                        10.226.14.253  
10030          18:5a:58:37:55:e1    Dynamic                        10.226.14.253  
10030          22:00:00:00:00:00    Dynamic    ge-1/1/7                           
Entries in access port:  1
Entries in network port: 3

Show the BGP EVPN route information.

admin@Leaf1# run show bgp evpn route
BGP table version is 19, local router ID is 10.226.14.207
Status codes: s suppressed, d damped, h history, * valid, > best, i - internal
Origin codes: i - IGP, e - EGP, ? - incomplete
EVPN type-1 prefix: [1]:[ESI]:[EthTag]:[IPlen]:[VTEP-IP]
EVPN type-2 prefix: [2]:[EthTag]:[MAClen]:[MAC]:[IPlen]:[IP]
EVPN type-3 prefix: [3]:[EthTag]:[IPlen]:[OrigIP]
EVPN type-4 prefix: [4]:[ESI]:[IPlen]:[OrigIP]
EVPN type-5 prefix: [5]:[EthTag]:[IPlen]:[IP]

   Network          Next Hop            Metric LocPrf Weight Path
                    Extended Community
Route Distinguisher: 10.226.14.24:3
*>i[2]:[0]:[48]:[00:00:20:20:00:fe]:[32]:[192.168.20.1]
                    10.226.14.201                 100      0 i
                    RT:65001:99 RT:65001:10020 ET:8 Rmac:50:9a:4c:e6:7e:71
*>i[2]:[0]:[48]:[00:00:20:20:00:fe]:[128]:[2002:0:0:1::1]
                    10.226.14.201                 100      0 i
                    RT:65001:99 RT:65001:10020 ET:8 Rmac:50:9a:4c:e6:7e:71
*>i[2]:[0]:[48]:[00:00:20:20:00:fe]:[128]:[fe80::529a:4c20:1e6:7e71]
                    10.226.14.201                 100      0 i
                    RT:65001:10020 ET:8
*>i[2]:[0]:[48]:[00:0c:29:77:8b:15]
                    10.226.14.201                 100      0 i
                    RT:65001:10020 ET:8
*>i[2]:[0]:[48]:[00:0c:29:77:8b:15]:[128]:[fe80::20c:29ff:fe77:8b15]
                    10.226.14.201                 100      0 i
                    RT:65001:10020 ET:8
Route Distinguisher: 10.226.14.24:4
*>i[2]:[0]:[48]:[00:00:30:30:00:fe]:[32]:[192.168.30.1]
                    10.226.14.201                 100      0 i
                    RT:65001:99 RT:65001:10030 ET:8 Rmac:50:9a:4c:e6:7e:71
*>i[2]:[0]:[48]:[00:00:30:30:00:fe]:[128]:[2003:0:0:1::1]
                    10.226.14.201                 100      0 i
                    RT:65001:99 RT:65001:10030 ET:8 Rmac:50:9a:4c:e6:7e:71
*>i[2]:[0]:[48]:[00:00:30:30:00:fe]:[128]:[fe80::529a:4c20:2e6:7e71]
                    10.226.14.201                 100      0 i
                    RT:65001:10030 ET:8
*>i[2]:[0]:[48]:[00:0c:29:77:8b:15]
                    10.226.14.201                 100      0 i
                    RT:65001:10030 ET:8
Route Distinguisher: 10.226.14.48:3
*>i[1]:[0]:[03:00:22:22:22:22:22:00:00:01]:[32]:[0.0.0.0]
                    10.226.14.202                 100      0 i
                    RT:65001:10020 ET:8
*>i[1]:[0]:[03:00:22:22:22:22:22:00:00:02]:[32]:[0.0.0.0]
                    10.226.14.202                 100      0 i
                    RT:65001:10020 ET:8
Route Distinguisher: 10.226.14.48:4
*>i[1]:[4294967295]:[03:00:22:22:22:22:22:00:00:01]:[32]:[0.0.0.0]
                    10.226.14.202                 100      0 i
                    RT:65001:10020 RT:65001:10030 ET:8 ESI-label-Rt:AA
*>i[4]:[03:00:22:22:22:22:22:00:00:01]:[32]:[10.226.14.202]
                    10.226.14.202                 100      0 i
                    ET:8 ES-Import-Rt:00:22:22:22:22:22 DF: (alg: 2, pref: 32767)
Route Distinguisher: 10.226.14.48:5
*>i[1]:[4294967295]:[03:00:22:22:22:22:22:00:00:02]:[32]:[0.0.0.0]
                    10.226.14.202                 100      0 i
                    RT:65001:10020 RT:65001:10030 ET:8 ESI-label-Rt:AA
*>i[4]:[03:00:22:22:22:22:22:00:00:02]:[32]:[10.226.14.202]
                    10.226.14.202                 100      0 i
                    ET:8 ES-Import-Rt:00:22:22:22:22:22 DF: (alg: 2, pref: 32767)
Route Distinguisher: 10.226.14.48:6
*>i[1]:[0]:[03:00:22:22:22:22:22:00:00:01]:[32]:[0.0.0.0]
                    10.226.14.202                 100      0 i
                    RT:65001:10030 ET:8
*>i[1]:[0]:[03:00:22:22:22:22:22:00:00:02]:[32]:[0.0.0.0]
                    10.226.14.202                 100      0 i
                    RT:65001:10030 ET:8
*>i[2]:[0]:[48]:[00:1e:c9:bb:c0:3c]:[32]:[192.168.30.42]
                    10.226.14.202                 100      0 i
                    ESI:03:00:22:22:22:22:22:00:00:01 VNI: 0
                    RT:65001:99 RT:65001:10030 ET:8 Rmac:18:5a:58:37:64:61
*>i[2]:[0]:[48]:[00:1e:c9:bb:c0:3c]:[128]:[2003:0:0:1::42]
                    10.226.14.202                 100      0 i
                    ESI:03:00:22:22:22:22:22:00:00:01 VNI: 0
                    RT:65001:99 RT:65001:10030 ET:8 Rmac:18:5a:58:37:64:61
*>i[2]:[0]:[48]:[00:1e:c9:bb:c0:3c]:[128]:[fe80::21e:c9ff:febb:c03c]
                    10.226.14.202                 100      0 i
                    ESI:03:00:22:22:22:22:22:00:00:01 VNI: 0
                    RT:65001:10030 ET:8
Route Distinguisher: 10.226.14.207:3
*> [1]:[0]:[03:00:22:22:22:22:22:00:00:01]:[128]:[::]
                    10.226.14.254                      32768 i
                    ET:8 RT:65001:10020
*> [1]:[0]:[03:00:22:22:22:22:22:00:00:02]:[128]:[::]
                    10.226.14.254                      32768 i
                    ET:8 RT:65001:10020
*> [2]:[0]:[48]:[00:00:20:00:00:fe]
                    10.226.14.254                      32768 i
                    ET:8 RT:65001:10020
*> [2]:[0]:[48]:[00:00:20:00:00:fe]:[32]:[192.168.20.2]
                    10.226.14.254                      32768 i
                    ET:8 RT:65001:10020 RT:65001:99 Rmac:04:f8:f8:20:67:7b
*> [2]:[0]:[48]:[00:00:20:00:00:fe]:[128]:[2002:0:0:1::2]
                    10.226.14.254                      32768 i
                    ET:8 RT:65001:10020 RT:65001:99 Rmac:04:f8:f8:20:67:7b
*> [2]:[0]:[48]:[00:00:20:00:00:fe]:[128]:[fe80::6f8:f810:120:677b]
                    10.226.14.254                      32768 i
                    ET:8 RT:65001:10020
Route Distinguisher: 10.226.14.207:4
*> [1]:[4294967295]:[03:00:22:22:22:22:22:00:00:01]:[128]:[::]
                    10.226.14.254                      32768 i
                    ET:8 ESI-label-Rt:AA RT:65001:10020 RT:65001:10030
*> [4]:[03:00:22:22:22:22:22:00:00:01]:[32]:[10.226.14.254]
                    10.226.14.254                      32768 i
                    ET:8 ES-Import-Rt:00:22:22:22:22:22 DF: (alg: 2, pref: 1)
Route Distinguisher: 10.226.14.207:5
*> [1]:[4294967295]:[03:00:22:22:22:22:22:00:00:02]:[128]:[::]
                    10.226.14.254                      32768 i
                    ET:8 ESI-label-Rt:AA RT:65001:10020 RT:65001:10030
*> [4]:[03:00:22:22:22:22:22:00:00:02]:[32]:[10.226.14.254]
                    10.226.14.254                      32768 i
                    ET:8 ES-Import-Rt:00:22:22:22:22:22 DF: (alg: 2, pref: 1)
Route Distinguisher: 10.226.14.207:6
*> [1]:[0]:[03:00:22:22:22:22:22:00:00:01]:[128]:[::]
                    10.226.14.254                      32768 i
                    ET:8 RT:65001:10030
*> [1]:[0]:[03:00:22:22:22:22:22:00:00:02]:[128]:[::]
                    10.226.14.254                      32768 i
                    ET:8 RT:65001:10030
*> [2]:[0]:[48]:[00:00:30:00:00:fe]
                    10.226.14.254                      32768 i
                    ET:8 RT:65001:10030
*> [2]:[0]:[48]:[00:00:30:00:00:fe]:[32]:[192.168.30.2]
                    10.226.14.254                      32768 i
                    ET:8 RT:65001:10030 RT:65001:99 Rmac:04:f8:f8:20:67:7b
*> [2]:[0]:[48]:[00:00:30:00:00:fe]:[128]:[2003:0:0:1::2]
                    10.226.14.254                      32768 i
                    ET:8 RT:65001:10030 RT:65001:99 Rmac:04:f8:f8:20:67:7b
*> [2]:[0]:[48]:[00:00:30:00:00:fe]:[128]:[fe80::6f8:f810:220:677b]
                    10.226.14.254                      32768 i
                    ET:8 RT:65001:10030
*> [2]:[0]:[48]:[00:1e:c9:bb:c0:3c]:[32]:[192.168.30.42]
                    10.226.14.254                      32768 i
                    ESI:03:00:22:22:22:22:22:00:00:01
                    ET:8 RT:65001:10030 RT:65001:99 Rmac:04:f8:f8:20:67:7b ND:Proxy
*> [2]:[0]:[48]:[00:1e:c9:bb:c0:3c]:[128]:[2003:0:0:1::42]
                    10.226.14.254                      32768 i
                    ESI:03:00:22:22:22:22:22:00:00:01
                    ET:8 RT:65001:10030 RT:65001:99 Rmac:04:f8:f8:20:67:7b ND:Proxy
*> [2]:[0]:[48]:[00:1e:c9:bb:c0:3c]:[128]:[fe80::21e:c9ff:febb:c03c]
                    10.226.14.254                      32768 i
                    ESI:03:00:22:22:22:22:22:00:00:01
                    ET:8 RT:65001:10030 ND:Proxy
Route Distinguisher: 10.226.14.208:3
*>i[1]:[0]:[03:00:22:22:22:22:22:00:00:01]:[32]:[0.0.0.0]
                    10.226.14.253                 100      0 i
                    RT:65001:10020 ET:8
*>i[1]:[0]:[03:00:22:22:22:22:22:00:00:02]:[32]:[0.0.0.0]
                    10.226.14.253                 100      0 i
                    RT:65001:10020 ET:8
Route Distinguisher: 10.226.14.208:4
*>i[1]:[4294967295]:[03:00:22:22:22:22:22:00:00:01]:[32]:[0.0.0.0]
                    10.226.14.253                 100      0 i
                    RT:65001:10020 RT:65001:10030 ET:8 ESI-label-Rt:AA
*>i[4]:[03:00:22:22:22:22:22:00:00:01]:[32]:[10.226.14.253]
                    10.226.14.253                 100      0 i
                    ET:8 ES-Import-Rt:00:22:22:22:22:22 DF: (alg: 2, pref: 32767)
Route Distinguisher: 10.226.14.208:5
*>i[1]:[4294967295]:[03:00:22:22:22:22:22:00:00:02]:[32]:[0.0.0.0]
                    10.226.14.253                 100      0 i
                    RT:65001:10020 RT:65001:10030 ET:8 ESI-label-Rt:AA
*>i[4]:[03:00:22:22:22:22:22:00:00:02]:[32]:[10.226.14.253]
                    10.226.14.253                 100      0 i
                    ET:8 ES-Import-Rt:00:22:22:22:22:22 DF: (alg: 2, pref: 32767)
Route Distinguisher: 10.226.14.208:6
*>i[1]:[0]:[03:00:22:22:22:22:22:00:00:01]:[32]:[0.0.0.0]
                    10.226.14.253                 100      0 i
                    RT:65001:10030 ET:8
*>i[1]:[0]:[03:00:22:22:22:22:22:00:00:02]:[32]:[0.0.0.0]
                    10.226.14.253                 100      0 i
                    RT:65001:10030 ET:8
*>i[2]:[0]:[48]:[00:1e:c9:bb:c0:3c]:[32]:[192.168.30.42]
                    10.226.14.253                 100      0 i
                    ESI:03:00:22:22:22:22:22:00:00:01 VNI: 0
                    RT:65001:99 RT:65001:10030 ET:8 Rmac:18:5a:58:37:55:e1 ND:Proxy
*>i[2]:[0]:[48]:[00:1e:c9:bb:c0:3c]:[128]:[2003:0:0:1::42]
                    10.226.14.253                 100      0 i
                    ESI:03:00:22:22:22:22:22:00:00:01 VNI: 0
                    RT:65001:99 RT:65001:10030 ET:8 Rmac:18:5a:58:37:55:e1 ND:Proxy
*>i[2]:[0]:[48]:[00:1e:c9:bb:c0:3c]:[128]:[fe80::21e:c9ff:febb:c03c]
                    10.226.14.253                 100      0 i
                    ESI:03:00:22:22:22:22:22:00:00:01 VNI: 0
                    RT:65001:10030 ET:8 ND:Proxy
Route Distinguisher: 24.24.24.24:2
*>i[5]:[0]:[24]:[172.16.0.0]
                    10.226.14.201            0    100      0 ?
                    RT:65001:99 ET:8 Rmac:50:9a:4c:e6:7e:71
*>i[5]:[0]:[24]:[192.168.20.0]
                    10.226.14.201            0    100      0 ?
                    RT:65001:99 ET:8 Rmac:50:9a:4c:e6:7e:71
*>i[5]:[0]:[24]:[192.168.30.0]
                    10.226.14.201            0    100      0 ?
                    RT:65001:99 ET:8 Rmac:50:9a:4c:e6:7e:71
*>i[5]:[0]:[64]:[2002:0:0:1::]
                    10.226.14.201            0    100      0 ?
                    RT:65001:99 ET:8 Rmac:50:9a:4c:e6:7e:71
*>i[5]:[0]:[64]:[2003:0:0:1::]
                    10.226.14.201            0    100      0 ?
                    RT:65001:99 ET:8 Rmac:50:9a:4c:e6:7e:71
*>i[5]:[0]:[64]:[2006:0:0:1::]
                    10.226.14.201            0    100      0 ?
                    RT:65001:99 ET:8 Rmac:50:9a:4c:e6:7e:71

Displayed 56 prefixes (56 paths)
  • No labels