Spine and Leaf Topology

The topology uses an EVPN-VXLAN Underlay using OSPF for the internal gateway protocol (IGP) and eBGP with unnumbered interfaces for advertising host and loopback routes. It also utilizes Dynamic Load Balancing (DLB) and Priority Flow Control (PFC) for PFC Dead Lock Detection requirements.

Leaf switches connect to both the spines layer (uplinks) and the end-hosts (downlinks, often GPUs/Servers).

Figure 4. PFC Dead Lock Detection and Recovery

Configure Leaf 1, Leaf 2, Leaf 3, Leaf 4, Spine 1 and Spine 2

The following configurations, enables PFC deadlock recovery on every priority interfaces on spines and leafs.

Configure global settings - PFC, QoS, DCB, DLB, and load balance rtag7 hashing.

Copy

data-center-bridging enable bridge 1priority-flow-control enable bridge 1
dynamic-load-balance enable
maximum-paths 64
load-balance rtag7 ipv4 dest-ipv4 src-ipv4 destl4-port srcl4-port protocol-id rocev2-dest-qpairs
load-balance rtag7 ipv6 dest-ipv6 src-ipv6 destl4-port srcl4-port rocev2-dest-qpairs next-hdr
 policy-map type queuing default default-out-policy
class type queuing default q6
 priority
 lossless
 exit
class type queuing default q7
priority
lossless
exit

Configure PFC default policy for all 8 traffic classes.

Copy

(config)#policy-map type queuing default default-out-policy
(config-cmap-que)#class type queuing default q0
(config-cmap-que)#priority
(config-cmap-que)#lossless
(config-cmap-que)#exit
(config)#class type queuing default q1
(config-cmap-que)#priority
(config-cmap-que)#lossless
(config-cmap-que)#exit
(config)#class type queuing default q2
(config-cmap-que)#priority
(config-cmap-que)#lossless
(config-cmap-que)#exit
(config)#class type queuing default q3
(config-cmap-que)#priority
(config-cmap-que)#lossless
(config-cmap-que)#exit
(config)#class type queuing default q4
(config-cmap-que)#priority
(config-cmap-que)#lossless
(config-cmap-que)#exit
(config)#class type queuing default q5
(config-cmap-que)#priority
(config-cmap-que)#lossless
(config-cmap-que)#exit
(config)#class type queuing default q6
(config-cmap-que)# priority
(config-cmap-que)# lossless
(config-cmap-que)# exit
(config)#class type queuing default q7
(config-cmap-que)#priority
(config-cmap-que)#lossless
(config-cmap-que)#exit

Configure PFC default lossless ECN policy.

Copy

(config)#policy-map type queuing default lossless_ecn_egress
(config-pmap-que-def)#class type queuing default q0
(config-pmap-c-que-def)#shape 10 gbps
(config-pmap-c-que-def)#priority
(config-pmap-c-que-def)#lossless
(config-pmap-c-que-def)#random-detect green min-threshold 500 max-threshold 600 yellow min-threshold 300 max-threshold 400 red min-threshold 100 max-threshold 200 packets ecn
 (config-pmap-c-que-def)#exit
 (config-pmap-que-def)#class type queuing default q1
 (config-pmap-c-que-def)#shape 10 gbps
 (config-pmap-c-que-def)#priority
 (config-pmap-c-que-def)#lossless
 (config-pmap-c-que-def)#random-detect green min-threshold 500 max-threshold 600 yellow min-threshold 300 max-threshold 400 red min-threshold 100 max-threshold 200 packets ecn
 (config-pmap-c-que-def)#exit
 (config-pmap-que-def)#class type queuing default q2
 (config-pmap-c-que-def)#shape 10 gbps
 (config-pmap-c-que-def)#priority
 (config-pmap-c-que-def)#lossless
  random-detect green min-threshold 500 max-threshold 600 yellow min-threshold 300 max-threshold 400 red min-threshold 100 max-threshold 200 packets ecn
 (config-pmap-c-que-def)#exit
 (config-pmap-que-def)#class type queuing default q3
 (config-pmap-c-que-def)#shape 10 gbps
 (config-pmap-c-que-def)#priority
 (config-pmap-c-que-def)#lossless
 (config-pmap-c-que-def)#random-detect green min-threshold 500 max-threshold 600 yellow min-threshold 300 max-threshold 400 red min-threshold 100 max-threshold 200 packets ecn
 (config-pmap-c-que-def)#exit
 (config-pmap-que-def)#class type queuing default q4
 (config-pmap-c-que-def)#shape 10 gbps
 (config-pmap-c-que-def)#priority
 (config-pmap-c-que-def)#lossless
 (config-pmap-c-que-def)#random-detect green min-threshold 500 max-threshold 600 yellow min-threshold 300 max-threshold 400 red min-threshold 100 max-threshold 200 packets ecn
 (config-pmap-c-que-def)#exit
 (config-pmap-que-def)#class type queuing default q5
 (config-pmap-c-que-def)#shape 10 gbps
 (config-pmap-c-que-def)#priority
 (config-pmap-c-que-def)#lossless
 (config-pmap-c-que-def)#random-detect green min-threshold 500 max-threshold 600 yellow min-threshold 300 max-threshold 400 red min-threshold 100 max-threshold 200 packets ecn
 (config-pmap-c-que-def)#exit
 (config-pmap-que-def)#class type queuing default q6
 (config-pmap-c-que-def)#shape 10 gbps
 (config-pmap-c-que-def)#priority
 (config-pmap-c-que-def)#lossless
 (config-pmap-c-que-def)#random-detect green min-threshold 500 max-threshold 600 yellow min-threshold 300 max-threshold 400 red min-threshold 100 max-threshold 200 packets ecn
 (config-pmap-c-que-def)#exit
 (config-pmap-que-def)#class type queuing default q7
 (config-pmap-c-que-def)#shape 10 gbps
 (config-pmap-c-que-def)#priority
 (config-pmap-c-que-def)#lossless
 (config-pmap-c-que-def)#random-detect green min-threshold 500 max-threshold 600 yellow min-threshold 300 max-threshold 400 red min-threshold 100 max-threshold 200 packets ecn
 (config-pmap-c-que-def)#exit

Configure routing map information to allow redistribution of routes. It is applied to incoming BGP routes, which sets the metric to 55555.

Copy

(config)#route-map HIG_MED permit 10
(config-route-map)#set metric 55555

Set the PFC mode to auto on the uplink interfaces.

On Leaf 1

Copy

(config)#interface cd1/1
(config-if)#priority-flow-control mode auto
(config-if)#ip address 10.1.12.1/24
(config-if)#ipv6 address 1012::1/64
(config-if)#shape rate 5 gbps burst 10000
(config-if)#ip ospf cost 100
(config-if)#ipv6 ospf cost 100 instance-id 0
(config-if)#ipv6 router ospf area 0.0.0.0 instance-id 0
(config-if)#lldp-agent
(config-if)#exit

(config)#interface cd3/1
(config-if)#priority-flow-control mode auto
(config-if)#ip address 10.1.11.1/24
(config-if)#ipv6 address 1011::1/64
(config-if)#shape rate 5 gbps burst 10000
(config-if)#ip ospf cost 100
(config-if)#ipv6 ospf cost 100 instance-id 0
(config-if)#ipv6 router ospf area 0.0.0.0 instance-id 0
(config-if)#lldp-agent
(config-if)#exit

(config)#interface cd2/1
(config-if)# priority-flow-control mode auto
(config-if)#ip address 20.1.11.1/24
(config-if)#ipv6 address 2011::1/64
(config-if)#shape rate 5 gbps burst 10000
(config-if)#ip ospf cost 100
(config-if)#ipv6 ospf cost 100 instance-id 0
(config-if)#ipv6 router ospf area 0.0.0.0 instance-id 0
(config-if)#lldp-agent
(config-if)#exit

(config)#interface cd4/1
(config-if)#priority-flow-control mode auto
(config-if)#ip address 20.1.12.1/24
(config-if)#ipv6 address 2012::1/64
(config-if)#shape rate 5 gbps burst 10000
(config-if)#ip ospf cost 100
(config-if)#ipv6 ospf cost 100 instance-id 0
(config-if)#ipv6 router ospf area 0.0.0.0 instance-id 0
(config-if)# lldp-agent
(config-if)# exit

(config)#interface cd29/1
(config-if)#priority-flow-control mode auto
(config-if)#ip address 30.1.11.1/24
(config-if)#ipv6 address 3011::1/64
(config-if)#shape rate 5 gbps burst 10000
(config-if)#ip ospf cost 100
(config-if)#ipv6 ospf cost 100 instance-id 0
(config-if)#ipv6 router ospf area 0.0.0.0 instance-id 0
(config-if)#lldp-agent
(config-if)#exit

(config)#interface cd32/1
(config-if)#priority-flow-control mode auto
(config-if)#ip address 30.1.12.1/24
(config-if)#ipv6 address 3012::1/64
(config-if)#shape rate 5 gbps burst 10000
(config-if)#ip ospf cost 100
(config-if)#ipv6 ospf cost 100 instance-id 0
(config-if)#ipv6 router ospf area 0.0.0.0 instance-id 0
(config-if)#lldp-agent
(config-if)#exit

On Leaf 2

Copy

(config)#interface ethernet0/1
(config-if)#priority-flow-control mode auto
(config-if)#ip address 10.1.13.1/24
(config-if)#ipv6 address 1013::1/64
(config-if)#shape rate 5 gbps burst 10000
(config-if)#ip ospf cost 100
(config-if)#ipv6 ospf cost 100 instance-id 0
(config-if)#ipv6 router ospf area 0.0.0.0 instance-id 0
(config-if)#lldp-agent
(config-if)#exit

(config)#interface ethernet32/1
(config-if)#priority-flow-control mode auto
(config-if)#ip address 10.1.14.1/24
(config-if)#ipv6 address 1014::1/64
(config-if)#shape rate 5 gbps burst 10000
(config-if)#ip ospf cost 100
(config-if)#ipv6 ospf cost 100 instance-id 0
(config-if)#ipv6 router ospf area 0.0.0.0 instance-id 0
(config-if)#lldp-agent
(config-if)#exit


(config)#interface ethernet30/1
(config-if)#priority-flow-control mode auto
(config-if)#ip address 20.1.13.1/24
(config-if)#ipv6 address 2013::1/64
(config-if)#shape rate 5 gbps burst 10000
(config-if)#ip ospf cost 100
(config-if)#ipv6 ospf cost 100 instance-id 0
(config-if)#ipv6 router ospf area 0.0.0.0 instance-id 0
(config-if)#lldp-agent
(config-if)#exit


(config)#interface ethernet33/1
(config-if)# priority-flow-control mode auto
(config-if)#ip address 20.1.14.1/24
(config-if)#ipv6 address 2014::1/64
(config-if)# shape rate 5 gbps burst 10000
(config-if)#ip ospf cost 100
(config-if)#ipv6 ospf cost 100 instance-id 0
(config-if)# ipv6 router ospf area 0.0.0.0 instance-id 0
(config-if)#lldp-agent
(config-if)#exit

(config)#interface ethernet10/1
(config-if)#priority-flow-control mode auto
(config-if)#ip address 30.1.13.1/24
(config-if)#ipv6 address 3013::1/64
(config-if)#shape rate 5 gbps burst 10000
(config-if)#ip ospf cost 100
(config-if)#ipv6 ospf cost 100 instance-id 0
(config-if)#ipv6 router ospf area 0.0.0.0 instance-id 0
(config-if)#lldp-agent
(config-if)#exit

(config)#interface ethernet11/1
(config-if)#priority-flow-control mode auto
(config-if)#ip address 30.1.14.1/24
(config-if)#ipv6 address 3014::1/64
(config-if)#shape rate 5 gbps burst 10000
(config-if)#ip ospf cost 100
(config-if)#ipv6 ospf cost 100 instance-id 0
(config-if)#ipv6 router ospf area 0.0.0.0 instance-id 0
(config-if)#lldp-agent
(config-if)#exit

On Leaf 3

Copy

(config)#interface cd2/1
(config-if)# priority-flow-control mode auto
(config-if)#ip address 10.1.16.1/24
(config-if)#ipv6 address 1016::1/64
(config-if)# shape rate 5 gbps burst 10000
(config-if)# ip ospf cost 100
(config-if)#ipv6 ospf cost 100 instance-id 0
(config-if)# ipv6 router ospf area 0.0.0.0 instance-id 0
(config-if)# lldp-agent
(config-if)# exit

(config)#interface cd0/1
(config-if)# priority-flow-control mode auto
(config-if)#ip address 10.1.15.1/24
(config-if)#ipv6 address 1015::1/64
(config-if)#shape rate 5 gbps burst 10000
(config-if)#ip ospf cost 100
(config-if)#ipv6 ospf cost 100 instance-id 0
(config-if)#ipv6 router ospf area 0.0.0.0 instance-id 0
(config-if)#lldp-agent
(config-if)#exit

(config)#interface cd10/1
(config-if)#priority-flow-control mode auto
(config-if)# ip address 20.1.15.1/24
(config-if)#ipv6 address 2015::1/64
(config-if)#shape rate 5 gbps burst 10000
(config-if)#ip ospf cost 100
(config-if)#ipv6 ospf cost 100 instance-id 0
(config-if)# ipv6 router ospf area 0.0.0.0 instance-id 0
(config-if)# lldp-agent
(config-if)# exit

(config)#interface cd5/1
(config-if)#priority-flow-control mode auto
(config-if)#ip address 20.1.16.1/24
(config-if)#ipv6 address 2016::1/64
(config-if)#shape rate 5 gbps burst 10000
(config-if)#ip ospf cost 100
(config-if)#ipv6 ospf cost 100 instance-id 0
(config-if)#ipv6 router ospf area 0.0.0.0 instance-id 0
(config-if)#lldp-agent
(config-if)#exit

(config)#interface cd3/1
(config-if)# priority-flow-control mode auto
(config-if)#ip address 30.1.15.1/24
(config-if)#ipv6 address 3015::1/64
(config-if)#shape rate 5 gbps burst 10000
(config-if)#ip ospf cost 100
(config-if)#ipv6 ospf cost 100 instance-id 0
(config-if)#ipv6 router ospf area 0.0.0.0 instance-id 0
(config-if)#lldp-agent
(config-if)#exit

(config)#interface cd4/1
(config-if)#priority-flow-control mode auto
(config-if)#ip address 30.1.16.1/24
(config-if)#ipv6 address 3016::1/64
(config-if)#shape rate 5 gbps burst 10000
(config-if)#ip ospf cost 100
(config-if)#ipv6 ospf cost 100 instance-id 0
(config-if)#ipv6 router ospf area 0.0.0.0 instance-id 0
(config-if)#lldp-agent
(config-if)#exit

On Leaf 4

Copy

(config)#interface ce1
(config-if)# priority-flow-control mode auto
(config-if)#ip address 10.1.17.1/24
(config-if)#ipv6 address 1017::1/64
(config-if)#ip ospf cost 100
(config-if)#ipv6 ospf cost 100 instance-id 0
(config-if)#ipv6 router ospf area 0.0.0.0 instance-id 0
(config-if)#lldp-agent
(config-if)#exit

(config)#interface ce5
(config-if)# priority-flow-control mode auto
(config-if)#ip address 10.1.18.1/24
(config-if)# ipv6 address 1018::1/64
(config-if)# ip ospf cost 100
(config-if)# ipv6 ospf cost 100 instance-id 0
(config-if)# ipv6 router ospf area 0.0.0.0 instance-id 0
(config-if)# lldp-agent
(config-if)# exit

Set PFC mode and DeadLock Detection Recovery timer on downlink interfaces.

On Leaf 1

Copy

(config)#interface cd6
(config-if)#description ***GPU ***
(config-if)#priority-flow-control mode on
(config-if)#priority-flow-control advertise-local-config
(config-if)#priority-flow-control enable priority 0 1 2 3 4 5 6 7
(config-if)#ip address 111.1.1.1/24
(config-if)#ipv6 address 111::1/64
(config-if)#ipv6 router ospf area 0.0.0.0 instance-id 0
(config-if)#lldp-agent
(config-if)#exit

(config)#interface cd31
(config-if)#description ***GPU ***
(config-if)#switchport
(config-if)#bridge-group 1
(config-if)#switchport mode trunk
(config-if)#switchport trunk allowed vlan add 111-115
(config-if)#priority-flow-control mode on
(config-if)#priority-flow-control advertise-local-config
(config-if)#priority-flow-control enable priority 0 1 2 3 4 5 6 7
(config-if)#lldp-agent
(config-if)#exit

(config)#interface xe65
(config-if)#description ***GPU ***
(config-if)#priority-flow-control mode on
(config-if)#priority-flow-control deadlock recovery-mode timer
(config-if)#priority-flow-control advertise-local-config
(config-if)#priority-flow-control enable priority 0 1 2 3 4 5 6 7
(config-if)#ip address 101.1.1.1/24
(config-if)#ipv6 address 101::1/64
(config-if)#ipv6 router ospf area 0.0.0.0 instance-id 0
(config-if)#lldp-agent
(config-if)#exit

The 'deadlock recovery-mode timer' command configured only on xe65 and not explicitly on cd6 or cd31.

On Leaf 2

Copy

(config)#interface ethernet3/1
(config-if)#description ***GPU ***
(config-if)#priority-flow-control mode on
(config-if)#priority-flow-control deadlock recovery-mode timer
(config-if)#priority-flow-control advertise-local-config
(config-if)#priority-flow-control enable priority 0 1 2 3 4 5 6 7
(config-if)#ip address 102.1.1.1/24
(config-if)#ipv6 address 102::1/64
(config-if)#ipv6 router ospf area 0.0.0.0 instance-id 0
(config-if)#lldp-agent
(config-if)#exit

(config)#interface ethernet6/1
(config-if)#description ***GPU ***
(config-if)#priority-flow-control mode on
(config-if)#priority-flow-control deadlock recovery-mode timer
(config-if)#priority-flow-control advertise-local-config
(config-if)#priority-flow-control enable priority 0 1 2 3 4 5 6 7
(config-if)#ip address 112.1.1.1/24
(config-if)#ipv6 address 112::1/64
(config-if)#ipv6 router ospf area 0.0.0.0 instance-id 0
(config-if)#lldp-agent
(config-if)#exit

(config)#interface ethernet8/1
(config-if)# description ***GPU ***
(config-if)# switchport
(config-if)# bridge-group 1
(config-if)#switchport mode trunk
(config-if)#switchport trunk allowed vlan add 211-215
(config-if)#priority-flow-control mode on
(config-if)#priority-flow-control deadlock recovery-mode timer
(config-if)#priority-flow-control advertise-local-config
(config-if)#priority-flow-control enable priority 0 1 2 3 4 5 6 7
(config-if)# lldp-agent
(config-if)#exit

On Leaf 3

Copy

(config)#interface cd7
(config-if)# description ***GPU ***
(config-if)#switchport
(config-if)#bridge-group 1
(config-if)#switchport mode trunk
(config-if)#switchport trunk allowed vlan add 311-315
(config-if)# priority-flow-control mode on
(config-if)# priority-flow-control deadlock recovery-mode timer
(config-if)#priority-flow-control advertise-local-config
(config-if)#priority-flow-control enable priority 0 1 2 3 4 5 6 7
(config-if)#lldp-agent
(config-if) exit

(config)#interface cd6
(config-if)#description ***GPU ***
(config-if)#priority-flow-control mode on
(config-if)#priority-flow-control deadlock recovery-mode timer
(config-if)#priority-flow-control advertise-local-config
(config-if)#priority-flow-control enable priority 0 1 2 3 4 5 6 7
(config-if)#ip address 113.1.1.1/24
(config-if)#ipv6 address 113::1/64
(config-if)#ipv6 router ospf area 0.0.0.0 instance-id 0
(config-if)#lldp-agent
(config-if)#exit

(config-if)#interface xe0
(config-if)# description ***GPU ***
(config-if)#priority-flow-control mode on
(config-if)# priority-flow-control deadlock recovery-mode timer
(config-if)# priority-flow-control advertise-local-config
(config-if)# priority-flow-control enable priority 0 1 2 3 4 5 6 7
(config-if)#ip address 103.1.1.1/24
(config-if)#ipv6 address 103::1/64
(config-if)#ipv6 router ospf area 0.0.0.0 instance-id 0
(config-if)# lldp-agent
(config-if)# exit

On Leaf 4

Copy

(config)#interface ce7
(config-if)#description ***GPU ***
(config-if)#priority-flow-control mode on
(config-if)#priority-flow-control deadlock recovery-mode timer
(config-if)# priority-flow-control advertise-local-config
(config-if)# priority-flow-control enable priority 0 1 2 3 4 5 6 7
(config-if)#ip address 114.1.1.1/24
(config-if)# ipv6 address 114::1/64
(config-if)#ipv6 router ospf area 0.0.0.0 instance-id 0
(config-if)#lldp-agent
(config-if)# exit

(config)#interface ce8
(config-if)#description ***GPU  ***
(config-if)#priority-flow-control mode on
(config-if)#priority-flow-control deadlock recovery-mode timer
(config-if)#priority-flow-control advertise-local-config
(config-if)#priority-flow-control enable priority 0 1 2 3 4 5 6 7
(config-if)#ip address 124.1.1.1/24
(config-if)#ipv6 address 124::1/64
(config-if)#ipv6 router ospf area 0.0.0.0 instance-id 0
(config-if)#lldp-agent
(config-if)#exit

(config)#interface xe34
(config-if)#description ***GPU ***
(config-if)#priority-flow-control mode on
(config-if)#priority-flow-control deadlock recovery-mode timer
(config-if)#priority-flow-control advertise-local-config
(config-if)#priority-flow-control enable priority 0 1 2 3 4 5 6 7
(config-if)#ip address 104.1.1.1/24
(config-if)#ipv6 address 104::1/64
(config-if)#ipv6 router ospf area 0.0.0.0 instance-id 0
(config-if)#lldp-agent
(config-if)#exit

Configure interfaces on spines that connects to leafs.

On Spine 1

Copy

(config-if)#interface ethernet1/1
(config-if)#priority-flow-control mode on
(config-if)#priority-flow-control deadlock recovery-mode timer
(config-if)#priority-flow-control advertise-local-config
(config-if)#priority-flow-control enable priority 0 1 2 3 4 5 6 7
(config-if)#load-interval 30
(config-if)#ip address 10.1.17.2/24
(config-if)#ipv6 address 1017::2/64
(config-if)#ip ospf cost 100
(config-if)#ipv6 ospf cost 100 instance-id 0
(config-if)#ipv6 router ospf area 0.0.0.0 instance-id 0
 (config-if)#lldp-agent
 (config-if)#exit

(config-if)#interface ethernet2/1
(config-if)#priority-flow-control mode on
(config-if)#priority-flow-control deadlock recovery-mode timer
(config-if)#priority-flow-control advertise-local-config
(config-if)#priority-flow-control enable priority 0 1 2 3 4 5 6 7
(config-if)#load-interval 30
(config-if)#ip address 20.1.11.2/24
(config-if)#ipv6 address 2011::2/64
(config-if)#ip ospf cost 100
(config-if)#ipv6 ospf cost 100 instance-id 0
(config-if)#ipv6 router ospf area 0.0.0.0 instance-id 0
(config-if)#lldp-agent
(config-if)#exit

(config)#interface ethernet3/1
(config-if)#priority-flow-control mode on
(config-if)#priority-flow-control deadlock recovery-mode timer
(config-if)#priority-flow-control advertise-local-config
(config-if)#priority-flow-control enable priority 0 1 2 3 4 5 6 7
(config-if)#load-interval 30
(config-if)#ip address 10.1.11.2/24
(config-if)#ipv6 address 1011::2/64
(config-if)#ip ospf cost 100
(config-if)#ipv6 ospf cost 100 instance-id 0
(config-if)#ipv6 router ospf area 0.0.0.0 instance-id 0
(config-if)#lldp-agent
(config-if)#exit

(config-if)#interface ethernet30/1
(config-if)#priority-flow-control mode on
(config-if)#priority-flow-control deadlock recovery-mode timer
(config-if)#priority-flow-control advertise-local-config
(config-if)#priority-flow-control enable priority 0 1 2 3 4 5 6 7
(config-if)#load-interval 30
(config-if)#ip address 20.1.13.2/24
(config-if)#ipv6 address 2013::2/64
(config-if)#ip ospf cost 100
(config-if)#ipv6 ospf cost 100 instance-id 0
(config-if)#ipv6 router ospf area 0.0.0.0 instance-id 0
(config-if)#lldp-agent
(config-if)#exit

(config-if)#interface ethernet31/1
(config-if)#priority-flow-control mode on
(config-if)#priority-flow-control deadlock recovery-mode timer
(config-if)#priority-flow-control advertise-local-config
(config-if)#priority-flow-control enable priority 0 1 2 3 4 5 6 7
 (config-if)#load-interval 30
 (config-if)#ip address 10.1.15.2/24
 (config-if)#ipv6 address 1015::2/64
 (config-if)#ip ospf cost 100
 (config-if)#ipv6 ospf cost 100 instance-id 0
 (config-if)#ipv6 router ospf area 0.0.0.0 instance-id 0
 (config-if)#lldp-agent
 (config-if)#exit

(config)#interface ethernet32/1
(config-if)#priority-flow-control mode on
(config-if)#priority-flow-control deadlock recovery-mode timer
(config-if)#priority-flow-control advertise-local-config
(config-if)#priority-flow-control enable priority 0 1 2 3 4 5 6 7
(config-if)#load-interval 30
(config-if)#ip address 20.1.15.2/24
(config-if)#ipv6 address 2015::2/64
(config-if)#ip ospf cost 100
(config-if)#ipv6 ospf cost 100 instance-id 0
(config-if)#ipv6 router ospf area 0.0.0.0 instance-id 0
(config-if)#lldp-agent
(config-if)#exit

(config)#interface ethernet62/1
(config-if)#priority-flow-control mode on
(config-if)#priority-flow-control deadlock recovery-mode timer
(config-if)#priority-flow-control advertise-local-config
(config-if)#priority-flow-control enable priority 0 1 2 3 4 5 6 7
(config-if)#load-interval 30
(config-if)#ip address 30.1.11.2/24
(config-if)#ipv6 address 3011::2/64
(config-if)#ip ospf cost 100
(config-if)#ipv6 ospf cost 100 instance-id 0
(config-if)#ipv6 router ospf area 0.0.0.0 instance-id 0
(config-if)#lldp-agent
(config-if)#exit

(config)#interface ethernet63/1
(config-if)#priority-flow-control mode on
(config-if)#priority-flow-control deadlock recovery-mode timer
(config-if)#priority-flow-control advertise-local-config
(config-if)#priority-flow-control enable priority 0 1 2 3 4 5 6 7
(config-if)#load-interval 30
(config-if)#ip address 30.1.15.2/24
(config-if)#ipv6 address 3015::2/64
(config-if)#ip ospf cost 100
(config-if)#ipv6 ospf cost 100 instance-id 0
(config-if)#ipv6 router ospf area 0.0.0.0 instance-id 0
(config-if)#lldp-agent
(config-if)#exit
(config)#interface ethernet64/1
(config-if)# priority-flow-control mode on
(config-if)#priority-flow-control deadlock recovery-mode timer
(config-if)# priority-flow-control advertise-local-config
(config-if)#priority-flow-control enable priority 0 1 2 3 4 5 6 7
(config-if)#load-interval 30
(config-if)#ip address 10.1.13.2/24
(config-if)#ipv6 address 1013::2/64
(config-if)#ip ospf cost 100
(config-if)#ipv6 ospf cost 100 instance-id 0
(config-if)#ipv6 router ospf area 0.0.0.0 instance-id 0
(config-if)#lldp-agent
(config-if)# exit

On Spine 2

Copy

(config)#interface cd1/1
(config-if)# priority-flow-control mode on
(config-if)# priority-flow-control deadlock recovery-mode timer
(config-if)#priority-flow-control advertise-local-config
(config-if)#priority-flow-control enable priority 0 1 2 3 4 5 6 7
(config-if)#ip address 10.1.12.2/24
(config-if)#ipv6 address 1012::2/64
(config-if)#ip ospf cost 100
(config-if)#ipv6 ospf cost 100 instance-id 0
(config-if)#ipv6 router ospf area 0.0.0.0 instance-id 0
(config-if)#lldp-agent
(config-if)#exit

(config)#interface cd3/1
(config-if)#priority-flow-control mode on
(config-if)#priority-flow-control deadlock recovery-mode timer
(config-if)#priority-flow-control advertise-local-config
(config-if)#priority-flow-control enable priority 0 1 2 3 4 5 6 7
(config-if)#ip address 10.1.18.2/24
(config-if)#ipv6 address 1018::2/64
(config-if)#ip ospf cost 100
(config-if)#ipv6 ospf cost 100 instance-id 0
(config-if)#ipv6 router ospf area 0.0.0.0 instance-id 0
(config-if)#lldp-agent
(config-if)#exit

(config)#interface cd4/1
(config-if)# priority-flow-control mode on
(config-if)# priority-flow-control deadlock recovery-mode timer
(config-if)# priority-flow-control advertise-local-config
(config-if)# priority-flow-control enable priority 0 1 2 3 4 5 6 7
(config-if)#ip address 20.1.12.2/24
(config-if)# ipv6 address 2012::2/64
(config-if)#ip ospf cost 100
(config-if)#ipv6 ospf cost 100 instance-id 0
(config-if)# ipv6 router ospf area 0.0.0.0 instance-id 0
(config-if)# lldp-agent
(config-if)# exit

(config)#interface cd7/1
(config-if)#priority-flow-control mode on
(config-if)#priority-flow-control deadlock recovery-mode timer
(config-if)#priority-flow-control advertise-local-config
(config-if)#priority-flow-control enable priority 0 1 2 3 4 5 6 7
(config-if)#ip address 20.1.16.2/24
(config-if)#ipv6 address 2016::2/64
(config-if)#ip ospf cost 100
(config-if)#ipv6 ospf cost 100 instance-id 0
(config-if)#ipv6 router ospf area 0.0.0.0 instance-id 0
(config-if)#lldp-agent
(config-if)#exit

(config)#interface cd25/1
(config-if)#priority-flow-control mode on
(config-if)#priority-flow-control deadlock recovery-mode timer
(config-if)#priority-flow-control advertise-local-config
(config-if)#priority-flow-control enable priority 0 1 2 3 4 5 6 7
(config-if)#ip address 30.1.14.2/24
(config-if)#ipv6 address 3014::2/64
(config-if)#ip ospf cost 100
(config-if)#ipv6 ospf cost 100 instance-id 0
(config-if)#ipv6 router ospf area 0.0.0.0 instance-id 0
(config-if)#lldp-agent
(config-if)#exit

(config)#interface cd27/1
(config-if)#priority-flow-control mode on
(config-if)#priority-flow-control deadlock recovery-mode timer
(config-if)#priority-flow-control advertise-local-config
(config-if)#priority-flow-control enable priority 0 1 2 3 4 5 6 7
(config-if)#ip address 30.1.16.2/24
(config-if)#ipv6 address 3016::2/64
(config-if)#ip ospf cost 100
(config-if)#ipv6 ospf cost 100 instance-id 0
(config-if)#ipv6 router ospf area 0.0.0.0 instance-id 0
(config-if)#lldp-agent
(config-if)#exit

(config)#interface cd29/1
(config-if)#priority-flow-control mode on
(config-if)#priority-flow-control deadlock recovery-mode timer
(config-if)#priority-flow-control advertise-local-config
(config-if)#priority-flow-control enable priority 0 1 2 3 4 5 6 7
(config-if)#ip address 20.1.14.2/24
(config-if)#ipv6 address 2014::2/64
(config-if)#ip ospf cost 100
(config-if)#ipv6 ospf cost 100 instance-id 0
(config-if)#ipv6 router ospf area 0.0.0.0 instance-id 0
(config-if)#lldp-agent
(config-if)#exit

(config)#interface cd30/1
(config-if)#priority-flow-control mode on
(config-if)#priority-flow-control deadlock recovery-mode timer
(config-if)#priority-flow-control advertise-local-config
(config-if)#priority-flow-control enable priority 0 1 2 3 4 5 6 7
(config-if)#ip address 10.1.14.2/24
(config-if)#ipv6 address 1014::2/64
(config-if)#ip ospf cost 100
(config-if)#ipv6 ospf cost 100 instance-id 0
(config-if)#ipv6 router ospf area 0.0.0.0 instance-id 0
(config-if)#lldp-agent
(config-if)#exit

(config)#interface cd31/1
(config-if)#priority-flow-control mode on
(config-if)# priority-flow-control deadlock recovery-mode timer
(config-if)# priority-flow-control advertise-local-config
(config-if)#priority-flow-control enable priority 0 1 2 3 4 5 6 7
(config-if)#ip address 30.1.12.2/24
(config-if)#ipv6 address 3012::2/64
(config-if)#ip ospf cost 100
(config-if)#ipv6 ospf cost 100 instance-id 0
(config-if)#ipv6 router ospf area 0.0.0.0 instance-id 0
(config-if)# lldp-agent
(config-if)#exit

(config)#interface cd32/1
(config-if)#priority-flow-control mode on
(config-if)#priority-flow-control deadlock recovery-mode timer
(config-if)#priority-flow-control advertise-local-config
(config-if)#priority-flow-control enable priority 0 1 2 3 4 5 6 7
(config-if)#ip address 10.1.16.2/24
(config-if)#ipv6 address 1016::2/64
(config-if)#ip ospf cost 100
(config-if)#ipv6 ospf cost 100 instance-id 0
(config-if)#ipv6 router ospf area 0.0.0.0 instance-id 0
(config-if)#lldp-agent
(config-if)#exit

Configure the Underlay routing OSPF for Spine and Leaf interconnections.

On Leaf 1

Copy

(config)#router ospf 100
(config-router)#ospf router-id 1.1.1.1
(config-router)#network 1.1.1.1/32 area 0.0.0.0
(config-router)#network 10.1.11.0/24 area 0.0.0.0
(config-router)#network 10.1.12.0/24 area 0.0.0.0
(config-router)#network 20.1.11.0/24 area 0.0.0.0
(config-router)#network 20.1.12.0/24 area 0.0.0.0
(config-router)#network 30.1.11.0/24 area 0.0.0.0
(config-router)#network 30.1.12.0/24 area 0.0.0.0
(config-router)#network 101.1.1.0/24 area 0.0.0.0
(config-router)#network 111.1.1.0/24 area 0.0.0.0
(config-router)#network 171.1.1.0/24 area 0.0.0.0
(config-router)#network 171.1.2.0/24 area 0.0.0.0
(config-router)#network 171.1.3.0/24 area 0.0.0.0
(config-router)#network 171.1.4.0/24 area 0.0.0.0
(config-router)#network 171.1.5.0/24 area 0.0.0.0

(config)#router ipv6 ospf
(config-router)#router-id 1.1.1.1

On Leaf 2

Copy

(config)#router ospf 100
(config-router)#ospf router-id 2.2.2.2
(config-router)#network 2.2.2.2/32 area 0.0.0.0
(config-router)#network 10.1.13.0/24 area 0.0.0.0
(config-router)#network 10.1.14.0/24 area 0.0.0.0
(config-router)#network 20.1.13.0/24 area 0.0.0.0
(config-router)#network 20.1.14.0/24 area 0.0.0.0
(config-router)#network 30.1.13.0/24 area 0.0.0.0
(config-router)#network 30.1.14.0/24 area 0.0.0.0
(config-router)#network 40.1.13.0/24 area 0.0.0.0
(config-router)#network 50.1.13.0/24 area 0.0.0.0
(config-router)#network 60.1.13.0/24 area 0.0.0.0
(config-router)#network 102.1.1.0/24 area 0.0.0.0
(config-router)#network 112.1.1.0/24 area 0.0.0.0
(config-router)#network 172.1.1.0/24 area 0.0.0.0
(config-router)#network 172.1.2.0/24 area 0.0.0.0
(config-router)#network 172.1.3.0/24 area 0.0.0.0
(config-router)#network 172.1.4.0/24 area 0.0.0.0
(config-router)#network 172.1.5.0/24 area 0.0.0.0

On Leaf 3

Copy

(config)#router ospf 100
 (config-router)#ospf router-id 3.3.3.3
 (config-router)#network 3.3.3.3/32 area 0.0.0.0
 (config-router)#network 10.1.15.0/24 area 0.0.0.0
 (config-router)#network 10.1.16.0/24 area 0.0.0.0
 (config-router)#network 20.1.15.0/24 area 0.0.0.0
 (config-router)#network 20.1.16.0/24 area 0.0.0.0
 (config-router)#network 30.1.15.0/24 area 0.0.0.0
 (config-router)#network 30.1.16.0/24 area 0.0.0.0
 (config-router)#network 103.1.1.0/24 area 0.0.0.0
 (config-router)#network 113.1.1.0/24 area 0.0.0.0
 (config-router)#network 173.1.1.0/24 area 0.0.0.0
 (config-router)#network 173.1.2.0/24 area 0.0.0.0
 (config-router)#network 173.1.3.0/24 area 0.0.0.0
 (config-router)#network 173.1.4.0/24 area 0.0.0.0
 (config-router)#network 173.1.5.0/24 area 0.0.0.0

On Leaf 4

Copy

(config)#router ospf 100
(config-router)#ospf router-id 4.4.4.4
(config-router)#network 4.4.4.4/32 area 0.0.0.0
(config-router)#network 10.1.17.0/24 area 0.0.0.0
(config-router)#network 10.1.18.0/24 area 0.0.0.0
(config-router)#network 104.1.1.0/24 area 0.0.0.0
(config-router)#network 114.1.1.0/24 area 0.0.0.0
(config-router)#network 174.1.1.0/24 area 0.0.0.0
(config-router)#network 174.1.2.0/24 area 0.0.0.0
(config-router)#network 174.1.3.0/24 area 0.0.0.0
(config-router)#network 174.1.4.0/24 area 0.0.0.0
(config-router)#network 174.1.5.0/24 area 0.0.0.0

On Spine 1

Copy

(config-router)#router ospf 100
(config-router)#ospf router-id 7.7.7.7
(config-router)#network 7.7.7.7/32 area 0.0.0.0
(config-router)#network 10.1.11.0/24 area 0.0.0.0
(config-router)#network 10.1.13.0/24 area 0.0.0.0
(config-router)#network 10.1.15.0/24 area 0.0.0.0
(config-router)#network 10.1.17.0/24 area 0.0.0.0
(config-router)#network 10.1.19.0/24 area 0.0.0.0
(config-router)#network 20.1.11.0/24 area 0.0.0.0
(config-router)#network 20.1.13.0/24 area 0.0.0.0
(config-router)#network 20.1.15.0/24 area 0.0.0.0
(config-router)#network 20.1.17.0/24 area 0.0.0.0
(config-router)#network 30.1.11.0/24 area 0.0.0.0
(config-router)#network 30.1.13.0/24 area 0.0.0.0
(config-router)#network 30.1.15.0/24 area 0.0.0.0
(config-router)#network 30.1.17.0/24 area 0.0.0.0
(config-router)#network 40.1.13.0/24 area 0.0.0.0
(config-router)#network 50.1.13.0/24 area 0.0.0.0
(config-router)#network 60.1.13.0/24 area 0.0.0.0

On Spine2

Copy

(config-router)#router ospf 100
(config-router)#ospf router-id 8.8.8.8
(config-router)#network 8.8.8.8/32 area 0.0.0.0
(config-router)#network 10.1.12.0/24 area 0.0.0.0
(config-router)#network 10.1.14.0/24 area 0.0.0.0
(config-router)#network 10.1.16.0/24 area 0.0.0.0
(config-router)#network 10.1.18.0/24 area 0.0.0.0
(config-router)#network 10.1.20.0/24 area 0.0.0.0
(config-router)#network 20.1.12.0/24 area 0.0.0.0
(config-router)#network 20.1.14.0/24 area 0.0.0.0
(config-router)#network 20.1.16.0/24 area 0.0.0.0
(config-router)#network 20.1.18.0/24 area 0.0.0.0
(config-router)#network 30.1.12.0/24 area 0.0.0.0
(config-router)#network 30.1.14.0/24 area 0.0.0.0
(config-router)#network 30.1.16.0/24 area 0.0.0.0
(config-router)#network 30.1.18.0/24 area 0.0.0.0

Configure the eBGP router in unnumbered mode on interfaces connecting to Spine and Leaf for advertising host and loopback.

On Leaf 1

Copy

(config)#router bgp 4294967201
(config-router)#bgp router-id 1.1.1.1
(config-router)#bgp log-neighbor-changes
(config-router)#neighbor underlay peer-group
(config-router)#neighbor underlay remote-as 4294967209
(config-router)#neighbor underlay shutdown
(config-router)#neighbor underlay authentication-key 0xb59db09d828b2528
(config-router)#neighbor underlay as-origination-interval 1
(config-router)#neighbor underlay advertisement-interval 0
(config-router)#neighbor underlay fall-over bfd
      (config-router)#exit

(config-router)#bgp unnumbered-mode
(config-router-unnum)#neighbor cd4/1 peergroup underlay
(config-router-unnum)#neighbor cd3/1 peergroup underlay
(config-router-unnum)#neighbor cd2/1 peergroup underlay
(config-router-unnum)#neighbor cd1/1 peergroup underlay
(config-router-unnum)#neighbor cd32/1 peergroup underlay
(config-router-unnum)#neighbor cd29/1 peergroup underlay
(config-router-unnum)#exit-unnumbered-mode

(config-router)#address-family ipv4 unicast
(config-router-af)#max-paths ebgp 10
(config-router-af)#redistribute connected
(config-router-af)#neighbor underlay activate
 
(config-router)#bgp v4-unnumbered-mode
(config-router-v4-unnum)#neighbor cd4/1 route-map HIG_MED in
(config-router-v4-unnum)#neighbor cd3/1 route-map HIG_MED in
(config-router-v4-unnum)#neighbor cd2/1 route-map HIG_MED in
(config-router-v4-unnum)#neighbor cd1/1 route-map HIG_MED in
(config-router-v4-unnum)#neighbor cd32/1 route-map HIG_MED in
(config-router-v4-unnum)#neighbor cd29/1 route-map HIG_MED in
(config-router-v4-unnum)#exit-v4-unnumbered-mode
(config-router-af)#exit-address-family
(config-router)#exit

On Leaf 2

Copy

(config)#router bgp 4294967202
(config-router)#bgp router-id 2.2.2.2
(config-router)#bgp log-neighbor-changes
(config-router)#neighbor underlay peer-group
(config-router)#neighbor underlay remote-as 4294967209
(config-router)#neighbor underlay shutdown
(config-router)#neighbor underlay authentication-key 0xb59db09d828b2528
(config-router)#neighbor underlay as-origination-interval 1
(config-router)#neighbor underlay advertisement-interval 0
(config-router)#neighbor underlay fall-over bfd

 (config)#bgp unnumbered-mode
 (config-router-unnum)#neighbor ethernet10/1 peergroup underlay
 (config-router-unnum)#neighbor ethernet0/1 peergroup underlay
 (config-router-unnum)#neighbor ethernet30/1 peergroup underlay
 (config-router-unnum)#neighbor ethernet33/1 peergroup underlay
 (config-router-unnum)#neighbor ethernet32/1 peergroup underlay
 (config-router-unnum)#neighbor ethernet11/1 peergroup underlay
 (config-router-unnum)#exit-unnumbered-mode
 
 (config)#address-family ipv4 unicast
 (config-router-af)#max-paths ebgp 10
 (config-router-af)#redistribute connected
 (config-router-af)#neighbor underlay activate
 
 (config)#bgp v4-unnumbered-mode
 (config-router-v4-unnum)#neighbor ethernet10/1 route-map HIG_MED in
 (config-router-v4-unnum)#neighbor ethernet0/1 route-map HIG_MED in
 (config-router-v4-unnum)#neighbor ethernet30/1 route-map HIG_MED in
 (config-router-v4-unnum)#neighbor ethernet33/1 route-map HIG_MED in
 (config-router-v4-unnum)#neighbor ethernet32/1 route-map HIG_MED in
 (config-router-v4-unnum)#neighbor ethernet11/1 route-map HIG_MED in
 (config-router-v4-unnum)#exit-v4-unnumbered-mode
 (config-router-v4-unnum)#exit-address-family
 (config-router-v4-af)#exit

On Leaf 3

Copy

(config)#router bgp 4294967203
(config-router)#bgp router-id 3.3.3.3
(config-router)#bgp log-neighbor-changes
(config-router)#neighbor underlay peer-group
(config-router)#neighbor underlay remote-as 4294967209
(config-router)#neighbor underlay shutdown
(config-router)#neighbor underlay authentication-key 0xb59db09d828b2528
(config-router)#neighbor underlay as-origination-interval 1
(config-router)#neighbor underlay advertisement-interval 0
(config-router)#neighbor underlay fall-over bfd

(config-router)#bgp unnumbered-mode
(config-router-unnum)#neighbor cd10/1 peergroup underlay
(config-router-unnum)#neighbor cd5/1 peergroup underlay
(config-router-unnum)#neighbor cd4/1 peergroup underlay
(config-router-unnum)#neighbor cd3/1 peergroup underlay
(config-router-unnum)#neighbor cd2/1 peergroup underlay
(config-router-unnum)#neighbor cd0/1 peergroup underlay
(config-router-unnum)#exit-unnumbered-mode
 
(config)#address-family ipv4 unicast
(config-router-af)#max-paths ebgp 10
(config-router-af)#redistribute connected
(config-router-af)#neighbor underlay activate
 
(config-router)#bgp v4-unnumbered-mode
(config-router-v4-unnum)#neighbor cd10/1 route-map HIG_MED in
(config-router-v4-unnum)#neighbor cd5/1 route-map HIG_MED in
(config-router-v4-unnum)#neighbor cd4/1 route-map HIG_MED in
(config-router-v4-unnum)#neighbor cd3/1 route-map HIG_MED in
(config-router-v4-unnum)#neighbor cd2/1 route-map HIG_MED in
(config-router-v4-unnum)#neighbor cd0/1 route-map HIG_MED in
(config-router-v4-unnum)#exit-v4-unnumbered-mode
(config-router-v4-unnum)#exit-address-family
(config-router-v4-af)#exit

On Leaf 4

Copy

(config)#router bgp 4294967204
(config-router)#bgp router-id 4.4.4.4
(config-router)#bgp log-neighbor-changes
(config-router)#neighbor underlay peer-group
(config-router)#neighbor underlay remote-as 4294967209
(config-router)#neighbor underlay shutdown
(config-router)#neighbor underlay authentication-key 0xb59db09d828b2528
(config-router)#neighbor underlay as-origination-interval 1
(config-router)#neighbor underlay advertisement-interval 0
(config-router)#neighbor underlay fall-over bfd

(config)#bgp unnumbered-mode
(config-router-unnum)#neighbor ce1 peergroup underlay
(config-router-unnum)# neighbor ce5 peergroup underlay
(config-router-unnum)# exit-unnumbered-mode
 
(config-router)#address-family ipv4 unicast
(config-router-af)#max-paths ebgp 10
(config-router-af)#redistribute connected
(config-router-af)#neighbor underlay activate

 (config-router)#bgp v4-unnumbered-mode
 (config-router-v4-unnum)#neighbor ce1 route-map HIG_MED in
 (config-router-v4-unnum)#neighbor ce5 route-map HIG_MED in
 (config-router-v4-unnum)#exit-v4-unnumbered-mode
 (config-router-v4-af)#exit-address-family

On Spine1

Copy

(config)#router bgp 4294967209
(config-router)#bgp router-id 7.7.7.7
(config-router)#bgp log-neighbor-changes
(config-router)#neighbor underlay peer-group
(config-router)#neighbor underlay remote-as 1
(config-router)#neighbor underlay shutdown
(config-router)#neighbor underlay authentication-key 0xb59db09d828b2528
(config-router)#neighbor underlay as-origination-interval 1
(config-router)#neighbor underlay advertisement-interval 0
(config-router)#neighbor underlay fall-over bfd
 
 (config-router)#bgp unnumbered-mode
 (config-router-unnum)#neighbor ethernet1/1 remote-as external
 (config-router-unnum)#neighbor ethernet1/1 peergroup underlay
 (config-router-unnum)#neighbor ethernet31/1 remote-as external
 (config-router-unnum)#neighbor ethernet31/1 peergroup underlay
 (config-router-unnum)#neighbor ethernet63/1 remote-as external
 (config-router-unnum)#neighbor ethernet63/1 peergroup underlay
 (config-router-unnum)#neighbor ethernet32/1 remote-as external
 (config-router-unnum)#neighbor ethernet32/1 peergroup underlay
 (config-router-unnum)#neighbor ethernet64/1 remote-as external
 (config-router-unnum)#neighbor ethernet64/1 peergroup underlay
 (config-router-unnum)#neighbor ethernet61/1 remote-as external
 (config-router-unnum)#neighbor ethernet61/1 peergroup underlay
 (config-router-unnum)#neighbor ethernet30/1 remote-as external
 (config-router-unnum)#neighbor ethernet30/1 peergroup underlay
 (config-router-unnum)#neighbor ethernet2/1 remote-as external
 (config-router-unnum)#neighbor ethernet2/1 peergroup underlay
 (config-router-unnum)#neighbor ethernet3/1 remote-as external
 (config-router-unnum)#neighbor ethernet3/1 peergroup underlay
 (config-router-unnum)#neighbor ethernet62/1 remote-as external
 (config-router-unnum)#neighbor ethernet62/1 peergroup underlay
 (config-router-unnum)#neighbor ethernet11/1 remote-as external
 (config-router-unnum)#neighbor ethernet11/1 peergroup underlay
 (config-router-unnum)#exit-unnumbered-mode
 
 (config-router)#address-family ipv4 unicast
 (config-router-af)#max-paths ebgp 10
 (config-router-af)#redistribute connected
 (config-router-af)#neighbor underlay activate
 
 (config-router)#bgp v4-unnumbered-mode
 (config-router-v4-unnum)#neighbor ethernet1/1 route-map HIG_MED in
 (config-router-v4-unnum)#neighbor ethernet31/1 route-map HIG_MED in
 (config-router-v4-unnum)#neighbor ethernet63/1 route-map HIG_MED in
 (config-router-v4-unnum)#neighbor ethernet32/1 route-map HIG_MED in
 (config-router-v4-unnum)#neighbor ethernet64/1 route-map HIG_MED in
 (config-router-v4-unnum)#neighbor ethernet61/1 route-map HIG_MED in
 (config-router-v4-unnum)#neighbor ethernet30/1 route-map HIG_MED in
 (config-router-v4-unnum)#neighbor ethernet2/1 route-map HIG_MED in
 (config-router-v4-unnum)#neighbor ethernet3/1 route-map HIG_MED in
 (config-router-v4-unnum)#neighbor ethernet62/1 route-map HIG_MED in
 (config-router-v4-unnum)#neighbor ethernet11/1 route-map HIG_MED in
 (config-router-v4-unnum)#exit-v4-unnumbered-mode
 (config-router-v4-af)#exit-address-family
 (config-router)#exit

On Spine2

Copy

(config)#router bgp 4294967209
(config-router)#bgp router-id 8.8.8.8
(config-router)#bgp log-neighbor-changes
(config-router)#neighbor underlay peer-group
(config-router)#neighbor underlay remote-as 1
(config-router)#neighbor underlay shutdown
(config-router)#neighbor underlay authentication-key 0xb59db09d828b2528
(config-router)#neighbor underlay as-origination-interval 1
(config-router)#neighbor underlay advertisement-interval 0
(config-router)#neighbor underlay fall-over bfd

 (config-router)#bgp unnumbered-mode
 (config-router-unnum)#neighbor cd29/1 remote-as external
 (config-router-unnum)#neighbor cd29/1 peergroup underlay
 (config-router-unnum)#neighbor cd3/1 remote-as external
 (config-router-unnum)#neighbor cd3/1 peergroup underlay
 (config-router-unnum)#neighbor cd30/1 remote-as external
 (config-router-unnum)#neighbor cd30/1 peergroup underlay
 (config-router-unnum)#neighbor cd25/1 remote-as external
 (config-router-unnum)#neighbor cd25/1 peergroup underlay
 (config-router-unnum)#neighbor cd4/1 remote-as external
 (config-router-unnum)#neighbor cd4/1 peergroup underlay
 (config-router-unnum)#neighbor cd1/1 remote-as external
 (config-router-unnum)#neighbor cd1/1 peergroup underlay
 (config-router-unnum)#neighbor cd31/1 remote-as external
 (config-router-unnum)#neighbor cd31/1 peergroup underlay
 (config-router-unnum)#neighbor cd32/1 remote-as external
 (config-router-unnum)#neighbor cd32/1 peergroup underlay
 (config-router-unnum)#neighbor cd27/1 remote-as external
 (config-router-unnum)#neighbor cd27/1 peergroup underlay
 (config-router-unnum)#neighbor cd7/1 remote-as external
 (config-router-unnum)#neighbor cd7/1 peergroup underlay
 (config-router-unnum)#neighbor cd9/1 remote-as external
 (config-router-unnum)#neighbor cd9/1 peergroup underlay
 (config-router-unnum)#exit-unnumbered-mode
 
(config-router)#address-family ipv4 unicast
(config-router-af)# max-paths ebgp 10
(config-router-af)#redistribute connected
(config-router-af)#neighbor underlay activate
 
(config-router)#bgp v4-unnumbered-mode
(config-router-v4-unnum)#neighbor cd29/1 route-map HIG_MED in
(config-router-v4-unnum)#neighbor cd3/1 route-map HIG_MED in
(config-router-v4-unnum)#neighbor cd30/1 route-map HIG_MED in
(config-router-v4-unnum)#neighbor cd25/1 route-map HIG_MED in
(config-router-v4-unnum)#neighbor cd4/1 route-map HIG_MED in
(config-router-v4-unnum)#neighbor cd1/1 route-map HIG_MED in
(config-router-v4-unnum)#neighbor cd31/1 route-map HIG_MED in
(config-router-v4-unnum)#neighbor cd32/1 route-map HIG_MED in
(config-router-v4-unnum)#neighbor cd27/1 route-map HIG_MED in
(config-router-v4-unnum)#neighbor cd7/1 route-map HIG_MED in
(config-router-v4-unnum)#neighbor cd9/1 route-map HIG_MED in
(config-router-v4-unnum)#exit-v4-unnumbered-mode
(config-router)#exit-address-family
(config-router)#exit

Validation

Check the PFC details on Spine1 interfaces.

Copy

Spine1-TH5-7001#show priority-flow-control details all

Admin Configuration
--------------------
interface           mode  advertise willing   cap   link delay  priorities
                                                     allowance
================================================================================
ethernet1/1         on    on        off       8     0           0 1 2 3 4 5 6 7
ethernet2/1         on    on        off       8     0           0 1 2 3 4 5 6 7
ethernet30/1        on    on        off       8     0           0 1 2 3 4 5 6 7
ethernet32/1        on    on        off       8     0           0 1 2 3 4 5 6 7
ethernet64/1        on    on        off       8     0           0 1 2 3 4 5 6 7

--------------------------------------------------------------------------------

Operational Configuration
-------------------------
interface           state cap   link delay  priorities
                               allowance
================================================================================
ethernet1/1         on    8     0           0 1 2 3 4 5 6 7
ethernet2/1         on    8     0           0 1 2 3 4 5 6 7
ethernet30/1        on    8     0           0 1 2 3 4 5 6 7
ethernet32/1        on    8     0           0 1 2 3 4 5 6 7
ethernet64/1        on    8     0           0 1 2 3 4 5 6 7

Change recovery-time to 100 and check the PFC deadlock status.

Copy

Spine1-TH5-7001#show priority-flow-control deadlock-status interface ethernet1/1

Deadlock Detection and Recovery Configuration
----------------------------------------------------------------------
interface            recovery     detection     detection     recovery
                     mode         multiplier    granularity   time
======================================================================
ethernet1/1          XON          100           10            100

---------------------------------------------------------------------------------------------------------

Deadlock Detection and Recovery Status
---------------------------------------------------------------------------------------------------------
interface           pri    state          detection    last detection             last recovery
                                          count        timestamp                  timestamp
=========================================================================================================
ethernet1/1           0    no deadlock            0    -                          -
ethernet1/1           1    no deadlock            0    -                          -
ethernet1/1           2    no deadlock            0    -                          -
ethernet1/1           3    no deadlock            0    -                          -
ethernet1/1           4    no deadlock            0    -                          -
ethernet1/1           5    no deadlock            0    -                          -
ethernet1/1           6    no deadlock            0    -                          -
ethernet1/1           7    no deadlock            0    -                          -

-------------------------------------------------------------------------------------

Remove recovery-time and check the PFC deadlock status.

Copy

#sh run int ethernet1/1
!
interface ethernet1/1
 description Connected-ce1-7042
 priority-flow-control mode on
 priority-flow-control advertise-local-config
 priority-flow-control deadlock recovery-mode pfc-state-xon detection-multiplier 100 time-granularity 10
 priority-flow-control enable priority 0 1 2 3 4 5 6 7
 load-interval 30
 ip address 104.1.1.2/24
 ipv6 address 1401::2/64
 mtu 9216
 ipv6 router ospf area 0.0.0.0 instance-id 0
!

No PFC deadlock detected.

Copy

-------------------------------------------------------------------------------------

Spine1-TH5-7001#show priority-flow-control deadlock-status

Deadlock Detection and Recovery Configuration
----------------------------------------------------------------------
interface            recovery     detection     detection     recovery
                     mode         multiplier    granularity   time
======================================================================
ethernet1/1          XON          100           10            -

---------------------------------------------------------------------------------------------------------

Deadlock Detection and Recovery Status
---------------------------------------------------------------------------------------------------------
interface           pri    state          detection    last detection             last recovery
                                          count        timestamp                  timestamp
=========================================================================================================