Virtualization - VMware Clusters and Blade Servers

Blade chassis help further consolidate the data-center footprint and provide an excellent platform to run virtualization; this further consolidates the data-center footprint.  I have noticed on a variety of installations the failure to ensure proper placement of the primary nodes when installing a VMware cluster on blade chassis technology. Primary placement is critical to ensure the availability of VMs in the event of a blade chassis failure. There are 5 primary nodes per cluster and these are selected as the nodes are added to the cluster. Primary nodes holds all cluster settings and node states and this is replicated to all primaries. Secondary nodes do not become primary nodes if a primary node were to fail. Heartbeats are sent from primary to primary nodes and from secondary to primary nodes. If a primary node fails and is not removed then no secondaries become primaries, but if a failed primary node is removed from the cluster than a secondary node  becomes a primary node - the selection of which secondary node becomes a primary node is random further complicating the balancing of primary nodes. The diagram below shows 3 blade chassis's running in RACK A leveraging VMware, the diagram below shows the problems that happen when improper placement of the primary nodes is not followed. In the case below the blade chassis's are HP C7000 series. The installation of ESX is completed in the order of the blade server slots assigned by the blade chassis and all 5 primary nodes end up on one blade chassis (the installation includes adding the nodes to the cluster). The issues identified in the diagram.

Virtualization - Zero Impact Maintenance

Leveraging Virtualization to provide zero impact maintenance during production hours.

 

Before performing maintenance on the physical node in question you need to migrate the VM over to another physical node. This step is non-disruptive to the production environment and the VM continues to provide services.

The migration happens fairly quickly with zero impact. You have to ability to migrate VM's either hot or cold but typically your VM's are running in production environment and cannot tolerate down time; therefore most administrators migrate them hot.

FCoE - Convergance

FCoE (Fibre Channel over Ethernet) allows companies to further converge their infrastructure reducing the complexity and costs.

• Reduction in Cables and Switches
• Reduction in Interface Cards
• Reduction in Power and Cooling

FCoE runs on your data network and removes the need to have a separate Fibre channel infrastructure. FcoE is not routable and will not extend across routed IP networks. FCoE runs on top of Ethernet and enhancements to Ethernet were required in order to prevent frame loss (PFC). The new enhancements to Ethernet are referred to as Lossless Ethernet.

• Priority Flow Control (802.1Qbb)

Additional enhancements to Ethernet may include (dependent on the vendor)

• Bandwidth Management (802.1Qaz)
• Congestion Management (802.1Qau)
• Shortest Path Bridging (802.1aq)

Servers leverage CNA (Converged Network Adapter) with contains both the NIC (Network Interface Card) and HBA (Host Bus Adapter). CNAs have 1 or more ports.

Note: The diagram below is NOT meant to show redundancy.

Introduction to QFabric

NX-OS VEM Physical and Virtual Ports

VEM (Virtual Ethernet Module)

VSM (Virtual Supervisor Module)

Nexus 1000v supports the following

-2 VSMs (High Availability)

-64 VEMs

-512 Active VLANs

-2048 Ports (Eth + vEth) 

-256 Port Channels

VEM supports the following

-216 Ports (vETH

-32 Physical NICs

-8 Port Channels

NX-OS Fibre Channel Module

The Nexus 5000 can run in two modes

Fabric Mode - The Nexus 5000 switch module runs as a typical switch in a fibre channel network. 

NPV (N-Port Virtualization) Mode - Does not operate as a typical FC switch. Operates like a NPIV enabled host within a fabric.

NX-OS FCoE Ports

Nexus 5000 Feature that needs to be licensed.

FCoE (Fibre Channel over Ethernet) - is leveraged to further unify I/O.  FCoE allows fibre channel to operate over ethernet by encapsulating Fibre Channel into ethernet. 

FCoE Ports

Virutal N_Port (VN_Port) - Node ports which exist on hosts or storage arrays and connect to a FC fabric. Operates over Ethernet links.

Virtual F_Port (VF_Port) - Switch or director ports that connct to node ports. Operates over Ethernet links.

Virtual E_Port (VE_Port) - Expansion port that is used to inter-connect two FC switches together. When two swithes are connected they form an ISL (interswitch link) Operates over Ethernet links.

NX-OS Virtual SPAN

Virtual SPAN empowers a network administrator to SPAN more than 1 VLAN and enables the network administrator the ability to selectively chose which VLAN goes to what destination SPAN port. Example: A network administrator wants to SPAN a trunk port with VLAN 10, 20, and 30 but wants to send VLAN 10 to SPAN port ethernet 1/1, send VLAN 20 to SPAN port ethernet 1/2. and send VLAN 30 to SPAN port ethernet 1/3. Virtual SPAN enables that flexibility. This also helps reduce the number of SPAN sessions required. 

NX-OS SPAN

Nexus 7000/5000 SPAN Sessions

SPAN Session Limit - 18

Nexus 1000V

SPAN Session Limit (SPAN and ERSPAN) - 64

Nexus 5000 SPAN Sessions

Can SPAN Ethernet,Fibre Channel,PortChannel,SAN PortChannel,VLAN,VSAN (Virtual Storage Area Network)

NX-OS ISSU

ISSU (In-Service Software Update) - Provides the ability to upgrade software without disrupting operations. The system performs the following steps to ensure a non disruptive upgrade:

1. Active and Standby Supervisors and all line cards BIOS's are upgraded
2. Standby Supervisor is upgraded and rebooted
3. Once the Standby Supervisor comes online with the upgraded version of NX-OS a stateful switchover is performed. All control plane traffic is now running on the former Standby Supervisor which is now the Active Supervisor.
4. The new Standby Supervisor (former Active Supervisor) is now upgraded.
5. Upgrade is performed on the Line Cards on are a time and then reloaded. The reload is non disruptive and is only performed on the CPU, no data plane components are impacted.
6. CMP (Connectivity Management Processor) on both Supervisors are upgraded.