Mark Minasi's Tech Forum
Register Calendar Latest Topics
 
 
 


Reply
  Author   Comment  
downtime

Senior Member
Registered:
Posts: 105
Reply with quote  #1 
Dear all,

I run a small Terminal Server Farm based on Windows Server 2012 R2. Every 2-3 months I get a situation where the Hyper-V Guests (also Windows Server 2012 R2) lose network connectivity.

When I check the NIC properties of the Guests the correct IP address is configured, but the NIC properties is saying "Unidentified network". Pinging to the default gateway fails.

The only "fix" is to reboot the Hyper-V Server.

Physical Server: HP Proliant DL380 G8
NIC x 2: HP Ethernet 1GB 4-port 331FLR adapter with Driver version 16.8.0.4
NICs are configured in a HP Network Team (NIC Teaming).
NIC Teaming Mode: Switch Independent
NIC Load Balancing: Dynamic.

I ran the following command on the Hyper-V server: get-netadapter | sort macaddress

NIC 1     HP Ethernet 1Gb 4-port 331FLR Adapter        A0-D3-C1-03-31-EC       100 Mbps
HP Network Team 1   Microsoft Network Adapter MP    A0-D3-C1-03-31-EC       200 Mbps
vEthernet (Microsoft N... Hyper-V Virtual E Ada #2      A0-D3-C1-03-31-EC        10 Gbps
NIC 2     HP Ethernet 1Gb 4-port 331FLR Adap...#2      A0-D3-C1-03-31-ED       100 Mbps

Should I be worried about the duplicate MAC addresses here?

The properties for the Hyper-V Virtual Switch is below:

Untitled.png 

I'm at a loss to explain why is happened. Have I configured something obviously wrong here?

Can anyone point me in the direction of recommended NIC settings (TCP Offload and things like that) for Hyper-V Server and Hyper-V Guests?

Any help would be much appreciated.

0
jsclmedave

Administrator
Registered:
Posts: 460
Reply with quote  #2 
Could this possibly be due to a timing issue?  I have heard if you sync the time wrong it will start to creep out of sync a little each day...  If I can find the article I will post it.
__________________
Tim Bolton @jsclmedave
Email: [string](0..20|%{[char][int](32+('527377347976847978324785847679797514357977').substring(($_*2),2))}) -replace ' '  

New to the forum? Please Read this
0
downtime

Senior Member
Registered:
Posts: 105
Reply with quote  #3 
The last time it failed was 20th Nov 2015. Same server.

Just some further information I may have forgot to mention: I have 2 Hyper-V servers running one Guest Terminal Server each + one member server, let's call them:

HyperV-SRV1 runs TS1 and MEMSERV1
HyperV-SRV2 runs TS2 and MEMSERV2.

It's TS2 that's failed twice. I am trying to ascertain if this is the server that always fails. I should tell you that today when TS2 lost network connectivity MEMSERV2 also lost network connectivity at the same time. Rebooting the Hyper-V server (HyperV-SRV2) "fixed" the problem.

On Monday I will check all the configurations between the Hyper-V servers and Guests to see if there are any differences.
0
wobble_wobble

Avatar / Picture

Associate Troublemaker Apprentice
Registered:
Posts: 887
Reply with quote  #4 
Check patch levels and firmware on both hosts
Check patch level on both guests.
You could disable TCP Chimney
https://support.microsoft.com/en-us/kb/951037
But also check AV

__________________
Have you tried turning it off and walking away? The next person can fix it!

New to the forum? Read this
0
downtime

Senior Member
Registered:
Posts: 105
Reply with quote  #5 
AV is Trend Micro Office Scan, version 11 which is fully patched AFAIK.

I will check (again) Trend recommendations for Hyper-V (file and folder exclusions, etc).

I will also check this:

http://social.technet.microsoft.com/wiki/contents/articles/2179.hyper-v-anti-virus-exclusions-for-hyper-v-hosts.aspx
0
wobble_wobble

Avatar / Picture

Associate Troublemaker Apprentice
Registered:
Posts: 887
Reply with quote  #6 
I have a few lists, hope it attaches OK

 
Attached Files
docx AV_Exclusions.docx (19.42 KB, 8 views)


__________________
Have you tried turning it off and walking away? The next person can fix it!

New to the forum? Read this

0
downtime

Senior Member
Registered:
Posts: 105
Reply with quote  #7 
Thanks.

I will check out the AV list and other recommendations on Monday.
0
downtime

Senior Member
Registered:
Posts: 105
Reply with quote  #8 
I checked the logs on the Hyper-V server around the time of the failure (Guests lose network connectivity) and found this error relating to duplicate MAC Addresses:

Untitled.png 

I ran the following command on the Hyper-V server: get-netadapter | sort macaddress

NIC 1     HP Ethernet 1Gb 4-port 331FLR Adapter        A0-D3-C1-03-31-EC       100 Mbps
HP Network Team 1   Microsoft Network Adapter MP    A0-D3-C1-03-31-EC       200 Mbps
vEthernet (Microsoft N... Hyper-V Virtual E Ada #2      A0-D3-C1-03-31-EC        10 Gbps

Looks like the HP Team is using the same MAC address as the first physical NIC as is the VEthernet Adapter.

I've seen a lot of other people with the same issue, but no fix. Is this normal?

I ran the command again today and it's the same as above, but I have no problems today, so maybe the above doesn't matter? Very confused now [frown]


0
dennis-360ict

New Friend (or an Old Friend who Built a New Account)
Registered:
Posts: 68
Reply with quote  #9 
I would check firmware levels as wobble says, i've had some bigtime problems with out-of the box hardware that were resolved by firmware updates.

Als also there is disabling VMQ for your NICs, thats only usefull on 10Gb nics and could be your issue. I have a internal KB written on the subject which is in dutch, but i will try to translate it for you, sorry for any weird translation or dutch words.

  • check if all 1Gbits VMQ (virtual machine qeueu) are turned off. You can ignore the teamed nics because they will get their settings from the member nics. Use Powershell: Get-NetAdapterVmq
  • Turn VMQ off on the nics: (Properties van NIC -> Configure -> Virtual Machine Que disable). Be awre this restarts your network nic, so you will loose connectivity for a moment. PS: Set-NetAdapterVmq -Name "LAN01 NIC1" -Enabled 0
  • Check with Get-NetAdapterVmq, it should only be enabled on the 10Gb nics/team. 

For the duplicate mac adresses, i don't like it although Ms says it can't hurt because the mac adress from one of the macadresses is being used by the team. As long it's in the team, it should matter. Al least this is what they say. The event log disagrees btw, and i've found you can get rid of them by doing the following (alos translated from my internal KB, so again sorry for any translation errors);
  • PS:
    Get-NetAdapter | select name, macaddress | sort macaddress
    check double mac adresses;
    • Rename the vSwitch in Hyperv to _old like "vSwitch_LiveMigrate" to "vSwitch_LiveMigrate_old"
    • Check the name : Get-NetAdapter | select name, macaddress | sort macaddress
    • Add-VMNetworkAdapter -ManagementOS -Name "vSwitch_LiveMigrate" -SwitchName vSwitch_LiveMigrate_old
      Remove-VMNetworkAdapter -ManagementOS -VMNetworkAdapterName vSwitch_LiveMigrate_old
      you will loose connectivity because of a new macadress and if you use DHCP, this will get you a different IP than before.
    • Rename vSwitch by removing _old, like "vSwitch_LiveMigrate_old" naar "vSwitch_LiveMigrate"
  • check duplicates:
    Get-NetAdapter | select name, macaddress | sort macaddress
  • This wille happen on the nicteam, the first nic of the team and on any teams that has the "share with management OS" has enabeld in the vSwitch.
  • I will replace those macs with mac adress from the prive range, like 06-00-00-00-xx-xx. But you can google for more, i believe they are in inthe 02- and 0A- range, but i'm guessing and know 06- is one range you can use.
  • Add the macadresses by changing the properties of the nic's, klick configure -> advanced -> MAC Address. Klick value and fill in "macadress".



__________________
-----
Home is where is sleep
0
downtime

Senior Member
Registered:
Posts: 105
Reply with quote  #10 
I did some checks on the NICs in the Hyper-V environment this week and found:

Driver version = 16.8.0.4
Firmware version = 1.34.0
Large Send Offload V2 (IPv4) = Enabled
Large Send Offload V2 (IPv6) = Enabled
TCP/UDP Checksun Offload (IPv4) = Rx and Tx Enabled
TCP/UDP Checksun Offload (IPv6) = Rx and Tx Enabled
VMQ Enabled? = Yes

There is an updated driver version 17.2.0.0
This is an updated firmware version 5.0.0.1

I will install both at an appropriate time and will also disable VMQ as per the recommendations.

I also noticed that "Large Send Offload V2 (IPv4)" and "TCP/UDP Checksum Offload (IPv4)" was disabled on one Guest, but enabled on another. I will enable these settings.

I will then monitor the situation. I will also look into the duplicate MAC address problem as described by dennis-360ict (thanks for that BTW).








0
Philip Walker

Still Checking the Forum Out
Registered:
Posts: 1
Reply with quote  #11 
Hey, did you manage to resolve this issue? I'm dealing with the same thing.

Thanks.
0
downtime

Senior Member
Registered:
Posts: 105
Reply with quote  #12 
Hello,

I have not experienced this issue since I disabled VMQ on the Hyper-V Host Servers. I have a template of which NIC settings are enabled or disabled on the Host and Guest. I will find the info tomorrow and post here.

Regards,

Stuart.
0
downtime

Senior Member
Registered:
Posts: 105
Reply with quote  #13 
OK, so this is what I did to solve this issue:

1, Make sure NIC Drivers and Firmware on the Hyper-V Host are up-to-date.

2, I have my NIC Settings on the Hyper-V Host like this:

Large Send Offload V2 (IPv4) = Enabled
Large Send Offload V2 (IPv6) = Enabled
TCP/UDP Checksum Offload (IPv4) = Rx and Tx Enabled
TCP/UDP Checksum Offload (IPv6) = Rx and Tx Enabled
VMQ Enabled = NO

3, I have my NIC Settings on the Guests like this:

Large Send Offload V2 (IPv4) = Enabled
Large Send Offload V2 (IPv6) = Enabled
TCP/UDP Checksum Offload (IPv4) = Rx and Tx Enabled
TCP/UDP Checksum Offload (IPv6) = Rx and Tx Enabled
(VMQ Enabled = Not applicable)

I am running HP Proliant DL380 G8 Servers.

This issue stopped for me as soon as these changes were made.

Regards,

Stuart.

 

0
Previous Topic | Next Topic
Print
Reply

Quick Navigation:

Easily create a Forum Website with Website Toolbox.