Useful PowerShell snippets for Storage Spaces Direct (S2D) operations

30 May

This is a very quirk post to just share some simple PowerShell snippets I have created or found elsewhere which I find very handy when it comes to troubleshooting, setup or operating Storage Spaces Direct (S2D).

1. Get the count of physical disk in each node:
When S2D is enabled in a cluster Get-Physicaldisk returns the physical disk from all cluster nodes. But when you want the disk count of each node the following script can be quite useful:

2. Get the processed and completed GBytes of running Storage Jobs (e.g. Rebuild Jobs):
This one comes from the Jaromir Kaspar and his ws2016lab Project on GitHub, which is btw a very nice  project to build a test lab in a highly automated way. So all credit goes to him for this script.
It shows you the amount of GBytes the running storage jobs have already processed and how much must still to be processed until the jobs are finished.

(original source: ws2016lab project)

SCVMM: When the deployment of new VM template suddenly fails

2 Mar

Recently I ran in a very strange behavior when deploying a VM template with Server 2016 through VMM 2012 R2. First of all to enable the full support of Windows Server 2016-based VMs in VMM 2012 you need at least Update Rollup 11 Hotfix 1 installed. But even after installing  the latest UR (UR12 in my case), the deployment of a Server 2016 VM has failed.

The Issue:
Everytime when a new VM is deployed from a Server 2016 VM template the process fails at specialize phase of the sysprep. However all other existing templates with Server 2012 were working as expected.

Because in in this phase also the domain join happens I decided to give another try with a VM template which has no domain join configured. And tada, the VM was deployed successfully. 

The root cause:
With this finding my assumption was that, when the VM template is configure for domain join, VMM adds something in the unattend.xml which Server 2016 does not like that much. So I inspected the unattend.xml file of a failed deployment and there I found the following section which has looked a litte bit strange:

 Somehow the Domain of the domain join account was missing. 

The Solution:
So I checked the VMM Run As Account which was specified as domain join credentials in the VM template. And as you can see, we have also no domain information here.  

After changing the username to “domain\vm domain join” the deployment went through smooth as it should. Inspecting the unattend.xml file showed that the domain is now also correctly filled in.

Conslusion:
When the deployment of a new VM Template in VMM suddenly fails at the domain join step, double check the run as account and be sure that there is also the domain name in the username field.
In my case it was a template with Server 2016. But I think chances are good as the same could also happens with new VM templates with another guest OS.

Install Azure Stack POC into a VM

1 Feb

Last week Microsoft released a first preview of the Microsoft Azure Stack. The software stack which allows you to run Azure in your own datacenter.

Official a physical server with quite a lot of CPU cores and memory is required to deploy the Azure Stack Technical Preview. Because I do not have any spare servers in my home lab to use exclusively for the Azure Stack Technical Preview I looked for an alternative and I tried to deploy it in a VM. And here is a short walkthrough how you do it and yes it actually works. Smile

Requirements:
First of all you need the following:

  • A Hyper-V Host installed with Windows Server 2016 TP4
    (TP4 is needed for nested virtualization feature)
  • The “Microsoft Azure Stack Technical Preview.zip” file which you can get from here: https://azure.microsoft.com/en-us/overview/azure-stack/
  • At least 32GB of RAM and 150GB of free Disk space available

Preparation:
Frist extract the Microsoft Azure Stack Technical Preview.zip on to the local hard drive of the Hyper-V Host. This will lead you to a folder with an .exe and 6 .bin files.

image

Run the Microsoft Azure Stack POC.exe to extract the actually data to deploy the Azure Stack Preview. This created the “Microsoft Azure Stack POC” folder.
image

Then copy the “WindowsServer2016Datacenter.vhdx” outside of the “Microsoft Azure Stack POC” folder and rename it to e.g. MicrosoftAzureStackPOCBoot.vhdx.

image

Mount (double click) the copied VHDX and copy the whole “Microsoft Azure Stack POC” folder into it.
image

Then dismount the VHDX through Explorer or by PowerShell (Dismount-VHD)image

Now it’s time to create a “litte” Winking smile VM with 32GB of RAM at minimum and as much vCPU as your hardware can suffer.
Note: Dynamic Memory must be disabled on this VM!
SNAGHTML3d386e9

Use the copied VHDX form above as the first disk (boot disk) of this VM and add 4 more empty data disks. (min. 140GB each)
image

image

Enable MAC address spoofing on the Network Adapter.
This is need for network connectivity of the nested VMs which the Azure Stack Setup will create.SNAGHTML696a02f

Lastly the nested visualization feature (new in TP4) must be enabled on the vCPU of the VM. Do this with the following PowerShell command:

Azure Stack Deployment:
Now start the VM and answer the question of the Windows Setup and the login with local Administrator account.

If you have less than 96GB RAM assigned to the VM you have to tweak the deployment script before you start the setup. Daniel Neumann has written an excellent blog post about the necessary modifications: http://www.danielstechblog.de/microsoft-azure-stack-technical-preview-on-lower-hardware/

Now, finally, you can run the PowerShell deployment script (Deploy Azure Stack.ps1) as it is described in the original documentation from Microsoft. The script will take several hours to finish. So better get you a cup of coffee or have a “little” break and hope everything goes well. Smile If it does, you will get a functional Azure Stack installation in a VM.

Update 09.03.2016:
Although the setup just works fine in the VM and you can even provisioning Subscriptions and Tenant VMs there are some serious issues with networking when using this nested setup. As soon as you connect to a fabric VM (with RDP or VM Console) the VM with the virtual Hyper-V Host will crash.
Many thanks to Alain Vetier for pointing this out and sharing his finding here!
See also his comments below.

Configuring Hyper-V Hosts with converged networking through PowerShell DSC

10 Dec

Lately I had to rebuild the Hyper-V Hosts in my home lab several times because of the release of the different TPs for Windows Server 2016. This circumstance (and because I am a great fan of PowerShell DSC Winking smile) gave me the Idea to do the whole base configuration of the Hyper-V Host, including the LBFO NIC Teaming, vSwitch and vNICs for the converged networking configuration, through PowerShell DSC.
But soon I realized that the DSC Resource Kit from Microsoft provides DSC resources only for a subset of the needed configuration steps. The result was some PowerShell modules with my custom DSC resources.

My custom DSC resources for the approach:
image

cLBFOTeam: To create and configure the Windows built-in NIC Teaming
cVNIC: To create and configuring virutal network adapters for Hyper-V host management
cPowerPlan: To set a desired Power Plan in Windows (e.g. High Performance Plan)

You can get the moduels from the PowerShell Gallery (Install-Module) or from GitHub. They will hopefully help everyone who has a similar intend  Winking smile

More to come:
Yes, I am not quite finished yet and I have more in the pipline. Winking smile
Currently I am also working on a fork of the xHyperV Module with a adopteted xVMSwitch resource with a paramter to specify the MinimumBandwidth mode of the Switch.
Futuremore I am also planing to add support for the SET (Swicht Embedded Teaming) in Windows Server 2016 to the xVMSwitch resource.

So you may soon read more about this topic here. In the meantime, happy DSCing! Smile

Registry keys to tune the data source colocation in DPM 2012

23 Aug

By default, DPM will create for every data source two volumes (a replica and a shadow copy volume). For Hyper-V and SQL Database DPM can colocation multiple data sources on a single replica an shadow copy volume. This is relatively well known setting. The option is especially useful for backup a large numbers of Hyper-V VMs.

What is less know, is the possibility to tune the initial size of the replica volume which DPM will choose when a new Protection Group with colocation is created. Continue reading

The connection between Hyper-V Network Virtualization (NVGRE) and MTU Size (and Linux)

26 Apr

In a network with Hyper-V Network Virtualization (using NVGRE encapsulation) the MTU (Maximum Transmission Unit) size is 42 Bytes smaller than in a traditional Ethernet network (where it is 1500 Bytes). The reason for this is the NVGRE encapsulation which needs the 42 Bytes to store his additional GRE Header in the packet. So the maximum MTU size with Hyper-V Network Virtualization is 1458 Bytes.

The problem with Linux: VMs:
For VMs running Windows Server 2008 or newer this should not be a Problem because Hyper-V has a mechanism which lowers the MTU size for the NIC of the VM automatically if needed. (Documented on the TechNet Wiki).
But with VMs running Linux you could run in a problem because the automatically MTU size reduction seem to not function correctly with Linux VMs:
https://support.microsoft.com/en-us/kb/3021753/
This has the effect that the MTU size in the Linux VMs stays at 1500 and therefore you can experience some very weird connection issues.

The Solution:
So there are two options to resolve this issue:

  • Set the MTU size for the virtual NICs of all Linux VMs manually to 1458 Bytes
  • Enable Jumbo Frames on the physical NICs on the Hyper-V Hosts. Then the there is no need to lower the MTU size in the VMs.
  • (wait for kernel updates for your Linux distribution which has the fix from KB3021753 implemented)

Beware of SCVMM UR5 if you have Hyper-V Clusters with Storage Spaces

23 Mar

The latest Update Rollup (UR5) for SCVMM 2012 R2 seems to have some issues with Live Migration in environments where the Hyper-V hosts have Cluster Shared Volumes (CSV) directly stored on clustered Storage Spaces. So basically this is the case if you are using one of the following configuration for your Hyper-V clusters:

  • Two Hyper-V host directly connected to a SAS JBOD.
  • Cluster in a Box Systems like these from DataON.

Note: The here described issue does not occur if the VMs are stored on a Scale-Out File Server or on a traditional Fiber Channel SAN

The Error:
Anyway the issue I have noticed in UR5 with one of the above listed configuration, is that when you try to live migrate a VM in SCVMM the Live Migration fails with the following Error:
Error20404

This only happens in SCVMM. Live Migration through Failover Cluser Manager still works fine.
After some troubleshooting I found also that the Live Migration works sometimes from e.g. Host 1 to Host 2 but not the other way. So this brings me to the conclusion that it must has something to do which host is the owner node of the CSV Volume. And some further tests has confirmed my assumption. If you change the owner node of a CSV which is stored on clustered Storage Spaces the whole disk is moved from on node to the other and this seems to confuse SCVMM.
As a result SCVMM inserts an invalid value (00000000-0000-0000-0000-000000000000) for “HostDiskID”, for one of the hosts, in the “tbl_ADHC_HostDisk” table in the SCVMM DB.
tbl_ADHC_HostDisk

During the Live Migration pre checks, SCVMM runs a query to find a disk with the ID “00000000-0000-0000-0000-000000000000” in the DB which obviously does not exists. So the pre checks and Live Migration Job fails immediately with error 20404

Solution / Conclusion:

Update April 29, 2015: UR6 for SCMM 2012 R2 is now released and includes a fix for this issue. Yeah! 🙂

After opening a Microsoft Support case and posting the issue on connect.microsot.com I got the confirmation from Microsoft that this behavior is a bug in UR5. It will probably fixed in UR6 which is expect for April.
So my advice for everyone, which is using one of the above configuration together with SCVMM, is to stay on UR4 and wait for UR6. If it’s not already too late. 😉

 

Open high ports (over 49151) on a Windows Server Gateway

19 Jan

In a cloud infrastructure with System Center Virtual Machine Manager (SCVMM), Hyper-V and Azure Pack the Windows Server Gateway could provide the tenants with the possibility to connect their virtual network, provided by Hyper-V Network Virtualization (HNV, NVGRE), with the Internet via NAT. The tenant has then also the possibility to open and forward inbound ports to his VMs. For example he can open Port 80 to run a Webserver which is public reachable over the internet.
Basically this works very well. But lately I had a situation where I had to forward TCP Port 60000. So I was going the Azure Pack Tenant Portal and was trying to add a new Network Rule like I did it several times before:
image

 

 

 

 

 

 

 

 

 

 

But then it happens. The operation failed with a strange error:image

Then I had a look in SCVMM and found this, not very instructive, error:
image

So I digged in a little deeper and discovered on the gateway VM that SCVMM adds the external address for the tenants with the port range 1-49151. So that’s explains why you can not forward Port over 49152 on multitenant Windows Server NAT Gateway:
image

Probably the SCVMM defines this port range for the external address because all ports above 49151 where per RFC6335 actually destined for dynamic ports or private ports. In Windows the this range is also specified for the dynamic client port range for outgoing connections. (RPC Communication and so forth)

Bonus, Possible solutions

Option 1, manual intervention with
PowerShell :
But the RRAS Role in Windows which is also used for multitenant NAT Gateway has no restriction which would hinder you define external address with the whole port range from 1-65535 with PowerShell. In fact when you set an external address with PowerShell the default values for PortStart and PortEnd is 1024 and 65535.
This means you can remove the external address set by scvmm and add the address again with PowerShell with the whole port range. This can achieved by the following steps:

  • Get all external IP Address with Get-NetNatExternalAddress and note the values form the parameter “NatName” and “IPAddress” from the definition which you want to change.
  • Remove the existing external Address definition:

    image
  • Add a New external Address with the same value for NatName and IPAddress but with the new port range:


    image

Afterward you can head again to the tenant portal and now you can add a Network Rule with a port greater that 49151 without any problem. Smiley

Option 2, Registry setting for SCVMM:
After some further research I found that SCVMM has a undocumented Registry setting where you can specify the end port for the external address definition on the NAT Gateway. By creating the following Registry Key SCVMM configures automatically the whole port range (1 to 65535) if a tenant activates the NAT Gateway for his VNet in the Tenant Portal.

image
HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Microsoft System Center Virtual Machine Manager Server\Settings\NATPortRangeEnd, REG_DWORD, Value: 65535

Disclaimer: Use these settings at your own risk! These where NO official solutions from Microsoft and changing these settings will lead probably to a unsupported situation! So be careful! Zwinkerndes Smiley