Useful PowerShell snippets for Storage Spaces Direct (S2D) operations

30 May

This is a very quirk post to just share some simple PowerShell snippets I have created or found elsewhere which I find very handy when it comes to troubleshooting, setup or operating Storage Spaces Direct (S2D).

1. Get the count of physical disk in each node:
When S2D is enabled in a cluster Get-Physicaldisk returns the physical disk from all cluster nodes. But when you want the disk count of each node the following script can be quite useful:

2. Get the processed and completed GBytes of running Storage Jobs (e.g. Rebuild Jobs):
This one comes from the Jaromir Kaspar and his ws2016lab Project on GitHub, which is btw a very nice  project to build a test lab in a highly automated way. So all credit goes to him for this script.
It shows you the amount of GBytes the running storage jobs have already processed and how much must still to be processed until the jobs are finished.

(original source: ws2016lab project)

Get insights about the performance of your Windows systems with Grafana

12 May

Ever dreamed about some mission control like dashboards to get a quick insight about the performance of your Windows systems? 😊

If yes, then you probably like a view like this:

So here is how you get such a dashboard for your system in 6 simple steps in under an hour:

Install a VM with Ubuntu Linux 16.04.2 LTS

Even when it is Linux, no rocket science is needed here 😊. Just download the ISO image from the Ubuntu Website, attach it to your VM and boot form it. After that you get asked some simple questions about time zone, keyboard and partition settings. The most you can accept with the defaults or choose simple your preferred languages etc. Quite easy.

Set time zone to UTC

Login in to your Ubuntu system and change the time zone to UTC. As the InfluxDB (the backend) uses UTC time internally it is a clever idea to set the time zone for the system also to UTC.
To do so run the following command. Then choose “Non of the above” > “UTC”.

Install InfluxDB

InfluxDB is the he backend of the solution where all data is stored. It is a database engine which is built form the ground up to store metric data and for doing real-time analytics.
To install InfluxDB run the following commands on the Linux VM:

Install Grafana

Grafana is the frontend which will generate your nice-looking dashboard with the data stored in the InfluxDB. To install Grafana run the following commands on the Linux VM:

Install Telegraf on your Windows system

Now we are ready to collect data from our systems with the Telegraf, a small agent which can collect data from many various sources. One of these source is Windows Perfmon Counters which we will use here.

1. Download the Windows version of the Telegraf agent
https://dl.influxdata.com/telegraf/releases/telegraf-1.2.1_windows_amd64.zip
2. Copy the content of the zip file to C:\Program Files\telegraf on your systems
3. Replace the telegraf.conf with this one. -> telegraf.conf
So all needed perform counters get collected which are needed for the example dashboard in the last step.
4.  Also in the telegraf.conf, update the urls  paramter so it point to the IP address of your Linux VM


5. Install Telegraf as service and start it

Create Dahsboards and have fun! 🙂

The last step is to create your nice dashboards in the Grafana web UI. A good starting point is the “Telegraf & Influx Windows Host Overview” dashboard which can directly imported from the grafana.net repository

Login into the Grafana Web UI -> http://<your linux VM IP>:3000 (Username: admin, Password: admin)

First Grafana need to know it’s data source. Click on the Grafana logo in the top left corner and select “Data Source” in the menu. Then click on “+ Add data source“.

Define an Name for the Data Source (e.g. InfluxDB-telegraf) and choose “InfluxDB” as Type.
The URL is http://localhost:8086 has we have installed the InfluxDB locally. “Proxy” as the access type is correct.
The telegraf agent will automatically create the data base “telegraf”. So enter “telegraf” as Database name. As user you can enter anything. InfluxDB does not need any credentials by default but the Grafana interface wants you to enter something. (otherwise you can not save the data source)

Now go ahead and import your first dashboard.  Select Dashboard > Import in the menu

Enter “1902” and click on “Load

Change the Name if you like and select the data source just created in the step above (InfluxDB-telegraf) and then click on Import.

And tada! 🙂

Further steps

Now the Telegraf / InfluxDB setup is collecting performance data of your windows machines. With Grafana the collected data can visualized in a meaningful way so the determination the health of your system gets easy.

To further customize the data and visualization to your specific needs you can:

Script to build streched file server cluster with Storage Replica

7 Mar

One possible scenario for use of Storage Replica in Windows Server 2016 is to build a stretched file server cluster based on two VMs on two different sites. With this configuration you can build a highly available file server across two sites without the need of replicated SAN or similarly. Instead you can simply use the Storage which is locally available at each site and leverage Storage Replica to replicate the data volumes inside the VMs. In case one of the Sites fails, the File Server Role will automatically fail over to the second site and the end user will probably not even notice it.

Recently I have made some tests with such a set up in my Homelab where I had the need to rebuild quickly the whole environment. Therefore I made a simple script with all the needed PowerShell commands.

You can get a copy of the Script at my GitHub Repository

The Script is intended to run on a third machine, like for example a Management Server which has the Windows Server 2016 RSAT Tools installed. Especially the Hyper-V, Failover Cluster and Storage Replica Cmdlets are required.

After you set the correct parameter values and you are really sure everything is right 😉 , you can run the script in one step. Or, probably the more interesting approach, is to open the script in the PowerShell ISE and run the individual steps one by one.
For this purpose the script has comments which mark the indivudaul steps:

So have fun with PowerShell and Storage Replica. A very nice combination! 🙂

SCVMM: When the deployment of new VM template suddenly fails

2 Mar

Recently I ran in a very strange behavior when deploying a VM template with Server 2016 through VMM 2012 R2. First of all to enable the full support of Windows Server 2016-based VMs in VMM 2012 you need at least Update Rollup 11 Hotfix 1 installed. But even after installing  the latest UR (UR12 in my case), the deployment of a Server 2016 VM has failed.

The Issue:
Everytime when a new VM is deployed from a Server 2016 VM template the process fails at specialize phase of the sysprep. However all other existing templates with Server 2012 were working as expected.

Because in in this phase also the domain join happens I decided to give another try with a VM template which has no domain join configured. And tada, the VM was deployed successfully. 

The root cause:
With this finding my assumption was that, when the VM template is configure for domain join, VMM adds something in the unattend.xml which Server 2016 does not like that much. So I inspected the unattend.xml file of a failed deployment and there I found the following section which has looked a litte bit strange:

 Somehow the Domain of the domain join account was missing. 

The Solution:
So I checked the VMM Run As Account which was specified as domain join credentials in the VM template. And as you can see, we have also no domain information here.  

After changing the username to “domain\vm domain join” the deployment went through smooth as it should. Inspecting the unattend.xml file showed that the domain is now also correctly filled in.

Conslusion:
When the deployment of a new VM Template in VMM suddenly fails at the domain join step, double check the run as account and be sure that there is also the domain name in the username field.
In my case it was a template with Server 2016. But I think chances are good as the same could also happens with new VM templates with another guest OS.

Be aware of DSC pull server compatibility issues with WMF 5.0 and 5.1

20 Feb

Apparently, there are some incompatibilities when WMF 5.0 computers wants to communicate with a DSC pull server running on WMF 5.1 or vice versa. This is especially the case when the “client” node and the pull server are not running the same OS version. For example, when you have a DSC pull server running on Server 2012 R2 (with WMF 5.0) and some DSC nodes running on Server 2016 (which as WMF 5.1 built in).

Currently I experienced two issues:

  1. A DSC pull client running on WMF 5.1 cannot send status reports when the DSC pull server is running still on WMF 5.0. This is because WMF 5.1 has invented the new “AdditinalData” parameter in the status report. I have reported this bug also on GitHub: https://github.com/PowerShell/PowerShell/issues/2921 
  2. A DSC pull client running von WMF 5.0 cannot communicate at all with a DSC pull server running on WMF 5.1.
     

Solution / Workaround for issue 1:
As the WMF 5.1 RTM no (again) available the simplest solution would be to upgrade the server and/or client to WMF 5.1. However, when you have to upgrade the DSC pull server then you must create a new EDB file and reregister all clients. Otherwise the issue preserve because the “AdditionalData” field is still missing in the database.

Solution / Workaround for issue 2:
The root cause of this issue can be found in the release notes of WMF 5.1:
“Previously, the DSC pull client only supported SSL3.0 and TLS1.0 over HTTPS connections. When forced to use more secure protocols, the pull client would stop functioning. In WMF 5.1, the DSC pull client no longer supports SSL 3.0 and adds support for the more secure TLS 1.1 and TLS 1.2 protocols.”

So, starting with WMF 5.1 the DSC pull server does not support TLS 1.0 anymore, but in reverse a DSC pull client running on WMF 5.0 is still using TLS 1.0 and can therefore not connect anymore to the DSC pull server.

The solution, without deploying WMF 5.1 to all pull clients, is to alter the behavior of the DSC pull server so he accepts again TLS 1.0 connections. This can be done by changing the following registry key on the DSC pull server:

Change Value from 0x0 to 0x1 and reboot the DSC pull server.
Afterward DSC pull clients running on WMF 5.0 can connect again to the DSC pull server.

How to enable CredSSP for PowerShell Remoting through GPO

19 Oct

In a domain environment CredSSP can easily enabled through a GPO. To do so there are three GPO settings to configure:

  1. Computer Configuration > Administrative Templates > Windows Components > Windows Remote Management (WinRM) > WinRM Client > Allow CredSSP Authentication (Enable)
    image
  2. Computer Configuration > Administrative Templates > Windows Components > Windows Remote Management (WinRM) >  WinRM Service > Allow CredSSP Authentication (Enable)
    image
  3. Computer Configuration > Administrative Templates  > System > Credential Delegation > Allow delegation of fresh credentials (add wsman/*<.FQDN of your domain>)
    image
  4. If in your environment are computers in an other, not trusted, AD domain to which you want connect using explicit credential and CredSSP you have to enabled also the following GPO setting.
    Computer Configuration > Administrative Templates  > System > Credential Delegation > Allow delegation of fresh credentials with NTLM-only server authentication (add wsman/*<.FQDN of your other domain>)
    image

Now you are ready to use CredSSP within your PowerShell remote sessions.

And a final word of warning! 😉
When you are using CredSSP your credentials were transferred to the remote system and your account is then a potential target for a pass-to-hash attack. Or with other words an attacker can steal your credentials. So only use CreddSSP with your PowerShell Remote session if you really have a need for it!

How to setup VMFleet to stress test your Storage Spaces Direct deployment

26 May

As an outcome of the Microsoft’s demos about Storage Spaces Direct at Ignite conference and the Intel Developer Forum, Microsoft published recently a bunch of PowerShell Script known as “VMFleet“.

VMFleet is a basically a collection of PowerShell script to easily create a bunch of VMs and run some stress tests in it (mostly with the DISKSPD tool) to test the performance of an underlying Storage Space Direct deployment or simply for demonstration purposes.

After you have downloaded the files it is not quite obviously how to get started as the included documentation does not give you some simple step by step guidelines how to get all up and running.

So I decided to write my own short guide with the needed steps to get the VMFleet up and running.

Update 03.01.2017: Since I wrote this post in May 2016, Microsoft has apparently extended the VMFleet with some new scripts to setup the environment. So I updated the setup steps below. Thanks to my colleague and coworker Patrick Mutzner for pointing that out!

Prerequisites:

  • A  VHDX file with installed Windows Server 2012 R2 / 2016 Core and the password of the local Administrator account set. (sysprep must not be run!)
    A fixed VHDX file is recommended to eliminate “warmup” effects when starting the test runs.
    Note: The VHDX should be at least 20GB big because the VMFleet Scripts will create a load test file of 10GB inside the VMs.
  • A functional Storage Spaces Direct Cluster (with existing Storage Pool and no configured Virtual Disks)

VMFleet Setup:

  1. First you need to create one Volume/CSV per node to store the test VMs.
  2. Then create an additional CSV to store the VMFleet scripts and for the collection of the test results from the VMs.
  3. Extract the ZIP file with the VMFleet scripts on one of the cluster nodes.
    (For example extract the downloaded ZIP file to C:\Source on the first cluster node)
  4. Run the install-vmfleet.ps1 Script. This will create the needed folder structure on the “collect” CSV. (The volume created at Step 2)
  5. Copy the DISKSPD.exe to C:\ClusterStorage\Collect\Control\Tools

    Note: Everything in this folder will later automatically be copied into every test VM. So you can also copy other files to this folder if you need them later inside the test VMs.

  6. Copy template VHDX to C:\ClusterStorage\Collect 
  7. Run the update-csv.ps1 (unter C:\ClusterStorage\Collect\Control) script to rename the mountpoints of the CSV volumes and to distribute the CSV even in the cluster so every node is the owner of one CSV.
    update-csv
  8. Now it’s time to create a bunch of test VMs on the four CSVs. This is done by the create-vmfleet.ps1 Script.

    As parameters you must specify the path to the template VHDX file, how many VM per Node you like to create, the Administrator password of the template VHDX and a Username and its Password which has access to the Cluster Nodes (HostAdmin in this example). This account will be used inside the VMs to connect back to the cluster so that the script running inside the VMs get access to  the collect CSV  (C:\ClusterStorage\collect).
  9. Because the create-vmfleet.ps1 creates the VMs with the default values from the New-VM Cmdlet you should run now the set-vmfleet.ps1 Script to change the count of vCPUs and the memory size of the test VMs to your desired values.
  10. (Optional) Now before you begin with the tests you can check your Storage Spaces Direct Cluster configuration with the test-clusterhealth.ps1 script. The Script will check on all nodes if there are any RDMA configuration errors and it will do some basic health checks of the Storage Spaces Direct setup.
    test-cluster 
  11. As the last preparation step start the watch-cluster.ps1 script. This will give you a nice dashboard like overview of the most interesting storage performance counters across all nodes in the cluster. So you get an overview of the storage load and performance.
    watch-cluster 
  12. At this Point you are ready to start you first test run. 
    Simply start all test VMs
     manually or with the start-vmfleet.ps1 script. After the VMs have booted up they will automatically look for the run.ps1 script on the collect CSV (at C:\ClusterStorage\Collect\Control). But by default the test run is in paused state. So to start the actual load test simply run the clear-pause.ps1 script. This will kick off the dskspd.exe in every VMs and you can observe how the numbers from the watch-cluster.ps1 will explode… 😉

To change the test pattern simply change the parameters in the run.ps1 Script and either restart all test VMs (with stop-vmfleet.ps1 and start-vmfleet.ps1 scripts) or pause and resume the tests with the set-pause.ps1 and clear-pause.ps1 scripts.

That’s it. VMFleet is ready. So have fun while testing your S2D deployment and while looking at (hopefully) some awesome IOPS numbers. 🙂

Configuring Hyper-V Hosts with converged networking through PowerShell DSC

10 Dec

Lately I had to rebuild the Hyper-V Hosts in my home lab several times because of the release of the different TPs for Windows Server 2016. This circumstance (and because I am a great fan of PowerShell DSC Winking smile) gave me the Idea to do the whole base configuration of the Hyper-V Host, including the LBFO NIC Teaming, vSwitch and vNICs for the converged networking configuration, through PowerShell DSC.
But soon I realized that the DSC Resource Kit from Microsoft provides DSC resources only for a subset of the needed configuration steps. The result was some PowerShell modules with my custom DSC resources.

My custom DSC resources for the approach:
image

cLBFOTeam: To create and configure the Windows built-in NIC Teaming
cVNIC: To create and configuring virutal network adapters for Hyper-V host management
cPowerPlan: To set a desired Power Plan in Windows (e.g. High Performance Plan)

You can get the moduels from the PowerShell Gallery (Install-Module) or from GitHub. They will hopefully help everyone who has a similar intend  Winking smile

More to come:
Yes, I am not quite finished yet and I have more in the pipline. Winking smile
Currently I am also working on a fork of the xHyperV Module with a adopteted xVMSwitch resource with a paramter to specify the MinimumBandwidth mode of the Switch.
Futuremore I am also planing to add support for the SET (Swicht Embedded Teaming) in Windows Server 2016 to the xVMSwitch resource.

So you may soon read more about this topic here. In the meantime, happy DSCing! Smile

Build a Windows Server 2016 TP3 VHD with minimal GUI

6 Sep

In the TP3 the installation option was changed. Therefore, when you create a VHD(X) directly from the ISO with the Convert-WindowsImage.ps1 Script you have choice to create a VHD with Core Server or the full GUI with Desktop Experience but nothing in between. To create a VHD with the minimal server interface (core server with Server Manger and mmc GUIs) or the Server Graphical Shell (without Desktop Experience) you have to add the corresponding features with DISM.

This is how you add the minimal server interface to a VHD with the core server installation:

  1. Create a Core Server VHD with the Convert-WindowsImage.ps1 Script
  2. Mount the Windows Server 2016 TP3 ISO (double click it)
  3. Mount the install.wim from the ISO with DISM
    (Index 4 for Datacenter, 2 for Standard Edition)
  4. Add the Server-Gui-Mgmt-Infra (for Minimal Server Interface) or the Server-Gui-Shell (for full Server Graphical Shell) feature to the VHD by specify the mounted WIM as source.
Update, 11/20/2015:
This does not work any more with the TP4 which is now public available as the feature “Server-Gui-Mgmt-Infra” is gone now. You can add the feature “Server-Gui-Mgmt” with DISM which gives you a similar experience. But the feature is not even listed in PowerShell (Get-WindowsFeature) so I think this is probably far form supported.
With other words: No “Minimal Server Interface” in TP4 anymore.
 

PowerShell DSC resource to enable/disable Microsoft Update

16 Jun

Ever get tired to manually set the check box for Microsoft Update in the above screen on a bunch of servers (e.g. in a new test lab setup or so)? Then this one is for you.

 I wrote recently, mostly as an exercise, a PowerShell DSC Module with a resource to enable (or disable) the Microsoft Update.

 I have then published the Module von GitHub to get another exercise. 😉
So if you interested you can get the Module from here:

https://github.com/J0F3/cMicrosoftUpdate

After you get the module, enabling the Microsoft Update settings will look like this:

Happy DSCing! 🙂