Automate GUIs with UIAutomation in PowerShell

In this post, I would like to describe how to utilize the Microsoft UIAutomation framework through PowerShell to automate GUIs. Since the UIAutomation assembly already comes with .Net it also can be accessed directly through PowerShell. Using UIAutomation directly actually turns out to be quite cumbersome therefore I will also show you how to use UIAutomation through the excellent FlaUI automation framework for .Net. (While there is also a PowerShell wrapper developed by Alexander Petrovskiy (https://github.com/apetrovskiy/STUPS) which even comes with its own macro recorder, the project comes with quite some dependencies and is not actively maintained. It is working and another great way to use UIAutomatino through PowerShell)

Within UIAutomat all GUI elements are part of a tree structure where each GUI’s elements are descendants of the parent window. In order to access GUI elements through UIAutomation we will need to use certain properties of the GUI elements to find the element within the element tree. To view those properties we will utilize another tool with which the properties of any GUI element can be viewed just by hovering the mouse over the element while pressing the “CRTL” key. Here is an example using the app with the “8” key from the built-in calculator app in Windows 10 :

The tool is called Visual UI Automation Verify and is part of the Windows SDK. While the source code is still available on CodePlex until 1st July 2021 there are also clones of it on GitHub.  You will need to download the source code and compile the release in order to get the UIAVerify.exe (this can be done using the free Visual Studio Community Edition (here is a link that describes the steps. Please let me know within the comments if you need a detailed walk-through on how to do this).

With the help of UIAVerify we can build a short script that will control the Windows 10 calculator app to add the numbers 5 and 9:

	Add-Type –AssemblyName UIAutomationClient
	Add-Type –AssemblyName UIAutomationTypes
	$calc = [Diagnostics.Process]::Start('calc')
	#wait for the UI to appear
	$null = $calc.WaitForInputIdle(5000)
	sleep –s 2
	$calcWindowId = ((Get-Process).where{$_.MainWindowTitle -eq 'Calculator'})[0].Id
	$root = [Windows.Automation.AutomationElement]::RootElement
	$condition = New-Object Windows.Automation.PropertyCondition([Windows.Automation.AutomationElement]::ProcessIdProperty, $calcWindowId)
	$calcUI = $root.FindFirst([Windows.Automation.TreeScope]::Children, $condition)
	function FindAndClickButton($name){
	$condition1 = New-Object Windows.Automation.PropertyCondition([Windows.Automation.AutomationElement]::ClassNameProperty, "Button")
	$condition2 = New-Object Windows.Automation.PropertyCondition([Windows.Automation.AutomationElement]::NameProperty, $name)
	$condition = New-Object Windows.Automation.AndCondition($condition1, $condition2)
	$button = $calcUI.FindFirst([Windows.Automation.TreeScope]::Descendants, $condition)
	$button.GetCurrentPattern([Windows.Automation.InvokePattern]::Pattern).Invoke()
	}
	#get and click the buttons for the calculation
	FindAndClickButton Five
	FindAndClickButton Plus
	FindAndClickButton Nine
	FindAndClickButton Equals
	#get the result
	$condition = New-Object Windows.Automation.PropertyCondition([Windows.Automation.AutomationElement]::AutomationIdProperty, "CalculatorResults")
	$result = $calcUI.FindFirst([Windows.Automation.TreeScope]::Descendants, $condition)
	$result.current.name

view raw UIAutomationCalculator.ps1 hosted with ❤ by GitHub

That’s probably one of the hardest ways to add two numbers using PowerShell :-). While this didn’t seem to be too difficult, it can rapidly get quite cumbersome if you want to do more advanced things.

Let’s switch to FlaUI to do just the same. We will start by downloading the required packages from NuGet. FlaUI (I could not get the UIA3 assembly for the latest version of FlaUI working in PowerShell therefore I’m downloading an older version) also has its own “Visual UI” (FlaUInspect) for which we can get a compiled version via chocolatey:

	$null = mkdir test
	cd test
	Install-Package –Name FlaUI.Core –ProviderName NuGet –RequiredVersion 1.3.1 –SkipDependencies –Destination $pwd –Source nuget.org
	Install-Package –Name FlaUI.CORE –ProviderName NuGet –RequiredVersion 1.3.0 –SkipDependencies –Destination $pwd –Source nuget.org
	Find-Package flauinspect | Install-Package

view raw FlaUIInstallation.ps1 hosted with ❤ by GitHub

Now we can use FlaUI to reproduce the same calculator automation (this is the content of a .ps1 script which should be put into the same folder as the previously downloaded packages):

	#assuming the script is in the same folder as the FlaUI assemblies
	Add-Type –Path "$PSScriptRoot\FlaUI.Core.1.3.0\lib\net45\FlaUI.Core.dll"
	Add-Type –Path "$PSScriptRoot\FlaUI.UIA3.1.3.1\lib\net45\FlaUI.UIA3.dll"
	$calc = [Diagnostics.Process]::Start('calc')
	#wait for the UI to appear
	$null = $calc.WaitForInputIdle(5000)
	sleep –s 2
	$calcWindowId = ((Get-Process).where{ $_.MainWindowTitle -eq 'Calculator' })[0].Id
	$uia = New-Object FlaUI.UIA3.UIA3Automation
	$cf = $uia.ConditionFactory
	$btnCondition = $cf.ByControlType('Button')
	$desktop = $uia.GetDesktop()
	$calc = $desktop.FindFirstDescendant($uia.ConditionFactory.ByProcessId($calcWindowId))
	$calc.FindFirstDescendant($btnCondition.And($cf.ByName('Five'))).AsButton().Click()
	$calc.FindFirstDescendant($btnCondition.And($cf.ByName('Plus'))).AsButton().Click()
	$calc.FindFirstDescendant($btnCondition.And($cf.ByName('Nine'))).AsButton().Click()
	$calc.FindFirstDescendant($btnCondition.And($cf.ByName('Equals'))).AsButton().Click()
	$calc.FindFirstDescendant($cf.ByAutomationId('CalculatorResults')).Properties.Name.Value

view raw FlaUICalculatorPowershell.ps1 hosted with ❤ by GitHub

This just scratches the surface of what can be done with UIAutomation, but it should give you some good ideas on how to get started.

---

Photo Credit: PapaPiper Flickr via Compfight cc