Automate GUIs with UIAutomation in PowerShell

In this post, I would like to describe how to utilize the Microsoft UIAutomation framework through PowerShell to automate GUIs. Since the UIAutomation assembly already comes with .Net it also can be accessed directly through PowerShell. Using UIAutomation directly actually turns out to be quite cumbersome therefore I will also show you how to use UIAutomation through the excellent FlaUI automation framework for .Net. (While there is also a PowerShell wrapper developed by Alexander Petrovskiy (https://github.com/apetrovskiy/STUPS) which even comes with its own macro recorder, the project comes with quite some dependencies and is not actively maintained. It is working and another great way to use UIAutomatino through PowerShell)

Within UIAutomat all GUI elements are part of a tree structure where each GUI’s elements are descendants of the parent window. In order to access GUI elements through UIAutomation we will need to use certain properties of the GUI elements to find the element within the element tree. To view those properties we will utilize another tool with which the properties of any GUI element can be viewed just by hovering the mouse over the element while pressing the “CRTL” key. Here is an example using the app with the “8” key from the built-in calculator app in Windows 10 :

The tool is called Visual UI Automation Verify and is part of the Windows SDK. While the source code is still available on CodePlex until 1st July 2021 there are also clones of it on GitHub.  You will need to download the source code and compile the release in order to get the UIAVerify.exe (this can be done using the free Visual Studio Community Edition (here is a link that describes the steps. Please let me know within the comments if you need a detailed walk-through on how to do this).

With the help of UIAVerify we can build a short script that will control the Windows 10 calculator app to add the numbers 5 and 9:

Add-Type AssemblyName UIAutomationClient
Add-Type AssemblyName UIAutomationTypes
$calc = [Diagnostics.Process]::Start('calc')
#wait for the UI to appear
$null = $calc.WaitForInputIdle(5000)
sleep s 2
$calcWindowId = ((Get-Process).where{$_.MainWindowTitle -eq 'Calculator'})[0].Id
$root = [Windows.Automation.AutomationElement]::RootElement
$condition = New-Object Windows.Automation.PropertyCondition([Windows.Automation.AutomationElement]::ProcessIdProperty, $calcWindowId)
$calcUI = $root.FindFirst([Windows.Automation.TreeScope]::Children, $condition)
function FindAndClickButton($name){
$condition1 = New-Object Windows.Automation.PropertyCondition([Windows.Automation.AutomationElement]::ClassNameProperty, "Button")
$condition2 = New-Object Windows.Automation.PropertyCondition([Windows.Automation.AutomationElement]::NameProperty, $name)
$condition = New-Object Windows.Automation.AndCondition($condition1, $condition2)
$button = $calcUI.FindFirst([Windows.Automation.TreeScope]::Descendants, $condition)
$button.GetCurrentPattern([Windows.Automation.InvokePattern]::Pattern).Invoke()
}
#get and click the buttons for the calculation
FindAndClickButton Five
FindAndClickButton Plus
FindAndClickButton Nine
FindAndClickButton Equals
#get the result
$condition = New-Object Windows.Automation.PropertyCondition([Windows.Automation.AutomationElement]::AutomationIdProperty, "CalculatorResults")
$result = $calcUI.FindFirst([Windows.Automation.TreeScope]::Descendants, $condition)
$result.current.name

That’s probably one of the hardest ways to add two numbers using PowerShell :-). While this didn’t seem to be too difficult, it can rapidly get quite cumbersome if you want to do more advanced things.

Let’s switch to FlaUI to do just the same. We will start by downloading the required packages from NuGet. FlaUI (I could not get the UIA3 assembly for the latest version of FlaUI working in PowerShell therefore I’m downloading an older version) also has its own “Visual UI” (FlaUInspect) for which we can get a compiled version via chocolatey:

$null = mkdir test
cd test
Install-Package Name FlaUI.Core ProviderName NuGet RequiredVersion 1.3.1 SkipDependencies Destination $pwd Source nuget.org
Install-Package Name FlaUI.CORE ProviderName NuGet RequiredVersion 1.3.0 SkipDependencies Destination $pwd Source nuget.org
Find-Package flauinspect | Install-Package

Now we can use FlaUI to reproduce the same calculator automation (this is the content of a .ps1 script which should be put into the same folder as the previously downloaded packages):

#assuming the script is in the same folder as the FlaUI assemblies
Add-Type Path "$PSScriptRoot\FlaUI.Core.1.3.0\lib\net45\FlaUI.Core.dll"
Add-Type Path "$PSScriptRoot\FlaUI.UIA3.1.3.1\lib\net45\FlaUI.UIA3.dll"
$calc = [Diagnostics.Process]::Start('calc')
#wait for the UI to appear
$null = $calc.WaitForInputIdle(5000)
sleep s 2
$calcWindowId = ((Get-Process).where{ $_.MainWindowTitle -eq 'Calculator' })[0].Id
$uia = New-Object FlaUI.UIA3.UIA3Automation
$cf = $uia.ConditionFactory
$btnCondition = $cf.ByControlType('Button')
$desktop = $uia.GetDesktop()
$calc = $desktop.FindFirstDescendant($uia.ConditionFactory.ByProcessId($calcWindowId))
$calc.FindFirstDescendant($btnCondition.And($cf.ByName('Five'))).AsButton().Click()
$calc.FindFirstDescendant($btnCondition.And($cf.ByName('Plus'))).AsButton().Click()
$calc.FindFirstDescendant($btnCondition.And($cf.ByName('Nine'))).AsButton().Click()
$calc.FindFirstDescendant($btnCondition.And($cf.ByName('Equals'))).AsButton().Click()
$calc.FindFirstDescendant($cf.ByAutomationId('CalculatorResults')).Properties.Name.Value

This just scratches the surface of what can be done with UIAutomation, but it should give you some good ideas on how to get started.

shareThoughts

Photo Credit: PapaPiper Flickr via Compfight cc

I'd love to hear what you think

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s