Convert-String for PowerShell is like FlashFill for Excel

16251434544_ac3c18c295_m

One of the many great additions that come with Powershell v5 is the Convert-String cmdlet (there is no official documentation at this point). Convert-String along with ConvertFrom-String expose FlashExtract functionality to the PowerShell world. While ConvertFrom-String allows for some very sophisticated template based parsing (Here, here and here are some good links to get started) does Convert-String work much more like the FlashFill feature added in MS Excel.
Convert-String can parse strings based on 1 or multiple examples. Let’s look at two simple use cases:
ConvertString
Code:

'Jeffrey Snover', 'Ed Wilson', 'Bruce Payette', 'Lee Holmes' | 
   Convert-String -Example 'First Last=>_ F. Last'

1..10 | 
   foreach { [math]::Round((get-random -Minimum 1e9 -Maximum 9e9)) } | 
   Convert-String -Example '0123456789=(012) 345-6789'

In addition to the input string the cmdlet only takes an argument on the ‘Example’ parameter. The ‘Example’ parameter expects one of the following argument forms (thank you dotPeek):

Description Syntax Example
A string of the form ORIGINAL=TRANSFORMED ‘ORIGINAL=TRANSFORMED’ ‘First Last=F. L’
An array of strings of the form ORIGINAL=TRANSFORMED ‘ORIGINAL1=TRANSFORMED1’, ‘ORIGINAL2=TRANSFORMED2’ (‘Jeffery Snover=J S’, ‘Lee Holmes=L H’)
A hashtable with BEFORE and AFTER keys @{BEFORE=ORIGINAL;AFTER=TRANSFORMED’} @{
BEFORE=’First Last’;
AFTER=’F Last’}
An array of hashtables with BEFORE and AFTER key pairs @{BEFORE=ORIGINAL;AFTER=TRANSFORMED’} @{
BEFORE=’Jeffery Snover’;
AFTER=’J S’},
@{
BEFORE=’Lee Holmes’;
AFTER=’L H’}

One more thing to note about the Example parameter syntax is, that ‘\’ can be used to escape characters (e.g. if you have ‘=’ in your pattern you have to use ‘\=’ instead).
The FlashExtract engine basically automatically figures out what you want based on examples. Attempting to parse more complex patterns requires more examples, where the examples should reflect the different variations of the pattern within the string to parse. Let’s look at some more examples for illustration purpose:

$txt = @'
Pete,47,Canada
Jake,23,France
Carol,33,Italy
Jane,18,Ireland
'@ -split "`n"
$ex = 'Pete,47,Canada=Pete is from Canada and 47 years old'
$txt | Convert-String -Example $ex

Whoops, this didn’t work out. This time around Convert-String was not able to recognize the full pattern throughout the entire input. Therefore we will need to provide more examples:

$txt = @'
Pete,47,Canada
Jake,23,France
Carol,33,Italy
Jane,18,Ireland
'@ -split "`n" 
$ex1 = 'Pete,47,Canada=Pete is from Canada and 47 years old'
$ex2 = 'Jake,23,France=Jake is from France and 23 years old'
$txt | Convert-String -Example ($ex1, $ex2)

Two examples are indeed sufficient in this case. Convert-String is also able to figure out some variations itself. Here for example the age is either one or two characters long:

$txt = @'
Pete,47,Canada
Jake,23,France
Carol,3,Italy
Jane,8,Ireland
'@ -split "`n"
$ex1 = 'Pete,47,Canada=Pete is from Canada and 47 years old'
$ex2 = 'Jake,23,France=Jake is from France and 23 years old'
$txt | Convert-String -Example ($ex1, $ex2)

Until now you might think that all this can also be quite easily achieved using previous PowerShell parsing capabilities and you would be right. While the examples so far can also make your life easier Convert-String can also do more advanced parsing which previously could be mainly done using Regular Expressions:

$txt = @'
Pete47Canada
Jake23France
Carol3Italy
Jane8Ireland
'@ -split "`n"
$ex1 = 'Pete47Canada=Pete is from Canada and 47 years old'
$ex2 = 'Jake23France=Jake is from France and 23 years old'
$txt | Convert-String -Example ($ex1, $ex2)

Pretty cool, huh?
Even more examples:

$txt = @'
Pete12/25/1966Canada
Jake3/14/1975France
Carol4/1/2012Italy
Jane8/17/2000Ireland
'@ -split "`n"
$ex1 = 'Pete12/25/1966Canada=Pete was born in 1966 and lives in Canada'
$ex2 = 'Carol4/1/2012Italy=Carol was born in 2012 and lives in Italy'
$txt | Convert-String -Example $ex1, $ex2

$txt = @'
Pete12/25/1966Canada
Jake3/14/1975France
Carol4/1/2012Italy
Jane8/17/2000Ireland
'@ -split "`n"
$ex1 = 'Pete12/25/1966Canada=Pete 12 Canada'
$ex2 = 'Carol4/1/2012Italy=Carol 4 Italy'
$txt | Convert-String -Example $ex1, $ex2 | 
    foreach { 
        $name, $month, $country = $_.Split()
        "$name was born in $((Get-Date -Month $month).ToString('MMMM')) and lives in $country" 
    }

$txt = @'
Pete$12.77Canada
Jake$13.8France
Carol$14.989Italy
Jane$17.99Ireland
'@ -split "`n"
$txt | Convert-String -Example 'Pete$12.77Canada=12,77'

Convert-String can make all of us string parsing rock stars!

shareThoughts


Photo Credit: Eric@focus via Compfight cc

Advertisements

Simplified syntax for calculated Properties with Select-Object

3798238251_59749f23cb_m

Following on my journey in an attempt to make PowerShell work exactly the way I would like it to work I had a look into the syntax for calculated properties with Select-Object. Calculated properties for Select-Object are basically syntactic accidents sugar to add custom properties to objects on the fly (examples are taken from here):

Get-ChildItem | Select-Object Name, CreationTime,  @{Name="Kbytes";Expression={$_.Length / 1Kb}}
Get-ChildItem | Select-Object Name, @{Name="Age";Expression={ (((Get-Date) - $_.CreationTime).Days) }}

Looking at the documentation for Select-Object we can see that the syntax for the calculated properties on the Property parameter permits different key names and value type combinations as valid arguments:

[TYPE]KEYNAME 1 [TYPE]KEYNAME 2 Example
[STRING]Name [STRING]Expression @{Name=”Kbytes”;Expression=”Static value”}
[STRING]Name [SCRIPTBLOCK]Expression @{Name=”Kbytes”;Expression={$_.Length / 1Kb}}
[STRING]Label [STRING]Expression @{Label=”Kbytes”;Expression=”Static value”}
[STRING]Label [SCRIPTBLOCK]Expression @{Label=”Kbytes”;Expression={$_.Length / 1Kb}}

Most of the people already familiar with PowerShell also know that the parameter also accepts abbreviated key names, just using the first letter:

Get-ChildItem | Select-Object Name, CreationTime,  @{n="Kbytes";e={$_.Length / 1Kb}}
Get-ChildItem | Select-Object Name, @{n="Age";e={ (((Get-Date) - $_.CreationTime).Days) }}

What I find confusing about this syntax is the fact that we need two key/value pairs in order to actually provide a name and a value. In my humble opinion it would make more sense if the Property parameter syntax for calculated properties would work the following way:

Get-ChildItem | Select-Object Name, CreationTime,  @{Kbytes={$_.Length / 1Kb}}
Get-ChildItem | Select-Object Name, @{Age={ (((Get-Date) - $_.CreationTime).Days) }}

Let’s see how this could be implemented with a little test function:

Now we can go ahead and create the proxy function to make Select-Object behave the same way.
First we will need to retrieve the scaffold for the proxy command. The following will copy the same to the clip board:

$Metadata = New-Object System.Management.Automation.CommandMetaData (Get-Command Select-Object)
$proxyCmd = [System.Management.Automation.ProxyCommand]::Create($Metadata) | clip

Below is the code of the full proxy command highlighting the modified lines (as compared to the scaffold code):

function Select-Object{
    [CmdletBinding(DefaultParameterSetName='DefaultParameter', HelpUri='http://go.microsoft.com/fwlink/?LinkID=113387', RemotingCapability='None')]
    param(
        [Parameter(ValueFromPipeline=$true)]
        [psobject]
        ${InputObject},

        [Parameter(ParameterSetName='SkipLastParameter', Position=0)]
        [Parameter(ParameterSetName='DefaultParameter', Position=0)]
        [System.Object[]]
        ${Property},

        [Parameter(ParameterSetName='SkipLastParameter')]
        [Parameter(ParameterSetName='DefaultParameter')]
        [string[]]
        ${ExcludeProperty},

        [Parameter(ParameterSetName='DefaultParameter')]
        [Parameter(ParameterSetName='SkipLastParameter')]
        [string]
        ${ExpandProperty},

        [switch]
        ${Unique},

        [Parameter(ParameterSetName='DefaultParameter')]
        [ValidateRange(0, 2147483647)]
        [int]
        ${Last},

        [Parameter(ParameterSetName='DefaultParameter')]
        [ValidateRange(0, 2147483647)]
        [int]
        ${First},

        [Parameter(ParameterSetName='DefaultParameter')]
        [ValidateRange(0, 2147483647)]
        [int]
        ${Skip},

        [Parameter(ParameterSetName='SkipLastParameter')]
        [ValidateRange(0, 2147483647)]
        [int]
        ${SkipLast},

        [Parameter(ParameterSetName='IndexParameter')]
        [Parameter(ParameterSetName='DefaultParameter')]
        [switch]
        ${Wait},

        [Parameter(ParameterSetName='IndexParameter')]
        [ValidateRange(0, 2147483647)]
        [int[]]
        ${Index})

    begin
    {
        try {
            $outBuffer = $null
            if ($PSBoundParameters.TryGetValue('OutBuffer', [ref]$outBuffer))
            {
                $PSBoundParameters['OutBuffer'] = 1
            }
            #only if the property array contains a hashtable property
            if ( ($Property | where { $_ -is [System.Collections.Hashtable] }) ) {
                $newProperty = @()
                foreach ($prop in $Property){
                    if ($prop -is [System.Collections.Hashtable]){
                        foreach ($htEntry in $prop.GetEnumerator()){
                           $newProperty += @{n=$htEntry.Key;e=$htEntry.Value}
                        }
                    }
                    else{
                        $newProperty += $prop
                    }
                }
                $PSBoundParameters.Property = $newProperty
            }
            $wrappedCmd = $ExecutionContext.InvokeCommand.GetCommand('Microsoft.PowerShell.Utility\Select-Object', [System.Management.Automation.CommandTypes]::Cmdlet)
            $scriptCmd = {& $wrappedCmd @PSBoundParameters }
            $steppablePipeline = $scriptCmd.GetSteppablePipeline($myInvocation.CommandOrigin)
            $steppablePipeline.Begin($PSCmdlet)
        } catch {
            throw
        }
    }

    process
    {
        try {
            $steppablePipeline.Process($_)
        } catch {
            throw
        }
    }

    end
    {
        try {
            $steppablePipeline.End()
        } catch {
            throw
        }
    }
<#
.ForwardHelpTargetName Microsoft.PowerShell.Utility\Select-Object
.ForwardHelpCategory Cmdlet
#>
}

Putting the above into your profile (You can read here and here on how to work with profiles) will make the modified Select-Object available in every session.
What do you think of the syntax for calculated properties?

shareThoughts


Photo Credit: ChrisK4u via Compfight cc

WMI query filters with PowerShell syntax instead of WQL

2465120031_ebb0a49e45_m
PowerShell comes already with tight integration to WMI with its built-in Get-WmiObject and Get-CimInstance cmdlets. One of the things that people already familiar with PowerShell syntax bothers about WMI is that it comes with its very own query language WQL. While WQL is very similar to SQL. Wouldn’t it be nicer if we could use the same operators and wild-card patterns we are already familiar with?
Well, for myself the answer is Yes:

PowerShellInsteadOfWQL
Let’s first build a proof of concept before creating a proxy function for Get-WmiObject. We can make us of the (newer) PowerShell parser to identify the different elements of the Filter for conversion and the WildcardPattern.ToWql method to convert the wild-card pattern for the LIKE operator:

All it takes are 35 lines of code to implement the functionality.
Now that we have the proof of concept working we can go ahead and dynamically create the proxy functions for Get-WmiObject and Get-CimInstance to keep it simple we just add an additional parameter (PowerShellFilter) that takes the PowerShell syntax, converts the PowerShell to WQL and passes it on to the ‘Filter’ parameter without worrying about adding an additional parameter set to mutually exclude the ‘Filter’ and ‘PowerShellFilter’ parameters. After retrieving the code for the proxy command using the command meta data we need to add the statements for the new parameter (considering the same parameter sets as for the existing ‘Filter’ parameter) at the the top of the param statement and insert the additional logic between the following lines (e.g. Get-WmiObject)…

if ($PSBoundParameters.TryGetValue('OutBuffer', [ref]$outBuffer))
{
$PSBoundParameters['OutBuffer'] = 1
}
$wrappedCmd = $ExecutionContext.InvokeCommand.GetCommand('Microsoft.PowerShell.Management\Get-WmiObject', [System.Management.Automation.CommandTypes]::Cmdlet)

… before creating the new function based on a script block made of the updated code:

Putting the above into your profile (You can read here and here on how to work with profiles) will enable you to use PowerShell syntax with Get-WmiObject and Get-CimInstance:

$state = 'Running'
Get-WmiObject -Class Win32_Service -PowerShellFilter {Name -like '*srv*' -and State -eq $state}
Get-CimInstance -ClassName Win32_Service -PowerShellFilter {Name -like '*srv*' -and State -eq 'Running'}

How do you like PowerShell syntax to query WMI?

shareThoughts


Photo Credit: Svedek via Compfight cc

Zen Coding for the PowerShell console and ISE

8381892061_eb3babe76a_m
First of all let’s clarify what Zen Coding actually is. According to their website: Emmet (formerly known as Zen Coding) is …

… a web-developer’s toolkit that can greatly improve your HTML & CSS workflow.

But what does this have to do with PowerShell? At least I find myself quite often trying to convert PowerShell output into HTML or even using the text manipulation capabilities of PowerShell to dynamically construct some static web content. Yes, I hear you shouting already isn’t that why we have ConvertTo-HTML and Here-Strings (and here)? Granted that those can make the job already pretty easy (I would definitely recommend you to have a look into Creating HTML Reports in PowerShell if you haven’t yet), but there is still an even better way to (dynamically) generate static HTML pages from within PowerShell. Before looking into the implementation details let’s have a look at some examples on what zen coding looks like (considering we would have a PowerShell function called zenCode that expands zen code expressions):

zenCode 'html:5'
<# output:
<!DOCTYPE html []>
<html lang="en">
	<head>
		<title></title>
		<meta http-equiv="content-type" content="text/html;charset=UTF-8" />
	</head>
	<body></body>
</html>
#>
zencode 'ul.generic-list>(li.item>lorem10)*4'
<# output:
<ul class="generic-list">
	<li class="item">Sapien elit in malesuada semper mi, id sollicitudin urna fermentum.</li>
	<li class="item">Sapien elit in malesuada semper mi, id sollicitudin urna fermentum.</li>
	<li class="item">Sapien elit in malesuada semper mi, id sollicitudin urna fermentum.</li>
	<li class="item">Sapien elit in malesuada semper mi, id sollicitudin urna fermentum.</li>
</ul>
#>
zencode 'div+div>p>span+em^bq'
<# output:
<div></div>
<div>
	<p>
		<span></span>
		<em></em>
	</p>
	<blockquote></blockquote>
</div>
#>
zencode 'html>head>title+body>table>tr>th{name}+th{ID}^(tr>(td>lorem2)*3)*2'
<# output:
<html>
	<head>
		<title></title>
		<body>
			<table>
				<tr>
					<th>name</th>
					<th>ID</th>
				</tr>
				<tr>
					<td>Dolor sit.</td>
					<td>Dolor sit.</td>
					<td>Dolor sit.</td>
				</tr>
				<tr>
					<td>Dolor sit.</td>
					<td>Dolor sit.</td>
					<td>Dolor sit.</td>
				</tr>
			</table>
		</body>
	</head>
</html>
#>

You can have a look at the Zen Coding Cheat Sheet for more examples The syntax might look a bit cryptic at the first glance but once you get the hang of it it’s pretty easy. While this is all already pretty cool I wanted a way to combine this with the PowerShell pipeline in order to do things like that:

zenCode 'ul>li.item{$_}'
<# output
<ul>
	<li class="item">5</li>
	<li class="item">4</li>
	<li class="item">3</li>
	<li class="item">2</li>
	<li class="item">1</li>
</ul>
#>
gps | zenCode 'html>head>title+body>table>tr>th{name}+th{ID}^(tr>td{$_.name}+td{$_.id})'
<# output (excerpt):
<html>
	<head>
		<title></title>
		<body>
			<table>
                                <tr>
					<td>conhost</td>
					<td>14956</td>
				</tr>
				<tr>
					<td>csrss</td>
					<td>600</td>
				</tr>
#>
gps | select -first 10 | zenCode 'html>head>title+body>table[border=1]>(tr>td{$_.Name}+td{$_.ID})+td{$_.Modules | select -first 10}'
<# output (excerpt):
<head>
		<title></title>
		<body>
			<table border="1">
				<tr>
					<th>Name</th>
					<th>ID</th>
					<th>Modules</th>
				</tr>
				<tr>
					<td>iexplore</td>
					<td>8048</td>
					<td></td>
				</tr>
				<tr>
					<td>iexplore</td>
					<td>8488</td>
					<td>
						<table>
							<tr>
								<th>pstypenames</th>
								<th>BaseAddress</th>
								<th>Container</th>
								<th>EntryPointAddress</th>
								<th>FileName</th>
								<th>FileVersionInfo</th>
								<th>ModuleMemorySize</th>
								<th>ModuleName</th>
								<th>Site</th>
								<th>Company</th>
								<th>Description</th>
								<th>FileVersion</th>
								<th>Product</th>
								<th>ProductVersion</th>
								<th>Size</th>
							</tr>
							<tr>
								<td>
									<table>
										<tr>
											<td>System.Diagnostics.ProcessModule</td>
										</tr>
										<tr>
											<td>System.ComponentModel.Component</td>
										</tr>
										<tr>
											<td>System.MarshalByRefObject</td>
										</tr>
										<tr>
											<td>System.Object</td>
										</tr>
									</table>
								</td>
								<td>18612224</td>
								<td></td>
								<td>18619984</td>
								<td>C:\Program Files (x86)\Internet Explorer\IEXPLORE.EXE</td>
								<td>File:             C:\Program Files (x86)\Internet Explorer\IEXPLORE.EXE
InternalName:     iexplore
OriginalFilename: IEXPLORE.EXE.MUI
FileVersion:      11.00.9600.16428 (winblue_gdr.131013-1700)
FileDescription:  Internet Explorer
#>

The solution to make this happen relies on the zencoding implementation Mad Kristensen has developed (https://github.com/madskristensen/zencoding) which he has also part made part of his awesome Web Essentials Visual Studio extension.
I’ve made a PowerShell function (Get-ZenCode alias zenCode) and also an ISE Add-on that expand Zen Code expressions and also work with the PowerShell pipeline. Get-ZenCode also supports output to a file (outPath parameter) and showing the result in the default browser (-Show Switch). I’m posting the source code for Get-ZenCode below but you can also download the same including full help and more examples from my GitHub page (the function requires the zenCoding.dll within a resources subfolder) . The ISE Add-On (including Get-ZenCode) is part of my ISEUtils Add-On. With the Add-On the expressions can be typed into the Editor (without the preceding zenCode call) and expanded by pressing CTRL+SHIFT+J (or the respective menu entry):
ExpandZenCode

How do you like Zen Coding?

shareThoughts


Photo Credit: jurvetson via Compfight cc

Simplified Where-Object for multiple conditions on the same property for PowerShell?

3107494832_60009ec22b_m
While PowerShell version 3 already introduced a (quite controversial) simplified syntax for the Where-Object cmdlet (alias where). It still doesn’t account for a quite common error PowerShell beginners encounter when using where with multiple conditions on the same property. As an example let’s say we would like to filter the range 1-10 to get only those numbers that are between 6 and 7. I’ve seen many people (yes that includes me) attempting to do it like below since it seems a logical translation of ‘where x is greater than 5 and lower than 8’.:

1..10 | where {$_ -gt 5 -and -lt 8}
#correct version
1..10 | where {$_ -gt 5 -and $_ -lt 8}

Granted that this failing makes mathematically total sense since it should say ‘where x is greater 5 than and x is lower than 8′ . I’d wish there would be a syntax supporting something like this:

1..10 | where {$_ (-gt 5 -and -lt 8)}
#or
Get-Process | where {$_.Name (-like 'power*' -and -notlike '*ise')}

The idea is that the parentheses would indicate that the preceding variable should be considered as the (left-hand) parameter for the operator. I came up with a crude proof of concept on how this could be done:

What do you think, would you also like to see this kind of syntax for Where-Object?
shareThoughts


Photo Credit: Ana Sofia Guerreirinho via Compfight cc