Extending the PowerShell range operator

tree15

The PowerShell range operator ‘..’ can be used to create lists of sequential numbers with an increment (or decrement) of one:

1..5
-1..-5

This is quite handy. Wouldn’t it be even better if it would also support stepwise lists (FIRST, SECOND..LAST similar to Haskell where the step width is determined by the first and the second list entry), day, month and letter ranges? :

monday..wednesday
march..may
#range of numbers from 2 to 15 with steps of 3 (5 - 2)
2:5..15
#range of numbers from 1 to 33 with steps of .2 (1.2 - 1)
1:1.2..33
#range of letters from a to z
a..z
#range of letters from Z to A
Z..A
#range of numbers from -2 to 1024 with steps of 6 (4 - -2)
-2:4..1kb

At least I though it would be.
While PowerShell does not support something like language extensions in order to change the behavior of the range operator directly, it is possible to get this working (admittedly it feels a bit like a hack) by overriding the ‘PreCommandLookupAction’ event. The ‘PreCommandLookupAction’ event is triggered after the current line is parsed but before PowerShell attempts to find a matching command.
I’ve encountered timed out IntelliSense/tab completion issues with the custom ‘PreCommandLookupAction’ therefore, I’ve updated the solution to use the ‘CommandNotFoundAction’ instead. The CommandNotFoundAction is triggered if PowerShell cannot find a command (who would have thought)


$ExecutionContext.SessionState.InvokeCommand.CommandNotFoundAction={
	param($CommandName,$CommandLookupEventArgs)
	#if the command consists of two dots with leading trailing words optionally containing (:,.) + leading -
	if ($CommandName -match '^(-*\w+[:.]*)+\.\.(-*\w+)+$'){
		$CommandLookupEventArgs.StopSearch = $true
		#associate new command
		$range = $CommandName.replace('get-','')
		$CommandLookupEventArgs.CommandScriptBlock={
			#no step specified
			if ($range -notlike '*:*') { 
				#check for month name or day name range
				$monthNames=(Get-Culture).DateTimeFormat.MonthNames
				$dayNames=(Get-Culture).DateTimeFormat.DayNames
				$enum=$null
				if ($monthNames -contains $range.Split("..")[0]){$enum=$monthNames}
				elseif ($dayNames -contains $range.Split("..")[0]){$enum=$dayNames}
				if ($enum){
					$start,$end=$range -split '\.{2}'
					$start=$enum.ToUpper().IndexOf($start.ToUpper()) 
					$end=$enum.ToUpper().IndexOf($end.ToUpper())
					$change=1
					if ($start -gt $end){ $change=-1 }
					while($start -ne $end){
						$enum[$start]
						$start+=$change
					}
					$enum[$end]
					return
				}
				#check for character range
				if ([char]::IsLetter($range[0])){
					[char[]][int[]]([char]$range[0]..[char]$range[-1])
					return
				}
				Invoke-Expression $range
				return 
			}
			$range = $range.Split(':')
			$step=$range[1].SubString(0,$range[1].IndexOf("..")) - $range[0]
			#use invoke-expression to support kb,mb.. and scientific notation e.g. 4e6
			[decimal]$start=Invoke-Expression $range[0]
			[decimal]$end=Invoke-Expression ($range[1].SubString($range[1].LastIndexOf("..")+2))
			$times=[Math]::Truncate(($end-$start)/$step)
			$start
			for($i=0;$i -lt $times ;$i++){
				($start+=$step)
			}
			
		}.GetNewClosure()
	}
}

Adding the code to your profile will make the “extended” range operator version available in all PowerShell sessions.
I’ve also create a separate Get-Range function which can be downloaded from my Github repository.

shareThoughts


Photo Credit: zachstern via Compfight cc

Advertisements

Work with files and folders that have paths longer than 260 characters in PowerShell using AlphaFS

tree

If you try to use one of the built-in ‘item’ cmdlets (i.e. Get-Command *item* -Module Microsoft.PowerShell.Management) on the FileSystem provider with a path that is longer than 260 characters (actually > 248 characters on a folder and > 260 on a file), you will not be able to do it. For example running the line below in order to create a folder structure on c: that is longer than 260 characters :

1..60 | foreach {$folderTree = 'c:\'} {$folderTree += "test$_\"}
$folderTree.Length
mkdir $folderTree

Will result in an error message:
screenShotFilePath
This is not only a PowerShell deficiency but stems probably from the good old Dos days and applies as well to the underlying .Net methods. You can read more about it here if you fancy. While there are ways around it (e.g. using subst or \\?\ prefix), those are not really nice solution.
This is where AlphaFS can help. According to the description on the project’s webpage

The file system support in .NET is pretty good for most uses. However there are a few shortcomings, which this library tries to alleviate. The most notable deficiency of the standard .NET System.IO is the lack of support of advanced NTFS features, most notably extended length path support (eg. file/directory paths longer than 260 characters).

it’s a perfect fit for the problem at hand. Let’s see how we can use AlphaFS through PowerShell starting off by downloading and un-compressing the release version from the project’s github repository:

cd c:\
#download and unzip
mkdir 'AlphaFS'
$destFolder = 'C:\AlphaFS'
$url = 'https://github.com/alphaleonis/AlphaFS/releases/download/v2.0.1/AlphaFS.2.0.1.zip'
Invoke-WebRequest $url -OutFile "$destFolder\AlphaFS.zip"
$shFolder = (New-Object -ComObject Shell.Application).NameSpace($destFolder)
$shZip = (New-Object -ComObject Shell.Application).NameSpace('C:\AlphaFS\AlphaFS.zip')
$shFolder.CopyHere($shZip.Items(),16)
del "$destFolder\AlphaFS.zip"

Now we can load the library, explore it and start accessing its members (we’ll use the .net v4 version):

$alphaFS = Add-Type -Path $destFolder\lib\net40\AlphaFS.dll -PassThru
$alphaFS | where IsPublic | select Name, FullName
$alphaFS | where IsPublic | select Name, FullName | where Name -like *Directory*
[Alphaleonis.Win32.Filesystem.Directory] | Get-Member -Static
[Alphaleonis.Win32.Filesystem.Directory]::CreateDirectory
#let's try it again
1..60 | foreach {$folderTree = 'c:\'} {$folderTree += "test$_\"}
[Alphaleonis.Win32.Filesystem.Directory]::CreateDirectory($folderTree)
#worked!
#delete the folder tree recursively
[Alphaleonis.Win32.Filesystem.Directory]::Delete($folderTree, $true)

The library has support for many other useful features like:

  • Creation of Hardlinks
  • Accessing hidden volumes
  • Transactional file operations (similar to the registry provider in PowerShell)
  • Copying and moving files with progress indicator
  • NTFS Alternate Data Streams
  • Accessing network resources (SMB/DFS)

The full documentation for AlphaFS can be found here.
Update: Recently I also came across the PowerShell usage wiki on the project’s GitHub page.

shareThoughts


Photo Credit: ekarbig via Compfight cc

How to convert Excel 97-03 (.xls) to new format (.xlsx) using office file converter (ofc.exe)

tree13

Usually one would use something like the code below in order to convert Excel 97-03 (.xls) files to the new format (.xlsx) via PowerShell through the Excel COM Interop interface:

function Remove-ComObject {
 # Requires -Version 2.0
 [CmdletBinding()]
 param()
 end {
         Start-Sleep -Milliseconds 500
         [Management.Automation.ScopedItemOptions]$scopedOpt = 'ReadOnly, Constant'
         Get-Variable -Scope 1 | where {
             $_.Value.PSTypenames -contains 'System.__ComObject' -and -not ($scopedOpt -band $_.Options)
         } | Remove-Variable -Scope 1 -Verbose:([Bool]$PSBoundParameters['Verbose'].IsPresent)
         [GC]::Collect()
     }
}

function Convert-XLStoXLSX($inputXLS, [switch]$keep){
	Add-Type -AssemblyName Microsoft.Office.Interop.Excel
	$xlFormat=[Microsoft.Office.Interop.Excel.XLFileFormat]::xlWorkbookDefault
	#remove old output file if existent
	$outputXLSX=[IO.Path]::ChangeExtension($inputXLS,'xlsx')
	if (test-path "$outputXLSX"){Remove-Item "$outputXLSX" -Force}
	$xl = New-Object -com "Excel.Application"
	$xl.displayalerts = $False 
        $xl.ScreenUpdating = $False
	$wb = $xl.workbooks.open($inputXLS) 
	$ws = $wb.worksheets.Item(1)
	$wb.SaveAs($outputXLSX,$xlFormat) 
	$wb.Close()
	$xl.Quit()
	Remove-ComObject
        if (!$keep){
	     Remove-Item $inputXLS -Force
        }
}

In my case since I’ve switched to Excel 2013 the above method (and actually all excel automation via COM Interop) is much slower than compared to Excel 2010. This caused me to look for alternative methods on how to automatically convert Excel files from the old format to the new format. The first promising candidate I found was excelcnv.exe, which comes as part of the office installation and can be found under ‘C:\Program Files (x86)\Microsoft Office\Office14’ (replace 14 with the appropriate version number). This is a tool to convert between old and new Excel format and vice versa. Here are some lines of PowerShell that utilize excelcnv.exe:

function ConvertTo-XLSX($xlsFile){
    $xlsxFile = [IO.PATH]::ChangeExtension($xlsFile,'xlsx')
    Start-Process -FilePath 'C:\Program Files (x86)\Microsoft Office\Office14\excelcnv.exe' -ArgumentList "-nme -oice ""$xlsxFile"" ""$xlsFile"""
}
#usage
ConvertTo-XLSX "$env:USERPROFILE\Desktop\Book1.xls"

While this converted the file successfully it also popped up a dialog after running it to convert multiple files that Excel did not launch correctly and asking whether to start in SafeMode. Not really something I was willing to accept.
The next option I came across was the Office File Converter (ofc.exe) which comes as part of Office Migration Planning Manager (OMPM). It requires you to install the Microsoft Office Compatibility Pack for Word, Excel, and PowerPoint File Formats. Ofc.exe can be used to convert any of the older office formats (i.e. ppt, doc, xls) to the new format details about the usage can be found here. The settings are controlled via an .ini file (ofc.ini). For my purpose I wanted it to convert all .xls files in a folder of my choice to .xlsx keeping the same file name and using the same folder without having to manually modify the .ini file for every conversion.
Those are the steps to setup ofc.exe to do just that through a PowerShell script:

  1. Download and install the Microsoft Office Compatibility Pack for Word, Excel, and PowerPoint File Formats
  2. Download OMPM
  3. Extract the OMPM download (comes as a self extracting archive)
  4. Copy the ‘Tools’ folder to a location of your choice (the script assumes C:\Scripts\ps1)
  5. Optionally: Rename the file ofc.ini in the Tools folder to something else (e.g. ofc_default.ini in order to keep it as a reference

Having the setup described above in place allows us to use PowerShell to create the .ini with the appropriate settings on the fly in order to instruct ofc.exe to copy the files .xlsx to the same folder along with some other default settings. The .xls files are deleted unless the -keep switch is used.:

function ConvertTo-XLSX($xlsFolder, [switch]$keep){
    $ofcFolder = 'C:\Scripts\ps1\Tools'
    $xlsxFile = [IO.PATH]::ChangeExtension($xlsFile,'xlsx')
@"
[RUN]
LogDestinationPath=$env:Temp
TimeOut=300
[ConversionOptions]
FullUpgradeOnOpen=1
MacroControl=0
CABLogs=1
[FoldersToConvert]
fldr=$xlsFolder
[ConversionInfo]
SourcePathTemplate=$('*\' * $xlsFolder.Split('\').Count)
DestinationPathTemplate=$xlsFolder
"@ | Out-File "$ofcFolder\ofc.ini"
    & "$ofcFolder\ofc.exe" "$ofcFolder\ofc.ini"
    if (!$keep){
        del "$xlsFolder\*.xls"
    }
}

The conversion through ofc.exe runs faster on my system than using the COM interop method, in addition, ofc also has the ability to convert office files within nested folder structures up to 10 level deep.

shareThoughts


Photo Credit: x1klima via Compfight cc

PowerShell Format-Table Views

trere11

Format views are defined inside the *format.ps1xml files and represent named sets of properties per type which can be used with any of the Format-* cmdlets. In this post I will focus mainly on the views for the Format-Table cmdlet:

Get-Process | select -Last 5 | Format-Table
Get-Process | select -Last 5 | Format-Table -View StartTime

screenShotFormatView1 Retrieving the format views for a particular type can be accomplished by pulling out the information from the respective XML files like so:

function Get-FormatView{
[CmdletBinding()]
param(
[Parameter(Mandatory=$true,
ValueFromPipeline=$true,
Position=0)]
$TypeName
)
$formatFiles = dir $psHome -Filter *.format.ps1xml
if ($TypeName -isnot [string]){
$TypeName = $Input[0].PSObject.TypeNames[0]
}
$formatTypes = $formatFiles |
Select-Xml //ViewSelectedBy/TypeName |
where { $_.Node.'#text' -eq $TypeName }

foreach ($ft in $formatTypes) {
$formatType = $ft.Node.SelectSingleNode('../..')
$props = $formatType.Name, ($formatType | Get-Member -MemberType Property | where Name -like '*Control').Name
$formatType | select @{n='Name';e={$props[0]}},
@{n='Type';e={'Format View'}},
@{n='Cmdlet';e={'Format-' + $props[1].Replace('Control','')}}

}
}

With this function the existing format views (for any of the Format-* cmdlets) can be easily discovered:

Get-Process | Get-FormatView | Format-Table -Auto

Creating a new Format-Table view can be done using the Add-FormatTableView function defined below:

function Add-FormatTableView {
[CmdletBinding()]
param (
[Parameter(ValueFromPipeline=$true,Mandatory=$true)]
$inputObject,

[Parameter(Position=0)]
$Label,

[Parameter(Mandatory=$true,Position=1)]
$Property,

[Parameter(Position=2)]
[int]$Width=20,

[Parameter(Position=3)]
[Management.Automation.Alignment] $Alignment = 'Undefined',

[Parameter(Position=4)]
$ViewName = 'TableView'
)
$typeNames = $input | group { $_.PSTypeNames[0] } -NoElement | select -ExpandProperty Name
$table = New-Object Management.Automation.TableControl
$row = New-Object Management.Automation.TableControlRow
$index = 0
foreach ($prop in $property){
if ($Label.Count -lt $index+1){
$currLabel = $prop
}
else{
$currLabel = $Label[$index]
}
if ($Width.Count -lt $index+1){
$currWidth = @($Width)[0]
}
else{
$currWidth = $Width[$index]
}
if ($Alignment.Count -lt $index+1){
$currAlignment = @($Alignment)[0]
}
else{
$currAlignment = $Alignment[$index]
}
$col = New-Object Management.Automation.TableControlColumn $currAlignment, (New-Object Management.Automation.DisplayEntry $prop, 'Property')
$row.Columns.Add($col)
$header = New-Object Management.Automation.TableControlColumnHeader $currLabel, $currWidth, $currAlignment
$table.Headers.Add($header)
$index++
}
$table.Rows.Add($row)
$view = New-Object System.Management.Automation.FormatViewDefinition $ViewName, $table
foreach ($typeName in $typeNames){
$typeDef = New-Object System.Management.Automation.ExtendedTypeDefinition $TypeName
$typeDef.FormatViewDefinition.Add($view)
[Runspace]::DefaultRunspace.InitialSessionState.Formats.Add($typeDef)
}
$xmlPath = Join-Path (Split-Path $profile.CurrentUserCurrentHost)  ($ViewName + '.format.ps1xml')
Get-FormatData -TypeName $TypeNames | Export-FormatData -Path $xmlPath
Update-FormatData -PrependPath $xmlPath
$xmlPath
}

Add-TableFormatView can be used to create custom table format views (you guessed it):

$fileName = Get-Process | Add-FormatTableView -Label ProcName, PagedMem, PeakWS -Property 'Name', 'PagedMemorySize', 'PeakWorkingSet' -Width 40 -Alignment Center -ViewName RAM
Get-Process | Format-Table -View RAM
#add this to the profile to have the format view available in all sessions
#Update-FormatData -PrependPath $fileName

screenShotFormatTableView3 This creates a custom format table view for the System.Diagnostics.Process type. The function determines the typename(s) based on the objects piped into it. Some details about the other parameters:

  • The Label property is optional if there is no label provided for a property the property name is used for the label.
  • The Width property is also optional and defaults to 20.
  • The Alginment can be defined per column/property or for all columns if only one alignment is specified, the default Alignment is ‘Undefined’.
  • If the ViewName is not specified it defaults to ‘TableView’. ‘TableView’ determine the default output format for the type (if no default is available) this can be handy to customize the output for custom objects.

The usage of the ‘TableView’ view name to modify custom object default output deserves another example:

#create a custom object array
$arr = @()
$obj = [PSCustomObject]@{FirstName='Jon';LastName='Doe'}
#add a custom type name to the object
$typeName = 'YouAreMyType'
$obj.PSObject.TypeNames.Insert(0,$typeName)
$arr += $obj
$obj = [PSCustomObject]@{FirstName='Pete';LastName='Smith'}
$obj.PSObject.TypeNames.Insert(0,$typeName)
$arr += $obj
$obj.PSObject.TypeNames.Insert(0,$typeName)
#add a default table format view
$arr | Add-FormatTableView -Label 'First Name', 'Last Name' -Property FirstName, LastName -Width 30 -Alignment Center
$arr 
#when using the object further down in the pipeline the property name must be used
$arr  | where "LastName" -eq 'Doe'

screenShotFormatTableView4
This can come in handy for custom object output since the output is still an object and not a format it can be used further down the pipeline (using the property name and not the label).

Get-FromatView and Add-FormatTableView can be downloaded from my GitHub repository.

shareThoughts


Photo Credit: jDevaun.Photography via Compfight cc

PowerShell Property Sets

tree11

What are PowerShell Property sets and how can they be used?

Property sets are named groups of properties for certain types that can be used through Select-Object. The property sets are defined in the *types*.ps1xml files that can be found in the $pshome directory. Most of the types have default property sets defined as DefaultDisplayPropertertySet within the PSStandardMembers MemberSet. Those determine the list of properties shown when an object of that type is output. An example:

$process = (Get-Process)[0]
$process.PSStandardMembers.DefaultDisplayPropertySet.ReferencedPropertyNames
$process | Format-List

screenShotPropertySet1

The list of default properties for a type can be modified by using Update-TypeData:

$process = (Get-Process)[0]
$typeName = $process.PSTypeNames[0]
Update-TypeData -TypeName $typeName -DefaultDisplayPropertySet Handles, Name, ID -Force
$process | Format-List

screenShotPropertySet

Some types have additional property sets defined that can be used with Select-Object:

Get-Process | select PSConfiguration -Last 5 | Format-Table -Auto

screenShotPropertySet2

The properties displayed when using the ‘PSConfiguration’ property set are defined in the types.ps1xml file but can be also accessed through Get-Member:

Get-Process | Get-Member -MemberType PropertySet

Property sets can also be added dynamically to any object instance by using Add-Member:

$ps = Get-Process
$ps | Add-Member -MemberType PropertySet 'Test' ('Path', 'Product', 'Company', 'StartTime')
$ps | select Test

Adding a property set to a type is unfortunately not as easy as just calling Update-TypeData, since the MemberType PropertySet is not supported:

$typeName = (Get-Process).GetType().FullName
Update-TypeData -TypeName $typeName -MemberType PropertySet -MemberName 'Test' -Value $props
Get-Process | select Test -First 5

screenShotPropertySet6

That does of course not mean it’s impossible, but it involves creating the appropriate type.ps1xml file ‘manually’:

function Add-PropertySet{
     <#    
    .SYNOPSIS
        Function to create property sets
    .DESCRIPTION
        Property sets are named groups of properties for certain types that can be used through Select-Object.
        The function adds a new property set to a type via a types.ps1xml file.
	.PARAM inputObject
		The object or collection of objects where the property set is added
	.PARAM propertySetName
		The name of the porperty set
	.EXAMPLE
		$fileName = dir | Add-PropertySet Testing ('Name', 'LastWriteTime', 'CreationTime')
        dir | Get-Member -MemberType PropertySet
        dir | select Testing
        #add this to the profile to have the property set available in all sessions
        Update-TypeData -PrependPath $fileName
	.EXAMPLE
		Get-Process | Add-PropertySet RAM ('Name', 'PagedMemorySize', 'PeakWorkingSet')
        Get-Process | select RAM
    #>
    [CmdletBinding()]
    param(
        [Parameter(ValueFromPipeline=$true,Mandatory=$true)]
        $inputObject,
    
        [Parameter(Mandatory=$true,Position=0)]
        $propertySetName,

        [Parameter(Mandatory=$true,Position=1)]
        $properties
    )
    $propertySetXML = "<Types>`n"
    $groupTypes = $input | group { $_.PSTypeNames[0] } -AsHashTable
    foreach ($entry in $groupTypes.GetEnumerator()) {
        $typeName = $entry.Key
        $propertySetXML += @"
    <Type>
        <Name>$typeName</Name>
        <Members>
            <PropertySet>
                <Name>$propertySetName</Name>
                <ReferencedProperties>
                    $($properties | foreach { 
                        "<Name>$_</Name>"    
                    })
                </ReferencedProperties>
            </PropertySet>
        </Members>
    </Type>
"@
    } 
    $propertySetXML += "`n</Types>"
    $xmlPath = Join-Path (Split-Path $profile.CurrentUserCurrentHost)  ($propertySetName + '.types.ps1xml')
    ([xml]$propertySetXML).Save($xmlPath)    
    Update-TypeData -PrependPath $xmlPath
    $xmlPath
}

With Add-PropertySet property sets can also be easily added to types:

Get-Process | Add-PropertySet RAM ('Name', 'PagedMemorySize', 'PeakWorkingSet')
Get-Process | select RAM -First 3 | Format-Table -Auto
Get-Process | Get-Member -MemberType PropertySet

screenShotPropertySet7

The property set is by default only available for the current PowerShell session. In order to have the property set available for every session the Update-TypeData call with the return value from the Add-PropertySet function (the types.ps1xml are created within the same directory as the profile and prefixed with the propertySetName) needs to be added to the profile:

#add this to your profile
Update-TypeData -PrependPath PATHRETURNEDFROMADD-PROPERTYSET

Add-PropertySet including full help can be download from my GitHub repository.

shareThoughts


photo credit: Happy Lazy Saturday via photopin (license)

Improve PowerShell commandline navigation

tree10

This post is about improving PowerShell in order to ease navigation on the commandline. There are some pretty cool solutions out there which I have added to my profile to be able to move around quicker:

Another one, that I came up with is in order to move down directory paths quickly similar to ‘cd ..’:

$ExecutionContext.SessionState.InvokeCommand.CommandNotFoundAction={
      param($CommandName,$CommandLookupEventArgs)
      #if the command is only dots
      if ($CommandName -match '^\.+$'){
            $CommandLookupEventArgs.CommandScriptBlock={
                  for ($counter=0;$counter -lt $CommandName.Length;$counter++){
                        Set-Location ..
                  }
            }.GetNewClosure()
      }
}

Adding those lines to your profile (I believe v3 is required for this to work) will enable you to move down directory paths quickly by using just dots where the number of dots determines how many levels you move down the folder hierarchy. If you are for example within “PS>c:\test1\test2\test3\test4” the command “…” will move you down 3 levels to “PS>c:\test1”.

shareThoughts


photo credit: dusk tusks via photopin (license)