Review of methods to download files using PowerShell

557483263_190baee82f_m
The goal of this post is to review and compare different methods to download files using PowerShell. As part of the review I would like to share (in addition to the inline source code you can also download a module (Get-FileMethods) that contains all functions via GitHub) some wrapper functions that follow the same pattern:

  • function name = Get-FileMETHODNAME
  • Parameters:
  • Name Description
    url URL to download from
    destinationFolder Defaults to “$env:USERPROFILE\Downloads”
    includeStats Switch parameter if specified the function will output stats for comparison purpose

  • If possible the function implements a progress bar including:
    • Remaining time (hh:mm:ss)
    • Elapsed time (hh:mm:ss)
    • Average download speed (Mb/s)/li>
    • Total download size (formatted according to size KB/MB/GB)
    • Currently downloaded size (formatted according to size KB/MB/GB)
  • After the download has finished the downloaded file is automatically unblocked
  • Provided that the includeStats is used the function outputs its name, the size of the file downloaded, and the time it took to download the file (the time is not representative and comparable as I’m currently using a quite bad 3G connection)

The following methods are compared and reviewed:

Method Function Name(s)
1. Invoke-WebRequest Get-FileInvokeWebRequest
2. Microsoft.VisualBasic.Devices.Network Get-FileVB
3. System.Net.WebClient Get-FileWCSynchronous, Get-FileWCAsynchronous
4. Background Intelligent Transfer Service Get-FileBitsTransferSynchronous, Get-FileBitsTransferAsynchronous


Get-FileSize filter
This is a little helper to convert file size units based on the number of actual bytes in a file, which I make use of within all the download functions:

 

↑Invoke-WebRequest

Invoke-WebRequest is a built-in cmdlet (since version 3) that can be used (amongst many other things) to download files.
Function:

Test:

$url = 'http://speedtest.reliableservers.com/10MBtest.bin'
Get-FileInvokeWebRequest $url -includeStats

Result:

We get a built-in progress bar showing only the currently downloaded bytes and the download of 10MB took around 1 minute:

ProgressInvokeWebRequest


resultInvokeWebRequest

↑Microsoft.VisualBasic.Devices.Network

 

The DownloadFile method of the Network class within the Microsoft.VisualBasic.Devices Namespace is an oldie but goldie from VisualBasic which can be also used (once the respective assembly is loaded) from within PowerShell:

Function:

Test:

$url = 'http://speedtest.reliableservers.com/10MBtest.bin'
Get-FileVB $url -includeStats

Result:

This method also contains a built-in progress bar which pops up in a separate window but doesn’t contain any additional information. The download of 10MB took 17 seconds:

progressBarVB

resultVB

↑System.Net.WebClient

 

The WebClient class provides two different means to download files. One works in synchronous mode (DownloadFile) and one in asynchronous mode (DownloadFileAsync). The difference between the two is, that in synchronous mode the execution of further commands halts until the download has finished while in asynchronous mode the execution continues since the download happens in the background (on another thread). Let’s first have a look at the synchronous one.

Function:

Test:

$url = 'http://speedtest.reliableservers.com/10MBtest.bin'
Get-FileWCSynchronous $url -includeStats

Result:

No progress bar (and no way to add one) and the download of 10MB took 35 seconds:
 

resultWCSync
Now the asynchronous version. The helper function is this time a bit more involved to get a proper progress bar by plugging into the respective events:

Function:

Test:

$url = 'http://speedtest.reliableservers.com/10MBtest.bin'
Get-FileWCAsynchronous $url -includeStats
 

Result:

This time we can add a custom progress bar including current and overall download speed and time. The download of 10MB took 33 seconds this time:

DownloadWCAsyncProgressbar

resultVB

GetFileWCAsync_Result

 

↑Background Intelligent Transfer Service

 

Background Intelligent Transfer Service (short BITS) is built into Windows since Windows 7. It is a file-transfer service designed to transfer files using only idle network bandwidth therefore BITS does not use all available bandwidth, so you can use it to download large files without affecting other network applications. BITS transfers are also very reliable and can continue when users change network connections or restart their computers. Therefore MS makes use of this technology to download Windows Updates (try Get-BitsTransfer -AllUsers | Select -ExpandProperty FileList to see a list of currently running BITS jobs). While BITS also provides a command-line interface…

bitsadmin /transfer ajob http://speedtest.reliableservers.com/10MBtest.bin %USERPROFILE%\Desktop\10MBtest.bin

… its use is deprecated since Windows 8. Instead the use of the BITS PowerShell module is encouraged. Similar to the WebClient class BITS also provides the user with a synchronous and asynchronous download method. Again we first take a look at the synchronous version:

Function:

Test:

$url = 'http://speedtest.reliableservers.com/10MBtest.bin'
Get-FileBitsTransferSynchronous $url -includeStats
 

Result:

This method provides a built-in progress bar but without much information. The download of 10MB took 1 minute 26 seconds which can be improved by specifying ‘Foreground’ to the ‘Priority’ paramater:

BitsSync_progressbar
BitsSync_Result

 

Again similar to WebClient the asynchronous mode provides means to add a custom progress bar. This time by using the properties of the job object:

Function:

Test:

$url = 'http://speedtest.reliableservers.com/10MBtest.bin'
Get-FileBitsTransferAsynchronous $url -includeStats
 

Result:

Also with the async mode for BITS we can add a custom progress bar including current and overall download speed and time. The download of 10MB took exactly the same as using BITS in synchronous mode (00:01:25):

BitsAsync_progressbar
BitsAsync_result

 

Conclusion

All methods have their advantages and disadvantages and the choice of course also depends on the actual task at hand (e.g. are we downloading a small file from the command prompt or are we downloading multiple files as part of a bigger scripts) and personal preferences.
The most versatile is the BITS asynchronous mode since it provides everything the other methods also provide plus the ability to run the download in the background using only idle network bandwidth and also options to pause and resume the download.

shareThoughts


Photo Credit: Nicolas Valentin via Compfight cc

PowerShell scope write-up

tree 5

I have seen and experienced myself quite some confusion and frustration around the application and understanding of scope using PowerShell. This is an attempt to write up a summary of the different aspects of how PowerShell handles scope which hopefully helps some people to understand it better.

TOC:
Definition and purpose of scope
Lexical vs. dynamic scoping
Types of scope
Scoping rules
Scope modifiers
Dot sourcing
Remote sessions and scope
Modules and scope
Recommendations

↑ Definition and purpose of scope

The scope of an object is the region of the program in which it is accessible. All objects have a scope that defines (from) where the name of the object can be used to refer to it.

Scopes help to protect objects from being unintentionally modified and also to avoid the risk of a name collision between two identically named objects.

Within PowerShell the following objects support scope:

  • Variables
  • Aliases
  • Functions
  • PowerShell Drives

↑ Lexical vs. dynamic scoping

PowerShell uses dynamic scoping as compared to most of the other languages that use lexical scoping. What is the difference? Lexical scope means that the scope is defined by the environment where an object was created. Dynamic scope means to use the environment where the object is called. An example should help to illustrate this:

$foo = 3
function Out-Foo { $foo }
function Set-Foo{
    $foo = 5
    Out-Foo
}
Out-Foo
Set-Foo
Out-Foo 

Running the example in PowerShell will output 3 5 3. Running a similar example within a programming language that is using lexical scoping will output 3 3 3. Lexical scoping treats the foo variable as having only one instance within the global scope. The dynamic scoping in PowerShell makes the modified value of foo that is set within the Set-Foo function available to the Out-Foo function.

↑ Types of scope

A new scope is created by:

  • running a script or function
  • creating a remote session
  • or by starting a new instance of PowerShell

When a new scope is created it will act as a child scope of the scope from where it was created. A child scope is created with a set of items. It includes all the items that have the AllScope option plus some variables that can be used to customize the scope, such as MaximumFunctionCount.

Global

All scopes are child scopes of the global scope (i.e. the mother of all scopes). Global is the scope that is in effect when PowerShell starts including Automatic variables, preference variables and the variables, aliases, and functions that are in the active profile. Items in global scope are visible everywhere.

Local

Only available in scripts or functions. Default scope for variables and aliases within scripts or functions.

Script

This scope is created while a script file runs. Items in script scope are visible everywhere inside the script and inside any scripts called within the parent script. Default scope for functions within scripts. Using the script scope modifier permits access to objects in the parent script scope.

Private

Items in private scope are not visible outside of the current scope. Private scope can be used to create a private version of an item with the same name in another scope. The Option parameter of the New-Variable, New-Alias can be used to set the value of the Option property to Private.

AllScope

Variables and aliases have an Option property that can take a value of AllScope. Items that have the AllScope property become part of any child scopes. Changes to the item in any scope affect all the scopes in which the variable is defined.

Numbered Scopes

A scope can be referred to by a number that describes the relative position of one scope to another. Scope 0 represents the current, or local, scope. Scope 1 indicates the immediate parent scope. Scope 2 indicates the parent of the parent scope, and so on.

↑ Scoping rules

  1. An item included in a scope is visible in the scope in which it was created and in any child scope, unless it is within private scope.
  2. The parent scope cannot access variables, which are defined in a child scope.
  3. An item created within a scope can be changed only in the scope in which it was created, unless one explicitly specifies a different scope by using a scope modifier. The only exception to this rule are aliases and variables that are created with the AllScope (link) option
  4. If you create an item in a scope, and the item shares its name with an item in a different scope, the original item might be hidden under the new item. But, it is not overridden or changed. In other words, if a child scope attempts to modify a parent scope element without using the special syntax, a new element of the same name is created within the child scope, and the child scope effectively “loses” access to the parent scope element of that name.

Some examples:

#foo has global scope if pasted to the console but script scope if run from within a script
$foo = 'I am global'
function bar { 
  #will read the global foo variable
  $foo 
  #will create a new variable $foo within the local scope since the child scope cannot modify the global foo variable
   $foo = 'I am local to the function'
   #access the local variable
   $foo
}
$foo
bar
$foo
<#output
I am global
I am global
I am local to the function
I am global
#>
$foo = 'I am global'
function bar { 
  $foo 
  #access the global/script scope by using a scope modifier
  $script:foo = 'Me too'
  #same as above using Get-Variable
  Get-Variable foo –Scope Script -ValueOnly
  $foo
}
$foo
bar
$foo
<#output
I am global
I am global
I am global
I am global
I am global
#>

↑ Scope modifiers

Scope modifiers (global, local, private, and script.) can be used for variables and functions:

function private:myFunction {}
$script:myVar

To find the items in a particular scope, use the Scope parameter of Get-Variable or Get-Alias. Use Get-Item to get the functions in a particular. Get-Item does not have a scope parameter but it returns all functions within the scope it is called from.

↑ Dot sourcing

A script or function can be added to the current scope by using dot sourcing. This will in effect make any functions, aliases, and variables that the script creates available in the current scope.

For example, to run the Sample.ps1 script from the C:\Scripts directory in the local scope, use the following command:
. C:\Scripts\sample.ps1

I would recommend using modules instead of dot sourcing in most of the cases but actually use it myself quite often in order to have a main function at the top of the script:

Function main {
	Function one
	Function two
}
Function one {
	“helper function”
}
Function two {
	“Another helper function”
}
#dot source the main function in order to make it “see” the other functions
. main

↑ Remote sessions and scope

A remote session has its own global scope, but a session is not a child scope of the session in which it was created. Child scopes within a remote session can be created by running a script.
To pass on an “external” variable into a remote session scriptBlock the $using qualifier can be utilized:

$foo = ‘test’
Invoke-Command –ComputerName bar	{
  $dir = 'c:\' + $using:foo
  Get-ChildItem	 $dir
}

↑ Modules and scope

Variables and Aliases in a module are by default not accessible outside the module (This can be controlled using Export-ModuleMember). The privacy of a module behaves like a scope, but adding a module to a session does not change the scope. And, the module does not have its own scope, although the scripts in the module do have their own scope.

Visibility

The Visibility property of a variable or alias determines whether it can be seen outside the container, such as a module, in which it was created. Visibility works for containers in the same way that the Private value of the Option property works for scopes.
The Visibility property takes can be set to either Private or Public. Items that have private visibility can be viewed and changed only in the container in which they were created. Because Visibility is designed for containers, it works differently in a scope. If you create an item that has private visibility in the global scope, you cannot view or change the item in any scope. If you try to view or change the value of a variable that has private visibility, Windows PowerShell returns an error message.
One can use the New-Variable and Set-Variable cmdlets to create a variable that has private visibility.

↑ Recommendations

1. Try to avoid using scope modifiers in order to modify items in a parent scope.
a. Better pass the item into a function through an argument by reference

$foo = 'some value'
function bar ([ref]$myFoo){
    #instead of $script:foo = 'new value'
    $myFoo.Value = 'new value'
}
$foo
bar ([ref]$foo)
$foo

<#output
some value
new value
#>

2. Use Set-Strict in order to prohibit references to uninitialized variables (see Get-Help Set-StrictMode for details)
3. Use modules instead of dot sourcing (see .dot sourcing for an exception)

shareThoughts


photo credit: Standing out from the Crowd. via photopin (license)