Accessing properties of Azure blobs with PowerShell

In the previous articles in this series talking about Azure blob storage and PowerShell, I’ve covered uploading and downloading blobs, copying blobs, and deleting blobs. Now let’s talk about how to read the properties of a blob, and how to update them.

I’m not going to discuss all of the properties of a blob, or the properties of a blob’s Properties. (Yes, a blob has a property called Properties and that property has properties of its own). (Yes, it’s a little confusing.) If you want to know all of the properties (and Properties properties) of a blob, please check out this article that I wrote for RedGate, and then come back here to see how to read and modify them using PowerShell.

Setup

As we’ve seen before, first you need a storage account context. This is the same as in the previous posts. To define a storage account context, you need the storage account name and key. Set the variables to storage account name and key. You can get these from the Storage Account in the Azure Portal.

$StorageAccountName = "yourStorageAccountName"
$StorageAccountKey = "yourStorageAccountKey"

Now let’s set up the storage account context.

$ctx = New-AzureStorageContext -StorageAccountName $StorageAccountName `
         -StorageAccountKey $StorageAccountKey

Now we need to set the name of the container that holds the blob that we want to use. Set this to your container name, and specify the name of the blob. I’m going to use the same values I was using in the previous post.

$ContainerName = “images”
$BlobName = “gizmodo_groundhog_texting.jpg”

Get a reference and check the properties

To examine or modify a blob’s properties, first you have to get a reference to the blob through the Azure Blob Service. You can use the cmdlet Get-AzureStorageBlob to access the blob – this returns an object of type ICloudBlob, which is generic. You then have to convert this to a CloudBlockBlob object to access the properties and Properties of the actual blob.

$Blob = Get-AzureStorageBlob -Context $ctx -Container $ContainerName -Blob $BlobName

$CloudBlockBlob = [Microsoft.WindowsAzure.Storage.Blob.CloudBlockBlob] $Blob.ICloudBlob

Now that we have a reference to the CloudBlockBlob object that represents the actual blob, we can look at the properties using the IntelliSense in PowerShell. In the command window, type in $CloudBlockBlob and a period (.), and it will show all the properties you can access. You can type in the variable with the property to see the value ($CloudBlockBlob.BlobType), or use the Write-Host command as displayed here.

Write-Host "blob type = " $CloudBlockBlob.BlobType
Write-Host "blob name = " $CloudBlockBLob.Name
Write-Host "blob uri = " $CloudBlockBlob.Uri

To see the Properties properties, fetch the attributes of the blob first. This will populate these properties.

$CloudBlockBlob.FetchAttributes()
Write-Host "content type = " $CloudBlockBlob.Properties.ContentType
Write-Host "size = " $CloudBlockBlob.Properties.Length

The files got uploaded with a default content type of “application/octet-stream”. Let’s change the content type to “image/jpg”. To do this, we can just assign a new value to the Content Type property, then call SetProperties to save the change. In the following example, I’m changing the content type, then using the Write-Host statements from above to see the new value.

$ContentType = "image/jpg"
$CloudBlockBlob.Properties.ContentType = $ContentType
$CloudBlockBlob.SetProperties()

Another important property is the metadata. This is a set of key/value pairs that you can attach to the blob. Here’s an example of how to set the metadata. This assigns both the key and the value for each pair. This then saves the changes by calling SetMetadata and then displays the results.

$CloudBlockBlob.Metadata["filename"] = "original file name"
#author is the key, “RobinS” is the value
$CloudBlockBlob.Metadata["author"] = "RobinS"
$CloudBlockBlob.SetMetadata()
$CloudBlockBlob.Metadata

To clear the metadata, you can call the Clear method on the Metadata, then save the changes.

#blank out the metadata
$CloudBlockBlob.Metadata.Clear()
#save the changes
$CloudBlockBlob.SetMetadata()
#view the saved data
$CloudBlockBlob.Metadata

Summary

In this post, I showed you how to access the properties of a blob in Azure Storage, and how to set the properties and the metadata. In the next post, I’ll show how to take a snapshot of a blob and then how to view the available snapshots for a blob.

Deleting and copying files in Azure Blob Storage with PowerShell

In my previous post, I showed you how to upload and download files to and from Azure blob storage using the Azure PowerShell cmdlets. In this post, I’ll show you how to delete blobs, copy blobs, and start a long-term asynchronous copy of a large blob and then check the operation’s status until it’s finished. I’m going to continue with the container and blobs that I used in the previous post.

Setup

First, you need a storage account context. This is the same as in the previous post. To define a storage account context, you need the storage account name and key. Set the variables to storage account name and key. You can get these from the Storage Account in the Azure Portal.

$StorageAccountName = "yourStorageAccountName"
$StorageAccountKey = "yourStorageAccountKey"

Now let’s set up the storage account context.

$ctx = New-AzureStorageContext -StorageAccountName $StorageAccountName `
         -StorageAccountKey $StorageAccountKey

Now we need to set the name of the container that holds the blob(s ) that we want to delete. Set this to your container name.

$ContainerName = “images”

Delete a blob

To delete a blob, you need the storage context, container name, and blob name. So let’s set the blob name, and then use the PowerShell cmdlet to delete a blob.

$BlobName = “GuyEyeingOreos.png”

Remove-AzureStorageBlob -Blob $BlobName -Container $ContainerName -Context $ctx

Now get the list of blobs and you’ll see that the one you deleted is no longer there.

Get-AzureStorageBlob -Container $ContainerName -Context $ctx | Select Name

Copy a blob

Now let’s copy a blob to another blob with a different name. We’ll use the Start-AzureStorageBlobCopy cmdlet to do this. The copy is done asynchronously. If it’s a huge file, we can check the progress of the copy. I’ll show you how to do that after this. Here, I’m going to copy one of my blobs to another blob, then get a list so I can see if it worked.

Set the blob name of the source blob, then set the blob name of the target blob. Then do the copy.

$BlobName = "gizmodo_groundhog_texting.jpg"
$NewBlobName = "CopyOf_" + $BlobName

Start-AzureStorageBlobCopy -SrcBlob $BlobName -SrcContainer $ContainerName `
        -DestContainer $ContainerName -DestBlob $newBlobName -Context $ctx

Get-AzureStorageBlob -Container $ContainerName -Context $ctx | Select Name

You can see the new blob in the listing now.

Copy a blob asynchronously and check its progress

What if you have a really large blob, like a VHD file, and you want to copy it from one storage account to another? Maybe you even want to copy it to a storage account in another region, so you can recreate the VM in that region?

You may have noticed that there’s not a CopyBlob method per se – the cmdlet is Start-AzureCopyBlob. This is because the blob copy runs asynchronously. For small blobs, you don’t generally need to check the status of a copy operations – it will probably finish before you have a chance to check the status, depending on the size. For large blobs, though, like a VHD file that is 127 GB, it will take a while to copy it. So you want to be able to kick off the copy and then check the status later. Let’s see how to do that.

Setup the variables for the operation

I have a VHD file in the container “vhds” in one storage account and I’m going to copy it to a different storage account. Let’s start by setting the variables for the container name and blob name for the source. We will use the same blob name for the source and the destination.

$sourceContainer = "vhds"
$BlobName = "testasynccopysvc-testasynccopy-2015-06-16.vhd "

Next I’ll set up the name and key for the destination storage account. Fill these in with your own information.

$destStorageAccountName = "yourStorageAccountName"
$destStorageAccountKey = "yourStorageAccountKey"

We need a storage account context for the destination storage account.

$destctx = New-AzureStorageContext –StorageAccountName $destStorageAccountName `
        StorageAccountKey $destStorageAccountKey

Let’s set up the destination container. I’ll set the name and then create the container.

$destContainer = "copiedvhds"
New-AzureStorageContainer -name $destContainer -Context $destctx

Asynchronous copy and status check

Now let’s kick off the copy operation, then check the status by calling Get-AzureStorageBlobCopyState. You have to assign this to a variable representing the destination blob so you can check the status of the resulting blob. Notice that this copy command is a little bit different than the previous one – this one specifically provides the destination context because it’s different from the source context.

$BlobResult = Start-AzureStorageBlobCopy -SrcBlob $BlobName -SrcContainer $sourceContainer `
        -DestContainer $destContainer -DestBlob $BlobName -Context $ctx `
        -DestContext $destctx

$status = $BlobResult | Get-AzureStorageBlobCopyState
# show the status
$status

This shows that it is just starting the copy operation – the bytes copied is 0. Wait a few seconds and check the status again…

Now it shows the number of  bytes copied is greater than 0.

This PowerShell code will loop continuously until the status is not “Pending”. It will retrieve the status, display it, sleep 10 seconds, then loop around and check the status again. When the status no longer equals “Pending”, it will exit the loop.

While ($status.Status -eq "Pending") {
    $status = $BlobResult | Get-AzureStorageBlobCopyState
    $status
    Start-Sleep 10
}

When it’s completed, the final status will look like the following:

You can see that it has copied all the bytes, and the copy operation was successful.

Summary

In this post, we went over how to delete a blob and how to copy a blob to another blob. We also learned how to do an asynchronous copy of a large blob from one storage account to another, and check the status repeatedly until it was finished. In my next post, we will learn how to take a snapshot of a blob using PowerShell.

Uploading and downloading files to Azure Blob Storage with PowerShell

In my previous post, I showed you how to set up PowerShell so you can use it to perform commands against blob storage. In this post, I’ll show you how to create a container in blob storage and then upload files from the local machine to blob storage, and how to download files from blob storage to the local machine.

Setup for transferring files

When using hardcoded values (such as storage account name), I tend to put them in variables, and then use the variable in the script. This makes it easier to change the hardcoded values, especially if they are used in multiple places, which makes your script more flexible for different storage accounts, containers, etc.

First, set up a folder with some pictures or files in it that you can upload. Then set up a folder to which you can download blobs. I’m going to use “D:\_Temp\_AzureFilesDownloaded” for my target directory for downloads, and “D:\_Temp\_AzureFilesToUpload” to hold pictures that I can upload.

Now run PowerShell ISE as an administrator like you did in the previous post. If PowerShell is on your task bar, you can right-click on it and select “Run ISE as administrator.” We’re going to write a script in the script window and run the commands as we need to. When we’re done, you’ll have a script you can use to upload and download files. I tend to do this because it’s good to create your own library of samples that you can reuse.

Next, let’s set up variable names for the storage account name and key. I’m going to use $StorageAccountName and $StorageAccountKey. (If you didn’t know it, the $ prefix indicates a variable name). Fill your storage account name and key in. You can get these from the Storage Account in the Azure Portal.

$StorageAccountName = "yourStorageAccountName"
$StorageAccountKey = "yourStorageAccountKey"

When you access a storage account from PowerShell, you need to use what’s called a context. You create this using the storage account name and key, and pass it in with each PowerShell command so it knows which account to use when running the command. Here’s the command:

$ctx = New-AzureStorageContext -StorageAccountName $StorageAccountName `
         -StorageAccountKey $StorageAccountKey

Note the backwards tick mark (`) after $StorageAccountName. This is a continuation character in PowerShell.

When accessing blobs, you have to specify which container the blob is in. So let’s specify a variable for container name. This can be a container that already exists or a new container (I’ll show you how to create a new container in a minute.)

$ContainerName = "images"

This is how you create a new container with public access to the blobs. If you try this and the container already exists, it will give you an error.

New-AzureStorageContainer -Name $ContainerName -Context $ctx -Permission Blob

So now we have the storage context and the container name set up. Set a variable for the local file directory:

$localFileDirectory = "E:\_Temp\_AzureFilesToUpload\"

Upload blobs from local folder

One of the files in my folder is called “SnowyCabin.jpg”. I’m going to set a variable for $BlobName – the actual name of the blob – and then append it with the $localFileDirectory to create the path to the local file. Then we’ll use the Set-AzureStorageBlobContent cmdlet to upload the file. To use that, you specify the path to the local file, the name of the container, the name of the blob, and the storage context.

$BlobName = "SnowyCabin.jpg"
$localFile = $localFileDirectory + $BlobName
Set-AzureStorageBlobContent -File $localFile -Container $ContainerName `
        -Blob $BlobName -Context $ctx

So now I’ve uploaded that one file. I can check blob storage and I’ll see the file there in the container called “images”. (To check blob storage, you can use one of the Azure Portals, Visual Studio Azure Explorer, or a storage explorer product like the Azure Management Studio from Cerebrata. I can also query the storage account for the blobs in that container using the cmdlet Get-AzureStorageBlob.

Get-AzureStorageBlob -Container $ContainerName -Context $ctx

Let’s upload some more blobs and do another list. I’m going to upload 3 more pictures and then get a list of blobs in the container. This is the same PowerShell as above, repeated 3 times with different files. I’ve added an argument “ | Select Name” to the end of Get-AzureStorageBlob. That first character is a pipe symbol; this takes the output from Get-AzureStorageBlob and selects just the name and displays it.

$BlobName = "BluebellsAndBeechTrees.jpg"
$localFile = $localFileDirectory + $BlobName
Set-AzureStorageBlobContent -File $localFile -Container $ContainerName `
        -Blob $BlobName -Context $ctx

$BlobName = "gizmodo_groundhog_texting.jpg"
$localFile = $localFileDirectory + $BlobName
Set-AzureStorageBlobContent -File $localFile -Container $ContainerName `
        -Blob $BlobName -Context $ctx

$BlobName = "GuyEyeingOreos.png"
$localFile = $localFileDirectory + $BlobName
Set-AzureStorageBlobContent -File $localFile -Container $ContainerName `
-Blob $BlobName -Context $ctx

Get-AzureStorageBlob -Container $ContainerName -Context $ctx | Select Name

Download blobs to local disk

First, we need to set a variable for the target directory on the local machine.

$localTargetDirectory = "D:\_Temp\_AzureFilesDownloaded"

To download a blob, use the cmdlet Get-AzureStorageBlobContent. I’m going to download 3 of the files I uploaded.

$BlobName = "BluebellsAndBeechTrees.jpg"
Get-AzureStorageBlobContent -Blob $BlobName -Container $ContainerName `
        -Destination $localTargetDirectory -Context $ctx

$BlobName = "gizmodo_groundhog_texting.jpg"
Get-AzureStorageBlobContent -Blob $BlobName -Container $ContainerName `
        -Destination $localTargetDirectory -Context $ctx

$BlobName = "GuyEyeingOreos.png"
Get-AzureStorageBlobContent -Blob $BlobName -Container $ContainerName `
        -Destination $localTargetDirectory -Context $ctx

Now if I look in that local directory, I see those files.

Summary

In this post, I showed you how to upload files to Azure blob storage from the local disk, and how to download files from Azure blob storage to the local disk. In my next post, I’ll show you how to delete a blob and how to copy a blob.

Azure Storage and PowerShell 101

I’ve been working for a while on a deep-dive course about Azure Storage for Opsgility. This will include video training, and Opsgility will offer live training classes using the material I’ve written. One of the things that I’ve found really difficult is doing anything remotely complicated with blob storage and PowerShell. For example, how do you take snapshots of a blob in PowerShell? You go ahead and bing that and let me know what you find. (whistling) Yeah, I didn’t find anything either. So I did a bit of trial and error, and thought I would blog some of the results in a series of posts, of which this is the first.

In this article, we’ll see how to install the PowerShell cmdlets and set up a default Azure subscription and Azure storage account. You must already have an Azure account. If you don’t have one, you can sign up for a free trial at http://azure.microsoft.com.

Let’s begin by installing the PowerShell cmdlets.

1. Open your browser and navigate to http://azure.microsoft.com.
2. Click Downloads and then click the Install link under Windows PowerShell.
3. Click Run when prompted and follow the instructions to install the cmdlets.

You can also install the cmdlets using the Web Platform Installer, or by downloading the msi from GitHub.

There are two ways to run PowerShell. There’s the command-line, and then there’s something called the ISE which is basically a window that allows you to write script and execute it one line at a time. You can also select a range of lines to run, or even run the entire script. I use this a lot to test my scripts.

I like to run PowerShell ISE from my task bar. If you want to try that, here’s how to set that up in Windows 8.1 or Windows 10.

1. Run Microsoft Azure PowerShell. To do this in Windows 8, you can go to the Start screen and type “powershell” and select it. In Windows 10, you can go to the Start menu and look for it. This runs the command-line version.
2. After it’s running, right-click on the icon in the task bar and select the option to pin this program to the taskbar.
3. Close the PowerShell command line window.
4. Right-click on the icon in the task bar, and you’ll see a bunch of options. Choose “Run ISE as Administrator”.

This should look like the following – you should have a white window on the top for the script and a blue window at the bottom. The window at the bottom will show the output from any parts of the script that you run. You can also type commands into the bottom window directly if you want to.

Before you can run anything, you need to set up the Azure account you’re going to use. To do this, you’re going to use the command “Add-AzureAccount”. You can type it into the script window and highlight it, then click icon for “Run Selection” (F8 also works). This is the icon with the red square around it I this image:

If you click the green play button, it runs the whole script.  We’re going to use this a lot in future articles.

For now, I’m going to put my command in the command window and run it – just type it in and press Enter. You will be prompted to enter your Azure subscription account information. After authentication, it will show the properties of the Azure account in the output window. My subscription name is “Azure Pass”. Write down your subscription name and set it as the current subscription and the default subscription by executing this command:

> Select-AzureSubscription –SubscriptionName “Azure Pass”

If you want to see a list of the subscriptions you have set up, you can ask to see the list and just show just the names of each subscription like this:

> Get-AzureSubscription | Select SubscriptionName

You can just see the subscription set as the current by using this command:

> Get-AzureSubscription –Current

If you don’t have a storage account, you’ll need to add one. You can go to the portal to do this, or you can do it right here in PowerShell. Here’s the command:

> New-AzureStorageAccount –StorageAccountName “nameofnewaccount” –Location “West US”

If the storage account already exists, you will get an error message. If the account was added successfully, the new storage account will be displayed in the output window.

Now set that storage account as the default account for this subscription to use. These defaults will be used the next time you run PowerShell. My new storage account is called “robinsnewstorage”.

> Set-AzureSubscription –SubscriptionName “Azure Pass”  -CurrentStorageAccountName “robinsnewstorage”

Now view the current Azure Subscription again and it will show the current storage account name.

In the next few articles, I’ll show you how to access the storage account and perform some actions against blob storage such as uploading and downloading files, deleting blobs, and copying blobs (including async copy). I’ll also show you how to access the properties of a blob. Then we’ll move onto the hard topics of snapshots, leases, and shared access signatures.