Azure Blob Storage, Click Once deployment, and recursive file uploads

In this blog post, I am going to show you how to upload an folder and all of its contents from your local computer to Azure blob storage, including subdirectories, retaining the directory structure. This can have multiple uses, but I want to call out one use that people still using Click Once deployment might appreciate.

I used to be the (only) Click Once MVP, and still blog about it once in a while. Click Once is a Microsoft technology that allows you to host the deployment of your client application, console application, or VSTO add-in on a file share or web site. When updates are published, the user picks them up automatically. This can be a very handy for those people still dealing with these technologies, especially since Microsoft removed the Setup & Deployment package feature from Visual Studio after VS2010 and replaced it with a lame version of InstallShield (example of lameness: it wouldn’t let you deploy 64-bit applications). But I digress.

I wrote a blog article showing how you can host your Click Once deployment in Azure blob storage very inexpensively. (It’s even cheaper now.) The problem is you have to get your deployment up to Blob Storage, and for that, you need to write something to upload it, use something like Cerebrata’s Azure Management Studio, or talk the Visual Studio team and ClickOnce support into adding an option to the Publish page for you. I tried the latter — what a sales job I did! “Look! You can work Azure into Click Once and get a bunch of new Azure customers!” “Oh, that’s a great idea. But we have higher priorities.” (At least I tried.)

Having been unsuccessful with my rah! rah! campaign, I thought it would be useful if I just provided you with the code to upload a folder and everything in it. You can create a new Windows Forms app (or WPF or Console, whatever makes you happy) and ask the user for two pieces of information:

  • Local folder name where the deployment is published. (For those of you who don’t care about ClickOnce, this is the local folder that you want uploaded.)
  • The name of the Blob Container you want the files uploaded to.

Outside of a Click Once deployment, there are all kinds of uses for this code. You can store some of your files in Blob Storage as a backup, and use this code to update the files periodically. Of course, if you have an excessive number of files, you are going to want to run the code in a background worker and have it send progress back to the UI and tell it what’s going on.

Show me the code

I’m going to assume you understand recursion. If not, check out the very instructive wikipedia article. Or put the code in and just step through it. I think recursion is really cool; I once wrote a program in COBOL that would simulate recursion that went up to 10 levels deep. (For you youngsters, COBOL is not a recursive programming language.)

In your main method, you need to add all of the following code (up until the next section).

First, you need to set up your connection to the Azure Storage Account and to the blob container that you want to upload your files to. Assuming you have the connection string to your storage account, here’s what you need to do.

First you’re going to get an instance of the CloudStorageAccount you’re going to use. Next, you get an reference to the CloudBlobClient for that storage account. This is what you use to access the actual blob storage. And lastly, you will get a reference to the container itself.

The next thing I always do is call CreateIfNotExists() on the container. This does nothing if the container exists, but it does save you the trouble of creating the container out in Blob Storage in whatever account you’re using if you haven’t already created it. Or if you have forgotten to create it. If you add this, it makes sure that the container exists and the rest of your code will run.

//get a reference to the container where you want to put the files, 
//  create the container if it doesn't exist
CloudStorageAccount cloudStorageAccount = CloudStorageAccount.Parse(connectionString);
CloudBlobClient cloudBlobClient = cloudStorageAccount.CreateCloudBlobClient();
CloudBlobContainer cloudBlobCOntainer = cloudBlobClient.GetContainerReference(containerName);
cloudBlobContainer.CreateIfNotExists();

I also set the permissions on the container. If this code actually creates the container, the default access is private, and nobody will be able to get to the blobs without either using a security token with the URL or having the storage account credentials.

//set access level to "blob", which means user can access the blob 
//  but not look through the whole container
//  this means the user must have a URL to the blob to access it
BlobContainerPermissions permissions = new BlobContainerPermissions();
permissions.PublicAccess = BlobContainerPublicAccessType.Blob;
cloudBlobContainer.SetPermissions(permissions);

So now we have our container reference (cloudBlobContainer) set up and the container is ready for use.

Next we need to get a list of files to upload, and then we need to upload them. I’ve factored this into multiple methods. Here are the top commands:

List<String> listOfiles = GetListOfFilesToUpload(folderPath);
string status = UploadFiles(listOfiles, folderPath);

After this finishes, all of your files are uploaded. Let’s look at the methods called.

GetListOfFilesToUpload(string folderPath)

This is the hard part – getting the list of files. This method calls the recursive routine. It starts with instantiating the list of files that will be the final result. Then it retrieves the list of files in the requested directory and adds them to the list. This is “the root”. Any files or directories in this root will be placed in the requested container in blob storage. [folderName] is the path to the local directory you want to uploaded.

//this is going to end up having a list of the files to be uploaded
//  with the file names being in the format needed for blob storage
List<string> listOfiles = new List<string>();
            
//get the list of files in the top directory and add them to our overall list
//these will have no path in blob storage bec they go in the root, so they will be like "mypicture.jpg";
string[] baseFiles = Directory.GetFiles(folderName);
for (int i = 0; i < baseFiles.Length; i++)
{
    listOfiles.Add(Path.GetFileName(baseFiles[i]));
}

Files will be placed in the same relative path in blob storage as they are on the local computer. For example, if D:\zAzureFiles\Images is our “root” upload directory, and there is a file with the full path of “D:\zAzureFiles\Images\Animals\Wolverine.jpg”, the path to the blob will be “Animals/Wolverine.jpg”.

Next we need to get the directories under our “root” upload directory, and process each one of them. For each directory, we will call GetFolderContents to get the files and folders in each directory. GetFolderContents is our recursive routine. So here is the rest of GetListOfFilesToUpload:

//call GetFolderContents (this routine) for each folder to retrieve everything under the top directory
string[] directories = Directory.GetDirectories(folderName);
for (int i = 0; i < directories.Length; i++)
{
    // an example of a directory : D:\zAzureFiles\Images\NatGeo; (root is D:\zAzureFiles\Images
    // topDir gives you the just the directory name, which is NatGeo in this example
    string topDir = GetTopDirectory(directories[i]);
    //GetFolderContents is recursive
    List<String> oneList = GetFolderContents(directories[i], topDir);
    //you have a list of files with blob storage paths for everything under topDir (incl subfolders)
    //  (like topDir/nextfolder/nextfolder/filename.whatever)
    //add the list of files under this folder to the list going in this iteration
    //eventually when it works its way back up to the top, 
    //  it will end up with a complete list of files
    //  under the top folder, with relative paths
    foreach (string fileName in oneList)
    {
        listOfiles.Add(fileName);
    }
}

And finally, return the list of files.

return listOfiles;

GetTopDirectory(string fullPath)

This is a helper method that just pulls off the last directory. For example, it reduces “D:\zAzureFiles\Images\Animals to “Animals”. This is used to pass the folder name to the next recursion.

private string GetTopDirectory(string fullPath)
{
    int lastSlash = fullPath.LastIndexOf(@"\");
    string topDir = fullPath.Substring(lastSlash + 1, fullPath.Length - lastSlash - 1);
    return topDir;
}

GetFolderContents(string folderName, string blobFolder)

This is the recursive routine. It returns List<string>, which is a list of all the files in and below the folderName passed in, which is the full path to the local directory being processed, like D:\zAzureFules\Images\Animals\.

This is similar to GetListOfFilesToUpload; it gets a list of files in the folder passed in and adds them to the return object with the appropriate blob storage path. Then it gets a list of subfolders to the folder passed in, and calls GetFolderContents for each one, adding the items returned from the recursion in to the return object before returning up a level of recursion.

This sets the file names to what they will be in blob storage, i.e. the relative path to the root. So a file on the local computer called D:\zAzureFiles\Images\Animals\Tiger.jpg would have a blob storage path of Animals/Tiger.jpg.

returnList is the List<String> returned to the caller.

List<String> returnList = new List<String>();
            
//process the files in folderName, set the blob path
string[] fileLst = Directory.GetFiles(folderName);
for (int i = 0; i < fileLst.Length; i++)
{
    string fileToAdd = string.Empty;
    if (blobFolder.Length > 0)
    {
        fileToAdd = blobFolder + @"\" + Path.GetFileName(fileLst[i]);
    }
    else
    {
        fileToAdd = Path.GetFileName(fileLst[i]);
    }
    returnList.Add(fileToAdd);
}

//for each subdirectory in folderName, call this routine to get the files under each folder
//  and then get the files under each folder, etc., until you get to the bottom of the tree(s) 
//  and have a complete list of files with paths
string[] directoryLst = Directory.GetDirectories(folderName);
for (int i = 0; i < directoryLst.Length; i++)
{
    List<String> oneLevelDownList = new List<string>();
    string topDir = blobFolder + @"\" + GetTopDirectory(directoryLst[i]);
    oneLevelDownList = GetFolderContents(directoryLst[i], topDir);
    foreach (string oneItem in oneLevelDownList)
    {
        returnList.Add(oneItem);
    }
}
return returnList;

UploadFiles(List<string> listOfFiles, string folderPath)

This is the method that actually uploads the files to Blob Storage. This assumes you have a reference to the cloudBlobContainer instance that we created at the top.

[listOfFiles] contains the files with relative paths to the root. For example “Animals/Giraffe.jpg”. [folderPath] is the folder on the local drive that is being uploaded. In our examples, this is D:\zAzureFiles\Images. Combining these gives us the path to the file on the local drive. All we have to do is set the reference to the location of the file in Blob Storage, and upload the file. Note – the FileMode.Open refers to the file on the local disk, not to the mode of the file in Blob Storage.

internal string UploadFiles(List<string> listOfFiles, string folderPath)
{
    string status = string.Empty;
    //now, listOfiles has the list of files you want to upload
    foreach (string oneFile in listOfFiles)
    {
        CloudBlockBlob blob = cloudBlobContainer.GetBlockBlobReference(oneFile);
        string localFileName = Path.Combine(folderPath, oneFile);
        blob.UploadFromFile(localFileName, FileMode.Open);
    }
    status = "Files uploaded.";
    return status;
}

Summary

So you have the following:

  • The code for the calling routine that sets the reference to the cloudBlobContainer and makes sure the container exists. This calls GetsListOfFilesToUpload and UploadFiles to, well, get the list of files to upload and then upload them.
  • GetListOfFilesToUpload calls GetFolderContents (which is recursive), and ultimately returns a list of the files as they will be named in Blob Storage.
  • GetFolderContents – the recursive routine that gets the list of files in the specified directory, and then calls itself with each directory found to get the files in the directory.
  • UploadFiles is called with the list of files to upload; it uploads them to the specified container.

If the files already exist in Blob Storage, they will be overwritten. For those of you doing ClickOnce, this means it will overlay the appname.application file (the deployment manifest) and the publish.htm if you are generating it.

One other note to those doing ClickOnce deployment – if you publish your application to the same local folder repeatedly, it will keep creating versions under Application Files. This code uploads everything from the top folder down, so if you have multiple versions under Application Files, it will upload them all over and over. You might want to move them or delete them before running the upload.

This post provided and explained the code for uploading a folder and all of its sub-items to Azure Blob Storage, retaining the folder structure. This can be very helpful for people using ClickOnce deployment and hosting their files in Blob Storage, and for anyone else wanting to upload a whole directory of files with minimal effort.

Tags: , , ,

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s


%d bloggers like this: