Introduction to the Semantic Logging Application Block (SLAB)

In my previous post, I discussed diagnostics logging and why it’s important, and how structured logging can help you troubleshoot problems and do some data mining. Now I want to talk about SLAB and what features it has, and provide some resources for you to use to follow up with. In my next post, I’ll offer up an example and talk about the things you have to consider when designing your structured logging.

Technologies for semantic logging

Here are the technologies involved:

Event Tracing for Windows (ETW)

• Is native to Windows.
• Has great performance and OK diagnostic tooling
• It has historically been difficult to publish events.

EventSource class

• New in .NET 4.5
• Helps make it easier to create structured logging
• It’s extensible, but only supports ETW out of the box

Semantic Logging Application Block (SLAB)

• This is a wrapper around ETW; it allows you to use ETW without having any specific knowledge of ETW.
• It rovides several destinations (sinks) for events published with EventSource.
• It has additional tooling support for writing events.

Basically, SLAB enables you to use the EventSource class to write log messages from your application. You set up a listener, and SLAB receives notifications whenever the application writes a message using an EventSource class. SLAB then writes the message to whatever sink you have set up.

SLAB can act as a stepping stone. When you use EventSource, there is no commitment to how you consume events. You can later decide to replace it with your own tooling or use something else.

SLAB Features – Sinks

SLAB has the following sinks/destinations built-in:

• Microsoft Azure Staroge Table
• SQL Database
• Flat File
• Rolling flat file
• Console
• Elastisearch

And the following formatters for text-based sinks:

• JSON
• XML
• plain text

SLAB Out-of-process service

SLAB can be used in-process – you start up the listener when your application starts up and let it run until the application ends. It also has an out-of-process service that you can host as a Windows Service (it needs to run in an account with elevated permissions). This service will get events out of the originating process using ETW and persist the events to the selected sink.

All sinks are supported, and the service is configuration-driven with support for reconfiguration. Also, the monitored application does not reference SLAB. It writes to EventSource, which is caught by the out-of-process service.

Benefits:

• Increased fault tolerance in case of application crash.
• You can monitor multiple processes from a single service.
• This moves the logging overhead from the application to a separate process. The overhead is still there, it’s just not part of your applicaton.

Drawbacks:

• There are more moving pieces.
• The internal buffers may overflow without notification.
• You have to make sure the process starts before the logging starts, and doesn’t end until after the logging ends.

SLAB Features – Event Source Analyzer

This helps you author and validate the Event Source derived class, and flags hard-to-detect errors relating to plumbing. You can run this inside a unit test to validate your classes.

SLAB Features – Observable-based

The event listener is IObservable.

The event sinks are IObservers.

This means you can use Reactive Extensions (Rx) to filter, pre-process, or transform the event stream before it’s persisted.

For example, you could use Rx to check the errors coming in, and if you get the same error 5 times in a row, it could send you an e-mail and not publish the same error for a few minutes. An example where this would be helpful is if your application is calling SQL Server and SQLServer is down, and as a result you’re about to have hundreds (or thousands) of error messages that are all the same come through the system. It would be useful to receive a notification that you might be having a serious problem. I had this problem once at the startup I worked at. We had a problem with SQL Azure, and I got over a thousand e-mails within a few minutes before we managed to shut down the service. Yikes! :-O

Resources

Here are some resources I found really helpful when learning about this topic.

Developer’s Guide

Enterprise Library 6

Enterprise Library 6 code + downloads + documentation

The last one has the Enterprise Library 6 Reference Docs, Quick Starts (this includes the reactive extension examples), and the Developer’s Guide and samples. They have a really good example of adding semantic logging to a Windows Forms app that calculates pi.

Summary

This post talked about the features of SLAB and ETW. In my next post, I will show you a class with a bunch of trace logging in it, and talk about how to “SLAB-ify” it.

Accessing your Azure Files from inside a VM

In the first post of this series on the Azure Files preview, I discussed what the feature entailed, and what you can do with it, as well as how to sign up for the preview. In the second post, I showed how to get the PowerShell cmdlets for the preview, and how to use them to manage your share and transfer files to/from your share. In this post, I’ll show you how to create a VM in Azure, show you two ways to map the share, and how to copy files to it.

How to create a VM in Azure

I realize many who read this post will already know this; I’m including it for those who are new to Microsoft Azure and want to try out the Azure Files feature.

First, log into your Azure account. Select Virtual Machines from the menu on the left.

Next, at the bottom, select + New, Compute, Virtual Machine, FromGallery.

On the next screen, select Windows Server 2012 R2 Datacenter and click the right arrow in the bottom right hand side of the screen.

On the next screen, fill in a name for the VM. Then specify a username and password. These credentials will be used when you RDP into your VM, so don’t forget them! When you’re finished, click on the arrow on the bottom right-hand side of the screen.

On the next screen, specify “Create a new cloud service” and provide a DNS name for the cloud service. This is kind of a wrapper for the VM. (You can deploy multiple VM’s into the same cloud service, and they will be load balanced under the same IP address.) In my case, I already have a storage account set up. If you don’t, or want to use a different one, you can ask it to generate one for you (it’s an option in the Storage Account dropdown list). For optimal performance, pick a region that is close to your location. When you’re done, click the arrow at the bottom right-hand side of the screen.

The next screen is used to select the VM Extensions to be installed. Leave the checkbox checked for “Install the VM agent”; for the purpose of this exercise, you don’t need any of the other extensions, so just leave them unchecked and click the checkmark in the circle at the bottom right-hand side of the screen to continue.

Now Azure will provision and start up your VM. At this point, you just wait until it says “Running” (like the first one in my list displayed below), and then you’re ready to continue. This seems to take a long time, but you’ll find the wait much more enjoyable if you take a quick run to the closest Starbucks. (I’ll be right back…)

Now click on your VM and it will bring up the Dashboard. (It might bring up the QuickStart screen; if it does, just continue to the Dashboard.)

At the bottom of the screen, you will see the Connect icon is lit up and waiting for you.

Access your Share from inside your VM

Click Connect at the bottom of the portal screen to RDP into your VM. When prompted, specify the username and password that you provided when creating the VM. Click through the security prompts. If it’s the first time you’ve logged into your VM, click Yes when prompted with this screen:

Now let’s attach our share. There are a couple of ways to do this. One is to use the NET USE command.

Open a command window. The easiest way to do this is to click the Windows start button and on the Modern interface, just start typing “command”. This will bring up the search box on the right, and you should see what you’re looking for; click on it to open the command window.

z:

Here is the command to use to connect your share:

C:\ net use [drive letter]: \\[storage account name].file.core.windows.net\[share name]
        /u:[storage account name]   [storage account key]

I’m going to mount the share I created in my previous post. It was called ‘nightbird’, and was on ‘nightbirdstorage3’. I’ve blurred out my storage account key in the following example:

Now open Windows Explorer, and you’ll see the share listed.

Now you can double-click on it and see what’s on it. On mine, I can see the images I uploaded and the folder I created in the previous post.

If you double-click on the folder Images, you can see the files in that folder. At this point, this is just like using any network share you’ve ever used on-premises.

Any changes you make from within the VM will of course appear if you go back and use the PowerShell commands to list the files on the share, whether you add, change, or delete files and/or directories.

At this point, if you go back to the command window, you can use the NET USE command to see what shares you have attached.

Another way to access the share

Instead of using the NET USE command, you can actually map the network drive from within Windows Explorer.

Bring up Windows Explorer, right-click on “This PC” and select “Map network drive:”.

Type in what would be the network UNC path to your share, which will be in this format:

\\[storage account name].file.core.windows.net\[share name]

Be sure that “Reconnect at sign-in” is checked. Click Finish to complete the drive mapping.

You will be prompted for username and password. This is for the share, so the username is the storage account name (nightbirdstorage3 in my case) and the password is the account key.

After doing this, it will open Windows Explorer, and show the share and the files and directories in the root.

So that’s the second way to map your network drive.

How do I put files on my share?

This is pretty simple; you can use the RDP session to do that. Just bring up Windows Explorer on your local computer. Select the directory and/or files that you want to copy and click Ctrl-C.

Switch to the RDP session. Using Windows Explorer, select where you want the files to go, then click Ctrl-V.

You can also copy the files on the share and paste them into Windows Explorer on the desktop to download them.

Another way to do this is when you click Connect in the portal, it will prompt you to save or open the RDP file. If you save it, you can then go find the RDP file, right-click on it and choose Edit. In the Local Resources tab, you can select More… under Local Devices, and then open up Drives and select a local drive. This will map the drive when you log into the VM, and you can access it as if it were local in the VM. I’m going to attach my D drive and then click OK, select the General tab on the main RDP window, and select Connect to connect to the VM.

Now after I log into my VM and bring up Windows Explorer, I can access that drive from inside the VM:

Now I can copy the files directly from my local computer to the file share accessed by my VM (and vice versa).

Regions and subscriptions and access, oh my!

An important thing to note is that your file share does not need to be in the same account as the VM’s you are going to use to attach it, it just needs to be in the same region.

I have multiple Microsoft accounts that have an Azure subscription. If I create a storage account in US West in one of my Azure accounts, and set up a file share, I can access that file share from any VM in any of my other Azure subscriptions that have VM’s in US West. This could bring up some interesting use cases.

Something to try

When you attach a file share to multiple VM’s, and one of the VM’s changes one of the files, a notification is sent to the other VMs that the file has changed and their view of the file share is updated. To see this work in action, you can follow these steps:

1. Create another VM in the same region as your file share.

2. RDP into the VM and attach the share.

3. RDP into the first VM and bring up the share folder.

4. Change one of the files on the share in one of the VMs.

5. Change over to the other RDP session and look at the files on the share, and you will see the update there as well.

Making the file share sticky

When I attach the share using NET USE, log out, restart the VM, and come back in, I have to provide credentials again to use the file share.

If I map the network drive using Windows Explorer, and check the box that says “Reconnect at sign-in”, then when I log out, restart the VM, and come back in, the share is still mapped and available without providing credentials.

This is tied to the user account used to map the drive. So be aware that you might need to map the drive under other user accounts. For more information, check out this article by the Microsoft Azure Storage team that addresses the stickiness of Azure File share mappings.

If you want to access that share from a web site, you would create a virtual directory on the VM hosting the web site. For example, if I wanted \\nightbirdstorage3.file.core.windows.net\nightbird\Images\ to be accessible as http://contoso.com/images, I would create a virtual directory that points to that folder on the share using that UNC path of the share. When I do that, it requests the credentials for accessing the share, and I can provide the storage account information at that time and it will be sticky if the role is rebooted.  (Thanks to Steve Evans for the tip.)

Summary

In this post, I showed you how to create a VM in Azure and attach your share using two different methods. I also showed how to copy files to the share from your local computer. In my next post, I’ll show you how to programmatically create a share, create directories, and upload/download files using the Storage Client Library.

Azure Files: how to manage shares, directories, and files

In my previous post, I discussed the new Azure Files preview, what it entailed, and what you can do with it, as well as how to sign up for the preview. Don’t forget that you also need to create a new storage account in order to get the Azure Files endpoint.

Unfortunately, there is no UI available yet to let you manage and look at your shares. I use Cerebrata’s Azure Management Studio to access my storage tables, queues, and blobs, and I expect them to have a new version supporting Azure Files before too long. In the meantime, you have multiple options: write some code and use the REST APIs, use PowerShell, RDP into your VMs and attach the share, or write a program using the Storage Client Library. In this post, I will show you how to use the PowerShell cmdlets to create your file share, upload files to it, list the files on it, etc.

Setting up the PowerShell cmdlets for the Azure Files preview

First, you need to download the PowerShell cmdlets for this preview. At the time of this writing, they can be found here. Download the zip file and save it to your local computer. Depending on your OS and security settings, you might need to unblock the zip file before you can access it. To do this, right-click on the file and check the properties; if there is an Unblock button, just click on it to unblock the file.

These Azure File cmdlets use Storage Client Library 4.0, so you need to run them in a different PowerShell session than the regular cmdlets. You need to unzip them, run PowerShell, and then set your default directory to the unzipped files. So you might want to think about that before unzipping the files, and put them in an easy-to-type location. For example, I’m going to put mine in a folder on the root of the data drive in my computer instead of MyDocuments. So I’ve created a folder called D:\zAzureFiles\ (so it shows up at the bottom of the directory listing), and unzipped the zip file into that folder. (If you don’t have a D drive, just create a folder on your C drive and use that).

When you have the files unzipped, run either Windows PowerShell or Windows Azure PowerShell. Change directories to the AzureStorageFile directory created when you unzipped the zip file.

If you do a DIR at this point, it will show you the files in the directory. What you want to do is load the Azure Files cmdlets. Here is the what the command looks like:

> import-module .\AzureStorageFile.psd1

When I type this into the Windows Azure PowerShell window, I get the list of cmdlets that were loaded.

Now that you have PowerShell up and running with the new cmdlets, you can fully manage your share(s).

Using the PowerShell cmdlets

The first thing you need to do is create a context that specifies the credentials for the storage account you want to use for your share. This is what the command looks like; you’ll want to make the appropriate substitutions.

> $ctx = New-AzureStorageContext [storageaccountname] [storageaccountkey]

Nobody in their right mind wants to type in their storage account key, so you’ll want to paste this. In case you’re a newbie at PowerShell, I will tell you that the PowerShell window is similar to a Command Window. After copying it into the Windows clipboard, you can right-click on the line in the PowerShell window where you want it to go, and it will paste it automatically (I didn’t even have to right-click and select Paste, I just right-clicked and it pasted it.)

I have a Surface Pro tablet, and when trying this, I had all kinds of trouble. I could not right-click and get it to paste with the Pen, the Touchpad, or using my finger (not even the middle one ). So here’s a free tip for those of you trying the same thing: Copy the string into the Windows clipboard, then select the PowerShell window, click Alt+Space. This will bring up a menu where you can type E for edit, and P for paste.

(This doesn’t seem like a big deal unless you create a demo on your desktop computer for a talk that you’re giving using your Surface Pro, and you don’t actually test the demo on the Surface Pro until midnight the night before the talk, and then you find that you can’t get it to do something simple like paste a big-ass string into a window. Not that I would ever do anything like that. *cough*. But I digress.)

Here’s my command creating the context in PowerShell, using my storage account called nightbirdstorage3 with my storage account key truncated so you can’t access my storage account and store your vacation pictures for free:

Now that you have the context set, you can create a file share in that storage account like this:

> $s = New-AzureStorageShare [sharename] –Context $ctx

I’m going to create a share called nightbird in my storage account:

Ta-dah! I now have a file share called nightbird in my nightbirdstorage3 storage account.

So how do you upload files onto your share? Using this command, you can upload one file to the share you’ve set as $s.

> Set-AzureStorageFileContent –Share $s –Source [local path to the file]

I’ll upload several images from the D:\zAzureFiles folder. Notice that on the last one, I’ve already uploaded that file, so it gives me an error message. You can use the –Force switch to tell it to overwrite the file if it already exists – just add it to the end of the command.

What if you don’t want them in the root folder of the share, but in a subfolder? You can create a directory on the share, and then specify that directory when uploading the files. To create a directory (where $s is the share):

> New-AzureStorageDirectory –Share $s –Path [foldername]

Then when you upload the file, you specify the path ($s is the share):

> Set-AzureStorageFileContent –Share $s –Source [path to local file] –Path [foldername]

I’ll create a directory called Images, and upload some files into it:

How do I see what is on my share? Here’s the command for that, followed by an example.

This gets the files in the root.

Get-AzureStorageFile $s

This gets the files in the specified folder.

Get-AzureStorageFile $s –Path [foldername]

Here is it in action:

Note in the first listing, which is the root, it shows both the Images folder and the files in the root directory. For the second command, it shows what is in the Images folder. There are no subfolders in the Images folder; if there were, it would show them in the listing.

Deleting a file from the share

These are very similar to the commands above. To delete a file, use Remove-AzureStorageFile instead of Set-AzureStorageFileContent (where $s is the share).

> Remove-AzureStorageFile –Share $s –Path [foldername]/[filename]

To remove a directory, use Remove-AzureStorageDirectory:

> Remove-AzureStorageDirectory –Share $s –Path Images

Note that the directory must be empty before you can delete it. Right now, they don’t have a way to delete a directory and all the files in it recursively like Directory.Delete(path, true) in .NET.

Downloading a file from the share to your local computer

To download a file from the share to the local computer, use Get-AzureStorageFileContent.

> Get-AzureStorageFileContent –Share $s –Path [path to file on the share] [path on local computer]

Here’s an example of downloading shark.jpg from the Images folder on the share into my local directory:

If the file already exists locally, it will give you an error. You can use the –Force switch to overwrite it if it already exists – just add it to the end of the command.

Uploading/downloading multiple files with one command

What if you want to upload a bunch of files? You have to either specify them separately, or use AzCopy version 2.4. Download AzCopy, then run the install. Then go search the C:\ drive to find out where it put the dang files. I copied the whole folder over to my D:\zAzureFiles directory for easier use. Use a Command Window and set the directory to the AzCopy directory. Then you can run AzCopy by typing in a command. Because of the likelihood that I’m not going to get it right the first time, I create a text file with the command in it, and save it as a .cmd file and execute it. Or copy the command from the text file and paste it into the command window and execute it. This makes it easier to edit, copy, and paste repeatedly when all you have to change is a directory, or to fix it if you have a problem.

Here are some examples of what you can do with this.

Upload all the files in a directory (designated by [localpath]) to the specified storage account and share name. The /s switch tells it to upload the directory recursively, which means it picks up the files in that directory, and all subdirectories and files under that original directory, retaining the directory structure.

AzCopy [localpath] https://[storageaccount].file.core.windows.net/[sharename]         
    /DestKey:[StorageAccountKey] /s

Here’s my example:

To make sure this worked, I’ll go back to PowerShell to pull a list of files in the directory and see if my new files are there.

You can see that there are two new files on the share – greatwhiteshark.jpg and macworld_bing.jpg – those came from my test directory.

Here’s how to upload files using a wild card. For example, you could upload all the jpg files.

AzCopy [localpath] https://[storageaccount].file.core.windows.net/[sharename]  *.jpg          
    /DestKey:[StorageAccountKey] /s

Here’s how to download the files from the file share to your local computer. This will download all of the files on the file share, including all subdirectories and the files in them.

AzCopy https://[storageaccount].file.core.windows.net/[sharename] [localpath]         
    /DestKey:[StorageAccountKey] /s

You can also copy just one file. This does not look recursively through the directories; it only looks in the one you specify.

AzCopy https://[storageaccount].file.core.windows.net/[sharename]/foldername [localpath]         
    /DestKey:[StorageAccountKey] /s [name of the file you want to download]

Note: It will not retain empty folders. So if you upload a directory structure that has an empty folder in it somewhere, that folder will not appear on the target. The same applies when download a directory structure.

Summary

In this post, I showed you how to obtain the PowerShell cmdlets for the Azure Files preview, and how to use them to manage your share and transfer files to/from your share. In my next post, I’ll show you how to create a VM, RDP into it, attach the file share, and access the files.

Overview of the new Azure Files Preview

One of the recent announcements from the Azure Storage team was about a new feature called Azure Files. This basically allows you to set up an equivalent to network shares in your Azure environment, to make files available to multiple VM’s simultaneously.

Some use cases

If you have a web application that outputs data that you need to input into a different web application, you could set up a file share and give access to both applications; one could write the data, the other could read it.

If you have a set of tools that you frequently use, you might put them on a shared drive. Then you can attach the drive and use the tools from multiple VM’s. As a developer, I would put the installation packages for tools that I use every day – Beyond Compare, FileZilla, etc – and when I spin up a VM with Visual Studio in it, I could then attach the share and install the tools in my VM.

If you have a team of developers using Visual Studio in VM’s, this makes it easy for them to share tools, and to enable all of the team members to use the same version. You could even put a OneNote notebook on the share and have the developers share notes about the project in that notebook.

Wasn’t there already a way to do this?

There was a convoluted way to do this with the current IAAS offering – hosting a file share backed by an IAAS data disk, and writing code to find the IAAS File Share from the rest of the VMs in your service, trying to handle high availability, etc. With the new feature, you just set up the shared drive, then attach it in your VMs. It’s that simple.

Since the Azure Files is built on the same underlying layers as Azure Storage, it provides the same availability, durability, and scalability that Azure Storage already provides for blobs, queues, and tables. The Azure Files can be locally redundant (3 replicas of the data within a single region) or globally redundant (3 replicas of the data in the same data center, and 3 replicas in another region).

How do I access the files on the share?

Azure Files provides two interfaces: SMB 2.1 and REST. The SMB protocol allows you to access the file share from VM’s. The interface for REST provides something you don’t have in on-premise file shares – the ability to access the files in the share with REST interface. For example, you might store files on the share that are used in multiple web applications, then use the network path to the share in your web applications to access those files. Those files can then be updated externally using REST.

The SMB 2.1 protocol is natively supported by OS APIs, libraries, and tools. This includes Windows (CreateFile, ReadFile, WriteFile…), .NET (FileStream.Read, FileStream.Write, etc.), and many others. It also supports standard file system commands for moving and renaming files and directories, as well as change notifications.

There is an SMB 3.0 protocol available, and I suspect that eventually Microsoft will support it. I think they choose 2.1 to start with because it is supported by the most tools and OSes. It is definitely supported by Windows Server 2008 R2, Windows Server 2012, and Windows Server 2012 R2. While not being completely supported on every version of Linux, it does work on some distributions, such as Ubuntu 13.10 and Ubuntu 14.04 LTS.

Note that the file and directory objects on the share are true file and directory objects. This is different from Blob storage, which has a flat structure. In Blob storage, you only have containers (the equivalent of a top-level directory). Anything that looks like it is in a folder within a container is actually part of the Blob file name, which is frequently parsed and visualized like a directory structure.

What are the specs?

These are the preview scalability targets for Azure Files:

• Up to 5TB per share
• A file can be up to 1 TB
• Up to 1000 IOPS (of size 8KB) per share
• Up to 60 MBps per share of data transfer for large IOs

Using the SMB protocol, Azure Files are only accessible to VMs in the same region as the file share. However, you can access the files from anywhere in the world using REST. (This may be because SMB 2.1 doesn’t have built-in encryption, so there’s no way to protect the files going out of the data center.)

One question I’ve seen asked a lot so far is “Can I attach the share to my desktop?” The answer is no, because the SMB protocol is not designed for the latency involved. This is going to be a huge feature request, so I’d be surprised if Microsoft didn’t come up with a solution at some point in the future.

How can I get access to this preview feature?

Go to the Azure Preview Portal and sign up for the Azure Files service using one (or more) of your existing Azure subscriptions. As subscriptions are approved, you will get an e-mail notification.

After being approved, log into your account and create a new storage account. It will not work with storage accounts created prior to signing up for the preview. When you create the new account, you will see the new endpoint displayed in the portal (http://storageaccountname.files.core.windows.net).

From there, you can create the share, put files into it, and connect to it from a VM.

Other resources

Check out this blog article by the storage team for more details. It also has a link to the video from the Tech Ed session.

Summary

The new Azure Files preview is something people have been asking for for a long time. To be able to set up a file share and access it from multiple VMs simultaneously brings a lot of new possibilities to Microsoft Azure, and should help people migrating from on-premise to the cloud, as well as those running hybrid implementations. In my next post, I’ll show you how to use PowerShell to maintain a file share and upload/download files.