This is the third and last blog post in a series about the Azure EA portal and the billing API. Part 1 showed you what analysis you can do in Excel with the billing data and PowerPivot. Part 2 was the boring, technical stuff, but the stuff everybody needs to do to get access to the data in an automated way. Part 3 will focus on how you can structure the reporting of your Azure Consumption.

The Times They Are A Changin’

Bob Dylan was right, you know. Entering the Cloud era means you have to adapt to new ways of what you are doing to achieve the benefits of the cloud without breaking your back. Don’t carry old requirements forward since they age pretty quickly. Imagine the cost model for the IBM mainframes of the 60’s and 70’s where you had one central computing resource for the entire company and the price you paid was proportional to the CPU consumption your programs used. That model didn’t carry forward in the desktop and server era that came later for obvious reasons and new cost models were developed. When moving to the cloud, it’s not a smart move to always try to use your current cost models as is since they mirror the past.

Don’t get me wrong here. I’m not going into a fight about CAPEX and OPEX or the TCO of cloud computing vs internal datacenters, or how such a comparison should be done to be fair. I’m simply arguing that such a big transition as moving workloads to the cloud is as a big change as moving from the mainframe to the servers era. You have to accept that the times are changing.

Azure EA Portal Hierarchy – a History lesson

An Azure subscription comes as a heritage from the MSDN and XBox world when Reddog was new and the Microsoft pioneers though of a way to make this commercialy available to customers before 2009. At that time you could only sign up for Azure subscriptions with a credit card (which caught much negative attention) and they were all stand-alone, pay-as-you-go subscriptions. Then came the possibility of enrolling Azure on the Enterprise Agreement (EA) that larger customers sign with Microsoft (Azure on EA made its debute in 2010) which meant that the company paid for all subscriptions bound to that EA enrollment.

In order to not have an entirely flat structure of subscriptions within the EA, the Account level was introduced. On the pay-as-you-go subscription, there was from the beginning two roles – Account Admin and Service Admin. Now with the EA bound subscriptions, the Account Admin moved up one level. If you have an Azure subscription that is bound to an EA, you can not change the billing details since that is something only the Account Admin can do.

The early guidelines from Microsoft on how to use the Account level within the EA portal were examples of Functinal, Business or Geography use of it. I always explained this as “think of the Account level as Cost Centers” since you can aggregate cost from all subscriptions up to the account level easily. Some hosters used one Account for one Customer making it easy to manage and see what the spending was for that account/customer.

Need for company context in the hierarchy

The Flat Earth Society have just as a hard time nowadays as do most enterprises have squeezing their Org Sheet into a three level structure that the EA portal made available. This led to clever naming schemes of the accounts since that meant you could filter your queries on that scheme. Imagine what happens when the next company reorg comes. To let customers add some of the of their own context into this, Microsoft recently (late 2014) introduced the concept of Department. The picture above is not entierly correct, since the Department is not a true layer on top of the Account level. It is instead an object you can create and add to each Account. In the EA Portal, you select one Department of the list of departments you have created to be associated with the Account.

The Department is a little bit more than just a name. You can add a Cost Center value too. This basically means a couple of things. The naming scheme of the Accounts can be totally decoupled from the cost reporting model. Use what ever names on the Accounts that makes sence in your coorporation. The second thing is that when the guys managing the company’s General Ledger decides to changed all Cost Center codes, you can do that without crying your eyes out. You just update the Cost Center value in one place.

This would be pretty useless if it wasn’t for that the Department and Cost Center values are included in the Billing Data CSV file. That means that you get your company’s context with you in the stream of reporting data that Azure can give you via the Billing API. Now you can filter and aggregate on your own values, like Cost Center codes.

But, if you scroll up and review the hierarchy picture above, you see that the Department level (with it’s accompanying Cost Center) is specified very high up in the food chain. This effectively means, NO, you can not tag a Cost Center code to individual resources within an Azure subscription in order to track costs for an individual VM. You can not even do it on the subscription level, but only on the level above it. Cost Center is really just a tool for not falling for the tempation of creating a beaurocratic naming scheme of Account names.

At this point some customers I’ve met and explained this to say that Microsoft sucks and is not Enterprise ready. Everyone entitled to an opinion, but remember that Bob Dylan is right and that swiming upstream is not the smartest thing to do. Don’t carry old requirements forward.

So what is the trick to get this right?

First of all, you have to understand what Subscriptions are. There is no reward for shoehorning every VM you ever found into a Subscription and packing them with things that have no relationship with each other. An Azure Subscription is both an econimical and a security boundry.Economical in the sense that some resources you provision within the Subscription will incure cost that you can not track in the old way. Good examples of this are cost for network egress which means that anything causing network traffic leaving an Azure datacenter inside the subscription will accumulate to the network egress cost. But who caused the network egress within the subscription will forever be a secret, because that is not something Azure tracks. Another example is the (rediculously small) cost for Storage I/O. Azure does not track who caused that I/O and it is just something that goes with the StorageAccount. You just have to accept that I/O occured stop trying to be Chief Inspector Clouseau.

My point is simple – don’t mix resources in Subscriptions that don’t share the same cost taxonomy. Internally at Microsoft, there is a cross charge mechanism on the subscription which means that what ever happens in the subscription is cross charged in the same way. It’s simple and easy and I can spend my time being productive and adding value to the company.

Resource Tags

The new Azure Portal (http://portal.azure.com) that (eventually) replaces the blue-and-white management portal introduced a concept of Resource Tags. The tags wouldn’t be such a fuzz unless these tags also show up as a column i the Billing Data CSV files. This means that you can tag resources in a Subscription, that means at the very lowest level, with your values. You could even see this as the asset tag strickers most of us have on the backside of our corporate laptops. With this you have a future possibility of aggregating cost across Subscriptions, Accounts and Departments. If I add Tags field to the Excel PowerPivot ROWS (see part 1), I can drill down to the cost of a certain tag and find the aggregated cost for that. You can assign the Tags in the Azure Portal but you will be dissapointed since the field will be blank in the Billing data currently. Eventually it will be there.

However, Don’t use tags to try to say Bob Dylan is wrong, because he isn’t. What tag would you set on the virtual network resource within a subscription? Oh, that’s right, Azure doesn’t track who causes the network traffic, so you can’t divide up that further.

Don’t rely on the Tags too much, though, because it is not a waterproof solution if you are planning to tag every thing at the lowest level. Imagine if you purchase and deploy an ISV solution, like EPIServers CMS system, and deploy it to your Azure Enrollment, you most probably will not have the possibility of trying to tag every Azure resource they used. You probably will just track cost on the Subscription level.

What would I do if I was given the task of setting up the cost model for Azure in a large Enterprise?

Is the available tools today for doing large volume analytics greater or smaller than the effort it takes for an implementation project to invent a scheme for assigning labels (didn’t want to reuse the word Tags here) on resources provisioned in Azure? Of course it’s greater! The power of todays analytical tools is a wet dream for a data scientists of the past. With the modern tools for data analytics it is easier to analyse the stream of data you get, in this case via the Billing API, than it is to develop complex procedures for ordering and provisioning Azure resources. You provision resources of different kinds into Azure subscriptions and Microsoft makes the Billing Data available to you in the way each resource type can. Using modern analytical tools you have an opportunity that didn’t exist in the past.

Another important thing to notice here. You can actually tell what IT costs, atleast in the Cloud. This means that down to the subscription level, without doing anything else, You can see your consumption and take actions on it. It is just like buying a new smartphone to your child. You never fully understand the mumbo-jumbo the sales person tells you in the store (atleast I don’t, congratz if you do), but you sure understand the invoice and the spending your child did. If that is above the budget, you have a constructive conversation about the value of money and make sure the next invoice is inline with the budget.

After all – I like Bob Dylan and I do believe he is right. The Times They Are A Changin’