This article generally talks about how to upgrade your hosted service instance count on the fly.

There are several solutions out there which achieve the idea of autoscaling to enable windows azure to meet demand from a set number of requests. Our sponsor Paraleap is in pole position there with a great piece of software to do this. How do they this? i.e. what underlies the ability of autoscaling.

Well, autoscaling (not the protected algorithm) but the management of services has invocations from the Service Management Api that we’ve been looking at its heart.

We’re going to discuss the test tool I mentioned in my last post and I’m going to drag this out by a couple of weeks until I finish the tool so that we end up with a four or five part series!

This first post is going to be highly practical and the second post is going to be a little more practical and then we’re going to relay a lot of theory.

So, I have a test tool which needs to change configuration on the fly if it changes the number of running instances. The configuration in question is in the ServiceConfiguration.cscfg and may look something like this:

<?xml version="1.0" encoding="utf-8"?>
<ServiceConfiguration serviceName="xxx" xmlns="http://schemas.microsoft.com/ServiceHosting/2008/10/ServiceConfiguration" osFamily="1" osVersion="*">
  <Role name="xxxx">
    <Instances count="1" />
    <ConfigurationSettings>
      <Setting name="DataConnectionString" value="DefaultEndpointsProtocol=https;AccountName=xxxx;AccountKey=" />
    </ConfigurationSettings>
  </Role>
</ServiceConfiguration>

It’s important to note the configuration that gets deployed is different because when you go through the deployment process in VS.NET there are some additions:

<?xml version="1.0" encoding="utf-16"?>
<ServiceConfiguration xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:xsd="http://www.w3.org/2001/XMLSchema" serviceName="" osFamily="1" osVersion="*"
xmlns="http://schemas.microsoft.com/ServiceHosting/2008/10/ServiceConfiguration">
<Role name="xxxx">
<ConfigurationSettings>
<Setting name="DataConnectionString" value="DefaultEndpointsProtocol=https;AccountName=xxxx;AccountKey=" />
<Setting name="IntelliTrace.IntelliTraceConnectionString"
value="BaseEndpoint=core.windows.net;Protocol=https;AccountName=xxxxx;AccountKey=n1n9w1tzexKVAVZbTkggxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxSu42QA
==" />
<Setting name="CloudToolsDiagnosticAgentVersion" value="1.6" />
</ConfigurationSettings>
<Instances count="1" />
<Certificates />
 </Role>
</ServiceConfiguration>

The key is to load the modified configuration XML and change the instances value as it’s needed. This is done quite simply using a couple of calls to the Service Management REST API.

I will post the code and the application in the coming weeks but for now let’s make do with explanation and effort. We begin with a call to build up a request to get the actual instance data which can later be modified. This should be familiar to all of the people tha attended the December user group session.

public void Run()
{
	var request = (HttpWebRequest) WebRequest.Create(String.Format("https://management.core.windows.net/{0}/services/hostedservices/{1}/deploymentslots/production", _subscriptionId, _serviceName));
        request.ClientCertificates.Add(_cert);
        request.Headers.Add("x-ms-version", "2011-10-01");
        request.BeginGetResponse(Callback, request);
        waitHandle.WaitOne();
}

We can parse the response simply using the following LINQ to Xml to get the number of role instances and their respective VM Size (the VM size is an instance property but will not vary between role instances):

XDocument doc = XDocument.Parse(xmlDocument);
InstanceCount = doc.Descendants(BuildDefaultNamespaceXmlEntity("RoleInstance")).Count();
string instanceSize = (string) doc.Descendants(BuildDefaultNamespaceXmlEntity("RoleInstance")).FirstOrDefault().Element(BuildDefaultNamespaceXmlEntity("InstanceSize"));
VmSize = (VMSize)Enum.Parse(typeof (VMSize), instanceSize);

Whilst the above invocation to the Service Management API used an HTTP GET request, the following uses a POST request. This operation is the same in both cases.

</pre>
/// <summary>
///<?xml version="1.0" encoding="utf-8"?>
///<ChangeConfiguration xmlns="http://schemas.microsoft.com/windowsazure">
/// <Configuration>base-64-encoded-configuration-file</Configuration>
/// <TreatWarningsAsError>true|false</TreatWarningsAsError>
/// <Mode>Auto|Manual</Mode>
///</ChangeConfiguration>
/// </summary>
<pre>public void Upgrade(string configurationFile, int instanceCount, string VmSize)
{
	// add an openfile and new details here
        string configuration = OpenFileAndReturnBase64AmmendedString(instanceCount, VmSize, configurationFile);
        string upgradePayload = BuildUpgradeXmlPayload(configuration);
        var request = (HttpWebRequest)WebRequest.Create(String.Format(
                "https://management.core.windows.net/{0}/services/hostedservices/{1}/deploymentslots/production?comp=config",
                _subscriptionId, _serviceName));
        request.ClientCertificates.Add(_cert);
        request.Headers.Add("x-ms-version", "2011-10-01");
        request.Method = "POST";
        Stream requestStream = request.GetRequestStream();
        using (var writer = new StreamWriter(requestStream))
        {
            writer.Write(upgradePayload);
        }
        request.BeginGetResponse((ar) =>
                                         {
                                             HttpWebRequest newRequest = (HttpWebRequest) ar.AsyncState;
                                             var response = (HttpWebResponse)newRequest.EndGetResponse(ar);
                                             RequestId = response.Headers["x-ms-request-id"];
                                             AsyncOperationTimer.Start();
                                             waitHandle.Release();
                                         }, request);
        waitHandle.WaitOne();
}

Bear in mind that the request encoding for the configuration file needs to be Base64 UTF8/16 encoded.

byte[] utfConfigData = Encoding.UT8.GetBytes(configuration);
string encodedData = Convert.ToBase64String(utfConfigData);

What’s important about the above is that an XML payload needs to be sent in the request body with the headers. The key here also is that there are various operations within the Service Management API which rely on an asynchronous response. Continuation tokens are commonplace in these instances in order to monitor state. The MSDN documentation goes through the number of operations which use async behaviour and also identifies the GetOperationStatus operation which will allow the token returned by the first operation to be used to get the operational status. The operation above is called ChangeDeploymentConfiguration and will not return anything in the body of the response but will return an HTTP header called x-ms-request-id which can be used to determine the state of the service operation in subsequent calls to GetOperationStatus. It’s worth pointing out that all async operations return a 200 OK HTTP status code.

AsyncOperationTimer = new T.Timer()
{
	AutoReset = true,
	Interval = 30000,
	Enabled = true
};

AsyncOperationTimer.Elapsed += (sender, args) =>
{
	if (RequestId != null)
	{
		var request = (HttpWebRequest) WebRequest.Create(String.Format(
			"https://management.core.windows.net/{0}/operations/{1}",
			_subscriptionId, RequestId));
		request.ClientCertificates.Add(_cert);
		request.Headers.Add("x-ms-version", "2011-10-01");
		var response = (HttpWebResponse) request.GetResponse();
		XDocument doc;
/* <?xml version="1.0" encoding="utf-8"?>
* <Operation xmlns="http://schemas.microsoft.com/windowsazure">
* <ID>request-id</ID>
* <Status>InProgress|Succeeded|Failed</Status>
* <!--Response includes HTTP status code only if the operation succeeded or failed -->
* <HttpStatusCode>http-status-code-for-asynchronous-operation</HttpStatusCode>
* <!--Response includes additional error information only if the operation failed -->
* <Error>
* <Code>error-code</Code>
* <Message>error-message</Message>
* </Error>
* </Operation>*/
		 using (var reader = new StreamReader(response.GetResponseStream()))
		 {
			doc = XDocument.Parse(reader.ReadToEnd());
		 }
		 string status = (string) doc.Descendants("Status").FirstOrDefault();
		 if (status == "Failed" || status == "Succeeded")
		 {
			RequestId = null;
			AsyncOperationTimer.Stop();
		 }
		 if (status != "Failed")
		 {
			CurrentWindow.ShowMessage("update to instance count failed!", DisplayType.MessageBox);
		 }
		 else if (status == "Succeeded")
		 {
			CurrentWindow.ShowMessage("successfully updated instance count", DisplayType.MessageBox);
		 }
	}
};

In order to facilitate getting a comprehensive response the code above sets up a Timer which can then be used to update some kind of state. If the update of the instance counts succeeded then we get a success message. Look at the XML in the above code snippet and it should be evident that there are the following three status codes.

  • InProgress
  • Succeeeded
  • Failed

These are the only relevant things we need to worry about and will determine the outcome for our operation.

I’ll try and get part II up in the next few days where I’ll be discussing the difficulty with re-provisioning services to a new VM Size and how this varies from what we’ve seen. The good news is that it can be done. The bad news is that the operation is not quick since the whole deployment needs to be done again! There is also one gotcha that you need to be aware with the instance count example above … Whilst the instance is reimaged if there are any necessary startup tasks these may not execute so they may need to be checked and done manually (this is my experience).

I’ve attached an image portion of the form showing the Azure bit. In the next part we’ll focus on the VmSize and reprovisioning 🙂

Cloud Testing Tool

Cloud Testing Tool