Monday, April 02, 2012

Software Delivery Bottlenecks - Deployment

Seems that the business waits on software developers, who have waited on system administrators, who have waited on infrastructure engineers to produce the ecosystem required for todays software applications, while waiting on the business to produce the contracts, by which the business can legally operate.

Looking at the most simplistic example of the macro tasks involved in a software project we have:
  • develop software 
  • runtime environment(s) 
  • deploy software

If we assume an admissibly arbitrary cost of 1, for our tasks, and fundamentally all tasks being equal, then we can create a simple formula to help describe the bottleneck in our software delivery process.

We do have to account for things we do many times or iterations. For instance, we will deploy software through more than 1 environment. While the software hasn't changed, but we still have the overhead of the deployment cost.


  • assumed cost of 1 arbitrary unit 
  • multiplier N, where N is equal to the number of iterations, which will be 5. 
  • multiplier E, where E is equal to the number of environments, which will be 4 (Development, QA, Staging, and Production)

The Cost of Doing Business

  • develop software (1 * N) = (1 * 5) = 5 
  • provision hardware (1 * E) = (1 * 4) = 4 
  • deploy software ((1 * E) * N) = ((1 * 4) * 5) = (4 * 5) = 20

  Software Delivery Bottleneck

Of course, this is a very simplistic view, contrived to drive the point home that deploying software can have a very high overall cost of a software delivery. Deployment is an area of the overall software delivery process that can bring an almost immediate value in economy of scale as the number of servers, and number of environments, increases.

Automate those things that have a high rate of repeatability. Don't be surprised, removing one bottleneck will typically force another bottleneck into the light and we address those one at a time.

People, Process, and Automation. 

Lather, Rinse, Repeat.

Wednesday, March 28, 2012

Windows PowerShell Retrospective

In interest of full disclosure, the opinion here is mine, and not widely shared by, well, anyone. With that out of the way, I like PowerShell, I really do, just I'd rather use Ruby.

Wikipedia on PowerShell
Windows PowerShell is Microsoft's task automation framework, consisting of a command-line shell and associated scripting language built on top of, and integrated with the .NET Framework. PowerShell provides full access to COM andWMI, enabling administrators to perform administrative tasks on both local and remote Windows systems.
PowerShell falls into the "scripting" language category, along with languages like Python, Ruby and Perl, and specifically targets the Windows platform. This means its a powerful tool for any type of Windows administration tasks.

So why then would anyone NOT use Windows PowerShell to administer a Windows system? You'd have to be a lunatic NOT to take advantage of Windows PowerShell, right? Right.

Taking advantage of PowerShell for discreet, Windows specific, tasks makes perfect sense as part of a toolchain targeting Windows. Basing an entire provisioning infrastructure around it probably does not. There are a plethora of existing tools to help you there.

"devops toolchain"
[image from]

Pick one. Get started with it. Enjoy life.

Things That Make Me Sad Inside

Stupid name. Really. I can't say "PowerShell" without hearing a Monster Truck voice in my head. I know you do, too. Granted, that was an easy one, so now that we've got the low hanging fruit out of the way, lets move on to other considerations when working with PowerShell.

PowerShell is dependent on .NET. No surprise there. However, PowerShell cmdlets written for previous versions may not work without a recompile targeting new .NET version, or some configuration changes. This is easily mitigated going forward with PowerShell 2.0 by leveraging a PowerShell vs. C# module based approach, which was not as convenient prior to 2.0.

PowerShell updates are tied to platform updates. Platform updates which may need to be requested of Infrastructure. You want to install Windows PowerShell (pre-Windows Server 2008 R2), you need this update. Look at all the goodness that you get with that update. You wanted just a PowerShell runtime? Sorry, its got dependency issues. Might also get a reboot out of that to boot. You read that right, this is not an XCOPY deployment we are talking about here. This update is something that is twiddling the bits of your server in an unknown manner. Do you have good feature test coverage on your applications to validate them after an update like that? Me either.

Enterprises are not generally a homogenous environment. This is where the wheels start to come off. Other scripting environments, even on Windows, might start to look more attractive. The major platform scripting languages (Python, Perl, Ruby) are each XCOPY deployable and have a side by side execution story, which PowerShell doesn't currently have.

Pipelining implementation goes a little too far. With most languages, pipelining is accomplished by piping ("|") strings into other processes. This is something every administrator is familiar with since the dawn of computing. Everything violates the Principal of Least Astonishment [PoLA] at one point or another, and with PowerShell,your first encounter with [PoLA] is when you start piping strings. Thats when you remember, you are not pipelining strings, but strings cleverly disguised as objects. Even with what appears to be STDOUT, aka, the console window.

PowerShell pipelining objects really isn't that big of a deal, except that it means you've implicitly broken every tool chain ever invented. Maybe thats a little overstated, but hopefully you get my point. Things just don't work like you would expect out of the gate, and you can spend an unfortunate amount of time refactoring your way around the pipelining implementation.

Even as awkward as pipelining feels at first, there is one feature of PowerShell that felt crippled because of it: remoting. If you've ever experienced secure shell (SSH), you know how convenient it is to SSH into a remote server, and read/edit a file.

Try doing that with PowerShell. Go ahead. I'll wait. Remote into a server and try to run the old edit.exe text editor. Its still on your system, I promise. Your PowerShell session will hang. This is because programs like edit.exe, vim, pico, emacs, etc. manipulate that console output in a tried and true fashion that just works on just about every platform (of course, you don't get SSH on Windows by default without installing something like OpenSSH, but thats another discussion). Very easy to work around this problem in a trusted, enterprise, environment by leveraging admin shares. Not necessarily as convenient, but there you have it.

It may seem like I'm bashing PowerShell (I meant to do that, then pun, that is) but I really am not. Its a very powerful addition to any Windows administrator's tool belt.  If you work on the Windows platform, you owe it to yourself to pick it up.  The ebook, "Master PowerShell", from Dr. Tobias Weltner, is an excellent resource to get started.

Saturday, October 16, 2010

Getting Started with WCF Routing Service, Part 1

"Change is inevitable. Change is constant."
  --  Benjamin Disraeli

In software development, everyone expects change.  Change is inevitable. Its constant. Its the law of the land. The Disraeli Principal and all that.  This is true through the development of software. 

Once software reaches maturity, and is released, then our expectations of change shift radically. A new law takes effect, one almost diametrically opposed to the first:

"Users hate change."
  --  anonymous

How we manage that change can make all the difference in the world.

In developing WCF services, we have a new option for mitigating change to a service endpoint, contract or behavior: the WCF Routing Service.

The WCF Routing Service is available with Microsoft .NET 4.0 and can be best described as a out of the box soap router for WCF. 

What can I do with a soap router you might ask? 

With a intermediate software router you can perform:

  • Protocol Bridging
  • Content Based Routing
  • Access Control
  • Unified Exception Handling
  • Logging
  • Load Distribution / Balancing
  • just about anything you can think of, both good and bad

Over the next few posts, I’ll be introducing myself to the scenarios supported by the WCF Routing Service and sharing what I find along the way, along with how I might use them in the real world scenarios I’m currently working in.  The features I’m most interested in are:

  • Part 2: Protocol Bridging
  • Part 3: Content Based Routing
  • Part 4: Access Control

If your looking for something now go check out the documentation on the WCF Routing Service

If your looking at how to kick it old school, then check out Building a WCF Router, Part 1 and Part 2 by Michele Bustamante.

Friday, August 06, 2010

[TestMethod] Code Snippet

Writing lots of tests today and I found myself typing a lot of repetitive code just inserting test methods, so I created the ‘tm’ code snippet.

Of course, as I installed it, I realized that there was already a ‘testm’ code snippet to do almost exactly the same thing.  I could/should have modified the original ‘testm’ to follow the naming convention for our tests.  The moral of that story is look before you leap.

This gist:

Will generate the following code snippet:

public void TestSuite_TestCondition_Assertion()


Thursday, April 08, 2010

GetSmarx C# vs. Ruby (and something about Shaving a Yak)

Steve Marx, of Windows Azure fame, recently posted “ Letting the Internet Choose My Wallpaper”, on how he allows the Internet to choose his wallpaper using Windows Azure and Service Bus.  Somewhat cool use of the technology showcased in a very easy to follow manner. 

Of course, the little devil in me thought how easy it would be to write a snippet of code to create an Azure Worker role to randomly change his wallpaper – anything worth doing is worth automating, right? 

The requirements were pretty simple. 

  1. http GET
  2. Parse the xhtml response (thanks, Steve!)
  3. Identify the wall paper links
  4. Randomly select a single link resource
  5. https GET of the resource (e.g.

Then I realized I don’t have the money to shell out just to poke Steve Marx, whom I don’t know.  Then I thought, well, I *do* have VS2010 Ultimate RC1 running, including the Testers edition (which rocks, btw), so then I thought I’d just plopped the code into a virtual user (VU) and let it (or multiple VU’s!) run every hour. 

But, you see, I’ve been hearing so much about Ruby, and how what an awesome language it is.  I’ve even started noodling with it on the side (shhh….don’t tell C#).  I do kind of like Ruby, but it can be raw.  Kinda of like how I view Linux – on the surface it looks great, especially now, but it can get rough quick if your anything more than just a casual user.

Anywayz….I then got distracted with creating a snippet of Ruby code to do the same thing I did in C# just to see if life was better. 

So, just like your optometrist…which do you like better?

1)  GetSmarx.cs


2) GetSmarx.rb

Of course, the C# version took about as long as it took to type it.  The Ruby version took much longer as I had to consult the almighty Google for much of it.  Its times like these (Foo Fighters FTW!) that makes the heart all warm when you realize what a rich ecosystem Ruby has underneath it.

And that is how Steve Marx was saved from being sent into an epileptic coma by his constantly changing desktop background due to my being so worn out by Shaving a Yak tonight that I just plain lost interest.

Good night, Steve.  Thanks for the Azure content.  Keep it coming!

Organize XAML Namespace Declarations with XmlnsDefinitionAttribute

Microsoft does it and you can do it, too!

Whenever you need to use a control from an assembly you need to provide an xml namespace for it to avoid any collisions with existing types. By default, you need to specifically reference the namespace for each control you need access to in your XAML.

This might lead to declarations like this in your XAML:


This can get ugly pretty fast if you have lots of controls in your XAML view!

However, notice how clean the Microsoft xml namespaces are? You get a lot of controls brought into scope by leveraging those simple namespaces. This is accomplished by using the assembly XmlnsDefinitionAttribute.


Specifies a mapping in an assembly between an XML namespace and a CLR namespace.

Which means it allows us to provide a facade, or an alias, for a namespace, or group of namespaces within an assembly. This makes it much easier to bring all the controls in a control library into scope.

For example, in the MaryKay.SamPortal.Common.UI library, all of the controls are surfaced through the namespace alias “urn:marykay-samportal-common-ui” using XmlnsDefinition in the assemblies AssemblyInfo.cs:

[assembly: XmlnsDefinition("urn:marykay-samportal-common-ui", "MaryKay.SamPortal.Common.UI.InfoBar.Views")]
[assembly: XmlnsDefinition("urn:marykay-samportal-common-ui", "MaryKay.SamPortal.Common.UI.RoleGroupPicker.Views")]
[assembly: XmlnsDefinition("urn:marykay-samportal-common-ui", "MaryKay.SamPortal.Common.UI.BetterPopup")]
[assembly: XmlnsDefinition("urn:marykay-samportal-common-ui", "MaryKay.SamPortal.Common.UI.TextEditor")]
[assembly: XmlnsDefinition("urn:marykay-samportal-common-ui", "MaryKay.SamPortal.Common.UI.Converters")]
[assembly: XmlnsDefinition("urn:marykay-samportal-common-ui", "MaryKay.SamPortal.Common.UI.Documents")]
[assembly: XmlnsDefinition("urn:marykay-samportal-common-ui", "MaryKay.SamPortal.Common.UI.SplashScreen")]
[assembly: XmlnsDefinition("urn:marykay-samportal-common-ui", "MaryKay.SamPortal.Common.UI.TemplateSelector")]
[assembly: XmlnsDefinition("urn:marykay-samportal-common-ui", "MaryKay.SamPortal.Common.UI.ModalDialog")]
[assembly: XmlnsDefinition("urn:marykay-samportal-common-ui", "MaryKay.SamPortal.Common.UI.ConsultantSearch.Views")]

and so forth..

This cleans up our required XAML declarations to simply:

<commonUI:InformationBar DataContext="{Binding InfoBar}"/>

Which makes our XAML much cleaner and easier to work with. It even supports intellisense!

Wednesday, March 31, 2010

PANDA – Packaging ANd Deployment Automation

Just read “CoApp: An open-source package management system for Windows”.

I really respect what CoApp is trying to do.  It can be amazingly difficult, and tedious, to deploy various applications types on the Windows platform.  Its not like the problem hasn’t been solved.  Its unfortunately been solved too many times, by too many.

Of course, anytime you have something amazingly tedious and difficult it needs to be automated. The Windows platform has needed this forever. 

Unfortunately, what Windows users got, at least for the desktop, was Windows Installer technology.  I never really liked this technology, for a variety of different reasons, but the biggest problem I have with them are that I just don’t care.  I don’t care to see your company’s logo, I don’t care to see a progress bar going across the screen, I don’t care to plug in options – I just don’t care.  I got frustrated that it would take someone with a PhD in Windows Installer technology to be able to produce a package you could install. 

There have been advancements recently, most notably MSDeploy, that seem like they should be able to fit the bill, but they still seem focused on web projects only.  I still have hopes for an extensibility model for MSDeploy that is something we can build on.  The Web Platform Installer is also a huge step forward – again, would be nice if it were opened up, which is where I think CoApp might come in.

You know your data center operations teams have also solved this problem of packaging, deployment, and dependency management, but its still likely to be focused on a couple of notable application types: web applications and Windows services.

I work for corporate America - which has a plethora of bad practices, some I probably perpetuate, and many I'm not proud of, but every once and a while we go and do something worthwhile.  For us, it was the development of PANDA.  PANDA, Packaging and Deployment Automation, is as much process as it is actual tooling.

PANDA actually started with the requirement of a repeatable deployment for web sites, windows services, click once applications and just basic file copies.  To solve this problem, we chose to rely heavily on convention, with a sprinkle of configuration metadata.

The idea of a PANDA ‘package’ was born, and that package had a specific structure.   In addition, it was decided that all applications would have specific target locations, based on conventions set by our operations team.   A PANDA package, being heavily file centric, might look like this:

  • /WindowsServices/{service}
  • /Web/{website}/{virtual root}
  • /ClickOnce/{app}
  • /Config
  • /Workflows
  • /Files/{bucket}

This package structure, combined with some magical application metadata, would allow us to write generic MSBuild scripts that would simply know how to execute what was in the package.  Samples of application metadata for a Windows service might be:

  • service name
  • display name
  • service account
  • executable
  • startup type
  • restart properties

The Windows service developer would include in their project, not an Installer project, but simply an xml file with a well known schema – a .deploy file.  In this case, it would be service.deploy.  Changing any of the properties of the service simply meant modifying the service.deploy file and executing the deployment.  The actual application code didn’t need to be recompiled each time.

Other application types would have their own metadata schema.

This PANDA package is then what all of our software builds produce.  We use Team Foundation Server, but instead of regular output from a build, build output gets rolled into a PANDA package.  If a developer needs to install those bits, either on their local machine, or a server, the experience is the same – they execute the deploy.bat found in the package.

Assuming that the deployment package had a consistent structure, we then built MSBuild targets that knew how to install whatever happened to be in the package.  Our deployment tools, and scripts, really just reused some community task libraries and a single custom library that was essentially a data access layer for our PANDA package.

The overall execution of the deployment had just a few high level MSBuild targets:

  • PreDeploy – users could provide their own MSBuild scripts, following the convention of *.predeploy and PANDA would execute them
  • StopServices – can’t copy over files in use
  • CopyFiles – a top-level target that….copies files, it actually chains together other file copy operations
    • CopyConfig, CopyServices, CopyWebs,CopyClickOnces,CopyWorkflow
  • Uninstall Services – it was easier to uninstall services
  • Install Services – since we uninstall, we pick up any changes to metadata (e.g. service name)
  • Start Services – respected the startup type in service metadata
  • PostDeploy – users could provide their own MSBuild scripts, following convention of *.postdeploy

Each target was smart enough that if there was nothing in the package that matched its criteria, then it didn’t do anything.  And that's it.  The MSBuild scripts just know how to execute the package, which has a consistent structure. 

Here is the breakdown of *.deploy files:

  • service.deploy – metadata for Windows services
  • site.deploy – web site metadata
  • apppool.deploy – app pool metadata
  • webapp.deploy – virtual directory metadata
  • clickonce.deploy- click once metadata
  • files.deploy – files manifests.  Only required for copying files that didn’t already fall into an application archetype – usually one off

A PANDA package could contain a single file, or single set of application components, or it could contain and aggregate of many application components, or indeed entire systems – all in a single PANDA package that has a one click installation (deploy.bat).

Other features of PANDA (not exhaustive):

  • application metadata can specify arbitrary target server role (e.g. ‘web server’, ‘app server’, ‘mylabenvironment’), so application bits only get copied and installed if target server role matches
  • environmental overrides – e.g. use different service account name for DEV, QA, Staging and Production environments.
  • use of hashed passwords – we request provisioning of service accounts by our security group.  We don’t actually receive a password, but the hash.  PANDA will decrypt the hash and install using actual accounts.  Passwords aren’t in the open and developers could care less what the actual value is.

Some notable PANDA shortfalls:

  • PANDA packages are not zipped.  We have the actual step to insert into the packaging process to zip up the bits, we just aren’t using it because we are blessed with near infinite bandwidth in our data centers.
  • PANDA tools need to be deployed to any server, or desktop, prior to attempting to execute a PANDA package.  We would like to eventually include the tools required for THAT package version

I didn’t realize I had typed this much until just now.. My intention was not really to toot the horn (toot! toot!) on PANDA – rather pointing out the need to have roll something like PANDA at all

Granted, PANDA is in its infancy.  My preference is that it stay there and not any more time be spent on enhancing it.  It meets the current needs of several different teams.  I’d rather it was killed off by something supported by either Microsoft, or the community – we don’t want to maintain it indefinitely, but probably would with no other choice.   I’ve often thought about approaching our management team about releasing PANDA into the community in hopes it would gain support there.

Would anyone else use something like PANDA?

Wednesday, February 10, 2010

@ntdebugging Q&A Question Opportunity

The team responsible for the  Advanced Windows Debugging and Troubleshooting blog is looking for your “burning debug questions” that they hope to answer in an upcoming Q&A article.  You can tweet your questions to @ntdebugging for consideration for the article.  If your remotely interested in Windows debugging, and anything debugging related, I highly recommend checking them out!

Unfortunately, I am finding it particularly hard to frame my question into 140 characters this morning, and short of a vertigo-like experience where I realize “I know Kung-Fu”, I thought I’d post my question here in hopes to get my question answered by the team that does know kung fu.

Realize that anything posted here, especially if false, is due to my imperfect understanding of experiences and knowledge imparted by Advanced Windows Debugging and Troubleshooting, Tess Ferrandez (and her many compatriots!), Jeffrey Richter,  and the Godfather of the Debugger, John Robbins

The Setup

.NET allocates memory for each generation’s heap (0, 1, 2, LOH) in segments to get a continuous block of memory when it attempts to satisfy an allocation request, after a collection, and can’t.  This memory for each heap will likely level off as the application “warms up”, except potentially for generation 2, and large object heap.  During a garbage collection, each heap (0, 1, 2) is swept and compacted, except for the large object heap (LOH), which is just swept. 

I understand the ‘sweep’ part of a collection to mean that the GC identifies which objects are no longer rooted and are available for collection (or finalization) and that ‘compact’ means that the addresses that are still alive in a heap are reorganized so that the heap has more continuous memory available to it.   As the budget for each segment within the heap is exceeded, .NET will allocate another segment in order to fulfill allocations if it can.

The Question

My question comes down to what happens to that memory in each heap, that might not be used by the application (committed) any longer, but is still reserved by .NET?  When is it released back to the OS? 

I’ve heard of rituals such as forcing Windows itself into low memory condition so that it asks each process to give up any reserved memory, or minimizing a process, and then restoring it, which essentially does the same thing.  These are just rumors and I’d prefer to know. 

There are also rumors of something involving chickens, but I’d rather not go there…no one likes rituals, they are unreliable and often messy (think of the chickens!).