Friday, 12 June 2009

Project assembly reference fun and games

We just had an issue with the build of one of our applications on our CI server. The build in question no longer contained an assembly in the bin folder where previously it had.

Pinpointing the issue

At first I couldn’t understand why the assembly was not in the bin folder as the assembly was clearly marked in Visual Studio as “Copy Local = true”. I then looked through the build history to see when the assembly was last published but instead of finding a point in time that it stopped working I found that the issue only occurred on one of our build agents. Two of the three build agents correctly publish the assembly but the other one did not.

The next step was to compare the configuration of the three build agents. The difference was that the two build agents that worked did not have the specific assembly in the GAC where the failing one did. But hang on a minute “Copy Local = true” should still copy the assembly into the bin folder whether referencing a assembly on the file system or in the GAC (try setting the System assembly to “Copy Local = true”).

Investigating the Issue

My first thought was to remove the assemblies from the GAC on the failing build agent but the voice in my head kept saying “This should work dam it” so instead I cracked open the project file. The first thing I noticed was that on some of the assembly references there was an inner element called private which was set to true where as others, including the affected assembly, had no element:

<Reference Include="Ass1">
<Reference Include="Ass2">

BUT they both show up Visual Studio as “Copy Local = true”. So to test how the project file works I set the System assembly to be local as well and sure enough the inner element “Private” with a value of true appears in the project file where before there was none. As expected when built the System.dll was output to the bin folder. However the curious thing was after reverting the System assembly back to “Copy Local = false” the element was not removed instead the value was changed to false.
So my final step was to add the “Private” element to the affected assembly reference and set its value to true, commit the changes to code repository and force a build on the build agent in question and low and behold it worked.

To see how the project file got into this state I tried adding assembly references to a dummy project where some assemblies were in the GAC and some were not. My results show that if you add an assembly reference from the file system then the Copy Local value is implied as no Private element is added to the assembly reference. The Visual Studio UI will show a value of false if the assembly is in the GAC and true if it is not. What this means is that the build output will be different for different environments.

The work around

The only way round this is either to edit the project file manually or to set the Copy Local value to false and then back to true again when adding a reference to the project which you do want included in the build output.


  1. Thanks so much for discovering this, I've been battling with it all day! Unfortunately I've found that this is only true for the executing assembly. If a referenced assembly uses a library that is registered in the GAC, there is nothing you can do to get it in the output directory. The only solution is to reference it from the executing assembly and then follow the instructions in the post.

  2. Stefan, Thanks for pointing that out.

  3. What about the "Targer framework"???
    If in your main project you have ".NET framework 4 Client Profile" and in the other projects just .Net framework 4 some of those will not be included.

    If you are targeting the .NET Framework 4 Client Profile, you cannot reference an assembly that is not in the .NET Framework 4 Client Profile. Instead you must target the .NET Framework 4.

  4. If an assembly is in the GAC, why would you ever want it copied locally? THat's kinda defeating the purpose of the GAC and it's benefits (side-by-side versioning, reducing memory footprint of apps by reducing "private bytes", security, CAS, etc.)

  5. I disagree DaveBlack. There are times where you need to deploy to a server you don't have control over and reference a library that is not in the server GAC. You can deploy an ASP.NET MVC site to a server without mvc if you bring along the proper dll's.

    Thanks for this tip Matt, wish it were not specific to the executing assembly as Stefan states.

    1. No worries Ryan, glad I could be of help.

  6. Dave sorry I never saw your post so haven't replied till now. The issue you have is in some cases, as Ryan has pointed out you do not have control over the environment you need to run in.

    Don't get me wrong the GAC is fine for some things but I am a great believer in that your code should compile and run on any machine with the least amount of involvement.

    The second thing is that your CI environment should be as clean as possible. The more things you install into the GAC the more potential there is for a knock on effect to something else breaking. For example if there is an app that requires xyz and it is installed in the GAC on the development and CI machines but that component is not allowed to be installed for some reason on a target machine you will not find out that it is missing until potentially it hits the Beta or Live environments.

  7. I agree with your points but only in the case of a client app. For a server app, you should have control over the environment. If you don't, then you should question why people think the environment(s) exist(s). Server Environments should serve the apps that run on them - period.

    Now there is always the competition between network engineering and development. That kind of control is not what I'm referring to here. I'm talking about application requirements. For a MS web app to run, IIS needs to be installed and configured in some manner. You may need WAS or MSMQ, 3rd party components, etc. These are effectively "environment requirements" of the application. If it runs on .NET 4.0, then .NET 4.0 needs to be installed, etc. If the app requires a service pack be installed on a server, then the SP should be installed. I agree with the KISS principle and simplicity is key.

    The biggest problem with your solution to copy locally is that it does not scale. I've worked for several very-large and well-known clients (whose names everyone in IT knows) where some apps had locally copied binaries. I'm talking about several dozen websites, web services, etc. This led to nothing but problems with deployment, configuration, versioning, memory footprints, and many obscure application errors that are red herrings as to the real cause. My solution was to move everything into the GAC. The only assemblies that I couldn't do this for were those for which we couldn't obtain a strong-named assembly. This solution solved the problems and made the deployment and configuration for each app much simpler. Overall memory consumption was reduced and performance was improved.

    Unfortunately, what usually ends up happening is that the tail wags the dog...development has to make app or deployment concessions because of some ridiculous constraint placed by the network engineering group. The whole reason for the servers is to run the apps - not the other way around...

    Just my $.02

  8. Thanks, this post just saved my day!

  9. Nice article. The copy to local as true will cause a lot of issues many times, such as API lib conflicts, loading issues, and so on. By the way, the Visual Smarter has a lot of widgets which can set reference properties automatically and can also help copy, add, remove, and update references automatically. The Multi-Reference Tweaker utility is also nice to try.