New Fun Blog – Scott Bilas

Take what you want, and leave the rest (just like your salad bar).

Automatically Choose 32 or 64 Bit Mixed Mode DLL’s

with 25 comments

Máncora

[Update: Stefan posted in a comment below a much simpler method. I recommend going with this instead of my considerably more complex method.]

Here’s a problem that was bugging me at my last job that I finally got around to solving last week on the bus: letting your top-level project run as AnyCPU and automatically choose a 32-bit or 64-bit low level  native/managed DLL based on environment bit width.

Say you’re using Shawn Hladky’s great P4.Net project (source) so your C# can speak Perforce – two great tastes that taste great together. P4.Net talks to the server using a native Perforce API. Now, unlike a C# EXE, which is usually AnyCPU (i.e. “let the jitter decide”), the native code that talks to the Perforce server must be compiled as either x86 or x64. This causes two big problems.

The first problem is that your app will crash with a confusing error if run on the wrong version of the .Net Framework! Say you’re using the x86 P4.Net and you run your app on Win7-x64. The jitter will compile the app in 64 bit, but on the first reference to P4.Net, it tries and fails to load the x86 DLL, and puts up an unhelpful error about a bad image format.

The easy workaround to this, of course, is to mark your EXE as x86 instead of AnyCPU. Solved. Unfortunately, you have to remember to do this with every single app that references P4.Net. Or references a DLL that indirectly references P4.Net. It’s like a virus in that way, but we can handle it.

Well, no. That leads to the second problem. What happens if you reference P4.Net (directly or indirectly) into an app that really does need 64 bit? Like, say, some memory-hoovering game build related tool that runs on the server farm? Well now you need a 64-bit version, not only of P4.Net, and not only of your EXE, but of every single DLL that is referenced on the path down to your new P4.Net_x64.DLL. Now we have a real problem. The pain in the ass to maintain kind of problem. Do we really want to have our tool chain output 32 and 64 bit versions of everything, just in case?

Had Microsoft supported fat binaries like NeXTSTEP did back in the early 90’s, we’d just have P4.Net with 32 and 64 bit in the same DLL, and go on with our lives. But no, we have to jump through hoops. This article is the story of how to jump through those hoops to make your bits go.

I’m actually a bit shocked that Microsoft hasn’t extended PE and their OS loaders to support fat binaries. There would be zero perf cost, and it’s been a long time since we had to worry about the size of binaries on disk (content overwhelms executable code size in nearly every app today).

About P4.Net

If I’m going to continue to use P4.Net for my example, I need to give a little more background. P4.Net is built from three components:

  1. p4api.lib: A native C++ API (headers and libs) provided by Perforce to talk to their server directly through sockets, without running p4.exe or using the COM object.
  2. p4dn.dll: A bridge assembly that statically links in p4api, and uses Managed C++ to export p4api as a low level set of .Net types.
  3. p4api.dll: A managed C# API that wraps up the low level p4dn and adds functionality to make it easier to work with. This is what everybody does an “add reference” on to talk to P4 from C#.

Note that p4api.dll is not strictly necessary as a separate assembly. The low level types exported by p4dn.dll could instead be kept internal, and all of that C# code from p4api.dll be written in Managed C++ and moved into p4dn.dll, entirely eliminating the need for p4api.dll.

Personally I was a fan of Managed C++, up until C# 3.0 where we started getting all kinds of nice language syntax to write better code more compactly. Today, I suppose I’d keep the extra DLL just for easier maintenance.

The Solution

In a nutshell, the solution is to trick the loader! Reference a p4dn.dll that does not exist, and use the AssemblyResolve event to intercept the load and reroute it to the correct bit size assembly.

It’s simple in concept but has a lot of details that took me a whole bus ride to figure out all the way (happily, there was a lot of traffic). Here is what I ended up doing to make it work how I wanted:

  1. Rename the x86 output of p4dn.dll to p4dn.proxy.dll.
  2. Update the x86 linker input settings to add __DllMainCRTStartup@12 to Force Symbol References.
  3. Build a new x64 configuration for p4dn, using the x86 configuration (well, ‘Win32’) as a template. Have it output to p4dn.x64.dll.
  4. Update the x64 linker input settings to add _DllMainCRTStartup to Force Symbol References.
  5. Add a post-build event to p4dn’s x86 configuration that deletes p4dn.x86.* and copies the p4dn.proxy.* to p4dn.x86.*.
  6. Update p4api to reference p4dn.proxy.dll. Not the csproj, but the actual DLL.
  7. Update the SLN settings to make p4api dependent on p4dn.
  8. Add a post-build step to p4api to delete p4dn.proxy.dll.
  9. Set all p4api and p4dn configurations to output to the same bin folder.
  10. Add a static constructor to P4API.P4Connection that registers an event handler on AppDomain.CurrentDomain.AssemblyResolve to pick the right DLL when the proxy is requested. I’ve pasted my code at the bottom of this post.

Once this is done, you’ll be able to have p4api as well as any assemblies that reference it set to AnyCPU. It will, upon first usage of the P4Connection class, fail to resolve the proxy and reroute to the correct bit width DLL.

A few notes on the above:

  • The _DllMainCRTStartup is required because, without it, I got a crash from uninitialized memory systems in the CRT DLL’s that p4dn was linked to. This happened regardless of static vs. dynamic linking. I didn’t bother to find out the real reason for it. The different symbol names for 32 bit vs. 64 bit are because the convention changed when Microsoft went to 64 bit.
  • The name of the DLL being referenced must match the original name of the DLL being built. That is, if you were to have p4dn outputting to p4dn.x86.dll then renaming it to proxy, and then referencing that, then it will actually look for the referenced DLL’s “true” name of p4dn.x86.dll and never call your hook.
  • In projects that reference p4api it’s best to set the references to non-private (clear the “Copy Local” flag) and have a post-build step that just copies whatever is in the p4api bin folder. That makes sure you get the exact files that you need. None of this will work if you accidentally end up with the proxy file existing.

Here’s the code for my hook function. Note that it attempts to catch problems with the post-build scripts.

[code lang="csharp"]
static P4Connection()
{
string assemblyDir = Path.GetDirectoryName(Assembly.GetExecutingAssembly().Location);
if ( File.Exists(Path.Combine(assemblyDir, "p4dn.proxy.dll"))
|| !File.Exists(Path.Combine(assemblyDir, "p4dn.x86.dll"))
|| !File.Exists(Path.Combine(assemblyDir, "p4dn.x64.dll")))
{
throw new InvalidOperationException("Found p4dn.proxy.dll which cannot exist. "
+ "Must instead have p4dn.x86.dll and p4dn.x64.dll. Check your build settings.");
}

AppDomain.CurrentDomain.AssemblyResolve += (_, e) =>
{
if (e.Name.StartsWith("p4dn.proxy,", StringComparison.OrdinalIgnoreCase))
{
string fileName = Path.Combine(assemblyDir,
string.Format("p4dn.{0}.dll", (IntPtr.Size == 4) ? "x86" : "x64"));
return Assembly.LoadFile(fileName);
}
return null;
};
}
[/code]

Microsoft, if you’re listening: FAT BINARIES.

April 18th, 2010 at 8:39 pm

Posted in .net,p4

25 Responses to 'Automatically Choose 32 or 64 Bit Mixed Mode DLL’s'

Subscribe to comments with RSS or TrackBack to 'Automatically Choose 32 or 64 Bit Mixed Mode DLL’s'.

  1. [...] my last post I wrote about building AnyCPU assemblies that can reference 32 or 64 bit DLL’s automatically [...]

  2. Thanks for the great article, Scott! Solved my problem wonderfully. I had a third scenario: I wanted my utility app to JIT to the local platform because I needed it to read the correct registry key (i.e. the real one if it was running a 64-bit version of the app, the WOW6432 one if it was running 32, and the 32-bit one if it was running under XP).

    I think Step 5 is incorrect: it should delete and move p4dn.x86.*, not p4api. Also, with versions of the perforce libraries greater than 07.2 I was getting linker errors. Either my P4API code is old (unlikely, I just downloaded it today), or there’s some kind of maintenance issue. I notice your article is from this year. Did you not have that problem at all?

    Again thanks, this saved my day!

    James Goldman

    20 Jul 10 at 7:02 pm

  3. Hi Scott,
    Thanks for posting this. I found myself crossing much the same paths as you did. I also work for a game company, which is also using a C# tool with P4.Net, and I have found myself with the task up upgrading this tool to work in 64 bit. Your post really helped me through some of the troubles of trying to build this library for 64 bit. Especially the part about _DllMainCrtStartup vs __DllMainCrtStartup@12. It probably would have taken me at least then entire day to get rid of that linker error, had you not posted the answer. I’m terrible when it comes to such low level C++ matters as entry points, calling conventions, and getting rid of linker errors when there are conflicting dependencies (all that “msvcrtd” business). If you can explain or point me to an explanation of this __DllMainCrtStartup business and why you need to “Force Symbol Reference” it, it would be much appreciated.
    Thanks Again!

    Mike Cline

    28 Jul 10 at 9:39 pm

  4. I tried implementing the 32/64 bit dll loading in my app via the AssemblyResolve event. Unfortunately, I’m crashing with a FileNotFoundException (“Could not load file or assembly”) before I ever have a chance to register my event handler (even though I register it at the top of “static void main”.

    I tried reproducing this behaviour in a small test app (I followed the instructions in this link http://stackoverflow.com/questions/108971/using-side-by-side-assemblies-to-load-the-x64-or-x32-version-of-a-dll which uses the same kind of technique) and could not reproduce the problem — the AssemblyResolve event always got hit if the DLL was not present.

    I’m not sure what the difference between my app and the test app is, or why it’s loading dll’s before main in the broken app.

    I’ve found a few other postings online where people complain of getting the same FileNotFoundException before their AssemblyResolve event is registered, and some of them laim that their code worked in VS2005 and not VS2008, so perhaps there is some bug they have introduced.

    Mike Cline

    29 Jul 10 at 3:16 pm

  5. this can be resolve in config file like:

    Ioan Toader

    9 Aug 10 at 8:09 am

  6. Ioan Toader

    9 Aug 10 at 8:11 am

  7. I cannot past xml code :(

    Ioan Toader

    9 Aug 10 at 8:11 am

  8. Everybody – my apologies for the late response.

    Ioan -

    Try wrapping your xml in code blocks, perhaps that would work. There’s instructions on how to do this at the bottom of the comment post area.

    James -

    Step 5 is correct, at least the way I’m doing it. I did mess up the naming of p4api vs. p4dn in the first version of this article though, and I went back and fixed that just now.

    The point of the proxy rename copy delete stuff is to have p4api reference a nonexistence proxy DLL, which redirects to the correct x86 or x64 based on platform. Easiest way to do this is output the proxy and use that during builds as a reference source, then post-build rename it to x86. In my final version I added a bit of code that actually checks to make sure that the proxy DLL does not exist in the same place the resolver is installed, to avoid accidentally using x86 from x64 systems if a build script is messed up – the event handler will only get called if the proxy does not exist.

    Regarding the versions, I did pull down the latest C++ API and latest P4.NET when I was figuring this out. The versions are from roughly the time as my posting of the article. What kind of linker errors were you seeing with the newer versions?

    Mike -

    I’m sorry but I never figured out why force referencing __DllMainCrtStartup resolves the memory system initialization issue. I remember that it was the end result of a lot of frustrated web searches and hacking and tweaking and messing with compiler settings. Once I got it working I didn’t look back, sorry.

    Regarding your FileNotFoundException, do you have first-chance exceptions set to catch it? If you run with the reference source wired up to your Visual Studio you will be able to catch it in the .NET Runtime at that exact point it fails, and look around in MS’s code to see what it’s doing to lead to this.

    But I’m betting the problem is that you are referencing the type in code before it has a chance to install the resolver event handler. Remember that .NET must pull in all referenced types before it can JIT a function. I’m guessing that you are using one or more types in your missing DLL from your static main function. If that’s the case, try this: move all the code in your static main except for the assembly resolve event installer to another static function in the same class. Then call that static function from static main after installing the event handler.

    Try messing around with a sample project with a dependent assembly and different combinations of type references and functions, watching when modules get loaded based on code that runs. This is an easy way to get familiar with how the loader works. Most of the time we don’t have to worry about this, but when altering how the loader works via events, it’s a good idea to pick up a few of the rules.

    Scott

    31 Aug 10 at 7:33 am

  9. Sadly, after messing around with trying to do it the right way for far too long, I decided to go with separate project files for 32 and 64 bit. This is not quite as terrible as it sounds since we auto-generate all of our csproj files anyhow (I don’t have to actually maintain two separate versions by hand). But it’s still pretty nasty.

    The other thing that I find quite lame, along the same lines, is that Visual Studio doesn’t make it very easy for you to have different debug vs release dll’s. So, if I build my solution in debug it writes over all of release dll’s and vice-versa. Therefore if you are switching back and forth between debug and release you are waiting for rebuilds more than you should have to. I ended up solving it much the same way — by having separate project files for debug vs release that contain references to DLL’s in different locations.

    -Mike

    Mike Cline

    31 Aug 10 at 3:23 pm

  10. Generating csproj files: thumbs-up! We did the same at my last gig for vcproj’s, it was nice. Never got around to updating the tool to generate csproj’s.

    Regarding debug vs. release.. At my current studio, we have stopped building release .NET assemblies almost entirely. Only in a couple very key performance sensitive assemblies do we bother with an optimized build (and then only for the version that is checked into the depot). The extra headache of two versions just isn’t worth the generally unnoticed perf boost. Especially with something like Perforce where 99.99999% of the processing time is spent waiting for packets to fly around the network.

    Scott

    31 Aug 10 at 4:00 pm

  11. I read the post fairly quickly so I might be missing something — but why not simply use paths instead? We put certain native DLLs in an x86/x64 directory and add the directory corresponding to the current execution environment to the PATH at startup. We haven’t really seen any problems with that approach so far…

    Stefan Boberg

    2 Sep 10 at 4:59 am

  12. Haha wow. That is 10x simpler than my approach. I cant think of any problems with the path method either. I wish I had thought of it before I went through this complicated mess!

    Thanks for the trick, I’ll update my post later today.

    Scott

    2 Sep 10 at 5:56 am

  13. @Stefan
    If I understand your approach correctly, it implies that the native DLLs have the same name regardless of the target platform.
    Concerning the extension of the PATH content I assume that the additional path should be put at the beginning of PATH as the other path may be already included (e.g. suggest the x86 dir is in PATH per default or per administrator setting the x64 dir must be put at the start of PATH to ensure correct loading).
    Am I wrong with this?

    @Scott
    I have to deal with 2 native DLL with suffixes _x86 and _x64 but in .NET 2.0. Can you outline what

    AppDomain.CurrentDomain.AssemblyResolve += (_, e) =>

    corresponds to? Thanks in advance.

    fgrt

    21 Sep 10 at 5:08 am

  14. Regarding PATH – I’d expect it would not matter where it goes in the path order..in fact I might even write some code to require that the x86/x64 subfolder is not in PATH at all, so that we are very explicit about it. There’s no reason for a user to have either of these in the system path – and having a default of x86 I think would be confusing and harder to debug when problems occur. The .Net “Fusion” stuff for sxs loading is a pain..have to enable a reg key and look at log files. Easier if we test in advance for the conditions we are expecting: x86 and x64 subfolders exist and have the DLL we are expecting, and that neither is in the system path already. Then throw a very specific message about what’s wrong.

    Regarding AssemblyResolve, I’m not sure I understand your question. Couldn’t you just adapt the AssemblyResolve event handler in my original post? Or is your question about .NET 2.0? I’m using an inline delegate but you could easily convert it to a member function instead. Is that what you mean?

    Scott

    21 Sep 10 at 8:59 am

  15. Yes, of course, I am just unaware of how (_, e) =>, I guess it is called a lambda expression, is to be converted into .NET 2.0 C# code; i.e. what is _, what is e, to what do the brackets () correspond to and at least what does => do. As far as I know lambda expressions are only a language specific shortcut for handwritten anonymous delegates.

    fgrt

    22 Sep 10 at 12:54 am

  16. It’s actually a compiler feature, not a runtime environment feature. You can take VS2010 and target .NET 2 yet still use lambdas. Once the anonymous delegate feature was added in the 2.0 runtime, all kinds of new things became possible.

    An example of a 4.0 feature that really does require the 4.0 runtime is covariant/contravariant types. Or extension methods in 3.0.

    Anyway, here’s some code with examples of all the types of callbacks I can think of. If I target 2.0 in my VS2010 these all still compile fine. So it’s really a question of which version of VS you’re using. Or, if using the command line compiler that comes with .NET, obviously you’ll be limited to the language level of the C# shipping at the same time.

    [code lang="csharp"]
    static P4Connection()
    {
    // old skool style
    AppDomain.CurrentDomain.AssemblyResolve +=
    new ResolveEventHandler(Resolve);
    AppDomain.CurrentDomain.AssemblyResolve +=
    Resolve;

    // anonymous delegates - much better
    AppDomain.CurrentDomain.AssemblyResolve +=
    delegate(object sender, ResolveEventArgs args)
    { throw new NotImplementedException(); };

    // lambdas - praise his noodly appendage
    AppDomain.CurrentDomain.AssemblyResolve +=
    (sender, args) =>
    { throw new NotImplementedException(); };
    }

    static Assembly Resolve(object sender, ResolveEventArgs args)
    { throw new NotImplementedException(); }
    [/code]

    Scott

    22 Sep 10 at 7:40 am

  17. @Scott:
    Thanks again for this post which helped me to do something similar I outlined on ‘social’ MSDN:
    Forums/en-US/vclanguage/thread/82d1e42a-9356-4f14-8d17-21052072ca5a.

    fgrt

    18 Oct 10 at 9:09 am

  18. @Scott: I created a fork of Shawn’s P4.Net on GitHub:

    https://github.com/milang/P4.net

    (the GitHub project is a .NET4-based-followup on my 2009 fork of Shawn’s P4.Net project, http://public.perforce.com:8080/guest/milan_gardian/p4netmerge/, that merged 32 bit p4dn + 64 bit p4dn + p4api assemblies into a single AnyCPU assemly :-) ).

    The GitHub project adds the following to Shawn’s version:

    - Support for .NET 2.0, 3.0, 3.5 (CLR 2)
    - Support for .NET 4.0 (CLR 4)
    - Single AnyCPU assembly — no need for the crazy dance (that we all went through) that you so nicely described in your blog article; instead, download the binaries (https://github.com/downloads/milang/P4.net/P4.Net_2.0.0.2.zip), pick a single P4API.dll (debug or release, .net2 or .net4), add reference to it from your project and that’s it.

    I invite everyone to check out the code, or even fork on GitHub and make further improvements :-).

    Milan

    9 Feb 11 at 9:38 pm

  19. Woo, awesome!

    Scott

    10 Feb 11 at 9:21 am

  20. Hey Scott, just wanted to mention how I ended up doing it at my last job. It was a method very similar to yours, but actually packed both the x86 and x64 p4dn binaries into p4api as resources. I then used the static constructor of the P4Connection to unpack the correct one into the name that the assembly resolver was looking for. This way we were able to bypass the need for the AssemblyResolve event handler while also removing an additional file from our deployment package.

    The assembly resolve succeeded because p4dn was not resolved until it was used – after the P4Connection static constructor ran.

    Greg

    11 May 11 at 5:22 pm

  21. [...] out Scott Bilias’s blog post on this http://scottbilas.com/blog/automatically-choose-32-or-64-bit-mixed-mode-dlls/. Note that he ends up preferring approach [...]

  22. Thanks for Great artical,it’s help me a lot i have convert my project from anycpu to x64 it convert all the dll except one dll that ComEventHandler.dll it get the error in windowsfoms .Is there any solution to resolve this error.

    thanks.

    victor

    6 Aug 12 at 9:20 pm

  23. The problem we have with paths is that somehow 64 bit dlls get copied to the 32 bit dll path or vice versa, causing the same problem all over again.

    Software Diplomat

    21 Jan 13 at 9:03 am

  24. Hi,

    I successfully used you method to link against x86 or x64 mixed mode assembly in my LZ4 compressor for .NET (http://lz4net.codeplex.com/)

    Thanks,

    Krashan

    5 Feb 13 at 3:28 pm

  25. Thanks, great starting point. I tweaked things a bit for my project.

    1. I wanted to include the C runtime dlls as well, which have the same name regardless of architecture. So I created x64 and x86 folders. I then copied the native dll along with the C rumtimes to the appropriate folder and loaded them from there.

    One other tweak: I added the assembly resolver code to my module initializer. Article on this here:

    http://einaregilsson.com/module-initializers-in-csharp/

    I’m not super happy with my project layout but at least everything works.

    Alnoor

    29 Jul 14 at 11:49 am

Leave a Reply