New Fun Blog – Scott Bilas

Take what you want, and leave the rest (just like your salad bar).

Archive for the ‘p4’ Category

Quickie: Building P4.Net

with one comment

In my last post I wrote about building AnyCPU assemblies that can reference 32 or 64 bit DLL’s automatically based on the environment. Just wanted to make a quick followup for people who are actually following my example, specifically trying to implement it for P4.Net. There are a few little snags if you’re not super familiar with native programming.

Make sure you do the following:

  • You must have the 64 bit compilers installed or you won’t be able to add 64-bit configurations to p4dn. Most likely you didn’t check the box for this when originally installing Visual Studio. Just go to add/remove programs, “change”, and check 64-bit under the C++ area.
  • The p4dn library requires Perforce’s p4api native library, which comes in many flavors. It is included in the distribution for P4.Net, but it is an old version. I recommend getting the latest API from their FTP site.
    • I used the p4api_vs2008_dyn and p4api_vs2008_dyn_vsdebug configurations from the ntx64 and ntx86 builds.
    • Why dynamic instead of static? .NET is all about DLL’s – using static just causes problems with the linker with conflicting runtimes.
  • Update the include and lib paths in the p4dn C++ project settings so that the library can find your p4api. If you don’t do this you’ll get #include errors about missing paths.

Written by Scott

April 25th, 2010 at 3:23 pm

Posted in .net,p4,quickie

Automatically Choose 32 or 64 Bit Mixed Mode DLL’s

with 27 comments

Máncora

[Update: Stefan posted in a comment below a much simpler method. I recommend going with this instead of my considerably more complex method.]

Here’s a problem that was bugging me at my last job that I finally got around to solving last week on the bus: letting your top-level project run as AnyCPU and automatically choose a 32-bit or 64-bit low level  native/managed DLL based on environment bit width.

Say you’re using Shawn Hladky’s great P4.Net project (source) so your C# can speak Perforce – two great tastes that taste great together. P4.Net talks to the server using a native Perforce API. Now, unlike a C# EXE, which is usually AnyCPU (i.e. “let the jitter decide”), the native code that talks to the Perforce server must be compiled as either x86 or x64. This causes two big problems.

The first problem is that your app will crash with a confusing error if run on the wrong version of the .Net Framework! Say you’re using the x86 P4.Net and you run your app on Win7-x64. The jitter will compile the app in 64 bit, but on the first reference to P4.Net, it tries and fails to load the x86 DLL, and puts up an unhelpful error about a bad image format.

The easy workaround to this, of course, is to mark your EXE as x86 instead of AnyCPU. Solved. Unfortunately, you have to remember to do this with every single app that references P4.Net. Or references a DLL that indirectly references P4.Net. It’s like a virus in that way, but we can handle it.

Well, no. That leads to the second problem. What happens if you reference P4.Net (directly or indirectly) into an app that really does need 64 bit? Like, say, some memory-hoovering game build related tool that runs on the server farm? Well now you need a 64-bit version, not only of P4.Net, and not only of your EXE, but of every single DLL that is referenced on the path down to your new P4.Net_x64.DLL. Now we have a real problem. The pain in the ass to maintain kind of problem. Do we really want to have our tool chain output 32 and 64 bit versions of everything, just in case?

Had Microsoft supported fat binaries like NeXTSTEP did back in the early 90’s, we’d just have P4.Net with 32 and 64 bit in the same DLL, and go on with our lives. But no, we have to jump through hoops. This article is the story of how to jump through those hoops to make your bits go.

I’m actually a bit shocked that Microsoft hasn’t extended PE and their OS loaders to support fat binaries. There would be zero perf cost, and it’s been a long time since we had to worry about the size of binaries on disk (content overwhelms executable code size in nearly every app today).

About P4.Net

If I’m going to continue to use P4.Net for my example, I need to give a little more background. P4.Net is built from three components:

  1. p4api.lib: A native C++ API (headers and libs) provided by Perforce to talk to their server directly through sockets, without running p4.exe or using the COM object.
  2. p4dn.dll: A bridge assembly that statically links in p4api, and uses Managed C++ to export p4api as a low level set of .Net types.
  3. p4api.dll: A managed C# API that wraps up the low level p4dn and adds functionality to make it easier to work with. This is what everybody does an “add reference” on to talk to P4 from C#.

Note that p4api.dll is not strictly necessary as a separate assembly. The low level types exported by p4dn.dll could instead be kept internal, and all of that C# code from p4api.dll be written in Managed C++ and moved into p4dn.dll, entirely eliminating the need for p4api.dll.

Personally I was a fan of Managed C++, up until C# 3.0 where we started getting all kinds of nice language syntax to write better code more compactly. Today, I suppose I’d keep the extra DLL just for easier maintenance.

The Solution

In a nutshell, the solution is to trick the loader! Reference a p4dn.dll that does not exist, and use the AssemblyResolve event to intercept the load and reroute it to the correct bit size assembly.

It’s simple in concept but has a lot of details that took me a whole bus ride to figure out all the way (happily, there was a lot of traffic). Here is what I ended up doing to make it work how I wanted:

  1. Rename the x86 output of p4dn.dll to p4dn.proxy.dll.
  2. Update the x86 linker input settings to add __DllMainCRTStartup@12 to Force Symbol References.
  3. Build a new x64 configuration for p4dn, using the x86 configuration (well, ‘Win32’) as a template. Have it output to p4dn.x64.dll.
  4. Update the x64 linker input settings to add _DllMainCRTStartup to Force Symbol References.
  5. Add a post-build event to p4dn’s x86 configuration that deletes p4dn.x86.* and copies the p4dn.proxy.* to p4dn.x86.*.
  6. Update p4api to reference p4dn.proxy.dll. Not the csproj, but the actual DLL.
  7. Update the SLN settings to make p4api dependent on p4dn.
  8. Add a post-build step to p4api to delete p4dn.proxy.dll.
  9. Set all p4api and p4dn configurations to output to the same bin folder.
  10. Add a static constructor to P4API.P4Connection that registers an event handler on AppDomain.CurrentDomain.AssemblyResolve to pick the right DLL when the proxy is requested. I’ve pasted my code at the bottom of this post.

Once this is done, you’ll be able to have p4api as well as any assemblies that reference it set to AnyCPU. It will, upon first usage of the P4Connection class, fail to resolve the proxy and reroute to the correct bit width DLL.

A few notes on the above:

  • The _DllMainCRTStartup is required because, without it, I got a crash from uninitialized memory systems in the CRT DLL’s that p4dn was linked to. This happened regardless of static vs. dynamic linking. I didn’t bother to find out the real reason for it. The different symbol names for 32 bit vs. 64 bit are because the convention changed when Microsoft went to 64 bit.
  • The name of the DLL being referenced must match the original name of the DLL being built. That is, if you were to have p4dn outputting to p4dn.x86.dll then renaming it to proxy, and then referencing that, then it will actually look for the referenced DLL’s “true” name of p4dn.x86.dll and never call your hook.
  • In projects that reference p4api it’s best to set the references to non-private (clear the “Copy Local” flag) and have a post-build step that just copies whatever is in the p4api bin folder. That makes sure you get the exact files that you need. None of this will work if you accidentally end up with the proxy file existing.

Here’s the code for my hook function. Note that it attempts to catch problems with the post-build scripts.

[code lang=”csharp”]
static P4Connection()
{
string assemblyDir = Path.GetDirectoryName(Assembly.GetExecutingAssembly().Location);
if ( File.Exists(Path.Combine(assemblyDir, "p4dn.proxy.dll"))
|| !File.Exists(Path.Combine(assemblyDir, "p4dn.x86.dll"))
|| !File.Exists(Path.Combine(assemblyDir, "p4dn.x64.dll")))
{
throw new InvalidOperationException("Found p4dn.proxy.dll which cannot exist. "
+ "Must instead have p4dn.x86.dll and p4dn.x64.dll. Check your build settings.");
}

AppDomain.CurrentDomain.AssemblyResolve += (_, e) =>
{
if (e.Name.StartsWith("p4dn.proxy,", StringComparison.OrdinalIgnoreCase))
{
string fileName = Path.Combine(assemblyDir,
string.Format("p4dn.{0}.dll", (IntPtr.Size == 4) ? "x86" : "x64"));
return Assembly.LoadFile(fileName);
}
return null;
};
}
[/code]

Microsoft, if you’re listening: FAT BINARIES.

Written by Scott

April 18th, 2010 at 8:39 pm

Posted in .net,p4

Upcoming Perforce Feature: Shelving

with 3 comments

Perforce announced on their blog that they have a new feature in beta testing called “shelve”. Great!

In short, “shelving” is taking a pending changelist and publishing it to the server without checking it into the depot.

As I mentioned in a previous post, this is one of the top features from git that was making me seriously consider checking it out. I started implementing this myself in my p4.exe wrapper, but I never gave it enough priority to get much work done. I’m excited to know that Perforce are adding it themselves. This is the first new feature I’ve seen from them in years that is worth upgrading for!

The things that interest me most about this ability are:

  • Easy trading of in-progress work in a formal, server-supported way that is done out-of-band of the depot, so it cannot break the rest of the team. Good for code reviews, small tests, moving the same pending changes among multiple machines/clients, etc.
  • Easy, incremental backup of works in progress. This is hot.
  • Easy ability to temporarily put aside a local change in order to work on something else that collides (typically as an emergency). This eliminates the main reason I needed to use several P4 clients on the same machine.

I hope Crucible will have support for this great new feature after it’s released. That would eliminate most of the need for the tool I wrote.

I’m looking forward to seeing this new feature in 2010.

Written by Scott

December 22nd, 2009 at 1:37 pm

Posted in p4