Random Encoding

Fody is a .Net weaver library that allows you to modify your compiled assemblies at build-time. It gives you most of the power of Reflection.Emit (aside from being able to generate dynamic code on arbitrary types at runtime) but without the dependency on dynamic runtime code generation. This is extremely important if you want your solution to work in Xamarin.Ios. Fody is a great tool, but writing the weaving code via the Mono.Cecil API can be frustrating due to a number of idiosynchracies with Mono.Cecil vs. Reflection.Emit. This blog post is an attempt to identify some of the most common problems you run into and what the solution might be.

Your Friends

PEVerify.exe Located in c:\Program Files (x86)\Microsoft SDKs\Windows\v10.0A\bin\NETFX 4.6 Tools (for Windows 10). This tool does a great job of identifying precisely where in your bytecode you have a problem (it gives you an offset which you can correlate with a decompiler such as ILSpy (see below)), and gives you a starting point from which to analyze your IL generation for problems.

ILSpy This is a decompiler. It's ideal (over, say, dotPeek) because it allows you to view the raw IL bytecode. Since this is what you're generating, it's far superior over a decompiler that only shows you C#.

ildasm.exe Located in c:\Program Files (x86)\Microsoft SDKs\Windows\v10.0A\bin\NETFX 4.6 Tools (for Windows 10). This is kind of like an IL-only version of ILSpy. Graphically it looks like it was written in the 90's, but it's far more accomodating of bad IL than ILSpy (which often throws exceptions when viewing corrupt assemblies). Thus if ILSpy fails for you, this is a good backup that allows you to still view the IL.

Member 'XXX' is declared in another module and needs to be imported

Whenever you get an error like this, it means you need to call ModuleDefinition.Import on that type or other member. Furthermore, you should avoid using TypeDefinition and MethodDefinition declarations and instead use TypeReference and MethodReference. When you need the definitions, you can call .Resolve() at that time. The reason this helps is that when you call Resolve(), the reference is always from the current module (keeping in mind that MethodDefinition implements MethodReference -- but it's a different reference). In contrast, using .Import preserves the details of how the reference is to a symbol in another module.

PEVerify "Unable to resolve token"

The Unable to resolve token error is a very common error, and diagnosing the precise problem can sometimes be tricky.

One of the most common scenarios in which you will encounter this error is when you try to instantiate a generic type when using a constructor that is bound (or obtained from) the unconstructed generic type. For example:

var func2Type = ModuleDefinition.Import(typeof(Action))

Create a generic version of it with concrete type parameters:

var int32 = ModuleDefinition.TypeSystem.Int32; var constructedFunc2Type = func2Type.MakeGenericInstanceType(int32, int32);

Intuitively, you'd think you could now get the Invoke method by getting it from the constructed type:

var invokeMethod = constructedFunc2Type.Methods.Single(x => x.Name == "Invoke");

However, this will fail. This is because the constructor you get from the constructed type is the same as you get for the unconstructed type. You must use GenericMethodDefinition instead:

It's important to call ModuleDefinition.Import on the reference, not the resolved reference (i.e. don't call .Resolve() to the argument you pass to Import()).

PEVerify "initlocals must be set for verifiable methods with one or more local variables."

If you're thinking in terms of C# and you have a class defined like this:

public class Outer<t> { public class Inner { } }

Intuitively you'd think that the type Inner defines no generic parameters, but implicitly has access to the generic parameter T defined in the enclosing class. This is true so far as C# goes, but is not how the type is actually represented by the CLR.

Under the hood, nested types do not know anything about the generic parameters of the enclosing type. Instead, the actual Inner class that is generated by the compiler actually re-declares type T so that it's better to visualize these types like:

public class Outer<t> { public class Inner<t> { } }

While if this were the actual C# code, you'd get a warning that the type parameter T on Inner<t> hides the type parameter in Outer<t>, in practice, this is how the type actually exists.

In other words, the notion that the inner type gets "for free" the references to the generic parameter in the outer type is a C# language trick, but not something that exists at the IL level. Therefore, when creating nested types, you must propagate the generic parameters yourself.

Now to Fody. If you want to dynamically define the Inner type in Fody, you must re-create the generic parameters of the enclosing type. So if you have a TypeDefinition for an enclosing type, enclosingType, then to define the nested type, you'd use syntax like:

var nestedType = new TypeDefinition(enclosingType.Namespace, "Nested", TypeAttributes.NestedPublic, Context.ObjectType); foreach (var parameter in enclosingType.GenericParameters) { var newParameter = new GenericParameter(parameter.Name, ProceedClass); foreach (var constraint in parameter.Constraints) { newParameter.Constraints.Add(constraint); } nestedType.GenericParameters.Add(newParameter); } enclosingType.NestedTypes.Add(nestedType);

Over the years, I've found that among advocates of TDD there is some disagreement over whether tests should be written before or after the actual code that is being tested. At least in my experience, I believe there are situations that warrant one approach over the other.

When I'm brainstorming and trying to come up with a good design, I don't want to write unit tests that might become obsolete in a couple of minutes. I want to try to solve a particular problem -- free-writing, so to speak -- in order to see if a particular design has merit. Often this will mean writing a ton of code before I write the first unit test. I think this approach is valid in this circumstance.

In contrast, when writing logic that is participating in an infrastructure that is already proven there is less need for the freedom to brainstorm. In these situations it makes sense to write the unit tests first. The reason here is what test-first TDD advocates have long stressed -- it's way harder to write unit tests for code you already think is perfect. It's easier by far to write the unit tests first, watch them fail, and then write the correct code that fixes the unit test.

The point here is that the order of writing unit tests should be predicated on the circumstances of the code that you are writing. If you are early in the design, it's quite possible you don't even know what classes you should be writing unit tests for. You're trying to figure out syntactically what would be pleasing for developers. You absolutely need the freedom to freewrite without thinking of all the ways things could succeed or fail -- in fact in this situation you're trying to see if any idea has even a chance of succeeding.

What follows is the documentation for my base state generator class that is used for building the state machines for yield and await operators for WootzJs.

The base class for C# state generators is used for async and yield functionality. The following is essentially my take on concepts that can be gleaned from a study of coroutines. The essential problem is that in normal control flow, a method has a beginning and an end -- at the end, flow jumps back to the caller immediately after the invocation of the method that has just been called. This is known as a function's epilogue. However, await and yield require control flow that is more flexible. It's easiest to understand by starting with the most flexible operator, await.

With await, you want to ask the awaiter to do its thing and when it's done (perhaps hours later) logically return the control flow to immediately after the await expression (the next argument in a function call, the next statement, etc.). In other words, to explicitly stimulate the function's epilogue on its own terms instead of implicitly when a method is simply finished.

In the meantime, the callstack is successively unwound; if the calling method is also async and awaiting, this bubbles up the callstack until a non-async method is reached. (Ultimately, the top of a callstack cannot be async, be it a Main method or a Run on another thread.) This allows the original thread to continue on its merry way and perform other actions. This is in stark contrast to a normal method invocation. Without async (or yield), the method must finish before the callstack unwinds back to the caller.

The way this is implemented is that when you start the awaiter, you provide the awaiter with a callback function. What's extremely special about this callback function is that its effect is essentially the function's epilogue -- it moves control flow back to an arbitrary point of a method. Needless to say, except for await and yield, this is not otherwise possible in C#.

This ability to create a callback that can return to an arbitrary point in the method is the key ingredient to both await and yield. When implementing a yield iterator, you are asking the caller to do some work, then the iterator to do some work, and then the caller, and then the iterator, and so on. Importantly each time work is handed off between the two parties, control flow must resume from where it left off last time. This is achieved on the caller side by normal imperative control flow. The caller does work and when it's ready to wait for a new element, it calls MoveNext() (like any other implementation of IEnumerator. Notably, the C# compiler's async method state machine uses the name MoveNext() as well). However, the implementation of an iterator's MoveNext() method is the callback ingredient we have been describing. This callback returns control flow to arbitary points in the iterator (wherever the yield operator was last used).

So, to implement await and yield, we must figure out a way to create this callback that can return to arbitrary points in a method. The following applies to both C# and Javascript. (In fact, the initial implementation was a C# -> C# translation, but is now a C# -> JS translation.) To create this callback we must rip apart the abstract syntax tree of the method doing the awaiting or yielding. The way this is usually done is by creating a state machine. Concretely, this means generating a switch statement that switches on an integer state variable. Each case statement represents some portion of logic (one or more statements, and sometimes even individual expressions). Modifying the value of this state variable effectively changes where you are in the original method. Thus it becomes possible to return from the function and, the next time it is invoked, logically continue execution from where you left off before.

Generally speaking, this is straightforward. However, control flow constructs (if, while, for, foreach, try/catch, etc.), must be implemented in a new way in order to be able to exit from the function and somehow resume back into the middle of a control flow. This can probably best be explained by example. We'll use yield in these examples because conceptually it's a bit more specific (caller and iterator) and thus easier to reason about. But all of this applies to await as well.

We'll start with the simplest possible yield iterator:

public IEnumerator<string> Empty() { yield break; }

This iterator will produce an enumerator that will return false on the first invocation of MoveNext() (a sequence with no elements). Let's convert it into a state machine. We'll use C# for consistency, but the generated output is obviously Javascript. Also, the implementation is simplified for clarity, but the real output is more sophisticated (implements IEnumerable, IDisposable, etc. Additionally, generated members such as state are actually translated prefixed with "$" to avoid collisions with user code).

class YieldBreakIterator : IEnumerator<string> { private int state; public string Current { get; private set; } public bool MoveNext() { switch (state) { case 0: return false; } } }

Here we have a simple state machine. Now let's change the iterator to:

public IEnumerator<string> ReturnFoo() { yield return "foo"; }

This iterator will be much the same as the first one, but it will first set the value of Current to "foo" before returning true.

class YieldReturnFooIterator : IEnumerator<string> { private int state; public string Current { get; private set; } public bool MoveNext() { switch (state) { case 0: state = 1; Current = "foo"; return true; case 1: return false; } } }

Here we can see that after the first invocation of MoveNext(), Current will contain the value "foo". The next invocation will arrive in state 2 and return false, ending the iteration.

The penultimate example is two yield statements in a row. This finally illustrates jumping into an arbitrary point in the iterator.

public IEnumerator<int> ReturnTenAnd42() { yield return 10; yield return 42; }

Translating to the state machine:

class YieldReturnTenAnd42Iterator : IEnumerator<int> { private int state; public int Current { get; private set; } public bool MoveNext() { switch (state) { case 0: state = 1; Current = 10; return true; case 1: state = 2; Current = 42; return true; case 2: return false; } } }

Now we get to see how the two statements in the iterator are broken down into two case statements. Each case statement is responsible for an individual yield return statement. Each invocation of MoveNext() moves you along, one statement at a time.

Finally, we will consider just one complex example -- the use of a for loop. The key difference here is that control flow now has to jump around between case statements. The for loop is just one example. All the control flow statement operations in C# require similar transformations.

public IEnumerator<int> ForLoop() { for (var i = 0; i < 2; i++) { yield return i; } }

And now, let's look at the state machine:

class ForLoop : IEnumerator<int> { private int state; private int i; public int Current { get; private set; } public bool MoveNext() { switch (state) { case 0: i = 0; goto 1; case 1: while (i < 2) { goto 2; } return false; case 2: state = 1; Current = i; i++; return true; } } }

Here we jump around between different case statements, and various expressions from the original iterator are decomposed into separate statements in the state machine.

The for loop initializer is pulled out and is expressed in the first line of case 0.

The for loop itself is converted into a while loop, using the same condition.

The body of the loop jumps to state 2 (not strictly necessary in this example to have a separate state, but in more complex examples this transformation becomes necessary).

State 2 is responsible for updating the Current property to the new value, apply the incrementor of the for loop, change the state to 1 (the top of the loop) and finally return true indicating that the next invocation should attempt to continue the loop.

When using reflection, it's trivial to determine the assemblies that are required (referenced) by that assembly. You just use AssemblyInfo.GetReferencedAssemblies(). However, if using Roslyn, it's not even possible to obtain the referenced assemblies of an AssemblySymbol, at least not via Roslyn. Instead, you must fall back to reflection to solve this problem.

So consider you have an IProject, a Compilation, and you want to determine, for each AssemblySymbol, what AssemblySymbols are referenced by that assembly. While not straightforward, this is still pretty easy to do. Ultimately, you need to combine both reflection and Roslyn to obtain the desired results:

public Dictionary<AssemblySymbol, AssemblySymbol[]> GetAssemblyReferences( IProject project, Compilation compilation) { var assemblyInfoByAssemblySymbol = project.MetadataReferences .ToDictionary( x => compilation.GetReferencedAssemblySymbol(x), x => Assembly.ReflectionOnlyLoadFrom(x.Display)); var referencedAssembliesByAssemblySymbol = assemblyInfoByAssemblySymbol .ToDictionary( x => x.Key, x => x.Value.GetReferencedAssemblies().Select(y => y.Name)); return referencedAssembliesByAssemblySymbol; }

At my current gig, we had a need for a UILabel that allowed for the following features:

Giving different words and phrases a different color from the surrounding text.

Allowing some words to appear like a hyperlink, and attach an Action event to such words that can respond to those words being tapped.

Indicate a maximum number of lines for the label, and if the text would exceed this limit, to then render the last line with a trailing ellipsis.

Those were the hard requirements, but in the end, I've written a control that allows for the following independently modified (as in, each word can potentially have their own set of stylings) attributes:

Text color

Font (and thus font size, bold, italics, etc.)

Text decoration (underline, strikethrough)

Tap events (underline a word and respond to when the user taps on it)

Text alignment (standard "paragraph" alignment options of left, right, and center)

Maximum lines (when surpassed, an ellipsis will be rendered at the end of the max line)

I had a pretty difficult time finding a good example of returning a string from C# to C in a callback. For our application, we have the core game mechanism written in a C library, but I would like to implement as much non-game functionality in C# (MonoTouch, etc.) since C# is a much richer language.

One feature I want to move from the C library to C# is the logic for determining the path for files to assets. There's a good deal of logic I'd like to apply on where assets live, and I don't want that logic to have to be implemented in C. To that end, I defined a callback in the C library. The header looks like:

typedef void (*ContentFilePathProvider)(const char* file, char* buffer, int bufferSize); extern void SetContentFilePathProvider(ContentFilePathProvider contentFilePathProvider); void GetContentFilePath(const char* file, char* buffer, int bufferSize);

Here we expect the C# code to import SetContentFilePathProvider and invoke it passing it a (static) method defined in C# to implement calculating the fully qualified path to an arbitrary file. The type of the delegate is dictated by the ContentFilePathProvider typedef. And on the C side, whenever we want the fully qualified path, we just call GetContentFilePath.

Before looking at the C# implementation, let's finish with the C implementation. First we need to create a global variable to store the callback:

static ContentFilePathProvider _ContentFilePathProvider = NULL;

The function to set the callback (called from C#):

void SetContentFilePathProvider(ContentFilePathProvider contentFilePathProvider) { _ContentFilePathProvider = contentFilePathProvider; }

This function is called anywhere in C we need to get the content file path for an asset:

void GetContentFilePath(const char* file, char* buffer, int bufferSize) { _ContentFilePathProvider(file, buffer, bufferSize); }

For example, elsewhere in our code, we have:

char fullPath[512] = {0}; GetContentFilePath(fileName, fullPath, 512); return Foo(fullPath);

We initialize the buffer to 512 bytes. This means the callback must ensure it does not stuff more than 512 bytes of path into it. Obviously this number might need to be tweaked. That is life in the unmanaged world.

Now, let's take a look at the C# impementation of the above contract. You'll need to import the C function for setting the callback, but first we need to specify the delegate type for the callback:

[UnmanagedFunctionPointer(CallingConvention.Cdecl)] public delegate void ContentFilePathProvider(string file, IntPtr buffer, int bufferLength);

I'm using a standard unmanaged C lib produced in Xcode, so the calling convention is Cdecl. The second parameter, buffer, is what really tripped me up.

The Google seemed to indicate it should be of type StringBuilder. I tried this but the result on the C side was that the text was gobbledygook. Clearly something wasn't right. Eventaully I found a terrific answer by Olivier Levrey on codeproject that led me to a working solution that uses IntPtr.

The last parameter indicates to the callee how much space there actually is in the buffer. You generally need the length of buffers in order to operate on them, if for overflow checking if nothing else.

Finally, the callback setter itself:

[DllImport(ImportSource)] private static unsafe extern void SetContentFilePathProvider( ContentFilePathProvider contentFilePathProvider);

The code to implement this is a bit cumbersome, but works fine:

[MonoPInvokeCallback(typeof(ContentFilePathProvider))] internal static void GetContentFilePath(string file, IntPtr buffer, int bufferLength) { var s = Config.ContentDir + Path.DirectorySeparatorChar + file; if (s.Length > bufferLength) throw new InvalidOperationException("File path is greater than length of buffer (" + bufferLength + ")"); // Convert from managed to unmanaged string IntPtr sPtr = Marshal.StringToHGlobalAnsi(s); // Create a byte array to receive the bytes of the unmanaged string var sBytes = new byte[s.Length + 1]; // Copy the the bytes in the unmanaged string into the byte array Marshal.Copy(sPtr, sBytes, 0, s.Length); // Copy the bytes from the byte array into the buffer passed into this callback Marshal.Copy(sBytes, 0, buffer, sBytes.Length); // Free the unmanaged string Marshal.FreeHGlobal(sPtr); }

The attribute up top (MonoPInvokeCallback) is extremely important if you intend to deploy to iOS. Without it you will get nasty AOT errors that will make you freak out. Especially since you will have otherwise tested this successfully in the simulator. And there is nothing that ruins my day more as when something works in the simulator but doesn't work on the device.

And to assign the callback:

SetContentFilePathProvider(GetContentFilePath);

Unfortunately Xamarin.iOS (née MonoTouch) does not expose the port property on NSUrl. To get around that, add the following extension method to a utility static class of yours (such as NSUrlExtensions):

public static int? GetPort(this NSUrl url) { NSValue port; if (typeof(NSUrl) == url.GetType()) { port = NSNumber.ValueFromPointer(Messaging.IntPtr_objc_msgSend(url.Handle, Selector.GetHandle("port"))); } else { port = NSNumber.ValueFromPointer(Messaging.IntPtr_objc_msgSendSuper(url.SuperHandle, Selector.GetHandle("port"))); } using (port) { if (port.PointerValue == IntPtr.Zero) return null; else return (int)(NSNumber)port.NonretainedObjectValue; } }

Now you can invoke url.GetPort() to obtain the port from a url. I think I got the conditional logic there correct -- it should be if (url.IsDirectBinding) { ... } but unfortunately IsDirectBinding is protected and I didn't really want to use reflection for this.

If you try to load a UIWebView offscreen (as in, before the web view has been added to any other views/controllers) in iOS 5.1, you might find that certain elements appear off screen.