Grudge Match: C# GC vs Objective-C ARC
Apple's Objective-C Automatic Reference Counting (ARC) is a big improvement in developer efficiency relative to the manual reference count system that used to be standard in Objective-C. However, it still has considerable shortfalls compared to the Garbage Collection (GC) systems used by Java, C#, and many other languages.
For example, with ARC, it's possible to:
Leak memory by creating strong reference cycles in blocks or delegates.
Inadvertently nil out weak references to view objects that aren't initially included in the view hierarchy.
Overrelease objects whose retain pattern confuses ARC, such as NSInvocation return values.
Of course, all of these issues can be avoided by a careful developer, but time spent thinking about weak and strong references or debugging issues that arise from their misuse is time not spent building useful functionality in your software.
The tradeoff for this extra work, however, is purportedly better performance. Apple has claimed that ARC is faster than GC, and even faster than manual reference counting. That being said, given the convenience of GC relative to ARC, I had to wonder: Is a little performance boost worth it?
In addition, Xamarin, an alternative cross-platform development environment based in C#, has a new, Generational GC engine. Maybe with advancements in GC technology, the gap between GC and ARC isn't that big anymore. I didn't find any useful data comparing Xamarin's C# GC and Apple's Objective-C ARC and decided to do a comparison.
I had a few simple goals for this comparison:
Isolate the performance of memory management, instead of comparing overall performance of Xamarin C# and Objective-C.
Allocate lots of small objects to allow ARC and GC to dominate CPU time. For sake of comparison and curiosity, however, I also wanted to see how relative performance would scale when allocating a smaller number of large objects.
Use standard Foundation objects, to objectively simulate the mix of memory allocations that will occur in an iOS app.
Operate on a background thread to avoid interference with the main runloop.
For both ARC and GC, three simple tests were performed. All tests iterated over the creation of 3 objects (NSMutableData, NSMutableArray, NSMutableSet), and assigned the objects to strong property references (Objective-C) or private fields (C#). The tests differed only in the number of loops performed, and the size (capacity) of the objects allocated:
Small Capacity - 50,000 iterations, 1 object capacity
Medium Capacity - 5,000 iterations, 1024 object capacity
Large Capacity - 500 iterations, 10,485,760 object capacity
I ran each test a number of times, and while the results varied a little, the results below were typical. All results are reported as the number of seconds to complete the test.
Ouch. Keep in mind that my goal in this test was to find out whether GC was close enough to ARC that ARC wasn't really worth the development headache. The results definitely dashed those hopes.
Remember that allocating a large number of small objects was the test that should most clearly demonstrate the relative performance of ARC and GC. In that test, ARC is 46 times faster than GC. Again, Ouch. In the medium test, ARC is actually 55 times faster. In the large test, where we might expect the task of allocation to overwhelm reference counting and garbage collection, ARC is still 15 times faster.
Given the staggering performance difference between the two, I wanted to know if there was something else going on. I decided to create another comparison that I thought might be more favorable to C#, although it wouldn't be nearly as direct a comparison of the memory management system. Instead of creating Foundation objects on both platforms, this test created C# / Mono objects (MemoryStream, ArrayList, Dictionary), in the C# app and equivalent Objective-C / Foundation (NSMutableData, NSMutableArray, NSMutableDictionary) objects in the Objective-C app. The same iteration counts and capacities as the original tests were used for the small, medium, and large tests.
These results start out looking a lot better for C# garbage collection. In this case, ARC is only 1.8 times faster in the small test. In the medium test, however, ARC is almost 12 times faster. In the large test, however, things got truly ugly for GC: The application crashed (every time) with an OutOfMemoryException.
One last test finally showed a ray of hope for the GC app, however. In my initial testing, all test runs (Foundation / Mono objects, small / medium / large capacities) were performed during a single launch of the application. I found that if the Mono / small test is run as soon as the application is launched, it actually runs a hair faster (0.85 seconds) than the ARC / small test.
Given the wide variety of results represented in this comparison, it's hard to draw a single conclusion. However, the following conclusions are fairly safe:
The performance gap between Objective-C ARC and Xamarin C# GC is not small. In some cases, performance is similar, but when memory usage is stressed, ARC is orders of magnitude faster.
While GC is generally a more "developer friendly" and "low maintenance" solution, it isn't foolproof. The ARC app never crashed due to a memory error, or for any other reason. The GC app, however, crashed every time the Mono / large test was performed.
GC was actually a bit faster than ARC in the case where small Mono objects were allocated immediately after application launch, but nearly twice as slow when run after other tests. This result makes sense, given the nature of ARC and GC. Because GC doesn't make any effort to clear memory when the application is busy and there isn't excessive memory pressure, it is actually able to "skip" some of the work of memory management. However, when multiple tests are run, memory pressure will build, forcing expensive GC operations, and causing erratic performance. ARC, on the other hand, will keep memory pressure low the entire time by immediately releasing memory, so performance is consistent and predictable.
The code used in this test won't win any awards for OO design, UX, or pretty much anything. Both apps are quick hacks, whose sole purpose was to test performance.
"Release" build profile used for C# and Objective-C versions
All other applications closed while the tests were run
Allocating arbitrary blocks of data on a loop and then not doing anything with them obviously isn't representative of real world application behavior. However, this comparison wasn't meant to look at real world behavior; It was meant to isolate the relative performance of ARC and GC.
I'm not a C# developer. My strategy for porting the Objective-C code to C# was to pretend I was porting to Java, then capitalize things and fix compilation errors.
I used the free Xamarin "Starter" edition. The starter edition didn't allow me to profile the application to see where it was spending time. It also didn't appear to have the option to actually turn on the fancy new generational garbage collector, so it was most likely using an older garbage collector.
In spite of trying to keep the playing field level by using as many Foundation / Objective-C objects and constructs as possible, there may be some level of "bridging" performance penalty that was to blame for the extremely disappointing performance in the Xamarin code.