Coding in My Sleep

Over the last few months as the wave of feature requests slowly trickled to a halt for the first time in two years. We've suddenly been able to think about something important. How much did we hack onto our code in order to 'make it work'?

The answer. A lot. This isn't uncommon, and it's entirely symptematic of dealing with post-MVP growth stages in your application. You go through cycles of growth and refactor, growth and refactor. And we entered a time where our need to refactor grew greatly.

I was spearheading a lot of the conversations around this and what it means. Because, when you get to this level, the stasis between the two spots, you need to start questioning every single one of your design choices from before, especially in cases of rapid growth. Why did we build our data model like this, and why did we structure our routing like this, and what's the real best workflow like? After spending a bunch of time developing the front end and the data requirements at breakneck speed, we're forced to look at what we did wrong, why we did it, and what was the best course of action to take in the future, and how can we can prevent it?

Meetings aren't always a joyeous part of the job (although I tend to like these high value ones), but, we scheduled many meetings to clarify what we're trying to do in the context of our application. Our main goal was 'we want to build this as to open source it without any skeletons or black magic as to how it works', even if we never released it as open source, it was a goal to create transparency in the architecture and simplify and standardize how we handled certain situations. We spent nearly a month of our time documenting, transpiling the concepts, and creating stories in our backlog of how we wanted to handle these things. It's not an easy task by any stretch of imagination, but a clearly important one when you're making the shift from an MVP to a full fledged application that's maintainable.

We realized we had a maintanance problem, well... basically the entire time. Originally we had 3 microapps that were built around each other (developed independently) and a single SSO application to handle syncing between the them, and that poor design choice made it very difficult to maintain, so, we took the first step and merged all of the code together, and DRY'ed it all up. We still had significant debt. Like, the national debt sized debt for our team. When I say it was 'that large', I really do mean it. When your team is small, which ours is, it's imperative that you make sure you don't grow outside of what's capable to maintain. In the 'olden days' we used to use lines of code measurement as a rule of thumb of how many developers you need, but a lot of it has to do with having the right mix of developers across many levels to effectively manage and delegate application issues. Our team had a glut of senior level developers, but a lack of mid-level and junior-level meant a lot of the high level application development got blocked by tasks that were best suited for mid and junior level developers. It's typically a problem in smaller teams that that happens to roll up and accumulate over a period of time and significantly hurts team velocity and quality. In a perfect world, senior developers wouldn't just be coding, but actively taking time to ensure standards are enforced, to cross-check code, to read through and comment on pull requests and mentoring. Because when you're looking at maintainability, what you're really looking at is building consistency, where you don't have to explore the wild wild west every single time you step out into a new story. Where any developer can step in without arcane knowledge and do their job is the biggest key to maintainability.

So, after talking about how we've developed a strategy on what's wrong, the answer then becomes: how do we fix it. And for a lot of it, it was mostly reorganizing data, about following standards. As a Rails shop, that meant going by the book with some things, realizing that we had disorganized models and controllers that did too much, that we were stuffing too much logic into views. But we also had significant application design issues, such as how we handled our copious amounts of user data through duplication, normalization and denormalization. We didn't have a specific strategy for it. One of our strategies was to use some polymorphism, and carry similar, meaningful data around between our data models. We shared user scores and action events, progress notes and conversationable items as a flexible, yet restricted type of data in a consistant way. This means: when we looked at how our users were progressing in their action logs, we weren't having to check multiple models, we asked a single model to handle all of this. That clearly leads to consistancy, because when you look at how workflow is driven, you're not often handling a single model. It's the base flaw of using something like RailsAdmin right out of the box and expecting it to grow with your application. What's best for your data isn't best for your administrators, and not all of the time best for your application. It requires a gentle, invisible hand to start staying: what if we do this? What's the implications of that?

As we defined it, maintainablity meant we were able to minimize time spent on exisiting code. For us, that meant we needed to get a more robust test suite. I can't ever suggest this is absolutely critical in the quest to create maintainable code. People may quibble on if you need to run unit tests on static compiled languages like C# and Java, and every single one of those people are wrong. Someone you may know, someone you may like, someone who may be you, can write side effects in an application that have inadvertant consequences. You may end up spending more time validating that your code doesn't affect other code's results than actually writing the code. You may be spread out amongst a half dozen files, hunting things down. And yet, this is exactly why I say you need to build tests for everything. Being able to say: "tell me if this change anything turned red" is a massive tool in maintainability. You're not building code that breaks things unexpectedly. But again, you're only as good as your test code, and your test code leads to more maintainability. BUT THAT'S OKAY. When you're making the change, like we were, to more maintianable, more stable code, you're actually spending less hours in the end on each story. You can say: i'm spending X hours here, and there's then going to be Y hours saved in the end because this touches so many parts of the application, that we're going to be able to test quickly without having to spend a massive amount of time doing high touch testing. It's a waste of time, honestly, to do it any other way.

That, though, obviously requires us to build methods and objects that respect testing, that are small, don't rely on side effects, and return states. It's not a hard thing, but something that we found helps create better maintainable code as well. We took a hard look at documentation, and one of the things that we were lacking were clear directives of what each function returned, and what it was supposed to do. We prefered Yardoc in Ruby and Codo in Javascript in order to get a better handle of what each function was. To go back in time and say: well, I don't know exactly how this works, or why we did it like this, also is essentially an audit of your system. It's a way to say: if I don't understand this, how it's written, how will anyone else? And those are big maintainability smells.

Salesforce and Ruby have a strange little twist together, considering that Heroku is owned by Salesforce and Heroku is a godsend for Rails developers everywhere for maintainable and manageable server environments. It's a strange pairing that I've learned about. Heroku maintains the fantastic databasedotcom Gem which is useful for querying the Salesforce REST API.

Originally we had a Salesforce Dev whip up something in Force.com to allow us to use the REST API to accept calls to natively convert Salesforce Leads. But somewhere along the lines, it was destroyed, overwritten by shoddy sandboxing and bad data, just 'disappeared' without a backup copy.

So, Sales kept pestering our Project Manager, asking to get this fixed, over, and over, and over again.

But, without having a Force.com developer. HOW? How would you do it? Easy: API THE SUCKER. Salesforce's API's are well documented, kinda crappy, but ultimately moldable and workable to do what you need them to do. Frankly, for the most part, moving data between a Ruby Application & Salesforce isn't even that hard! Heroku's making it even easier to do that, to move information from web portals to your Salesforce databases. But, converting a Lead? Couldn't be done without accessing the old SOAP API.

And thankfully, I have a lot of good SOAP experience working with old API's from other companies and have Savon stuffed in my back pocket for situations like this. So I went to fire up the good ol Bundler Gem Builder, and named it Soapforce, because, what else would you name a Gem that works with the Soap half of Salesforce?

And then I went to push the shell to Github, and there was already a Soapforce on Github, and granted it wasn't updated that recently, I was using a Rails 3.2.x application. So I vetted the gem, and said: well. This just took out about 6 hours that I would have been Gem Building. (cue Self Highfive). It uses Savon, and has a function called

def call_soap_api(method, message_hash={})

Perfect!

So, you'd authenticate, and then go

call_soap_api 'convert_lead', ...

From there, we checked out the SOAP documentation for the convert lead process. 

Ah! The joy, the guts of SOAP. If you check your WSDL (or the sample Partner WSDL that Soapforce has in it's Library), you can find LeadConvert, which gives you the basic structure of your message_hash

<complexType name="LeadConvert">   <sequence>     <element name="accountId" type="tns:ID" nillable="true"/>     <element name="contactId" type="tns:ID" nillable="true"/>     <element name="convertedStatus" type="xsd:string"/>     <element name="doNotCreateOpportunity" type="xsd:boolean"/>     <element name="leadId" type="tns:ID"/>     <element name="opportunityName" type="xsd:string" nillable="true"/>     <element name="overwriteLeadSource" type="xsd:boolean"/>     <element name="ownerId" type="tns:ID" nillable="true"/>     <element name="sendNotificationEmail" type="xsd:boolean"/>   </sequence> </complexType>

Originally the Hash was set up with

{'accountId' => nil, 'contactId' => nil, 'convertedStatus' => 'Closed', 'doNotCreateOpportunity' => true}

And of course, it threw fun errors back at me like Element X invalid at this location because I wasn't mimicing the XML correctly, obviously.

{'LeadConvert' => {'accountId' => ...}}

Simple things that I wasn't expecting, that it wasn't expecting just a bunch of elements, but a sequence of items in XML.

With that though, the action actually became easy enough, we passed in our accountId and contactId that we made in the step before with data from our portal using databasedotcom, and then connected through Soapforce to our database, and bang boom, I was converting leads rather easily from then out.

I saw it posted about a few times on Stack Overflow without good direction on how to solve the issue, so I took it the next step and voila (or as some of my friends like to say: Whala!), you're now well connected with the SOAP half of Salesforce, and you get access to the following actions:

[login, describe_s_object, describe_s_objects, describe_global, describe_data_category_groups, describe_data_category_group_structures, describe_layout, describe_softphone_layout, describe_search_layouts, describe_search_scope_order, describe_tabs, create, update, upsert, merge, delete, undelete, empty_recycle_bin, retrieve, process, convert_lead, logout, invalidate_sessions, get_deleted, get_updated, query, query_all, query_more, search, get_server_timestamp, set_password, reset_password, get_user_info, send_email_message, send_email, perform_quick_actions, describe_quick_actions, describe_available_quick_actions]

We're still digesting today's lengthy decision in the Oracle v. Google appeal, but we're disappointed—and worried. The heart of the appeal was whether Oracle can claim a copyright on Java APIs and, if so, whether Google infringed that copyright. According to the Federal Circuit today, the answer to both questions was a qualified yes—with the qualification being that Google may have a fair use defense.