remoteinterview.io - Official Blog

We have also updated Java and Node.js to newer versions.

In Tests, these languages are enabled by default when you add question from library or create a new programming question. But you will need to manually enable them for any existing programming questions you have. It’s easy, here’s how: Click Edit (pencil icon) on each of your questions:

And then head over to Languages tab and enable the languages you want to enable:

Click Next and finally save the changes. That’s it!

1. Describe the project requirements

A good take-home project is usually somewhat related to a small module of what your company actually does or problems you have already faced and solved. Be sure to reduce the problem to the most basic form of it so the applicant does not waste time on things you don't really care about. A good project should be a couple of hours of work.

2. Evaluating the submissions

Usually applicants either host a github repo of their submission or email the zipped project. Both methods have their pros and cons. But the primary timesuck is trying to run the projects locally and writing notes about the project. We try to improve this by providing online sandbox to explore and run the project code. Collaborative note-taking allows you to discuss and take any notes about the submission.

The reason is simple. They screen candidates based on CVs and not on actual skills.

Now, relying on CVs or resumes has two major problems:

1 - It's easy to get misled by a good looking CV (we all tend to get attracted by shiny objects, don’t we?)

2 - Great candidates often don’t put enough time to polish their profiles, and so companies easily miss out those talented people (been there, done that)

For programming jobs, we believe you should not screen candidates based on CVs or Resumes. You should screen them on the basis of their actual programming skills.

Introducing RemoteInterview Tests

RemoteInterview Tests is the most effective way to screen your candidates. Instead of collecting CVs, companies can create programming challenges and quizzes to identify top-skilled candidates. Once the test is configured, scoring is completely automated. You don't have to raise another finger, just sit back, relax and do something more productive with your time (ain't that cool now?)

We privately launched RemoteInterview Tests a few months ago, and to our own astonishment, on average it saves up to 90% of time and spendings! One of our large clients in North America used to spend more than 150 hours of their engineers time to hire just one programmer. After using RemoteInterview.io Tests, they now spend just about 15 hours to do similar hiring. They are saving hundreds of productive hours of their engineers.

Here are some screenshots:

It's really easy to get started. Just signup for free trial at https://remoteinterview.io/signup

A lot of our competitors must have faced similar problem but none of them have disclosed anything as if this was their secret sauce.

We decided to do just that and release all our findings for anyone to use. Since we heavily believe in open-source and plan to continue contributing back.

  The Actual Idea

In an ideal world, a simple solution would be a server which compiles the code on user’s behalf and returns the output. Next we had to support multiple languages. For that the server should have all those compilers installed and “API-fied” so the client can send over the code and get the results back.

But there is one problem, you can’t just trust any code your user submits. The main problem is (wait for it!) security. What if the not-so-malicious-looking submitted code is very evil and tries to damage or take control of the server. Also we had to allow native binaries to be run.

Even if you could somehow resolve the security issue using intricate commands to restrict resources and execution time, there was still this issue of efficiency. Focusing on the “complexity is your enemy” phrase, we had to keep efficiency consideration in our minds too! In short we were adamant on making life difficult for ourselves.

The untrusted program needs to have some restrictions, like it shouldn’t have access to server files, should not be able to open ports, should not consume a lot of resources and should terminate after some time. The amount of parsing needed to turn this code into a ‘good-boy’ was interesting and challenging. But like developers usually do, we managed to come up with a solution. And we feel happy about it.

Early Brainstorming

- Chroot jail: What was that first thing that occurred to us? Well, no points for guessing, it was none other than the  chroot jail. But before we could proceed any further we realised this may not be the best way to go. As stated by Josh Bressers on  SecurityBlog . Seeing tutorials teaching us how to break out of a chroot jail proved to be the final nail in the coffin and we decided to try something else.

- Ideone: Does exactly what we wanted. But then things that do exactly what you want don’t really come for free (our solution does though). Ideone, though ideal, turned out to be a rather costly solution in our initial usage. Not to mention that the time Sphere Engine takes to compile a small source code is significant. Needless to say, we decided to move on. 

- Using VmWare/Virtual Box: As safe as you could expect. Does the job pretty well. But in case this virtual machine was compromised, detecting it and recreating it was not ideally time-efficient. Also VirtualBoxes are handled best using a GUI, which again is a constraint, given server communication is done mostly through terminal. So we packed our bags once more and decided to move to the next possible solution.

  The Breakthrough

We had to be patient, and eventually we managed to find a way through to get the job done. Thanks to Docker.io we managed to build a virtual machine image of Ubuntu and decided to use its instances (the containers) to run the untrusted code. Using Docker provided us with the following benefits

- If any malicious code was to attempt to destroy the system, its effects would remain inside the container it is working in.

- Containers don't have to be running all the time, they are created when needed, and destroyed when they have completed their job

- Each container is a running instance of one Image, we had all our compilers installed inside the image.

- Once the output file is created. We send its contents back to the client.

Tada! It sounded simple yet effective,  we decided to give it a go.

EDIT: As many of you have (rightfully) pointed out that our considerations regarding "security" part of Docker were somewhat misleading, Docker is good for achieving isolation but not so much in terms of security. However, so far, Docker has managed to do the job for us pretty well. Pull requests are always welcome. Also we are trying to find out ways that could improve the security aspects of the compilation further and looking into integrating SELinux and AppArmor into our code.

Working Our Way Through

Our task at hand was divided into two phases.

1. Create a Dockerfile which installs all the relevant compilers in the container.

2. Make an API/Supervisor which receives code from user, runs it in a Docker container, and gets back the output.

Creating a virtual image inside Docker is easy given that your internet connection is having a good day. You create a Dockerfile and run it with Docker using the “build” command. See this link for a better idea.

Once we completed the Docker layer we moved on to writing a supervisor using shell script and Node.js for API. The supervisor creates new container, compiles the given program using selected compiler. The compilation and execution is carried out inside Docker container. The output is then redirected to a file from where the supervisor reads the contents and returns them to the client app.

The execution time is quick, the containers usage is sleek and the Virtual image itself is pretty lightweight.

Here is a visual for the whole system.

  Saying Goodbye

With everything said and done all that is left for you is to experience the finished product and see it working here and here. Alternatively, you can find the source code at github and let us know of what you think. Thanks in advance for any bug reports!

That would be that,

Cheers!

The HackerNews community has earned my respect and are my inspiration since I have been getting my 'morning fix' there. It has been my most frequently visited site for more than a year now.

So when it came to releasing our own product, making a post at HN meant a lot to us from the beginning.

Our product, remoteinterview.io was ready for private beta and we wanted to attract our early adapters through our landing page. The response we got was so good and I actually got addicted to reading and responding every comment as well as checking analytics (of all sorts).

THE FAILED ATTEMPT

When we got our page ready, we immediately posted it to HN on a Monday at 1pm EDT with a really bad tag line. I guess the timing was also bad and we never made it to the front page. We realized this in few minutes and deleted the post immediately.

WELL-PLANNED ATTEMPT

This time, we thought about a better tagline and posted on Tuesday at about 8:30am EDT. Luckily we immediately got some comments and without wasting any time, we quickly engaged with every commenters to keep the thread alive. In just about 10 minutes, we were on the front page!!

Some Analytics (Using Heap)

Before HackerNews (mostly from FB and Twitter followers):

And then HN happened!!:

We were all glued to our screens, reading comments and responding to them all. 

After sometime, we started going down the list and eventually to the second page. We stopped following the post when no more comments were coming and went for a meal. But...

THE EPIC COMEBACK

I checked HN after an hour or so (I regularly do anyway) and saw our post jumping in the Top-20 on the front-page. It stayed there for almost whole day.

THE PROS

Our aim was to make noise and collect early beta users. The total number of confirmed email subscriptions (at MailChimp) have reached 1000+ as of now.

Here are some demographics about subscribers

  Also, the comments were simply great.

THE CONS (ALSO PROS)

We are using FireBase as our raw email collection, since MailChimp is a double opt-in process. So, that much traffic caused us to max our Free package and we had to buy a paid package. We had to buy one anyway but we were trying to be resourceful ;)

Please note that we don't encourage spamming HN (you will regret it anyway), if you really feel like your post is worth HN or you have worked on something and want genuine opinion of some very cool people only then do it.

remoteinterview.io is easy and effective. It lets you watch your candidate code during the interview so you can know his thought process: 

The feature I personally love most is the powerful rating system. Throughout the interview, this tool lets you rate your candidate on things that matter. For example, candidate’s understanding of the tasks, approach to a solution, speed, knowledge of algorithms, etc. 

We are using cutting edge web technologies to provide real time audio, video and text chat to make communication easy between your candidate and panelists. Yes, you can have multiple people from your company participate in the interview as panelists.

Once you are done with the interview, you can see the performance of all your candidates at a single glance, and simply pick your favorite rock star coder! 

If you are building a remote team of developers, this tool is for you! We have already launched it for private beta access and got more than thousand signups within 24 hours! We were also featured on HN and reddit.