#reactive frameworks

(via https://www.youtube.com/watch?v=EvZ70gm1foE)

Also check out  :   Learn and Understand AngularJS tutorials" : 

http://bit.ly/angularjstutorials

AJAX was a hack to get some semblance of reactivity from what was originally a system meant to markup academic documents. Modern applications require more than that and so we, the industry, turned to Reactive Frameworks.

Meteor is a Javascript client-database-server framework that takes care of the plumbing in making your application reactive. On the front-end it uses Blaze and on the back-end it uses the Distributed Data Protocol(DDP). Using oplog tailing and MongoDB, Meteor allows real-time monitoring and propagation of changes from the client to the server and vice-versa. It runs on top of Node.js and eliminates callback hell with Fibers. It is the most popular reactive framework with a growing ecosystem and access to both the Node.js ecosystem and its own package repository, Atmosphere.

While Meteor has its own API to talk to the server, it doesn’t care what is said. This abstraction allows the back-end to be written in a system-oriented language when optimization is needed. The Distributed Data Protocol is like “REST for websockets” which allows live updates of data changes to be represented as JSON.

Meteor is more than a framework though, it’s an application akin to an OS. It provides all the parts necessary to build an app on the web while abstracting a lot of plumbing on how things talk to what and how resources are used. For example, Fibers treat function calls like threads and allocate and schedule specific cpu time for them. DDP acts like a more reactive version of HTTP. The Oplog tailing and MongoDB implementation act as the file system.

Sadly Meteor doesn’t play well with all hosting providers, as there are reports that it doesn’t work out-of-the-box with Heroku. On the flip side, they have their own integrated, hosting service. 

Meteor is tightly integrated with MongoDB, and although there are existing adapters and workarounds to use other databases, they won’t work as reliably as MongoDB will.

What is MeteorJS?

MeteorJS is a web application framework that simplifies web application creation, especially for real-time applications.  So what does that mean for us today? As developers who want to create real-time applications, it is difficult to efficiently and correctly push data through the web, maintain data consistency and make it perform fast (from the users point of view). Instead of handling data consistency between client and server, Meteor abstracts all that away with its DDP protocol - optimized for real-time web applications.   

What are some strengths?

Perhaps Meteor’s biggest strength is its low-latency live updates for web applications. Before, applications would have to constantly busy poll the server to make sure it’s data is consistent. But that solution is hardly scalable or feasible given the amount of data transferred, speed, and the number of users for many applications. But now Meteor uses epilog tailing, and tight integration with MongoDB (and tracking the database’s replication log) to observe queries without having to requery the database.   Secondly, Meteor only sends the minimum amount of data necessary to keep update speeds fast. Thirdly, Meteor uses pre-fetching of data and dirty-writes to the cache on the client side (such as on a mobile app) to keep the user response fast.

What are some weaknesses?

One is that Meteor is not backwards compatible with other application communication interfaces such as AJAX. That also means that the interfaces Meteor is communicating with must also work well together which is often beyond Meteor’s control.   Secondly, Meteor still uses object-data binding which from Pete’s lecture on React seems to be a worse methodology for building web-applications. I could not find much information about Meteor’s take on it but was looking forward to Max giving his side on the topic.  

Related Technologies

On February 11, Matt Debergalis talked to us about Meteor.js, a new reactive JavaScript platform designed to build web applications, much like Rails or Django, but in a more innovative and modern way. The main points of his talk, as seen in the blog posts written by the various groups, were on Meteor’s unique usage of oplog tailing and DDP. In addition, we analyze how Meteor fulfills the basic principles of a reactive system and how it compares to several other frameworks.

Meteor has proven itself as a popular choice lately as a framework for all sorts of web applications that possess reactive functionalities. It boasts low latency updates and minimal bandwidth usage, as well as reliability through DDP with a poll-and-diff format in conjunction with MongoDB. DDP, known also as “REST for websockets,” allows for live updates and content delivery in real time as opposed to traditional REST that requires the client to make a request to receive new information. In addition to Meteor’s use of websockets via DDP, it also keeps track of database changes with the poll-and-diff technique that determines the differences between what is in the database and what is presented to the client. Another reason for Meteor’s popularity is that it uses JavaScript across the stack; Meteor’s backend, as well as its Blaze frontend both use it. Despite Meteor’s popularity, it is still a fairly young technology and still has kinks to work out, such as bugs and poll-and-diff performance costs, as well as a lack of supporting frameworks for things such as testing, and API changes.

Oplog tailing, something relatively new to Meteor, is a way for clients to get information from the server in real-time. While a more typical strategy to do this would be to route queries through code, which begins to have problems when one tries to update the database directly, or when more complex queries are made, Meteor instead uses something called oplog tailing, which makes storing and updating data on the database easier. To implement this, special queries called “observable queries”, which are queries that can be changed when Meteor notices that the data has been changed, are used. As opposed to poll-and-diff (which Meteor implemented previously), which passes large amounts of data to the client, which then must be diffed to find any changes which may have occurred, oplog tailing is when Meteor receives the changelog from MongoDB, then uses that to see which queries would need to be updated, and then promptly updates them, resulting in real-time updates to the user in the application.

Reactive systems have grown recently out of a large demand for highly responsive applications. The recent transformation of the web as a place for rich, interactive applications rather than simply a store of static documents has prompted this growth and demand. Since building out such reactive systems today requires much thought and design to ensure a secure interaction, developers look to reactive frameworks as tools to easily create applications while abstracting away many of the details.

Reactive systems ideally should strive to follow a few key principles. These systems should be very responsive to the user, providing results in nearly real-time with low latency so as to make the user’s experience seamless. Consistent and instantaneous feedback is a cornerstone of many reactive frameworks such as Meteor. Resilience is another key feature, as any system might become unresponsive after an unexpected failure. Isolating components in a system allows parts to fail without bringing down the system as a whole. Making a system highly available requires replication and redundancy of data. Meteor uses it’s client-side data cache as a form of replication to make their system responsive and resilient.

Reactive systems must be scalable and remain responsive under increased workloads. These systems should be able to intelligently allocate resources to ensure that there are no bottlenecks which would require the entire system to stall. Finally, reactive systems should rely on an asynchronous message-driven approach to communicating within the system. Implementing non-blocking message queues within a reactive system helps enable load management and flow control.

In practice these reactive system principles can be seen in many of the reactive frameworks available today. Sails.js offers built-in asynchronous message communication via pub-sub connections over WebSockets. It also offers much more flexibility over Meteor since developers can choose which database to use via its powerful Waterline ORM. Sails’ scalability comes from leveraging the full power of Node.js on the back-end and being able to interop with many front-end web frameworks. However, this also means that responsive UI’s are not built-in to Sails unlike Meteor.

The current HTTP protocol of delivering websites does not match the current way we interact with web applications. Today, the majority of web applications need a way to sync the server and client while the user is continuously interacting with the product and updating information. However, HTTP was designed to send static documents from the server to the client without a way to maintain state or even send data from the client to the server in a straightforward fashion. As a result, several technologies were created to circumvent the flaws of HTTP to enable one to build modern web apps, but recently, Meteor.js decided to rethink the whole process and create the Distributed Data Protocol (DDP) to enable developers to build robust apps that easily sync the frontend with the backend.

Meteor.js is a full javascript platform that aims to simplify how developers write web apps. Through its Distributed Data Protocol, any information updated on the client side is automatically propagated to the server without the need to write additional code to ensure both the server and client are in the same state. In addition, Meteor also built an in-memory database into the client so that accessing models on the server and client are through the same API. Due to this data framework, accessing database information has never been easier. Also, since Meteor provides both a frontend and backend framework, projects are entirely built in Javascript, which makes it easier for developers to learn the stack versus other mixed-languages ones.

Meteor has a widely different philosophy of web development than most other frameworks. It moves away from the traditional AJAX structure and more towards a reactive one, which may be hard for programmers to switch to. In addition, Meteor only completely supports MongoDB, which may not be the database of choice for many developers. In addition, unlike other frameworks, Meteor requires the app to be built around it. For example, one may try out React for a small feature on their site, but Meteor must be used as the base of the application by its nature. Thus, startups might just be using Meteor while bigger or established companies avoid it. For these reasons, Meteor may not be a good tool for the project.

Meteor is revolutionizing the way we develop applications. Meteor.js is a very modern javascript based development platform with a collection of libraries that combines the utility of various projects such as Node.js, Ember.js, Mongo, etc.

Interoperability with any client that uses the DDP protocol

Meteor created DDP (Distributed Data Protocol) as a means of communicating through websockets. REST has become outdated due to the technological advancements that have done away with past limitations which made REST so viable in the past. Instead of having to follow the rigid format for which REST was designed for, clients can easily communicate and interact with any service over DDP using brief EJSON messages. It is simple and easy because neither the Meteor client or the Meteor service knows what exactly it is talking to, but all they need to know is that both are talking in DDP. This allows a great deal of abstraction as a client could be written purely in Meteor while the service could be written in C.

Real Time Updating

Given the fact that users are starting to expect the latest updates almost immediately on their screen on whatever service they are using, maintaining real time updating on something which has many clients which modify a shared data source can be a real challenge. Meteor makes use of two techniques called op logging and poll diffing in an attempt to portray the illusion of perfect data consistency among its servers and all its unknowing clients. By comparing the log which is written to each time an update is made to the server, it is able to reactively update the user based on the differences rather than querying the server for the entire document to update, reminiscent of the shadow diffing concept talked about in the previous week. When all else fails, Meteor has no shame in falling back on the computation heavy poll diffing, staying true to the saying “if it works, it works”. Lastly, Meteor makes use of simple guesswork to generate the illusion of zero latency in the face of it’s clients using a speculative cache which it relies on for information while its truth cache has yet to return with a hit from the server database.

Weaknesses

When building a web app, there’s a whole lot of factors to consider, including the server-side language, the type of database you’ll be using, the front-end framework to use - and, of course, how each of these pieces will be interacting with each other. Sometimes, though, you just want to bypass this mess and start building.

And by sometimes, we mean most of the time.

Reactive frameworks help solve this problem by allowing a single programmer to control the entire application stack in a tightly integrated package. The resulting app treats HTTP more dynamically, establishing two-way flows of data rather than a simple transaction of documents across the network.

Ember is one big player in the reactive framework game, providing light, powerful code on the front-end for large web applications. It quickly updates templates (based on Handlebars.js) when the underlying data changes. The focus on developers helps reactive apps feel more like writing with an API than wrestling with the server. Ember doesn’t have explicit solutions to server-side code, however, and also goes through a number of rapid iterations, which can result in difficulties for smaller, more static applications. It truly demands a large, dynamic application that has the team and flexibility to take advantage of it.

Another option is Elm. It's functional and concise. Immutability and type inference, for example, help a developer write short, fast, and maintainable code. It's reactive. With Elm, you can create interactive applications without callbacks or shared state. Finally, it's compatible with HTML, CSS, and JavaScript. Elm uses what it calls Signals. A Signal is a value that changes over time. Two examples are Mouse.position and Window.dimensions. With these abstractions, a developer does not need to manually handle events. Thus, there isn't a need to use callbacks or shared mutable memory. The benefit of this is that, because it leads to an architecture that centralizes state, it's difficult for the model to get out of sync. What's the problem with callbacks? In callback heavy frameworks, such as AJAX or Node.js, applications are passed around as callbacks. This makes them difficult to read–it forces the developer to jump around the codebase–and difficult to maintain–changes to seemingly unrelated functions may cause things to break. For updates, Elm uses a virtual DOM. To this, it adds purity and immutability. This combination results in fast performance. Purity means that functions will always have the same outputs given particular inputs. With the virtual DOM in Elm, if the current virtual DOM and the new virtual DOM are the same–that is, if they are referentially equal–they must also be structurally equal. This is because all values in Elm are immutable.

Other reactive frameworks lend themselves to rapid development. Sails.js is a flexible and scalable MVC framework that was built primarily to support Node.js on the backend and works with various front-end frameworks on both web and native applications. Aside from being able to support various front-end frameworks, Sails uses Waterline for object-relational mapping which allows it to support sourcing data from different types of databases such as MySQL, MongoDB, Redis, etc. As it was built to support Node.js, Sails also utilizes Express for handling HTTP requests and WebSockets for real-time communication, with both remaining to be the most established Node.js modules for their use cases. Being fully written in JavaScript, this and all its other strengths places Sails in the space of a good ‘filler technology’ for people who have a great deal of skill with other Javascript frameworks. Where Meteor forces a paradigm onto the entire stack, Sails permits the integration of any front-end, and any database - even if they don’t exist yet. This allows for a great deal of flexibility, but also mastery over a number of moving parts. Another strength of this framework is the ease of development, getting started is quick and RESTful APIs are surprisingly easy to create with Sails’ auto-generation capability. One downside is its relative youth over other frameworks - the code base is still growing, as well as the documentation and the developer community around it.

In the middle of the playing field are players like Meteor. Meteor allows for rapid but scalable development that is production-ready. Meteor’s code is readable, light, and ideal for producing excellent apps with quickly. This comes at the cost of some flexibility; Meteor can feel monolithic and inflexible as apps become more mature. With its extensive documentation and commitment to simplicity, it’s allows for full production in a single language, and with tons of “background magic” that allows programmers to work more efficiently. Furthermore, Meteor’s pledge of simplicity allows for some programming best practices to shine through, which can be seen in its handling of database systems; where some frameworks might try to impose their own rules on the database, Meteor plays well with the already well-established ACID properties existing within database languages. For example, Meteor uses the Distributed Data Protocol, which runs over a WebSocket, to move data between different parts of the app. This allows it to deliver live updates as data changes. This means the programmer has less to worry about, but also means that Meteor isn’t trying to force properties that have already been done well.