Creating Pagination in a semi-stateless manner
Let's walkthrough developing the process of paginating results from a server. As we go along we'll identify places where we feel we need state, and I'll show you, how I deal with that problem without being overly dogmatic.
Except where otherwise state, all code examples are fully functional. Just open a developer console on the lodash docs page and you can copy and paste to verify the results.
Our result set of numbers 0-100 sits on the server
We want to view a subset of that data on our screen, on the client side.
server = _.range(0,100) client = { subset: [] // <--- server data goes here }
We want to take different slices of the data using an offset and limit property. The offset moves us further into the results, and the limit says how many we want to retrieve.
server = _.range(0,100) client = { subset: [], offset: 0, // <--- Start at the first result limit: 10 // <--- How many we will retrieve }
Now to fetch the data from the server, we need to take a slice based on our offset and limit property.
server = _.range(0,100) client = { subset: [], offset: 0, limit: 10, //assign a slice of the server data to the subset property current: function(){ return this.subset = server.slice(this.offset, this.offset+this.limit) } }
We can now use this new function.
client.current() //=> [0,1,2,3,4,5,6,7,8,9]
We've barely started, yet we have run into some problems.
This simple application is already difficult to reason about,
We have to have store what the state of the client is (in our brains!), in order to verify the result is correct.
We are accessing external state via this.subset, this.offset, this.limit and server, inside the function. So calling the same function twice will not guarantee getting the same result twice.
We are modifying external state via this.subset, which means we could clobber the hard work of some other function, and make debugging a nightmare.
We can't easily compose multiple functions together to create new and interesting functionality, because each function could be doing bizarre things that are unrelated to our current process.
At this stage, I think we can live with all these problems for such a trivial application. But with a few small modifications we can keep our state, but also keep our pure functions.
server = _.range(0,100) client = { subset: [], offset: 0, limit: 10, pure: { //Pure function: takes a subset of data from a given server //based on offset and limit parameters current: function(server,offset,limit){ return server.slice(offset, offset+limit) } }, //Stateful function: assigns a subset of server data to //this.subset property current: function(){ return this.subset = this._current(server,this.offset,this.limit) } }
Now our pure function is namespaced as pure.current, and our stateful function can make use of its functionality accordingly. The actual functionality is in pure.current. current is just providing the state and performing an assignment.
Okay! Now our code is composable, testable, less error prone, but still stateful! Huzzah!
server = _.range(0,100) client = { subset: [], offset: 0, limit: 10, pure: { //Pure function: takes a subset of data from a given server //based on offset and limit parameters current: function(server,offset,limit){ return server.slice(offset, offset+limit) }, //increment the offset by the limit next: function(offset,limit){ return offset+limit } }, //Assigns an increment offset property to this.offset //and then assigns the latest subset data based on the new offset next: function(){ this.offset = this.pure.next(this.offset,this.limit) return this.current() }, //Assigns a subset of server data to //this.subset property current: function(){ return this.subset = this.pure.current(server,this.offset,this.limit) } }
All we've done is add a pure.next that simply increments offset by limit in a stateless manner. And we've now got a stateful next function that calls a pure function with internal state as arguments, and then assigns the result to client.offset.
For convenience client.next() then updates the client.subset property with a call toclient.current(). But if you wanted to be more explicit, you could leave them as separate calls.
Just a quick example on why this is so great.
[client.offset, client.limit] //=> [0, 10] <--- current state of client client.pure.next(client.offset, client.limit) //=> 10 client.pure.next(client.offset, client.limit) //=> 10 // <--- Still 10! client.pure.current(_.range(0,100),client.offset,client.limit) //=> [0,1,2,3,4,5,6,7,8,9] client.pure.current(_.range(0,100),client.offset,client.limit) //=> [0,1,2,3,4,5,6,7,8,9] // <--- Still 0-9 //looking into the future statelessly client.pure.current( _.range(0,100), client.pure.next( client.offset, client.limit ), // <--- Get the next offset client.limit ) //=> [10,11,12,13,14,15,16,17,18,19] <-- Always will be!
This means we can test the functionality of our application without fear of breaking anything. That last one was a little verbose, but in test code that is a good thing. All state is there for the eyes to see!
Next we are going to add the prev and pure.prev functions. But I want to point out, that since our pure functions are pure, we will start to notice common patterns that we can factor out of our code, that would be harder to accomplish if our code was touching external state.
server = _.range(0,100) client = { subset: [], offset: 0, limit: 10, pure: { //Pure function: takes a subset of data from a given server //based on offset and limit parameters current: function(server,offset,limit){ return server.slice(offset, offset+limit) }, //increment the offset by the limit next: function(offset,limit){ return offset+limit }, //decrement the offset by the limit prev: function(offset,limit){ return offset-limit } }, //Assigns an decremented offset property to this.offset //and then assigns the latest subset data based on the new offset prev: function(){ this.offset = this.pure.prev(this.offset,this.limit) return this.current() }, //Assigns an increment offset property to this.offset //and then assigns the latest subset data based on the new offset next: function(){ this.offset = this.pure.next(this.offset,this.limit) return this.current() }, //Assigns a subset of server data to //this.subset property current: function(){ return this.subset = this.pure.current(server,this.offset,this.limit) } }
Let's zoom in on pure.next and pure.prev for a moment.
//client.pure {} //increment the offset by the limit next: function(offset,limit){ return offset+limit }, //decrement the offset by the limit prev: function(offset,limit){ return offset-limit } //...
They are very similar. All that is changing is the polarity.
And let's look at the stateful versions.
prev: function(){ this.offset = this.pure.prev(this.offset,this.limit) return this.current() }, next: function(){ this.offset = this.pure.next(this.offset,this.limit) return this.current() },
They are even easier to refactor. All that is changing is which pure function is being called. We could replace each of these function with a single function called change( 'next'|'prev' ). That might look like this.
change: function( directionName ){ this.offset = this.pure[directionName](this.offset,this.limit) return this.current() }, prev: function(){ return this.change('prev') }, next: function(){ return this.change('next') }
We could do this. But keep in mind, indirection via multiple refactored functions can actually make it harder to reason about your code. So only pull that trigger when you feel complexity is going to be decreased. Here is a great read on this process Compression Oriented Programming by Casey Muratori.
Now our client can paginate statefully through the server data, but usually in pagination we don't completely remove our current dataset, we add a little bit of new data, and remove a little bit of old. This is often referred to as "infinite scroll". Let's see how hard it is to implement this change to our current app.
server = _.range(0,100) client = { subset: [], offset: 0, limit: 5, padding: [1,1], // <--- [paddingAbove, paddingBelow] pure: { //Pure function: takes a subset of data from a given server //based on offset, limit and padding parameters current: function(server,offset,limit,padding){ return server.slice(Math.max(offset-padding[0],0), offset+limit+padding[1]) }, //increment the offset by the limit next: function(offset,limit){ return offset+limit }, //decrement the offset by the limit prev: function(offset,limit){ return offset-limit } }, //Assigns an decremented offset property to this.offset //and then assigns the latest subset data based on the new offset prev: function(){ this.offset = this.pure.prev(this.offset,this.limit) return this.current() }, //Assigns an increment offset property to this.offset //and then assigns the latest subset data based on the new offset next: function(){ this.offset = this.pure.next(this.offset,this.limit) return this.current() }, //Assigns a subset of server data to //this.subset property current: function(){ return this.subset = this.pure.current(server,this.offset,this.limit,this.padding) } }
Here we've added a padding property to client. We've modified the pure.current to addpadding[1] to the limit and subtract padding[0] from the offset. We've also ensured that the subtraction is always >= 0.
We could have done this many different ways. We could have not told pure.current about thepadding and left that up to next and prev. And that could have worked. But I chose to make the most minimal change to the client code as possible. And I wanted as little code duplication as possible.
We also could have performed this with in the stateful current function. But I would advise against that, as our stateful functions are a lot easier to maintain and understand if they only provide and assign state, and leave the logic to the pure functions.
Let's see our padding in action. Note I've updated the limit parameter from 10 to 5 to make it clearer what is going on.
Let's look at the stateful code in action.
[client.limit, client.padding ] //=> [ 5, [1,1] ] client.current() //=> [0,1,2,3,4,5] // <--- Note 5 is padding client.next() //=> [4,5,6,7,8,9,10] <-- Note the overlap of [4,5] from the previous call client.next() //=> [9,10,11,12,13,14,15] <-- Overlap of [9,10] client.prev() //=> [4,5,6,7,8,9,10] <-- Overlap of [9,10]
So it works! And we can adjust our padding very easily. We can pad more on the top than the bottom depending on our scroll direction or vice versa by just setting client.padding = [paddingTop, paddingBottom] and subsequent calls to client.current(), client.next() and client.prev() will obey that constraint.
We are finished! We can infinitely scroll through a range of numbers. But creating a simple API is not a simple task.
There could be confusion for the user of this API when they have set the limit to 5 with padding of[1,1] and they end up with a subset.length of 7. This confusion stems from the idea that limit is the maximum length of the subset. But is instead the number of items to fetch from the server. Thepadding is added to the limit and subtracted from the offset, so the max length is really a function ofoffset,padding and limit.
maximumLength: function(padding,offset,limit){ return offset-padding[0] + limit+padding[1] }
It doesn't really make sense to set the maximum length, because then the padding is applied internally, and seeing as you have limitied the size, all you are really doing is shifting the offset to the left.
maximumLength = 5 padding = [1,1] limit = 5 offset = 10 server.slice( offset-padding[0], //shift to the left by the offset limit+padding[1] //expand the limit by the padding <-- pointless ).slice(maximumLength) //ignore the expansion of the limit as we only use up to the max of 5
But say you really just want 5 items, and you want each page to flow into the next page seamlessly. What do you do?
Just ignore padding altogether and move the offset forwards by less than the limit.
overlap = 2 client.limit = 5 client.padding = [0,0] client.offset += client.limit - overlap
And if you are scrolling in the oppositie direction, just subtract the offset instead
overlap = 2 client.limit = 5 client.padding = [0,0] client.offset -= client.limit + overlap
Your "server" is just an Array!
In this example, the server was just a generated array of numbers from 0-100. But that was so we could focus on the algorithm of the pure functions instead of AJAX calls.
That said, it makes sense to have your server code in a separate module and to just think of it as another client side collection of data while interacting with it. That way, the server module can act like an array and handle internal caching and prefetching to make our client module super fast.
Here is a simple implementation of what our server might look like. Note this below code isn't actual code that will work, it is more of an illustration of the concept.
server = { data: [], slice: function(start,end){ if( data[start] && data[end] ){ return Promise.resolve(data.slice(start,end)) } else { return fetch(start,end) } }, handleResponse: function(start,end,response){ return data.splice(start,end,response) }, fetch: function(){ return $.get('https://api.number.com/range?start='+start+'&end='+end) .then(this.handleResponse.bind(this,start,end)) } }
And we'd need to update our client code to call server.slice(start,end).then( ... ) instead ofserver.slice(start,end) But then we'd be able to treat our server as an array, and everything would be great!
I am not fully convinced by functional programming's strict adherence to statelessness. I think there is more of a continuum. If you can separate stateful activity from stateless algorithmic activity you can save yourself the vast majority of headaches of both statefullness and functional purity. I personally don't use a pure namespace in my own code. I tend to just keep the majority of functions stateless except for the assignment functions. But I think it could be a good pattern to follow, particular if thepure object was bound to a frozen context to make accidental calls to this harmless.
It seems to me there are plenty of occasions where state is appropriate, particularly for performance. But I have also seen extremely stateful code that is very hard to interact with, and can take a lot of mental fortitude to even attempt to open and contemplate the file. That is why I have taken the middle road, keep your algorithm stateless, but don't shy away from state.
The great thing about this implementation of the client. Is the user has a choice to work statefully or statelessly. Their application could call the pure functions directly, and maintain their own pagination state somewhere else entirely. And they could also just call client.next() and let the object handle it's own concern.
I don't know if this is the best way, but I feel it is practical and applicable and easy to follow in real world scenarios.
