Why not use Django Paginator for displaying search results?

[abarysh]

Currently, some tables are paginated and filtered in the browser (e.g., /genes/3335/similar/ and /datasets/65/similar/), whereas others are paginated and filtered on the server (e.g., /datasets/?query=MMS).

I would like to re-discuss when we use either type and potential improvements.

The reason for using server-side pagination was to speed up the loading of large tables (e.g., /datasets/?query=growth). However, now that I look at the speed of /genes/3335/similar/, I see that loading a table with ~5000 rows is not that problematic. Most likely, our past issues with performance were due to us calling object methods for each row of the table. Now we have mostly solved that issue with Django tricks (e.g., STRING_AGG, select_related, etc.). Hence my question -- do we really need to do server-side (instead of client-side) pagination and filtering? It works ok, but, to an attentive eye, it creates inconsistent expectations about responsiveness across different pages of the website (tables that look very similar behave slightly differently).

I do see one big reason for server-side pagination of search results (e.g., /datasets/?query=growth) -- the ability to save (and/or send) a link to a specific page of the results. That is possible with Django's won Paginator (https://docs.djangoproject.com/en/3.1/topics/pagination/) that passes info about order and page number via the URL parameters. Our current implementation doesn't have this feature. As a result, once you leave the search results page (e.g., /datasets/?query=growth), you can never come back to it -- if you click the "back" button on your browser, the ordering and page number are reset to default.

So, my questions are:

• Do we still need server-side pagination?
• If yes, is there an advantage in using Django Paginator (instead of our current custom implementation)?

[vsoch]

Some quick answers:

Server Side Pagination

I do think we still need the pagination. The factors that made the queries slow were running custom functions across hundreds to a few thousand records, which we avoid now because we only need to do so for 10 entities. Even without that, the rendering in the browser because of the sheer size of the page was also slow. We currently have a strategy that will scale, and it would be a lot of work to go back, and would likely be slow again.

Django Paginator

We actually started with this (you might remember the double pagination - that was left over from Django paginator) and this would work fine, but we'd then lose the ability to sort that DataTables offers. Instead of using the DataTables api we would have to implement our own sorting (as you pointed out, and see here https://stackoverflow.com/questions/45403849/django-sorting-table-with-pagination) and it would be a ton of work that I don't see would add much value over what we have now. Arguably, we could try to add that the url updates based on the data tables query, and the query also parses the url.

So overall, these feel like things that could be done or tried at some point, but there are so many other things to work on I don't think this should be a priority right now.