Paginate query responses and document assets

[ejanalysis]

Implement pagination for /query responses instead of a fixed first-page cap. Adds page and limit parameters, bounds limit at 500 rows per page, returns a consistent results plus pagination envelope, validates invalid pagination inputs with HTTP 400 responses, and documents the response contract in the README.\n\nAlso keeps the static assets route documentation from the original PR.

[ejanalysis]

/query needs either pagination, a limit on number of row returned, or a way to select fewer columns. Otherwise broad cutoffs like 0.80 can fail with a 500 code from response size

[ejanalysis]

The limit does not have to be 100 -- change it if you think it should be higher, before merging.

[ericnost]

Thanks for this @ejanalysis! Yes, the API needs to be more like an API.... I'm not sure capping at 100 or even 500 rows is a great solution. Pagination is, but it will take some development and testing. This, generated with the help of Gemini through a Google search, seems like the bones of what we want. We'd be adding a page parameter, then on each call to the endpoint, the API would do the whole calculation, returning the results corresponding to the requested page (e.g. if the limit is 100 per page, then page 2 would return results 101-200, or 100-199 depending on indexing).

#* Get a paginated list of items
#* @param page The page number (default 1)
#* @param limit Number of items per page (default 100)
#* @get /data
function(page = 1, limit = 100) {
  
  # Ensure parameters are numeric
  page  <- as.numeric(page)
  limit <- as.numeric(limit)
  
  # Calculate index range
  start_idx <- ((page - 1) * limit) + 1
  end_idx <- page * limit
  
  # Assuming 'my_dataset' is available in your R environment
  total_rows <- nrow(my_dataset)
  
  # Handle out-of-bounds pages gracefully
  if (start_idx > total_rows) {
    return(list(
      data = list(),
      metadata = list(
        current_page = page,
        total_items = total_rows,
        total_pages = ceiling(total_rows / limit)
      )
    ))
  }
  
  # Extract the requested slice of data
  paginated_data <- my_dataset[start_idx:min(end_idx, total_rows), ]
  
  # Return combined data and metadata
  list(
    data = paginated_data,
    metadata = list(
      current_page = page,
      limit = limit,
      total_items = total_rows,
      total_pages = ceiling(total_rows / limit)
    )
  )
}

This example has limit as a parameter, but I might just hard code it to something like 500. Or, leave it as a parameter, but make it no more than 500.

[ejanalysis]

I just assigned the rewrite (per your comments) and testing of this to Codex, so let's see what it can come up with.

[Copilot]

Pull request overview

This PR updates the /query endpoint to support explicit pagination (page/limit) and returns a consistent { results, pagination } response envelope, with corresponding README documentation and new unit tests for the pagination helper/endpoint wrapper.

Changes:

• Added page and limit parameters to /query, including input validation and HTTP 400 mapping for invalid pagination inputs.
• Introduced query_pagination.R helpers (paginate_query_results(), query_endpoint_response()) and wired them into rest_controller.r.
• Documented the paginated /query response contract in README.md and added testthat coverage.

Reviewed changes

Copilot reviewed 4 out of 4 changed files in this pull request and generated 3 comments.

File	Description
tests/test-query-pagination.R	Adds tests covering pagination behavior, validation, and 400 mapping for invalid inputs.
rest_controller.r	Wires /query to the new pagination-aware response helper and documents new params.
README.md	Documents /query pagination parameters and the {results, pagination} response envelope.
query_pagination.R	Implements pagination helpers, input validation, and endpoint response assembly.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[ejanalysis]

Addressed the remaining review comments on #32. Changes pushed in 1d8491f: added integer-overflow validation for pagination inputs, fixed empty-result pagination metadata, and imported pandas in the README query example. Verification: Rscript tests/test-query-pagination.R, parse check, and Plumber /query route smoke all passed.

— Codex