Pagination

SinglebaseCloud's Datastore API provides comprehensive pagination support for efficiently handling large result sets. Pagination allows you to retrieve data in manageable chunks, improving performance and user experience while reducing memory usage and network overhead.

How Pagination Works

Request Parameters

Control pagination using these parameters in your collection.find requests. You can use either the limit/offset approach or the page/per_page approach:

Option 1: Using limit/offset

{
  "op": "collection.find",
  "collection": "products",
  "match": { "status": "active" },
  "limit": 50,
  "offset": 0,
  "sort": "_created_at desc"
}

Option 2: Using page/per_page

{
  "op": "collection.find",
  "collection": "products",
  "match": { "status": "active" },
  "page": 1,
  "per_page": 50,
  "sort": "_created_at desc"
}

Pagination Parameters

Parameter	Type	Default	Description
limit	Integer	100	Maximum number of documents to return
offset	Integer	0	Number of documents to skip
page	Integer	1	Current page number (1-based)
per_page	Integer	100	Number of documents per page
sort	String	"_created_at desc"	Sort order for consistent pagination

Parameter Relationships:

• When using page and per_page, the API automatically calculates: offset = (page - 1) * per_page and limit = per_page
• You cannot mix approaches - use either limit/offset OR page/per_page, not both
• If both approaches are provided, page/per_page takes precedence

Parameter Limits

• Maximum limit/per_page: 1000 documents per request
• Minimum limit/per_page: 1 document per request
• Maximum offset: No hard limit, but performance degrades with very high values
• Minimum page: 1 (pages are 1-based)
• Default behavior: If no pagination parameters specified, returns up to 100 documents (equivalent to page=1, per_page=100)

Pagination Response

Response Structure

Every paginated response includes both data and metadata:

{
  "data": [
    {
      "_key": "550e8400e29b41d4a716446655440000",
      "_userkey": "user123",
      "_created_at": "2024-01-15T10:30:00.000Z",
      "_modified_at": "2024-01-15T10:30:00.000Z",
      "name": "Product 1",
      "price": 29.99
    }
    // ... more documents
  ],
  "meta": {
    "pagination": {
      "page": 1,
      "per_page": 50,
      "total_pages": 10,
      "size": 500,
      "count": 50,
      "has_next": true,
      "next_page": 2,
      "has_prev": false,
      "prev_page": null,
      "page_showing_start": 1,
      "page_showing_end": 50
    }
  }
}

Pagination Metadata

Field	Type	Description
page	Integer	Current page number (1-based)
per_page	Integer	Documents per page (same as limit or per_page parameter)
total_pages	Integer	Total number of pages available
size	Integer	Total number of documents matching the query
count	Integer	Number of documents in current response
has_next	Boolean	Whether there are more pages after this one
next_page	Integer/null	Next page number, null if no next page
has_prev	Boolean	Whether there are previous pages
prev_page	Integer/null	Previous page number, null if no previous page
page_showing_start	Integer	Position of first document in current page
page_showing_end	Integer	Position of last document in current page

Pagination Patterns

Basic Page Navigation (using page/per_page)

// First page
const firstPage = await singlebase.find('products', {
  match: { category: 'electronics' },
  page: 1,
  per_page: 25,
  sort: 'name asc'
});

console.log(`Page ${firstPage.meta.pagination.page} of ${firstPage.meta.pagination.total_pages}`);
console.log(`Showing ${firstPage.meta.pagination.count} of ${firstPage.meta.pagination.size} products`);

// Navigate to next page
if (firstPage.meta.pagination.has_next) {
  const secondPage = await singlebase.find('products', {
    match: { category: 'electronics' },
    page: 2, // Simply increment page number
    per_page: 25,
    sort: 'name asc'
  });
}

Basic Page Navigation (using limit/offset)

// First page
const firstPage = await singlebase.find('products', {
  match: { category: 'electronics' },
  limit: 25,
  offset: 0,
  sort: 'name asc'
});

console.log(`Page ${firstPage.meta.pagination.page} of ${firstPage.meta.pagination.total_pages}`);
console.log(`Showing ${firstPage.meta.pagination.count} of ${firstPage.meta.pagination.size} products`);

// Navigate to next page
if (firstPage.meta.pagination.has_next) {
  const secondPage = await singlebase.find('products', {
    match: { category: 'electronics' },
    limit: 25,
    offset: 25, // Skip first 25 documents
    sort: 'name asc'
  });
}

Page-Based Navigation

// Using page/per_page (recommended)
function getPage(pageNumber, pageSize = 50) {
  return singlebase.find('articles', {
    match: { published: true },
    page: pageNumber,
    per_page: pageSize,
    sort: 'published_date desc'
  });
}

// Using limit/offset (alternative)
function getPageWithOffset(pageNumber, pageSize = 50) {
  const offset = (pageNumber - 1) * pageSize;
  
  return singlebase.find('articles', {
    match: { published: true },
    limit: pageSize,
    offset: offset,
    sort: 'published_date desc'
  });
}

// Get specific pages
const page1 = await getPage(1);      // page: 1, per_page: 50
const page3 = await getPage(3);      // page: 3, per_page: 50
const page5 = await getPage(5, 20);  // page: 5, per_page: 20

Infinite Scroll Implementation

class InfiniteScroller {
  constructor(collection, query, pageSize = 50) {
    this.collection = collection;
    this.query = query;
    this.pageSize = pageSize;
    this.currentPage = 1;
    this.hasMore = true;
    this.allData = [];
  }

  async loadNext() {
    if (!this.hasMore) return [];

    const response = await singlebase.find(this.collection, {
      ...this.query,
      page: this.currentPage,
      per_page: this.pageSize
    });

    this.allData.push(...response.data);
    this.currentPage++;
    this.hasMore = response.meta.pagination.has_next;

    return response.data;
  }

  async loadAll() {
    while (this.hasMore) {
      await this.loadNext();
    }
    return this.allData;
  }
}

// Usage
const scroller = new InfiniteScroller('posts', {
  match: { status: 'published' },
  sort: '_created_at desc'
});

const firstBatch = await scroller.loadNext(); // Load page 1
const secondBatch = await scroller.loadNext(); // Load page 2

Performance Considerations

Optimal Page Sizes

Small Pages (10-25 items):

• Fast response times
• Good for real-time applications
• Higher request overhead
• Better for mobile devices

Medium Pages (50-100 items):

• Balanced performance and efficiency
• Good for most web applications
• Recommended default approach

Large Pages (500-1000 items):

• Fewer requests needed
• Higher memory usage
• Longer response times
• Good for batch processing

Sorting for Consistent Pagination

Always use consistent sorting to prevent duplicate or missing results:

// Good: Consistent sorting
const results = await singlebase.find('orders', {
  match: { status: 'pending' },
  sort: '_created_at desc, _key asc', // Secondary sort for tie-breaking
  page: 1,
  per_page: 100
});

// Avoid: No sorting (inconsistent results)
const inconsistentResults = await singlebase.find('orders', {
  match: { status: 'pending' },
  page: 1,
  per_page: 100
  // No sort specified - results may vary between requests
});

High Page Number Performance

Large page numbers can impact performance when using limit/offset internally:

// Less efficient for high page numbers (uses large offset internally)
const page100 = await singlebase.find('logs', {
  match: { level: 'error' },
  page: 100, // Internally: offset = 99 * 50 = 4950
  per_page: 50
});

// More efficient: Use cursor-based pagination for large datasets
const cursorResults = await singlebase.find('logs', {
  match: { 
    level: 'error',
    '_created_at:$lt': lastSeenTimestamp 
  },
  per_page: 50,
  sort: '_created_at desc'
});

Advanced Pagination Techniques

Cursor-Based Pagination

For large datasets or real-time data, use cursor-based pagination:

class CursorPaginator {
  constructor(collection, query, pageSize = 50) {
    this.collection = collection;
    this.baseQuery = query;
    this.pageSize = pageSize;
    this.lastCursor = null;
  }

  async getNextPage() {
    const query = { ...this.baseQuery };
    
    // Add cursor condition for subsequent pages
    if (this.lastCursor) {
      query.match = {
        ...query.match,
        '_created_at:$lt': this.lastCursor
      };
    }

    const response = await singlebase.find(this.collection, {
      ...query,
      per_page: this.pageSize,
      sort: '_created_at desc'
    });

    // Update cursor to last item's timestamp
    if (response.data.length > 0) {
      const lastItem = response.data[response.data.length - 1];
      this.lastCursor = lastItem._created_at;
    }

    return response;
  }
}

// Usage for real-time feed
const paginator = new CursorPaginator('activity_feed', {
  match: { user_id: 'user123' }
});

const page1 = await paginator.getNextPage(); // Latest activities
const page2 = await paginator.getNextPage(); // Older activities

Search Result Pagination

Paginate through search results while maintaining query consistency:

async function paginatedSearch(searchTerm, page = 1, pageSize = 20) {
  const results = await singlebase.find('documents', {
    match: {
      '$or': [
        { 'title:$regex': searchTerm },
        { 'content:$regex': searchTerm },
        { 'tags:$in': [searchTerm] }
      ]
    },
    page: page,
    per_page: pageSize,
    sort: 'relevance_score desc, _created_at desc'
  });

  return {
    results: results.data,
    pagination: results.meta.pagination,
    searchTerm: searchTerm
  };
}

// Search with pagination
const searchResults = await paginatedSearch('javascript', 1, 25);
console.log(`Found ${searchResults.pagination.size} results for "${searchResults.searchTerm}"`);

Best Practices

Choose the Right Pagination Style

// Use page/per_page for UI pagination (simpler, more intuitive)
const userFacingResults = await singlebase.find('products', {
  match: { category: 'electronics' },
  page: currentPage,
  per_page: 20,
  sort: 'name asc'
});

// Use limit/offset for programmatic access or when you need precise control
const batchResults = await singlebase.find('products', {
  match: { category: 'electronics' },
  limit: 1000,
  offset: processedCount,
  sort: 'name asc'
});

Consistent Sorting

Always specify sort order to ensure predictable pagination:

// Include secondary sort for consistent ordering
sort: 'priority desc, _created_at desc, _key asc'

Page Size Guidelines

• Mobile: 10-25 items per page
• Desktop: 50-100 items per page
• API/Batch: 500-1000 items per page
• Never exceed: 1000 items per page (API limit)

Error Handling

async function safePagination(collection, query, page, pageSize) {
  try {
    // Validate page bounds
    if (page < 1) page = 1;
    if (pageSize > 1000) pageSize = 1000;
    
    const response = await singlebase.find(collection, {
      ...query,
      page: page,
      per_page: pageSize
    });
    
    return response;
  } catch (error) {
    if (error.code === 'INVALID_PAGE') {
      // Handle page beyond available data
      return await singlebase.find(collection, {
        ...query,
        page: 1, // Reset to first page
        per_page: pageSize
      });
    }
    throw error;
  }
}

Pagination enables efficient handling of large datasets while providing users with smooth navigation experiences. Choose the appropriate pagination strategy based on your data size, user interface requirements, and performance needs.