Datastore

Singlebase's Datastore is our document database that organizes data into collections containing JSON documents. This flexible NoSQL approach allows you to store structured data without predefined schemas while maintaining consistency through automatic base properties and access controls.

Collections

What is a Collection?

A collection is a logical container for related documents, similar to a table in relational databases but without rigid schema requirements. Collections provide organization, access control, and query optimization for your data.

Collection Properties

Naming Conventions:

• Use lowercase letters, numbers, and underscores
• Start with a letter or underscore
• Maximum length: 64 characters
• Examples: users, orders, product_catalog, user_sessions

Collection Features:

• Schema-free: Documents within a collection can have different structures
• Automatic indexing: Common query patterns are automatically optimized
• Access control: Permission management at the collection level
• Scalability: Collections automatically scale with data volume

Common Collection Patterns

// User-related collections
"users"              // User profiles and account information
"user_preferences"   // User settings and customizations
"user_sessions"      // Active user sessions

// Business data collections
"products"           // Product catalog information
"orders"             // Purchase orders and transactions
"inventory"          // Stock levels and warehouse data

// Content collections
"blog_posts"         // Published articles and content
"comments"           // User comments and interactions
"media_files"        // File metadata and references

Documents

Document Structure

Every document in Singlebase is a JSON object with automatic base properties and user-defined fields.

Base Properties

All documents automatically include these system-managed properties:

{
  "_key": "550e8400e29b41d4a716446655440000",
  "_userkey": "user_abc123xyz789",
  "_created_at": "2024-01-15T10:30:00.000Z",
  "_modified_at": "2024-01-15T14:25:30.000Z",
  
  // Your custom fields here
  "name": "John Doe",
  "email": "john@example.com",
  "status": "active"
}

Base Property Descriptions

Property	Type	Description
_key	String (UUID4)	Unique document identifier, automatically generated
_userkey	String	Owner identification, set to current authenticated user
_created_at	DateTime (ISO 8601)	Document creation timestamp
_modified_at	DateTime (ISO 8601)	Last modification timestamp, automatically updated

Document Ownership and Privacy

Default Privacy Model:

• Documents are private by default to the document owner (_userkey)
• Only the user who created the document can access, modify, or delete it
• Other users cannot read, update, or interact with documents they don't own

Access Control:

// User A creates a document
const userADocument = await singlebase.insert('notes', {
  title: "My Private Note",
  content: "This is confidential information"
});
// _userkey is automatically set to User A's identifier

// User B cannot access User A's document
const userBQuery = await singlebase.find('notes', {
  _key: userADocument._key
});
// Returns empty result - User B has no access

Primary Key (_key)

Key Characteristics:

• Format: UUID4 string without hyphens
• Generation: Automatically created by the system
• Uniqueness: Guaranteed unique across all collections and databases
• Immutability: Cannot be changed after document creation
• Length: 32 characters including without hyphens

Example Keys:

550e8400e29b41d4a716446655440000
f47ac10b58cc4372a5670e02b2c3d479
6ba7b8109dad11d180b400c04fd430c8

Document Examples

User Profile Document:

{
  "_key": "550e8400e29b41d4a716446655440000",
  "_userkey": "user_john123",
  "_created_at": "2024-01-15T10:30:00.000Z",
  "_modified_at": "2024-01-15T10:30:00.000Z",
  
  "email": "john@example.com",
  "display_name": "John Doe",
  "phone": "+1234567890",
  "preferences": {
    "theme": "dark",
    "notifications": true,
    "language": "en"
  },
  "profile_complete": true
}

Product Document:

{
  "_key": "f47ac10b58cc4372a5670e02b2c3d479",
  "_userkey": "user_store_admin",
  "_created_at": "2024-01-10T09:15:00.000Z",
  "_modified_at": "2024-01-12T16:45:00.000Z",
  
  "name": "Wireless Headphones",
  "description": "High-quality wireless headphones with noise cancellation",
  "price": 199.99,
  "currency": "USD",
  "category": "electronics",
  "tags": ["audio", "wireless", "noise-cancelling"],
  "inventory": {
    "stock": 45,
    "warehouse": "US-WEST-1"
  },
  "specifications": {
    "battery_life": "30 hours",
    "connectivity": "Bluetooth 5.0",
    "weight": "250g"
  }
}

Order Document:

{
  "_key": "6ba7b8109dad11d180b400c04fd430c8",
  "_userkey": "user_customer456",
  "_created_at": "2024-01-15T14:20:00.000Z",
  "_modified_at": "2024-01-15T14:20:00.000Z",
  
  "order_number": "ORD-2024-001",
  "status": "pending",
  "total": 249.98,
  "currency": "USD",
  "items": [
    {
      "product_key": "f47ac10b-58cc-4372-a567-0e02b2c3d479",
      "quantity": 1,
      "price": 199.99
    }
  ],
  "shipping": {
    "address": "123 Main St, City, State 12345",
    "method": "standard",
    "cost": 9.99
  },
  "payment": {
    "method": "credit_card",
    "last_four": "1234",
    "status": "authorized"
  }
}

Data Types and Validation

Supported Data Types

Singlebase documents support standard JSON data types:

{
  "string_field": "text value",
  "number_field": 42,
  "float_field": 3.14159,
  "boolean_field": true,
  "null_field": null,
  "array_field": ["item1", "item2", "item3"],
  "object_field": {
    "nested_property": "nested value",
    "nested_number": 100
  },
  "date_string": "2024-01-15T10:30:00.000Z"
}

Field Naming Conventions

Recommended Practices:

• Use snake_case for field names (user_name, created_date)
• Keep names descriptive but concise
• Avoid reserved prefixes (_ prefix is reserved for system fields)
• Use consistent naming across similar documents

Reserved Field Names:

• _key - Document primary key
• _userkey - Document owner identifier
• _created_at - Creation timestamp
• _modified_at - Modification timestamp

Document Operations

Creating Documents

// Insert with automatic base properties
const newDocument = await singlebase.insert('users', {
  email: 'jane@example.com',
  display_name: 'Jane Smith',
  status: 'active'
});

// Response includes auto-generated properties
{
  "_key": "generated-uuid4-here",
  "_userkey": "current-user-id",
  "_created_at": "2024-01-15T10:30:00.000Z",
  "_modified_at": "2024-01-15T10:30:00.000Z",
  "email": "jane@example.com",
  "display_name": "Jane Smith",
  "status": "active"
}

Querying Documents

// Find documents owned by current user
const userDocuments = await singlebase.find('notes', {
  status: 'active'
});
// Only returns documents where _userkey matches current user

// Find specific document by key
const specificDocument = await singlebase.find('users', {
  _key: '550e8400e29b41d4a716446655440000'
});
// Only returns document if current user owns it

Updating Documents

// Update document (automatically updates _modified_at)
const updatedDocument = await singlebase.update('users', {
  _key: '550e8400e29b41d4a716446655440000',
  display_name: 'John Smith',
  last_login: '2024-01-15T15:30:00.000Z'
});

// _modified_at is automatically updated to current timestamp

Privacy and Access Control

Default Security Model

Owner-Only Access:

• Each document is automatically associated with its creator via _userkey
• Users can only access documents they own
• No cross-user document access without explicit permission mechanisms

Query Isolation:

// User A's query
const userAResults = await singlebase.find('posts', {
  category: 'technology'
});
// Returns only posts where _userkey = userA's ID

// User B's query (same collection, same criteria)
const userBResults = await singlebase.find('posts', {
  category: 'technology'
});
// Returns only posts where _userkey = userB's ID

Privacy Benefits

Data Isolation:

• Automatic tenant separation without complex permissions
• No accidental data leakage between users
• Simplified security model for most applications

GDPR Compliance:

• Clear data ownership boundaries
• Easy user data identification for deletion requests
• Built-in data minimization through user-scoped access

Best Practices

Collection Organization

Group Related Data:

// Good: Logical grouping
"user_profiles"      // User account information
"user_preferences"   // User settings
"user_activity"      // User action logs

// Avoid: Everything in one collection
"users"              // Mixed user data, settings, and logs

Consider Query Patterns:

// Optimize for common queries
"active_orders"      // Frequently queried orders
"order_history"      // Archived completed orders

Document Design

Embed Related Data:

{
  "_key": "[INTERNAL-UNIQUE-KEY]",
  "order_id": "ORD-123",
  "customer": {
    "name": "John Doe",
    "email": "john@example.com"
  },
  "items": [
    {"product": "Widget", "quantity": 2}
  ]
}

Reference When Appropriate:

{
  "_key": "[INTERNAL-UNIQUE-KEY]",
  "order_id": "ORD-123",
  "customer_key": "550e8400e29b41d4a716446655440000",
  "product_keys": ["f47ac10b-58cc-4372-a567-0e02b2c3d479"]
}

Performance Optimization

Index-Friendly Queries:

• Use consistent field names across documents
• Query on commonly accessed fields
• Avoid deep nesting for frequently queried fields

Efficient Data Structure:

• Store frequently accessed data at the document root level
• Use arrays for ordered data, objects for key-value pairs
• Consider document size limits for large datasets

---

Datastore API

The Datastore API provides a comprehensive set of operations for managing documents in collections. This RESTful API allows you to perform CRUD operations, search, count, and manage document lifecycle with features like archiving and restoration.

Operations Overview

OP	Description
collection.find	Retrieve documents from a collection
collection.insert	Insert new documents into a collection
collection.update	Update existing documents in a collection
collection.upsert	Update existing or insert new documents
collection.count	Count documents matching criteria
collection.delete	Delete documents (recoverable)
collection.archive	Archive documents (hide from queries)
collection.restore	Restore archived or deleted documents

---

1. Find Documents

collection.find

Retrieve one or multiple documents from a collection based on matching criteria.

Request Body

{
  "op": "collection.find",
  "collection": "users",
  "match": {
    "status": "active",
    "age:$gte": 18
  },
  "filter": {
    "name": 1,
    "email": 1,
    "created_at": 1
  },
  "sort": "_created_at desc",
  "limit": 50,
  "offset": 0,
  "lookup":{},
  "project":{}
}

Parameters

• op (string, required): Must be "collection.find"
• collection (string, required): Name of the collection to query
• match (object, required): Criteria for matching documents
• filter (object, optional): Criteria to do a post filtering on the retrieved documents for complex operations
• sort (string, optional): Sort order (default: "_created_at desc")
• limit (integer, optional): Maximum number of results (default: 100)
• offset (integer, optional): Number of results to skip (default: 0)

Match Operators

• field:$gt: Greater than
• field:$gte: Greater than or equal
• field:$lt: Less than
• field:$lte: Less than or equal
• field:$ne: Not equal
• field:$in: Value in array
• field:$nin: Value not in array

Filter Operators

• field:$gt: Greater than
• field:$gte: Greater than or equal
• field:$lt: Less than
• field:$lte: Less than or equal
• field:$ne: Not equal
• field:$in: Value in array
• field:$nin: Value not in array

Response

{
  "data": [
    {
      "_key": "user_123",
      "_created_at": "2024-01-15T10:30:00Z",
      "name": "John Doe",
      "email": "john@example.com",
      "status": "active"
    }
  ],
  "meta": {
    "pagination": {
      "page": 1,
      "per_page": 50,
      "total_pages": 3,
      "size": 150,
      "count": 50,
      "has_next": true,
      "next_page": 2,
      "has_prev": false,
      "prev_page": null,
      "page_showing_start": 1,
      "page_showing_end": 50
    }
  }
}

Example Use Cases

Find all active users:

{
  "op": "collection.find",
  "collection": "users",
  "match": { 
    "status": "active" 
  }
}

Find recent orders with pagination:

{
  "op": "collection.find",
  "collection": "orders",
  "match": { 
    "_created_at:$gte": "2024-01-01T00:00:00Z" 
  },
  "sort": "_created_at desc",
  "limit": 20,
  "offset": 0
}

---

2. Insert Documents

collection.insert

Insert one or multiple new documents into a collection.

Insert Single Document

{
  "op": "collection.insert",
  "collection": "users",
  "data": {
    "name": "Jane Smith",
    "email": "jane@example.com",
    "age": 28,
    "status": "active"
  },
  "captcha": "optional_captcha_code"
}

Insert Multiple Documents

{
  "op": "collection.insert",
  "collection": "products",
  "data": [
    {
      "name": "Laptop",
      "price": 999.99,
      "category": "electronics"
    },
    {
      "name": "Mouse",
      "price": 29.99,
      "category": "electronics"
    }
  ]
}

Parameters

• op (string, required): Must be "collection.insert"
• collection (string, required): Target collection name
• data (object/array, required): Single document object or array of documents
• captcha (string, optional): Captcha verification for anonymous insertions

Response

{
  "data": [
    {
      "_key": "auto_generated_key_123",
      "_created_at": "2024-01-15T10:30:00Z",
      "name": "Jane Smith",
      "email": "jane@example.com",
      "age": 28,
      "status": "active"
    }
  ]
}

---

3. Update Documents

collection.update

Update existing documents in a collection using different strategies.

Update Single Document by Key

{
  "op": "collection.update",
  "collection": "users",
  "data": {
    "_key": "user_123",
    "status": "inactive",
    "last_login": "2024-01-15T10:30:00Z"
  }
}

Update Multiple Documents by Key

{
  "op": "collection.update",
  "collection": "users",
  "data": [
    {
      "_key": "user_123",
      "status": "premium"
    },
    {
      "_key": "user_456",
      "status": "premium"
    }
  ]
}

Update Multiple Documents by Match Criteria

{
  "op": "collection.update",
  "collection": "orders",
  "match": {
    "status": "pending",
    "created_at:$lt": "2024-01-01T00:00:00Z"
  },
  "data": {
    "status": "expired",
    "&updated_at": "@now()"
  }
}

Response

{
  "data": [
    {
      "_key": "user_123",
      "_updated_at": "2024-01-15T10:30:00Z",
      "status": "inactive",
      "last_login": "2024-01-15T10:30:00Z"
    }
  ]
}

---

4. Upsert Documents

collection.upsert

Update existing documents or insert new ones if no matches are found.

Request Body

{
  "op": "collection.upsert",
  "collection": "user_settings",
  "match": {
    "user_id": "user_123",
    "setting_type": "preferences"
  },
  "update": {
    "theme": "dark",
    "notifications": true,
    "updated_at": "2024-01-15T10:30:00Z"
  },
  "insert": {
    "user_id": "user_123",
    "setting_type": "preferences",
    "theme": "light",
    "notifications": false,
    "created_at": "2024-01-15T10:30:00Z"
  }
}

Parameters

• op (string, required): Must be "collection.upsert"
• collection (string, required): Target collection name
• match (object, required): Criteria to find existing documents
• update (object, required): Data to update when document exists
• insert (object, required): Data to insert when no document matches

Use Cases

User preferences management:

{
  "op": "collection.upsert",
  "collection": "preferences",
  "match": { 
    "user_id": "user_123" 
  },
  "update": { 
    "last_login": "2024-01-15T10:30:00Z" 
  },
  "insert": { 
    "user_id": "user_123", 
    "created_at": "2024-01-15T10:30:00Z" 
    }
}

---

5. Count Documents

collection.count

Count the number of documents in a collection that match specific criteria.

Request Body

{
  "op": "collection.count",
  "collection": "users",
  "match": {
    "status": "active",
    "age:$gte": 18
  }
}

Parameters

• op (string, required): Must be "collection.count"
• collection (string, required): Collection to count documents in
• match (object, required): Criteria for filtering documents to count

Response

{
  "data": {
    "count": 1247
  }
}

Example Use Cases

Count active users:

{
  "op": "collection.count",
  "collection": "users",
  "match": { "status": "active" }
}

Count orders from last month:

{
  "op": "collection.count",
  "collection": "orders",
  "match": {
    "_created_at:$gte": "2024-01-01T00:00:00Z",
    "_created_at:$lt": "2024-02-01T00:00:00Z"
  }
}

---

6. Delete Documents

collection.delete

Soft delete documents from a collection. Deleted documents can be restored using the reference ID.

Request Body

{
  "op": "collection.delete",
  "collection": "users",
  "match": {
    "status": "inactive",
    "last_login:$lt": "2023-01-01T00:00:00Z"
  }
}

Parameters

• op (string, required): Must be "collection.delete"
• collection (string, required): Collection containing documents to delete
• match (object, required): Criteria for selecting documents to delete

Response

{
  "data": {
    "_ref": "del_ref_abc123xyz789"
  }
}

The _ref value can be used to restore deleted documents later.

Example Use Cases

Delete inactive users:

{
  "op": "collection.delete",
  "collection": "users",
  "match": { "status": "deleted" }
}

Delete old temporary files:

{
  "op": "collection.delete",
  "collection": "temp_files",
  "match": { "_created_at:$lt": "2024-01-01T00:00:00Z" }
}

---

7. Archive Documents

collection.archive

Archive documents to hide them from queries while keeping them in storage.

Request Body

{
  "op": "collection.archive",
  "collection": "orders",
  "match": {
    "status": "completed",
    "_created_at:$lt": "2023-01-01T00:00:00Z"
  }
}

Parameters

• op (string, required): Must be "collection.archive"
• collection (string, required): Collection containing documents to archive
• match (object, required): Criteria for selecting documents to archive

Response

{
  "data": {
    "_ref": "arch_ref_def456uvw012"
  }
}

Difference Between Archive and Delete

• Archive: Documents are hidden from queries but remain accessible for restoration
• Delete: Documents are marked for deletion and can be permanently removed after a period

---

8. Restore Documents

collection.restore

Restore previously archived or deleted documents using their reference ID.

Request Body

{
  "op": "collection.restore",
  "collection": "users",
  "match": {
    "_ref": "del_ref_abc123xyz789"
  }
}

Parameters

• op (string, required): Must be "collection.restore"
• collection (string, required): Original collection of the documents
• match (object, required): Must contain the _ref from delete/archive operation

Response

{
  "data": {
    "data": true
  }
}

A true value indicates successful restoration.

---

Error Responses

All endpoints may return error responses with the following structure:

{
  "error": {
    "code": "VALIDATION_ERROR",
    "message": "Invalid collection name",
    "details": {}
  }
}

Common Error Codes

• AUTHENTICATION_FAILED: Invalid or missing API key/token
• COLLECTION_NOT_FOUND: Specified collection doesn't exist
• VALIDATION_ERROR: Invalid request parameters
• RATE_LIMIT_EXCEEDED: Too many requests
• INTERNAL_ERROR: Server-side error

---

Migration and Schema Evolution

Schema-Free Benefits

Gradual Migration:

// Add new field to existing documents
await singlebase.update('users', {
  _key: user._key,
  phone_verified: false  // New field added
});

// Old documents without the field still work
const allUsers = await singlebase.find('users', {});
// Mix of documents with and without phone_verified field

Version Management:

{
  "_key": "user-key",
  "schema_version": 2,
  "email": "user@example.com",
  "profile": {
    // Updated structure in v2
    "personal": {"name": "John Doe"},
    "contact": {"phone": "+1234567890"}
  }
}

Best Practices

1. Efficient Querying

• Use specific match criteria to reduce result sets
• Implement pagination for large datasets
• Use filters to return only needed fields

2. Data Management

• Regularly archive old documents to improve performance
• Use meaningful collection names
• Implement proper error handling for restore operations

3. Security

• Always include both API key and bearer token
• Validate captcha for anonymous operations
• Use specific match criteria to avoid unintended operations

4. Performance

• Batch operations when updating multiple documents
• Use count operations for analytics instead of fetching all data
• Consider archiving vs. deletion based on compliance requirements