Projection

Projection and Data Transformation

Projection in DDO allows you to transform data into a new shape or structure without modifying the original data. Unlike mutations that change existing documents, projections create new representations of your data based on templates you provide.

Understanding Projection

Projection is particularly useful for:

Creating API responses with specific data structures
Generating reports and summaries
Transforming data for different views or interfaces
Computing derived values from existing data
Reshaping nested data structures

Basic Projection Syntax

Projection QLOs use template expressions rather than operators:

# Source data
user_data = {
    "firstName": "John",
    "lastName": "Doe",
    "email": "john.doe@example.com",
    "age": 30,
    "orders": [
        {"total": 100, "status": "completed"},
        {"total": 150, "status": "pending"}
    ]
}

# Projection QLO
projection = {
    "displayName": "{{ firstName }} {{ lastName }}",
    "contact": "{{ email }}",
    "&age": "age",                                    # Copy as-is with native type
    "isAdult": "?{{ age >= 18 }}",                   # Boolean result
    "&orderCount": "@length(orders)"                  # Number result
}

# Result:
{
    "displayName": "John Doe",        # String
    "contact": "john.doe@example.com", # String  
    "age": 30,                        # Number (native type)
    "isAdult": True,                  # Boolean (native type)
    "orderCount": 2                   # Number (native type)
}

Simple Field Mapping

Map and rename fields from source to target structure:

# Source: database user record
source = {
    "user_id": "12345",
    "first_name": "Jane",
    "last_name": "Smith",
    "email_address": "jane@example.com",
    "created_timestamp": "2024-01-15T10:30:00Z"
}

# Projection: API response format
projection = {
    "id": "{{ user_id }}",
    "name": "{{ first_name }} {{ last_name }}",
    "email": "{{ email_address }}",
    "joined": "{{ @format_date(created_timestamp, 'YYYY-MM-DD') }}"
}

# Result:
{
    "id": "12345",
    "name": "Jane Smith", 
    "email": "jane@example.com",
    "joined": "2024-01-15"
}

Nested Structure Creation

Create complex nested structures from flat or differently structured data:

# Source: flat user data
source = {
    "userId": "u123",
    "firstName": "Alice",
    "lastName": "Johnson", 
    "email": "alice@company.com",
    "phone": "+1-555-0123",
    "street": "123 Main St",
    "city": "Portland",
    "state": "OR",
    "zipCode": "97201",
    "department": "Engineering",
    "role": "Senior Developer",
    "salary": 95000
}

# Projection: structured profile
projection = {
    "id": "{{ userId }}",
    "personal": {
        "fullName": "{{ firstName }} {{ lastName }}",
        "contact": {
            "email": "{{ email }}",
            "phone": "{{ phone }}"
        },
        "address": {
            "street": "{{ street }}",
            "city": "{{ city }}",
            "state": "{{ state }}",
            "zipCode": "{{ zipCode }}"
        }
    },
    "professional": {
        "department": "{{ department }}",
        "position": "{{ role }}",
        "&compensation": "salary",
        "level": "{{ @if_else(salary >= 90000, 'Senior', 'Junior') }}"
    }
}

Working with Arrays and Collections

Transform arrays and collections within projections:

# Source: order data with items
source = {
    "orderId": "ord_456",
    "customer": {"name": "Bob Wilson", "email": "bob@example.com"},
    "items": [
        {"productId": "p1", "name": "Laptop", "price": 999.99, "quantity": 1},
        {"productId": "p2", "name": "Mouse", "price": 29.99, "quantity": 2}
    ],
    "shipping": {"method": "standard", "cost": 9.99}
}

# Projection: order summary
projection = {
    "orderNumber": "{{ orderId }}",
    "customer": {
        "name": "{{ customer.name }}",
        "email": "{{ customer.email }}"
    },
    "summary": {
        "&itemCount": "@length(items)",
        "&subtotal": "@sum(@map(items, 'price * quantity'))",
        "shipping": "?{{ shipping.cost }}",
        "&total": "@add(@sum(@map(items, 'price * quantity')), shipping.cost)"
    },
    "&itemDetails": "@map(items, '{\"product\": name, \"qty\": quantity, \"total\": price * quantity}')",
    "shippingMethod": "{{ shipping.method }}"
}

Conditional Projections

Create different outputs based on conditions:

# Source: user account data
source = {
    "userId": "u789",
    "name": "Charlie Brown",
    "accountType": "premium",
    "credits": 150,
    "lastActive": "2024-01-10T14:30:00Z",
    "preferences": {"theme": "dark", "notifications": True}
}

# Projection: different views based on account type
projection = {
    "userId": "{{ userId }}",
    "displayName": "{{ name }}",
    "accountInfo": {
        "type": "{{ accountType }}",
        "status": "{{ @if_else(accountType == 'premium', 'Premium Member', 'Standard User') }}",
        "&credits": "credits",
        "hasCredits": "?{{ credits > 0 }}"
    },
    # Conditional fields based on account type
    "features": "{{ @if_else(accountType == 'premium', '{\"advancedReports\": true, \"prioritySupport\": true}', '{\"basicReports\": true}') }}",
    "limits": {
        "&maxProjects": "@if_else(accountType == 'premium', 50, 5)",
        "&storageGB": "@if_else(accountType == 'premium', 100, 10)"
    },
    "activity": {
        "lastSeen": "{{ @time_ago(lastActive) }}",
        "&isRecentlyActive": "@diff_dates(@now(), lastActive, 'days') <= 7"
    }
}

Aggregation and Statistics

Create summary statistics and aggregated data:

# Source: sales data
source = {
    "salesPerson": {"name": "Diana Prince", "region": "West"},
    "sales": [
        {"date": "2024-01-01", "amount": 1200, "product": "Software"},
        {"date": "2024-01-03", "amount": 800, "product": "Hardware"},
        {"date": "2024-01-05", "amount": 1500, "product": "Software"},
        {"date": "2024-01-07", "amount": 950, "product": "Services"}
    ]
}

# Projection: sales report
projection = {
    "salesperson": {
        "name": "{{ salesPerson.name }}",
        "region": "{{ salesPerson.region }}"
    },
    "performance": {
        "&totalSales": "@sum(@map(sales, 'amount'))",
        "&averageSale": "@avg(@map(sales, 'amount'))",
        "&salesCount": "@length(sales)",
        "&highestSale": "@max(@map(sales, 'amount'))",
        "&lowestSale": "@min(@map(sales, 'amount'))"
    },
    "breakdown": {
        "&byProduct": "@group_by(sales, 'product')",
        "&softwareSales": "@sum(@map(@filter(sales, 'product == \"Software\"'), 'amount'))",
        "&hardwareSales": "@sum(@map(@filter(sales, 'product == \"Hardware\"'), 'amount'))"
    },
    "targets": {
        "&quarterlyTarget": "15000",
        "&progressPercent": "@round(@div(@sum(@map(sales, 'amount')), 15000) * 100)",
        "&onTrack": "@sum(@map(sales, 'amount')) >= 4500"  # 30% of quarterly target
    }
}

Dynamic Field Generation

Generate fields dynamically based on data content:

# Source: product catalog
source = {
    "productId": "prod_123",
    "name": "Smart Watch",
    "categories": ["electronics", "wearable", "fitness"],
    "attributes": {
        "color": ["black", "white", "blue"],
        "size": ["small", "medium", "large"],
        "features": ["heart-rate", "gps", "waterproof"]
    },
    "pricing": {"base": 299.99, "currency": "USD"},
    "inventory": {"inStock": 45, "reserved": 5}
}

# Projection: product catalog entry
projection = {
    "id": "{{ productId }}",
    "title": "{{ name }}",
    "categorization": {
        "&primaryCategory": "@slice(@sort_by(categories, 'length'), 0, 1)[0]",
        "&allCategories": "categories",
        "&categoryCount": "@length(categories)"
    },
    "variants": {
        "&colorOptions": "attributes.color",
        "&sizeOptions": "attributes.size",
        "&totalCombinations": "@mul(@length(attributes.color), @length(attributes.size))"
    },
    "features": {
        "&featureList": "attributes.features",
        "&hasGPS": "@includes(attributes.features, 'gps')",
        "&isWaterproof": "@includes(attributes.features, 'waterproof')"
    },
    "pricing": {
        "displayPrice": "${{ pricing.base }}",
        "&numericPrice": "pricing.base",
        "currency": "{{ pricing.currency }}"
    },
    "availability": {
        "&available": "inventory.inStock - inventory.reserved",
        "&inStock": "inventory.inStock > inventory.reserved",
        "stockLevel": "{{ @if_else(inventory.inStock - inventory.reserved > 10, 'High', @if_else(inventory.inStock - inventory.reserved > 0, 'Low', 'Out of Stock')) }}"
    }
}

Projection with JMESPath Queries

Use JMESPath for complex data extraction within projections:

# Source: complex organizational data
source = {
    "company": "Tech Corp",
    "departments": [
        {
            "name": "Engineering",
            "employees": [
                {"name": "Alice", "role": "Senior Dev", "salary": 95000, "skills": ["Python", "React"]},
                {"name": "Bob", "role": "Junior Dev", "salary": 65000, "skills": ["JavaScript", "Vue"]}
            ]
        },
        {
            "name": "Marketing", 
            "employees": [
                {"name": "Carol", "role": "Manager", "salary": 85000, "skills": ["Strategy", "Analytics"]},
                {"name": "David", "role": "Specialist", "salary": 55000, "skills": ["Content", "SEO"]}
            ]
        }
    ]
}

# Projection: company analytics
projection = {
    "companyName": "{{ company }}",
    "workforce": {
        "&totalEmployees": "@length(@search(departments, '[].employees[]'))",
        "&departmentCount": "@length(departments)",
        "&avgSalary": "@avg(@map(@search(departments, '[].employees[]'), 'salary'))"
    },
    "departments": {
        "&engineeringSize": "@length(@search(departments, '[?name==`Engineering`].employees[]'))",
        "&marketingSize": "@length(@search(departments, '[?name==`Marketing`].employees[]'))",
        "&seniorStaffCount": "@length(@search(departments, '[].employees[?contains(role, `Senior`) || contains(role, `Manager`)]'))"
    },
    "skills": {
        "&allSkills": "@unique(@search(departments, '[].employees[].skills[]'))",
        "&techSkills": "@unique(@search(departments, '[?name==`Engineering`].employees[].skills[]'))",
        "&marketingSkills": "@unique(@search(departments, '[?name==`Marketing`].employees[].skills[]'))"
    },
    "compensation": {
        "&totalPayroll": "@sum(@map(@search(departments, '[].employees[]'), 'salary'))",
        "&highestPaid": "@max(@map(@search(departments, '[].employees[]'), 'salary'))",
        "&engineeringPayroll": "@sum(@map(@search(departments, '[?name==`Engineering`].employees[]'), 'salary'))"
    }
}

Advanced Projection Patterns

Special Field Selection with `_` Key

DDO provides powerful field selection capabilities using the special _ key in projections:

Include All Fields

Use {"_": "**"} to include all fields from the source data, plus any additional mappings:

# Source data
source = {
    "userId": "u123",
    "firstName": "John",
    "lastName": "Doe", 
    "email": "john@example.com",
    "age": 30,
    "department": "Engineering"
}

# Projection: include all fields plus computed ones
projection = {
    "_": "**",                                    # Include all original fields
    "fullName": "{{ firstName }} {{ lastName }}", # Add computed field
    "&isAdult": "age >= 18"                      # Add boolean field
}

# Result: All original fields + computed fields
{
    "userId": "u123",
    "firstName": "John", 
    "lastName": "Doe",
    "email": "john@example.com",
    "age": 30,
    "department": "Engineering",
    "fullName": "John Doe",      # Added
    "isAdult": True              # Added
}

Include Specified Fields Only

Use {"_": "{field1, field2, ...}"} to include only specific fields plus mappings:

# Source data
source = {
    "userId": "u123",
    "firstName": "John",
    "lastName": "Doe",
    "email": "john@example.com", 
    "age": 30,
    "department": "Engineering",
    "salary": 95000,
    "ssn": "123-45-6789"
}

# Projection: include only safe fields for API response
projection = {
    "_": "{userId, firstName, lastName, email, department}", # Only these fields
    "fullName": "{{ firstName }} {{ lastName }}",           # Plus computed field
    "&isStaff": "department != null"                        # Plus boolean
}

# Result: Only specified fields + computed fields
{
    "userId": "u123",
    "firstName": "John",
    "lastName": "Doe", 
    "email": "john@example.com",
    "department": "Engineering",
    "fullName": "John Doe",      # Added
    "isStaff": True              # Added
    # salary and ssn are excluded
}

Exclude Specified Fields

Use {"_": "?{field1, field2, ...}"} to include all fields except specified ones:

# Source data (same as above)
source = {
    "userId": "u123",
    "firstName": "John",
    "lastName": "Doe",
    "email": "john@example.com",
    "age": 30, 
    "department": "Engineering",
    "salary": 95000,
    "ssn": "123-45-6789",
    "internalNotes": "High performer"
}

# Projection: exclude sensitive fields
projection = {
    "_": "?{salary, ssn, internalNotes}",        # Exclude these fields
    "displayName": "{{ firstName }} {{ lastName }}", # Add computed field
    "&canEdit": "department == 'Engineering'"    # Add permission field
}

# Result: All fields except excluded ones + computed fields
{
    "userId": "u123",
    "firstName": "John",
    "lastName": "Doe",
    "email": "john@example.com", 
    "age": 30,
    "department": "Engineering", 
    "displayName": "John Doe",   # Added
    "canEdit": True              # Added
    # salary, ssn, internalNotes are excluded
}

Field Selection Patterns

Dynamic Field Selection

Combine field selection with template expressions for dynamic projections:

# Source: user data with varying fields
source = {
    "userId": "u456",
    "profile": {
        "firstName": "Alice",
        "lastName": "Smith",
        "email": "alice@company.com",
        "phone": "+1-555-0123",
        "address": {"city": "Portland", "state": "OR"}
    },
    "preferences": {"theme": "dark", "notifications": True},
    "metadata": {"lastLogin": "2024-01-15T10:30:00Z", "ipAddress": "192.168.1.1"}
}

# Projection: public profile (exclude metadata)
projection = {
    "_": "?{metadata}",                          # Include all except metadata
    "contactInfo": {
        "email": "{{ profile.email }}",
        "phone": "{{ profile.phone }}",
        "location": "{{ profile.address.city }}, {{ profile.address.state }}"
    },
    "&hasNotifications": "preferences.notifications"
}

Nested Field Selection

Apply field selection to nested objects:

# Source: complex user object
source = {
    "user": {
        "id": "u789",
        "personal": {
            "firstName": "Bob",
            "lastName": "Wilson", 
            "email": "bob@example.com",
            "phone": "+1-555-0456",
            "ssn": "987-65-4321"
        },
        "professional": {
            "title": "Senior Developer",
            "department": "Engineering",
            "salary": 110000,
            "startDate": "2020-03-15"
        },
        "internal": {
            "performanceRating": "Excellent",
            "notes": "Top performer",
            "nextReview": "2024-06-01"
        }
    }
}

# Projection: employee directory entry
projection = {
    "id": "{{ user.id }}",
    "personal": {
        "_": "?{ssn}",                           # Include personal info except SSN
        "fullName": "{{ user.personal.firstName }} {{ user.personal.lastName }}"
    },
    "work": {
        "_": "?{salary}",                        # Include work info except salary
        "yearsOfService": "?{{ @diff_dates(@now(), user.professional.startDate, 'years') }}"
    }
    # internal section completely excluded
}

Conditional Field Inclusion

Use field selection with conditional logic:

# Source: order data
source = {
    "orderId": "ord_123",
    "customer": {
        "id": "cust_456",
        "name": "Charlie Brown",
        "email": "charlie@example.com",
        "tier": "premium"
    },
    "items": [
        {"productId": "p1", "name": "Laptop", "price": 999.99, "cost": 600.00},
        {"productId": "p2", "name": "Mouse", "price": 29.99, "cost": 15.00}
    ],
    "payment": {
        "method": "credit_card",
        "cardLast4": "1234",
        "total": 1029.98,
        "processorFee": 30.90
    },
    "internal": {
        "profit": 384.08,
        "commission": 51.50
    }
}

# Projection: customer receipt (exclude internal data and costs)
projection = {
    "_": "?{internal}",                          # Exclude all internal data
    "customer": {
        "_": "{name, email}",                    # Only customer name and email
        "&isPremium": "customer.tier == 'premium'"
    },
    "items": "?{{ @map(items, '{\"name\": name, \"price\": price}') }}", # Items without cost
    "payment": {
        "_": "?{processorFee}",                  # Payment info without fees
        "last4": "{{ payment.cardLast4 }}"
    },
    "summary": {
        "&itemCount": "@length(items)",
        "orderTotal": "{{ payment.total }}"
    }
}

Merging Behavior

When using the _ key, the selected fields are merged with your explicit mappings:

# Source data
source = {
    "id": "item_123",
    "name": "Smart Phone",
    "price": 699.99,
    "category": "electronics",
    "inStock": True,
    "description": "Latest smartphone with advanced features",
    "internalCode": "INT_789",
    "costPrice": 450.00
}

# Projection with field selection and additional mappings
projection = {
    "_": "?{internalCode, costPrice}",           # All fields except internal ones
    "displayPrice": "${{ price }}",              # Format price for display
    "availability": "{{ @if_else(inStock, 'Available', 'Out of Stock') }}",
    "&searchText": "@lowercase(name + ' ' + category)", # For search functionality
    # Note: if there's a conflict, explicit mappings override selected fields
    "name": "{{ @titlecase(name) }}"             # Override the original name field
}

# Result: Merged fields with explicit mappings taking precedence
{
    "id": "item_123",                            # From field selection
    "name": "Smart Phone",                       # From explicit mapping (overrides)
    "price": 699.99,                            # From field selection  
    "category": "electronics",                   # From field selection
    "inStock": True,                            # From field selection
    "description": "Latest smartphone...",       # From field selection
    "displayPrice": "$699.99",                  # From explicit mapping
    "availability": "Available",                # From explicit mapping
    "searchText": "smart phone electronics"     # From explicit mapping
    # internalCode and costPrice excluded by field selection
}

Complex Field Selection Examples

Multi-level Data Sanitization

# Source: sensitive user data
source = {
    "users": [
        {
            "id": "u1",
            "profile": {
                "name": "John Doe",
                "email": "john@example.com", 
                "phone": "+1-555-0123",
                "ssn": "123-45-6789"
            },
            "account": {
                "balance": 1500.00,
                "creditScore": 750,
                "accountNumber": "ACC123456"
            },
            "preferences": {"theme": "dark"}
        }
    ]
}

# Projection: safe user list for frontend
projection = {
    "&sanitizedUsers": "@map(users, '{\"id\": id, \"profile\": {\"_\": \"?{ssn}\", \"displayName\": profile.name}, \"account\": {\"_\": \"?{accountNumber, creditScore}\"}, \"preferences\": preferences}')"
}

API Response Versioning

# Different API versions with different field requirements
base_data = {
    "productId": "p123",
    "name": "Wireless Headphones",
    "price": 199.99,
    "description": "High-quality wireless headphones",
    "specifications": {"battery": "20h", "weight": "250g"},
    "vendor": {"name": "AudioTech", "contactEmail": "vendor@audiotech.com"},
    "internal": {"cost": 120.00, "margin": 79.99}
}

# V1 API - Basic fields only
v1_projection = {
    "_": "{productId, name, price, description}",
    "formattedPrice": "${{ price }}"
}

# V2 API - Include specifications 
v2_projection = {
    "_": "?{internal}",                          # All except internal
    "specs": "?{{ specifications }}",
    "vendor": {
        "_": "?{contactEmail}",                  # Vendor without contact
        "company": "{{ vendor.name }}"
    }
}

# V3 API - Full details for partners
v3_projection = {
    "_": "**",                                   # Include everything
    "partnerId": "{{ @md5(vendor.contactEmail) }}", # Add partner identifier
    "&profitMargin": "internal.margin / price * 100"
}

The _ key provides flexible field selection that works seamlessly with DDO's template system, allowing you to create clean, secure, and appropriately scoped data projections for different use cases and audiences.

Template Composition

Build complex templates by composing simpler ones:

projection = {
    # Base user info
    "user": {
        "name": "{{ firstName }} {{ lastName }}",
        "contact": "{{ email }}"
    },
    
    # Computed properties using other projected fields
    "summary": "User {{ firstName }} {{ lastName }} has {{ @length(orders) }} orders",
    
    # Complex nested calculations
    "analytics": {
        "&lifetimeValue": "@sum(@map(orders, 'total'))",
        "tier": "{{ @switch(@sum(@map(orders, 'total')), {'< 1000': 'Bronze', '< 5000': 'Silver', '>= 5000': 'Gold'}, 'Bronze') }}",
        "&loyaltyScore": "@round(@div(@length(orders), @diff_dates(@now(), firstOrderDate, 'months')))"
    }
}

Conditional Structure

Create entirely different structures based on data:

projection = {
    # Always include basic info
    "id": "{{ userId }}",
    "type": "{{ userType }}",
    
    # Conditional sections based on user type
    "profile": "{{ @if_else(userType == 'business', '{\"company\": \"' + companyName + '\", \"industry\": \"' + industry + '\"}', '{\"firstName\": \"' + firstName + '\", \"lastName\": \"' + lastName + '\"}') }}",
    
    # Different metrics for different user types
    "metrics": {
        "&primaryMetric": "@if_else(userType == 'business', revenue, orderCount)",
        "label": "{{ @if_else(userType == 'business', 'Revenue', 'Orders') }}"
    }
}

Projection provides a powerful way to reshape and transform your data without modifying the source, making it ideal for creating views, reports, API responses, and data exports in exactly the format you need.

Projection

Projection and Data Transformation​

Understanding Projection​

Basic Projection Syntax​

Simple Field Mapping​

Nested Structure Creation​

Working with Arrays and Collections​

Conditional Projections​

Aggregation and Statistics​

Dynamic Field Generation​

Projection with JMESPath Queries​

Advanced Projection Patterns​

Advanced Projection Patterns​

Special Field Selection with _ Key​

Include All Fields​

Include Specified Fields Only​

Exclude Specified Fields​

Field Selection Patterns​

Dynamic Field Selection​

Nested Field Selection​

Conditional Field Inclusion​

Merging Behavior​

Complex Field Selection Examples​

Multi-level Data Sanitization​

API Response Versioning​

Template Composition​

Conditional Structure​

Projection and Data Transformation

Understanding Projection

Basic Projection Syntax

Simple Field Mapping

Nested Structure Creation

Working with Arrays and Collections

Conditional Projections

Aggregation and Statistics

Dynamic Field Generation

Projection with JMESPath Queries

Advanced Projection Patterns

Advanced Projection Patterns

Special Field Selection with `_` Key

Include All Fields

Include Specified Fields Only

Exclude Specified Fields

Field Selection Patterns

Dynamic Field Selection

Nested Field Selection

Conditional Field Inclusion

Merging Behavior

Complex Field Selection Examples

Multi-level Data Sanitization

API Response Versioning

Template Composition

Conditional Structure