Export all work order data to data warehouse

Hi Fatima — great to see this level of planning going into the pipeline. Let me address each point with what we've seen work at similar scale.

Cursor-based pagination

We don't currently support cursor-based pagination (I know, I know — it's on the roadmap). What we do have is offset-based pagination with a stable sort on updated_at. The pattern that works for incremental sync is:

GET /v1/work_orders?updated_since={last_sync_timestamp}&sort=updated_at&order=asc&limit=500

The key gotcha: if a record gets updated during your pagination walk, it can appear twice (once on the current page, once on a later page). We recommend deduplicating on id in your staging layer.

Rate limits

100 req/min for standard keys, 500 req/min for enterprise. At 50k records with 500/page, you're looking at ~100 requests for a full sync — well within limits. The pain point is historical backfill: 5 years × 50k = 250k records = 500 requests, which is fine, but you'll want to throttle to avoid hammering the API.

Deleted records

We soft-delete (status = archived). Hard deletes are rare and require support intervention. For your warehouse, I'd recommend treating archived records as "deleted" in your BI layer but keeping them in cold storage for compliance.

Warehouse connectors

No native connector today. API-only, though we're in early talks with Fivetran about a partner-built connector — nothing committed.

Happy to share a Python sample that implements the incremental pattern with backoff/retry if that would help.

YMMV on the offset pagination — we tried exactly what Eli described and hit edge cases during our fiscal year-end when dispatchers were bulk-updating hundreds of records. The stable sort on updated_at held, but we saw some records with identical timestamps get reordered across pages.

Our workaround: we added a secondary sort on id and dedupe aggressively in staging. Heads up that the API doesn't document this clearly, but sort=updated_at,id does work.

Also worth noting: custom fields come back as a nested object in the JSON, not flat columns. Your ETL will need to handle that flattening unless you want variant schemas in Snowflake.

"custom_fields": {
  "cf_123": {"label": "Equipment Type", "value": "HVAC-R"},
  "cf_456": {"label": "Warranty Expiry", "value": "2026-03-15"}
}

We ended up building a type-2 slowly-changing dimension table for the custom fields specifically since they change more often than the base work order data.

Eli, Devon — thank you for these detailed responses. The timestamp collision edge case is precisely the type of operational risk our governance framework requires us to document.

A follow-up question regarding the updated_since parameter: I have confirmed in our testing that this filter operates on the updated_at column. However, does this include system-generated updates (e.g., webhook delivery confirmations, background sync operations) or only user-initiated mutations? From a data lineage perspective, we need to distinguish between substantive business changes and technical bookkeeping.

Additionally, regarding archived records: Is there a separate endpoint or parameter to retrieve archived work orders, or must we include status=archived explicitly in our query and merge streams?

Good catches — let me clarify both.

System-generated updates: updated_at is currently bumped by any write to the record, including system operations. We don't expose a separate "business_updated_at" timestamp, though I've logged that as a feature request (internal ref: DW-2847). For now, you'd need to diff the payload yourself or maintain a hash of business-relevant fields to detect substantive changes.

Archived records: By default, the endpoint excludes archived. Use include_archived=true to get them in the same stream. I'd recommend:

GET /v1/work_orders?updated_since={ts}&include_archived=true&sort=updated_at,id

Then filter downstream in your warehouse transform.

One more gotcha I should've mentioned: the updated_since parameter is inclusive of the boundary timestamp, so your next sync should use {max(updated_at) from previous batch} plus 1 microsecond to avoid overlap. Or use strict greater-than if your orchestration supports it.

• Include _meta.request_id in your logging for debugging pagination anomalies
• Consider the API Pagination for Large Data Exports workaround pattern
• Rate limit headers: X-RateLimit-Remaining, X-RateLimit-Reset

Docs on incremental sync: API Rate Limits and Best Practices (section on backpressure)