Skip to content

Syncing data

How syncing works

Each submodule fetches data from the Blizzard API and stores it locally in Drupal. There are two sync patterns:

Catalog sync (reference data)

Realms, classes, races, specializations, power types, and reputation factions are synced by fetching an index endpoint from Blizzard, then creating or updating a local taxonomy term for each item.

These syncs use conditional requests — they send If-Modified-Since from the previous sync. If Blizzard responds with "304 Not Modified", no data is written and the operation completes instantly. Use --force to bypass this.

Queue-based sync

Large catalogs — achievements, mounts, pets, titles, toys — are synced one entity per Advanced Queue job rather than in a single request. The module enqueues the jobs; the queue processor (cron or a dedicated worker) drains them. On the dashboard, Sync now drains the queue inline as a Drupal batch with a progress bar.

Sync vs refresh

  • Sync = update a catalog or reference-data vocabulary from its Blizzard index endpoint. Affects the "what exists" dictionary; never touches per-character data directly.
  • Refresh (characters and guilds) = re-fetch a single profile from /profile/wow/character/{realm}/{name} or the guild equivalent, update the owning entity, and fire registered CharacterDataProvider / GuildDataProvider plugins so their per-owner rows land too.

Both flow through the same BattleNetClient, so both count against rate limits.

For a fresh installation, sync in this order:

  1. Reference data — realms, classes, races, specializations, power types, reputation factions
  2. Achievement catalog — categories, then achievements
  3. Collection catalogs — mounts, pets, titles, toys
  4. Characters and guilds — imported individually with wow:character-lookup / wow:guild-lookup; their collection and reputation data is pulled automatically by registered providers.

Characters reference realms, classes, and races; collection tracking references the catalog entities. Syncing in this order avoids temporary gaps.

Automating with cron

For ongoing maintenance:

  • Cron processes the Advanced Queue automatically and runs the TTL sweep at most once every 23 hours.
  • Scheduled drush commands re-sync reference data periodically (weekly or monthly is sufficient — this data rarely changes).
  • Character refreshes happen on demand (e.g., when a player visits their profile) or via a scheduled task you wrap around CharacterSync::refresh(). There is no built-in bulk-refresh command.

A CharacterSync::refresh() that hits a 403 / 404 from Blizzard returns NULL rather than deleting the entity; stale characters are cleaned up by the TTL sweeper, not by the refresh path.

Rate limiting

The module enforces both Blizzard rate-limit ceilings (36,000 requests per hour and 100 per second) across every PHP process (web, drush, queue workers). Counters are stored in Drupal State and check+increment is serialized under a lock. If a request would exceed either ceiling, BattleNetClient::get() throws a RuntimeException — queue workers naturally pace themselves through the Advanced Queue processor, and ad-hoc drush calls surface the exception to the operator. See API compliance.

Data freshness and the 30-day rule

Blizzard's API Terms of Use require all cached data to be refreshed or deleted within 30 days. Two freshness fields exist for technical reasons:

  • Content entities (characters, guilds, achievements, mounts, pets, titles, toys, items, and their per-owner junction rows) use a last_fetched base field.
  • Taxonomy reference data (realms, classes, races, specializations, power types, reputation factions) uses a wow_last_fetched configurable field — Drupal's taxonomy_term entity can't host arbitrary base fields, so submodules ship it as config.

The wow.ttl_sweeper service runs daily under cron (hook_cron throttled to once per 23 hours) and deletes content entities whose last_fetched exceeds 30 days, in chunks of 50 to keep memory bounded. Both catalog entities (mounts, pets, titles, etc.) and per-owner junction rows (collected mounts, achievement progress, etc.) are swept independently — junctions need their own TTL because Blizzard's API may return a partial collection on a character refresh and because un-collected items disappear from the API but linger locally until swept.

Reference data is not swept — operators keep it fresh with the scheduled sync commands (drush wow:sync-realms, etc.). For a one-shot sweep:

drush wow:sweep-ttl

See API compliance for the full contract.