# Scheduled work and jobs

Set up background work and operate scheduled execution.

## Scheduler and jobs

For manual VPS deployments, install the scheduler orchestrator once per server.
Register every deployed project; scheduling is required even without application-specific tasks:

```bash
sudo php /var/www/site/core/cli/nimbly.php scheduler:install
sudo php /var/www/site/core/cli/nimbly.php scheduler:project:add site /var/www/site
sudo php /var/www/site/core/cli/nimbly.php scheduler:cron:install --user=www-data
```

This creates `/etc/cron.d/nimbly-scheduler`, which runs
`/usr/local/bin/nimbly-scheduler-orchestrator` every minute as `www-data`, and
`/etc/logrotate.d/nimbly-scheduler`, which rotates the scheduler log weekly. The
orchestrator then dispatches registered project schedulers independently. In Docker
containers the scheduler runs automatically, so no host cron setup is required.

Run `schedule:init` once to create project schedule files in `ext/cli/`:

```bash
./nimbly schedule:init
```

This generates three files:

| File | Used when |
|---|---|
| `ext/cli/schedule.inc` | `APP_ENV=dev` or no env-specific file found |
| `ext/cli/schedule.prod.inc` | `APP_ENV=prod` |
| `ext/cli/schedule.stage.inc` | `APP_ENV=stage` |

The prod and stage schedules include `ext:sync`, which commits and pushes data changes in `ext/` back to the repository every hour. The dev schedule does not. Customize each file independently for environment-specific tasks.

When migrating an existing VPS, first install the orchestrator, register the
projects, run `scheduler:run --dry-run`, run the wrapper once
manually as `www-data`, and confirm `/var/log/nimbly-scheduler.log`. Back up the
old `www-data` crontab under `/etc/nimbly/backups/`, then remove only migrated
per-project `schedule:run` lines. Do not remove unrelated cron jobs.

Admin `git pull` remains available under Settings as a simple self-managed update option. For release-managed sites, prefer a repeatable deploy path: pull the intended git revision, run setup/build/lint, and make sure the scheduler is active.

---

## `schedule:run`
Runs due scheduled commands for one project. On manual VPS servers with multiple
Nimbly projects, do not create one cron entry per project. Use the server-level
scheduler orchestrator instead and let it call each registered project's
`schedule:run` sequentially.

Projects may define schedules in `ext/cli/`. The scheduler selects files in this order:

1. `SCHEDULE_FILE` from `.env` or the process environment, as an explicit file path.
2. `ext/cli/schedule.<env>.inc`, where `<env>` comes from `SCHEDULE_ENV`, `APP_ENV`, or `NIMBLY_ENV`.
3. `ext/cli/schedule.inc`.
4. `core/cli/schedule.inc`.

Environment aliases are normalized: `production` → `prod`, `staging` → `stage`, `development` and `local` → `dev`.

Use environment-specific schedule files when staging or development must run background jobs but must not run production-only tasks such as member reminders.

Schedule tasks may use `every_minutes` for intervals longer than one minute:

```php
[
    'id' => 'uptime-poll',
    'command' => 'uptime:poll',
    'every_minutes' => 2,
]
```

The value must be a positive integer. A new task is due immediately; after it
runs, it becomes due when the clock enters the next interval bucket. Existing
`every`, `daily_at`, and optional `timezone` schedules remain supported.

Scheduler last-run state is stored in `ext/data/.state/schedule`. Existing installs with older `ext/data/.config/schedule` state are read as a migration fallback until the scheduler writes the new state file.

## `scheduler:*`
Runs multiple project schedulers from one server-level cron. The project schedule
files remain project-local in `ext/cli/`; the server registry is the only source
of truth for which projects are orchestrated.

The orchestrator uses these server-managed files by default:

| Path | Purpose |
|---|---|
| `/usr/local/bin/nimbly-scheduler-orchestrator` | Wrapper run by cron |
| `/etc/nimbly/scheduler-projects.json` | Registered projects and defaults |
| `/etc/cron.d/nimbly-scheduler` | System cron entry |
| `/var/log/nimbly-scheduler.log` | Scheduler output log |
| `/etc/logrotate.d/nimbly-scheduler` | Weekly rotation of the log, eight weeks kept |

Install it once per server from any current Nimbly core checkout:

```bash
sudo php core/cli/nimbly.php scheduler:install
sudo php core/cli/nimbly.php scheduler:project:add site-name /var/www/site-name
sudo php core/cli/nimbly.php scheduler:cron:install --user=www-data
```

Useful management commands:

```bash
php core/cli/nimbly.php scheduler:project:list
sudo -u www-data php core/cli/nimbly.php scheduler:run --dry-run
sudo -u www-data /usr/local/bin/nimbly-scheduler-orchestrator
php core/cli/nimbly.php scheduler:cron:status
```

`scheduler:run` takes a short global lock and dispatches two workers per
registered project: one runs `jobs:run 10` for prompt queue delivery, and the
other runs the project's `schedule:run`. Workers log their completion and exit
code to the host scheduler log. A long application task therefore does not
delay another project's minute tasks or its own queued alerts. Job runners use
a per-project lock so the direct queue worker and mandatory `jobs-run` task do
not dispatch the same job concurrently. The project scheduler retains its own
lock, so later cron ticks do not duplicate a long schedule run. Legacy
`default_delay_after_seconds` values in the registry are ignored.

`scheduler:run` itself always dispatches every registered project's workers
immediately (it never waits before spawning), so a slow project cannot delay
the next cron tick's dispatch. Concurrency is instead capped inside each
`scheduler:worker`: before running the project's real `jobs:run`/`schedule:run`,
a worker blocks on a shared slot semaphore, and releases it when done. This
keeps a large fleet from pushing every project's worker onto a single-CPU host
at the same instant. The cap defaults to 5 simultaneous workers across the
whole host and is configurable with `NIMBLY_SCHEDULER_MAX_CONCURRENCY`; the
slot lock files themselves live under `NIMBLY_SCHEDULER_SEMAPHORE_DIR`
(default: the system temp directory).