Solr on DigitalOcean¶

This page is written for an AI coding agent working with the person who maintains a Drupal site. It creates a DigitalOcean Droplet, runs Solr on it behind HTTPS and basic authentication, and connects the Solr VDB Provider to it. A person can follow it as well.

Agents can read this page as Markdown: https://git.drupalcode.org/project/ai_vdb_provider_solr/-/raw/1.0.x/docs/ai/digitalocean.md

Rules for the agent¶

Before you run anything

Ask before anything that costs money or changes DNS. Creating the Droplet, enabling backups and adding a DNS record each need the person's yes, with the size, region and hostname stated.
Never read, print, log or commit the Solr password. It is made on the Droplet, in a file only root can read. Commands that need it read it there. The person moves it into the site's secret store.
Do not handle the DigitalOcean token. If doctl is not signed in, the person runs doctl auth init themselves.
Run every check, and stop when one fails. Report what failed and the output; do not work around it.
Ask before deleting anything, and name each resource first.

What this builds¶

Drupal site ──HTTPS 443──▶ Cloud Firewall ──▶ Droplet
                                               ├─ Caddy: TLS from Let's Encrypt,
                                               │  passes on /solr/<core>/ only
                                               └─ Solr 10: basic authentication,
                                                  one core from _default

Solr and Caddy run in Docker Compose under /srv/solr on Ubuntu 24.04.
Solr's own port, 8983, is bound to the Droplet's loopback address. Its admin UI is reached through an SSH tunnel.
The Cloud Firewall allows SSH from the person's address only, and HTTP and HTTPS from anywhere. Port 80 is for Let's Encrypt and redirects.

If the Drupal site runs on a Droplet in the same region, Solr can stay off the internet: see On a private network.

Decisions to ask for¶

Ask the person for each value, offering the default. Write them to ~/.solr-vectors.env, outside the site's repository. The file holds no secrets.

Value	Default	Notes
`SOLR_NAME`	`solr-vectors`	Names the Droplet, its tag and its firewall.
`SOLR_REGION`		The region nearest the Drupal host, such as `nyc3`, `sfo3`, `tor1`, `lon1`, `ams3`, `fra1`, `sgp1` or `syd1`.
`SOLR_SIZE`	`s-2vcpu-4gb`	See Sizing. `s-1vcpu-2gb` is enough for a trial.
`SOLR_HEAP`	`1g`	Java heap. The rest of the memory caches the vectors.
`SOLR_CORE`	`vectors`	The core's name.
`SOLR_USER`	`drupal`	The basic authentication user.
`SOLR_HOST`		A hostname for the Droplet, such as `solr.example.com`. See DNS.
`ADMIN_IP`		The person's public IPv4 address, for SSH. They can find it at their router, or with `curl -4 -fsS https://icanhazip.com`.
`SSH_KEY`		The fingerprint of an SSH key already in their DigitalOcean account.
Quantization	`scalar`	`none`, `scalar` or `binary`; set in Drupal. Scalar holds about four times as many vectors in the same memory.
Password store	environment variable	Where the Drupal host keeps the password: an environment variable such as `SOLR_PASSWORD`, or a file outside the web root.

~/.solr-vectors.env

# Solr on DigitalOcean. No secrets in this file.
SOLR_NAME=solr-vectors
SOLR_REGION=nyc3
SOLR_SIZE=s-2vcpu-4gb
SOLR_HEAP=1g
SOLR_CORE=vectors
SOLR_USER=drupal
SOLR_HOST=solr.example.com
ADMIN_IP=203.0.113.10
SSH_KEY=3b:16:bf:e4:8b:00:8b:b8:59:8c:a9:d3:f0:19:45:fa

Shell variables may not last from one command to the next, so start each command below with . ~/.solr-vectors.env.

1. Check doctl¶

doctl account get
doctl compute ssh-key list --format ID,Name,FingerPrint

If doctl account get fails, ask the person to run doctl auth init. If their key is not listed, they can add it:

doctl compute ssh-key import "$USER" --public-key-file ~/.ssh/id_ed25519.pub

2. Create the tag and the firewall¶

The firewall applies to every Droplet with the tag, so it protects the Droplet from its first boot.

. ~/.solr-vectors.env
doctl compute tag create "$SOLR_NAME"
doctl compute firewall create \
  --name "$SOLR_NAME" \
  --tag-names "$SOLR_NAME" \
  --inbound-rules "protocol:tcp,ports:22,address:$ADMIN_IP/32 protocol:tcp,ports:80,address:0.0.0.0/0,address:::/0 protocol:tcp,ports:443,address:0.0.0.0/0,address:::/0" \
  --outbound-rules "protocol:tcp,ports:all,address:0.0.0.0/0,address:::/0 protocol:udp,ports:all,address:0.0.0.0/0,address:::/0 protocol:icmp,address:0.0.0.0/0,address:::/0" \
  --format ID,Name,Status

If the Drupal host has fixed outbound addresses, ask whether to allow port 443 from those addresses only.

3. Create the Droplet¶

Confirm the size and region with the person first: this starts billing. Add --enable-backups if they want DigitalOcean's daily backups, at extra cost (see Backups).

. ~/.solr-vectors.env
doctl compute droplet create "$SOLR_NAME" \
  --image ubuntu-24-04-x64 \
  --size "$SOLR_SIZE" \
  --region "$SOLR_REGION" \
  --ssh-keys "$SSH_KEY" \
  --tag-names "$SOLR_NAME" \
  --enable-monitoring \
  --wait \
  --format ID,Name,PublicIPv4,PrivateIPv4,Status
echo "SOLR_IP=$(doctl compute droplet get "$SOLR_NAME" --format PublicIPv4 --no-header)" >> ~/.solr-vectors.env

Wait for SSH, and accept the new host key:

. ~/.solr-vectors.env
for wait in 5 8 13 21 34; do
  ssh -o StrictHostKeyChecking=accept-new -o ConnectTimeout=5 root@"$SOLR_IP" true && echo "SSH answers." && break
  echo "Waiting ${wait}s for SSH."
  sleep "$wait"
done

4. Install Docker¶

. ~/.solr-vectors.env
ssh root@"$SOLR_IP" 'bash -s' <<'EOF'
set -euo pipefail
cloud-init status --wait
apt-get update
DEBIAN_FRONTEND=noninteractive apt-get install -y docker.io docker-compose-v2
docker compose version
EOF

5. Point a hostname at it¶

Caddy needs a hostname that resolves to the Droplet before it can get a certificate. Ask the person which applies:

The domain's DNS is at DigitalOcean (doctl compute domain list shows it). With their yes, add the record, here solr in example.com:

. ~/.solr-vectors.env
doctl compute domain records create example.com \
  --record-type A --record-name solr --record-data "$SOLR_IP" --record-ttl 300

The DNS is elsewhere. The person adds an A record for SOLR_HOST with the Droplet's address, SOLR_IP.
A trial, with no domain. sslip.io resolves a name made of the address to the address itself. Set SOLR_HOST in ~/.solr-vectors.env to the output of the command below. Let's Encrypt's rate limits are shared by every user of sslip.io, so do not rely on it for a production site.
```
. ~/.solr-vectors.env
echo "$(echo "$SOLR_IP" | tr . -).sslip.io"
```

Check that the name resolves to the Droplet before going on:

. ~/.solr-vectors.env
[ "$(dig +short "$SOLR_HOST" | tail -n1)" = "$SOLR_IP" ] && echo "$SOLR_HOST resolves to $SOLR_IP." || echo "$SOLR_HOST does not resolve to $SOLR_IP yet."

6. Make the password and Solr's security.json¶

This makes a random password in /srv/solr/password, readable by root only, and writes Solr's security.json with its hash. Solr stores a password as base64(sha256(sha256(salt + password))), a space, and base64(salt).

. ~/.solr-vectors.env
ssh root@"$SOLR_IP" "SOLR_USER=$SOLR_USER bash -s" <<'EOF'
set -euo pipefail
install -d -m 700 /srv/solr
install -d -o 8983 -g 8983 -m 750 /srv/solr/var /srv/solr/var/data
umask 077
[ -s /srv/solr/password ] || openssl rand -base64 48 | tr -d '\n/+=' | cut -c1-40 > /srv/solr/password
python3 - <<'PY'
import base64, hashlib, json, os
user = os.environ["SOLR_USER"]
password = open("/srv/solr/password", "rb").read().strip()
salt = os.urandom(32)
digest = hashlib.sha256(hashlib.sha256(salt + password).digest()).digest()
security = {
    "authentication": {
        "class": "solr.BasicAuthPlugin",
        "blockUnknown": True,
        "realm": "solr",
        "credentials": {
            user: base64.b64encode(digest).decode() + " " + base64.b64encode(salt).decode(),
        },
    },
}
with open("/srv/solr/var/data/security.json", "w") as f:
    json.dump(security, f, indent=2)
PY
chown 8983:8983 /srv/solr/var/data/security.json
echo "Wrote security.json for $SOLR_USER."
EOF

Running this again keeps the password and rewrites security.json; restart Solr afterwards with docker compose restart solr in /srv/solr. blockUnknown refuses every request without the user's credentials. With one user and no authorization plugin, that user may do everything, which the provider needs: it changes the core's schema.

7. Start Solr and Caddy¶

. ~/.solr-vectors.env
ssh root@"$SOLR_IP" "SOLR_CORE=$SOLR_CORE SOLR_HEAP=$SOLR_HEAP SOLR_HOST=$SOLR_HOST bash -s" <<'EOF'
set -euo pipefail
cat > /srv/solr/compose.yaml <<YAML
services:
  solr:
    image: solr:10.0.0
    command: solr-precreate ${SOLR_CORE}
    environment:
      SOLR_HEAP: ${SOLR_HEAP}
    volumes:
      - /srv/solr/var:/var/solr
    ports:
      - 127.0.0.1:8983:8983
    restart: unless-stopped
  caddy:
    image: caddy:2
    ports:
      - 80:80
      - 443:443
    volumes:
      - /srv/solr/Caddyfile:/etc/caddy/Caddyfile:ro
      - caddy_data:/data
      - caddy_config:/config
    restart: unless-stopped
volumes:
  caddy_data:
  caddy_config:
YAML
cat > /srv/solr/Caddyfile <<CADDY
${SOLR_HOST} {
  @core path /solr/${SOLR_CORE} /solr/${SOLR_CORE}/*
  handle @core {
    reverse_proxy solr:8983
  }
  handle {
    respond 404
  }
}
CADDY
cd /srv/solr
docker compose up -d
docker compose ps
EOF

solr-precreate makes the core from the _default configset on the first start, and leaves it alone after that. Caddy passes on requests for the core only; Solr's admin API and UI answer 404 from outside.

8. Check it¶

On the Droplet, so the password stays there:

. ~/.solr-vectors.env
ssh root@"$SOLR_IP" "SOLR_CORE=$SOLR_CORE SOLR_USER=$SOLR_USER SOLR_HOST=$SOLR_HOST bash -s" <<'EOF'
auth="$SOLR_USER:$(cat /srv/solr/password)"
for wait in 2 3 5 8 13 21 34; do
  if curl -fsS -u "$auth" "http://127.0.0.1:8983/solr/$SOLR_CORE/admin/ping?wt=json" | grep -q '"status":"OK"'; then
    echo "Solr's $SOLR_CORE core answers."
    break
  fi
  echo "Waiting ${wait}s for Solr."
  sleep "$wait"
done
for wait in 2 3 5 8 13 21 34; do
  if curl -fsS -u "$auth" "https://$SOLR_HOST/solr/$SOLR_CORE/admin/ping?wt=json" | grep -q '"status":"OK"'; then
    echo "https://$SOLR_HOST/solr/$SOLR_CORE answers with its certificate."
    exit 0
  fi
  echo "Waiting ${wait}s for Caddy's certificate."
  sleep "$wait"
done
echo "The core did not answer over HTTPS."
docker compose -f /srv/solr/compose.yaml logs --tail 50
exit 1
EOF

If 443 is limited to the Drupal host's addresses, the HTTPS check from the Droplet fails; skip it and rely on step 10.

From outside, without credentials, the core must refuse and everything else must be missing:

. ~/.solr-vectors.env
curl -s -o /dev/null -w "core without credentials: %{http_code} (want 401)\n" "https://$SOLR_HOST/solr/$SOLR_CORE/admin/ping"
curl -s -o /dev/null -w "admin API: %{http_code} (want 404)\n" "https://$SOLR_HOST/solr/admin/cores"
curl -s -o /dev/null -w "Solr's port: %{http_code} (want 000)\n" --connect-timeout 5 "http://$SOLR_IP:8983/solr/"

9. Give Drupal the password¶

Ask the person to copy the password into the Drupal host's secret store, under the name they chose, such as the environment variable SOLR_PASSWORD. They can read it with the command below. Do not run it yourself.

. ~/.solr-vectors.env && ssh root@"$SOLR_IP" cat /srv/solr/password

Then create a Key that points at it. The Key holds the variable's name, not the password. Run Drush the way the site runs it, such as ddev drush or terminus drush <site>.<env> --.

drush key:save solr_password --label="Solr password" --key-type=authentication \
  --key-provider=env \
  --key-provider-settings='{"env_variable":"SOLR_PASSWORD","base64_encoded":false,"strip_line_breaks":true}' \
  --key-input=none

For a file outside the web root, use --key-provider=file --key-provider-settings='{"file_location":"/path/to/solr_password","strip_line_breaks":true}'. If the host has a secret store with its own Key provider, use that provider. Do not use the config provider: it puts the password in exported configuration.

10. Connect the provider¶

Use the values from ~/.solr-vectors.env; Drush may run where that file is not.

drush config:set ai_vdb_provider_solr.settings scheme https -y
drush config:set ai_vdb_provider_solr.settings host solr.example.com -y
drush config:set ai_vdb_provider_solr.settings port 443 -y
drush config:set ai_vdb_provider_solr.settings path / -y
drush config:set ai_vdb_provider_solr.settings core vectors -y
drush config:set ai_vdb_provider_solr.settings username drupal -y
drush config:set ai_vdb_provider_solr.settings password_key solr_password -y
drush config:set ai_vdb_provider_solr.settings quantization scalar -y

Check that Drupal reaches the core:

drush php:eval '$solr = \Drupal::service("ai.vdb_provider")->createInstance("solr"); echo $solr->ping() ? "Solr answers.\n" : "Solr does not answer.\n"; echo "Solr ", $solr->getClient()->solrVersion() ?? "version unknown", "\n";'

If Solr does not answer, check that PHP sees the variable, without showing it. Never use drush key:value-get here: it prints the password.

drush php:eval 'echo getenv("SOLR_PASSWORD") === false ? "SOLR_PASSWORD is not set.\n" : "SOLR_PASSWORD is set.\n";'

Drush and the web server can see different environments; the settings form at /admin/config/ai/vdb_providers/solr pings from the web server when it saves.

Neither the settings nor the Key hold the password, so both can be exported with the site's configuration. Environments that use other Solr servers can override the settings in settings.php, such as $config['ai_vdb_provider_solr.settings']['host'] = 'localhost';.

11. Create the AI Search server¶

The person, or an agent with a browser, at /admin/config/search/search-api/add-server:

Name the server.
Choose the AI Search backend and an Embeddings Engine.
Choose Apache Solr as the Vector Database.
Under Vector Database Configuration, give a Collection name and choose the Similarity Metric.
Leave Include raw embedding vector in results off.
Save. The collection is created in the core.

Then add an index on the server, and index every enabled index:

drush search-api:index

The server's page shows the core, the Solr version, the vector field and the number of chunks.

On a private network¶

When the Drupal site runs on a Droplet in the same region, the two can talk over the region's VPC, and Solr needs no hostname, certificate or Caddy:

Skip step 5, and leave Caddy out of compose.yaml.
Publish Solr on the Droplet's private address instead of loopback: - <PrivateIPv4>:8983:8983.
In the firewall, replace the 80 and 443 rules with protocol:tcp,ports:8983,tag:<the Drupal Droplet's tag>.
Keep basic authentication.
In step 8, check the private address in place of 127.0.0.1, and skip the HTTPS checks.
In step 10, use scheme http, the private address as host, and port 8983.

Running it¶

Sizing¶

A collection's vectors should fit in the memory left after Solr's heap and the system, so the system can cache them. Each dimension of a vector takes 4 bytes unquantized, about 1 byte scalar, or 1 bit binary, and the search graph adds to that. For example, 100,000 chunks of 1,536 dimensions take about 600 MB unquantized and about 150 MB scalar.

To resize, power the Droplet off, and with the person's yes:

. ~/.solr-vectors.env
doctl compute droplet-action power-off "$(doctl compute droplet get "$SOLR_NAME" --format ID --no-header)" --wait
doctl compute droplet-action resize "$(doctl compute droplet get "$SOLR_NAME" --format ID --no-header)" --size s-4vcpu-8gb --wait
doctl compute droplet-action power-on "$(doctl compute droplet get "$SOLR_NAME" --format ID --no-header)" --wait

Raise SOLR_HEAP in /srv/solr/compose.yaml to match, then docker compose up -d in /srv/solr.

Backups¶

The vectors come from the site's content: a new Droplet and a full reindex rebuild them, at the cost of the embedding calls. DigitalOcean's backups avoid that cost:

. ~/.solr-vectors.env
doctl compute droplet-action enable-backups "$(doctl compute droplet get "$SOLR_NAME" --format ID --no-header)"

Updates¶

Ubuntu installs security updates on its own; it does not reboot on its own. To update Solr or Caddy, change the image in /srv/solr/compose.yaml, then:

. ~/.solr-vectors.env
ssh root@"$SOLR_IP" 'cd /srv/solr && docker compose pull && docker compose up -d && docker compose ps'

Read Solr's upgrade notes before a major version. Caddy renews the certificate on its own.

Logs and the admin UI¶

. ~/.solr-vectors.env
ssh root@"$SOLR_IP" 'cd /srv/solr && docker compose logs --tail 100'

For Solr's admin UI, open a tunnel and browse to http://localhost:8983/solr/, signing in as SOLR_USER:

. ~/.solr-vectors.env
ssh -N -L 8983:127.0.0.1:8983 root@"$SOLR_IP"

Removing it¶

Only with the person's yes, naming each resource. This deletes the core and every collection in it.

. ~/.solr-vectors.env
doctl compute droplet delete "$SOLR_NAME"
doctl compute firewall delete "$(doctl compute firewall list --format ID,Name --no-header | awk -v n="$SOLR_NAME" '$2 == n { print $1 }')"
doctl compute tag delete "$SOLR_NAME"

Remove the DNS record, the Key and the Search API server as well.