Labelled copies of audio and video¶

ai_disclosure_av prepares labelled copies of audio and video files. The original upload is never changed. A copy is a separate file, called a rendition, and the video and audio formatters show it once it is ready.

This page describes the infrastructure: the settings, the queue, the storage and the formatter swap. The two renderers that ship with it are described in the Media section: the icon in the picture of a video, see The icon in the picture, and a spoken disclaimer in front of audio, see A spoken disclaimer first. Renderers plug into the interface described under Writing a renderer.

It is a separate submodule: installing ai_disclosure does not enable it, and the main module gains no dependency. It needs the core media and file modules and the FFmpeg binaries on the server. It needs no Composer package: processes run through Symfony Process, which ships with core.

drush en ai_disclosure_av

Requirements¶

The submodule needs two things on the server and in the file system.

ffmpeg and ffprobe on the server.
A private file system for the copies, see Where the copies are stored.
A working cron, or the Drush commands below.

Without ffmpeg and ffprobe nothing is queued: a job that can only fail is not created. The features that need them stay on their pages, disabled, with one line under each: "FFmpeg is not installed, so making labelled copies is off", and a link to the requirements. This covers the settings of this tab, the icon schedule and the spoken disclaimer language on the media form, and the Record and Generate with AI buttons. The two path fields stay editable, so you can fix the path. Once the binaries run, the files are queued the next time their media is saved, or when ai-disclosure-av:requeue runs.

The status report lists what the submodule found:

FFmpeg and ffprobe: whether the binary runs, and its version. An error when it does not.
FFmpeg filters: whether this build of FFmpeg has the overlay and concat filters, which the renderers build on. A warning when one is missing.
Storage of the labelled copies: whether the file system of the copies exists and is writable. An error when it does not, a warning when it is the public one.
Labelled copies: how many labelled copies are ready, waiting, failed or skipped. Informational when no module makes labelled copies.
Clips: a warning when an audio file chose a language that has lost its clip, or that is no longer active on the site, and so follows the default rule. Informational when an active language has no clip yet.
Generate with AI (speech): whether a provider for the Generate with AI button is available. Informational when it is not.

Settings¶

The "Video and audio" tab, under Media, at /admin/config/content/ai-disclosure/video-audio, needs the "administer ai disclosure" permission.

FFmpeg path and ffprobe path. A name found in the PATH, such as ffmpeg, which is the default, or a full path. Saving runs <binary> -version and refuses a path that prints no version.
Store labelled copies in. Private files, the default, or public files.
Longest file to label. In seconds, 3600 by default. A longer file gets no labelled copy and keeps its original.
Video. The schedule and the size of the icon, see The icon in the picture.
Server load. FFmpeg threads, the video preset and a cron time budget, see Low-resource servers.

Where the copies are stored¶

A copy is written to {scheme}://ai_disclosure/renditions/{fid}-{hash}.{ext}, where fid is the ID of the original file.

The default is the private file system, because a copy of a file that only some users can see must only be served to those users. When the site has no private file system, the form says so and offers only public files, with a warning: anyone who knows or guesses the URL of a copy can download it, whoever may view the media. The status report carries the same warning. The install does not fail either way, because the private file system is a setting of the site and may be added later. Until then the renditions fail with a message that names the directory.

Set $settings['file_private_path'] in settings.php to use the private file system.

Even when the storage is set to public files, the copy of an original that is not public is written to the private file system, because a public URL skips the access check. When the site has no private file system, that copy is not made and the status messages report the failure.

When the storage changes and copies or spoken disclaimer clips already exist, the form asks before saving. "Move" moves them to the new file system in a batch, as file entities, so their IDs and the media that use them stay the same. "Leave" keeps them where they are, still served, and only new files go to the new file system.

A copy of a non-public file always stays private. Choosing the public scheme makes the URLs of the other copies reachable by anyone who knows them, whoever may view the media.

A private copy is served by Drupal, and a user may download it when they can view a media that uses the original file. That check runs in hook_file_download(), and it denies the download when no such media exists.

When a copy is made¶

No work is done while a page is built, so a visitor never waits for a copy. The work happens in two steps.

Queueing. When a media item is saved, the submodule looks at the audio and video files of its source field, with a MIME type of video/* or audio/*. For each file it asks ai_disclosure.file_resolver what the file must say, then asks the renderers which one applies, and computes the hash. If there is no row for that file and hash, it adds a queued row and a queue item. The same happens when another entity with a disclosure field is saved and shares a file with a media, because the disclosure of such an entity can decide the file.

When no renderer applies any more, for example because the label is no longer required, the renditions of the file and their files are removed. An old copy is never served for a file that no longer needs one.

When the disclosure changes but the file still needs a label, the newest ready copy keeps being served, and the download check allows it, until the copy for the new disclosure is ready. A file that shows a label with an old sentence is safer than one that shows none. A page view queues the new copy if no row exists for it.

Working. The queue worker ai_disclosure_av_render runs on cron, with 120 seconds per run. For each item it:

Checks that the hash is still the current one. If the file or its disclosure changed meanwhile, the row is skipped and the newer row takes over.
Runs ffprobe to read the duration and the streams.
Skips the file when the duration is above the maximum. The row is skipped and the message says why.
Asks the renderer again, now with the probe. A renderer that no longer applies makes the row skipped.
Runs the renderer into a temporary file, moves it into the rendition directory and sets the row ready with the FFmpeg version.
Removes the older renditions of the same file, rows and files.

When anything throws, the row is failed and holds the message, usually the end of the output of ffmpeg. Nothing is retried by itself, because a render that failed once will fail again until something changes. Saving the media again does not retry it either. ai-disclosure-av:requeue does.

Drush¶

drush ai-disclosure-av:process [--limit=N]
drush ai-disclosure-av:requeue

ai-disclosure-av:process works the queue now, without waiting for cron. --limit stops after N renditions.

ai-disclosure-av:requeue goes through every audio and video file used by a media. It queues the files whose hash is new and queues again the renditions that failed or were skipped. Use it after a grade, a profile or a setting of a renderer changed. Those changes are not followed by themselves, only when the media is next saved, which is the lazy part of the design: saving a grade does not queue thousands of renders.

The commands are found by Drush 12.5 and later and by Drush 13. Drush is not a dependency. Older Drush 12 versions do not find them, use cron there.

The hash¶

A rendition is named by a hash: the SHA-256 of these values, cut to 16 hex characters.

the file ID, and the changed time and size of the file
the icon, the label requirement and the modality of the resolved disclosure
the IPTC URI of the grade, and its disclosure sentence
the ID of the renderer and its settingsVersion()
for a renderer that implements FileVersionedRendererInterface, its fileVersion() of the file. The video renderer returns its schedule and its icon position. Other renderers keep the hashes they had

A change of any of them gives a new hash, so a new rendition is made and the old one is removed once the new one is ready. Until then the old one is served.

The table¶

The table ai_disclosure_rendition has one row per file and hash, unique on the two:

Column	Holds
`id`	the row ID, also the payload of the queue item
`fid`	the ID of the original file
`hash`	the hash
`renderer`	the ID of the renderer
`uri`	the URI of the copy, empty until it is ready
`status`	`queued`, `ready`, `failed` or `skipped`
`ffmpeg_version`	the version that made the copy
`message`	why it failed or was skipped
`created`, `changed`	Unix timestamps

ai_disclosure_av.storage reads and writes it. Deleting a file deletes its rows and their files.

Low-resource servers¶

Only one copy is made at a time: a lock stops cron, Drush and a second cron run from starting FFmpeg while another copy is being made. The one that finds the lock taken leaves the item in the queue. On a small or shared server, three settings under Server load on the settings page limit the rest:

FFmpeg threads (0). Passes -threads N to every encode. 0 lets FFmpeg decide.
Video encoding speed (very fast). The x264 preset of the video copies: medium, fast, very fast or ultra fast. A faster preset needs less time and CPU and makes a bigger file. It does not apply to the MPEG-4 encoder used when FFmpeg has no libx264.
Cron time budget (0 s). A cron run starts no new copy once this time has passed. The copy that is running is not stopped. 0 sets no limit of its own: cron gives the queue 120 s, so a longer budget changes nothing.

None of them changes the hash, so a copy that exists stays valid.

A site that cannot render at all can render on another machine and upload the result. Both commands work for the video and the audio renderer.

drush ai-disclosure-av:render-command <rendition id>
drush ai-disclosure-av:attach <rendition id> <path>

render-command prints the FFmpeg commands the site would run for a queued or failed rendition, and the files to copy. Nothing is rendered. The commands read the files by their names, in the directory they run in, and write the output file. The icon variant of a video is measured on the site before the commands are printed, with a few quick reads of single frames.

$ drush ai-disclosure-av:render-command 12
# Files to copy to one directory on the other machine:
# /var/www/private/video/talk.mp4
# /var/www/html/modules/contrib/ai_disclosure/assets/icons/png/ai_generated--black--960.png
# Run, in that directory:
ffmpeg -nostdin -hide_banner -loglevel error -y -i talk.mp4 -i ai_generated--black--960.png -filter_complex '[1:v]scale=212:-1[ic];[0:v][ic]overlay=W-w-32:32:enable='\''lt(t,5)+lt(mod(t,60),5)'\''[v]' -map '[v]' -map '0:a?' -c:v libx264 -preset veryfast -crf 20 -pix_fmt yuv420p -c:a copy -movflags +faststart+use_metadata_tags -metadata 'comment=This is synthetic media produced by AI.' 7-3f9c1a2b4d5e6f70.mp4
# Then bring back 7-3f9c1a2b4d5e6f70.mp4 and run: drush ai-disclosure-av:attach 12 <path to 7-3f9c1a2b4d5e6f70.mp4>

Copy the files, run the commands, then give the result back to the site:

drush ai-disclosure-av:attach 12 /tmp/7-3f9c1a2b4d5e6f70.mp4

attach checks that the rendition is still queued or failed and still current, that the file is not empty and has the extension the renderer writes. Then it moves the file into the rendition directory and marks the rendition ready, so the status event and the audit trail work as for a local copy. It refuses otherwise, for example when the file or its disclosure changed since the commands were printed: run ai-disclosure-av:requeue and print them again. An audio copy takes up to three commands, and the first ones write temporary files next to the output, which can be deleted after the last one.

What the reader sees¶

The core "Video" and "Audio" file formatters, which a media of those types uses, show the copy instead of the original as soon as a row is ready. The swap happens in hook_preprocess_file_video() and hook_preprocess_file_audio(), so it holds wherever the formatter is used. It changes the src and the type of the source. Nothing else about the markup changes.

Every status change invalidates the cache tag ai_disclosure_rendition:{fid}, and the tag is added to every video or audio element, ready or not, so a page that showed the original is rebuilt when the copy becomes ready.

Events¶

The submodule dispatches three events. Their names are constants of Drupal\ai_disclosure_av\Event\AiDisclosureAvEvents.

Constant	Event object	Dispatched
`STATUS_CHANGE`	`RenditionStatusEvent`	after a rendition is queued, changes status or is removed
`VIDEO_SCHEDULE_CHANGED`	`VideoScheduleChangedEvent`	after the video icon schedule of a file changed
`AUDIO_DISCLAIMER_CHANGED`	`AudioDisclaimerChangedEvent`	after the disclaimer languages of a file changed

The two change events fire only when the new value differs from the old one. Each carries the ID of the file, from getFileId().

The audit trail¶

Every status change dispatches RenditionStatusEvent. When the ai_disclosure_audit submodule is enabled it writes one row per change to its audit trail, through its AuditLogger. Neither submodule needs the other: with the audit submodule off, nothing is written and nothing fails.

Writing a renderer¶

A renderer is a service tagged ai_disclosure_av.renderer that implements Drupal\ai_disclosure_av\RenditionRendererInterface:

services:
  my_module.video_renderer:
    class: Drupal\my_module\MyVideoRenderer
    arguments: ['@ai_disclosure_av.binaries']
    tags:
      - { name: ai_disclosure_av.renderer }

Method	Does
`id()`	the machine name. It is part of the hash and is stored, so it never changes
`settingsVersion()`	a string that changes when the settings of the renderer change. It is part of the hash
`extension()`	the extension of the files it writes, without a dot
`applies($file, $resolved, $probe)`	whether it makes a copy of this file
`render($input, $output, $context)`	writes the copy to `$output`

applies() is called twice. At queue time, in a web request, $probe is NULL and no process may run. In the worker $probe is the answer of ffprobe and FALSE makes the row skipped. A video renderer will usually test MediaOverlay::burnsIn() of the resolved disclosure, and in the worker that the file has a video stream. When several renderers are installed, the first that applies wins, highest tag priority first, so give them conditions that do not overlap.

render() runs in the worker. It reads the original at $input and writes $output, a path in the temporary directory that ends with the extension. It throws Drupal\ai_disclosure_av\RenditionException when it fails, and the message is stored with the row. $context carries the file, the resolved disclosure and the probe.

Start processes with ai_disclosure_av.binaries:

$result = $this->binaries->run([$this->binaries->ffmpeg(), '-i', $input, ..., $output], 600);
if ($result['exit'] !== 0) {
  throw new RenditionException(MediaProbe::tail($result['stderr']));
}

run() takes the command as an array of separate parts, never a shell string, and a timeout in seconds. Do the same in your own code.

The probe, ai_disclosure_av.probe, returns duration, size, format, has_video, has_audio, width, height and the list of streams.

Limits¶

Only files that are local to the server can be rendered, because ffmpeg reads a path.
Rendering takes about as long as the file does or more, and runs inside the cron time of the queue. A single render can outlast it. Give the cron user enough time, or use the Drush command from a shell.
A file longer than the maximum duration is never rendered.
The size of the icon is computed from the coded short edge of the video. A clip that carries a rotation in its metadata is rotated before the filter runs, so the icon of a portrait phone clip can be smaller than the share set.
A WAV copy cannot hold the IPTC tag.
The speech of Generate with AI has the voice and the language the provider gives, which the module cannot choose.
A rendition that failed is not retried until ai-disclosure-av:requeue runs.
A change of a grade or a profile is picked up when a media is next saved or when ai-disclosure-av:requeue runs.
Uninstalling the submodule does not remove the copies. Delete the ai_disclosure/renditions directory when you no longer need them.