Skip to content
Language:DeutschEnglish Last updated: 2026-11-15

Digital archive: overview

Anton ingests digital records with verification, identifies their formats and records how well they can be preserved. Anton changes a master only where the archive has set it up or asked for it: at ingest through a format rule or the processing of a QR upload, afterwards through a migration (see Storage). Storage itself and its redundancy are provided by the operating infrastructure; in installations with a connected long-term archive, that archive takes over bitstream preservation. This page shows how the tasks are divided and refers to the detail pages. It also answers the question of which artefact is a backup and which is a publication view.

Division of tasks

Digital preservation consists of several layers. Anton covers the archival ones, operations cover storage:

Task Anton Operations Connected long-term archive
Checksums on transfer verifies every file of a SIP against the package
Checksum per file calculates MD5 and SHA-512 on upload and stores them
Format identification and risk PRONOM ID, NARA assessment, preservation planning provides Siegfried or Fido
Master and access copies keeps the master, changes it only in preingest (format rules, upload processing) and by a migration with media:migrate, produces web and thumb
Integrity check provides media:check, media:snapshot and media:repair, logs the checks sets the schedule as a cron job, sets up the backup for media:repair takes over fixity of the masters
Redundant storage copies at several locations bitstream preservation
Format decisions shows formats that need action

With Anton as a Service, k & r provides operations. On premises, it is the operating institution.

The chain

Transfer

During SIP ingest according to eCH-0160, Anton verifies the checksum declared in the metadata.xml for every file in the package: the hash is recalculated over the actual file and compared. If it deviates, validation fails. The algorithm comes from the package itself, so it is not fixed to MD5. In addition, Anton remembers the checksum of the SIP ZIP and rejects packages that have already been loaded.

Checks at every ingest

Whichever way a file comes in (upload, import, SIP, agate), Anton checks it before it becomes a medium:

  • Viruses: Where the virus check is set up, every file is first scanned with ClamAV. A finding goes to quarantine, the import stops. media:virus-scan checks the holdings too, against the latest signatures. A finding there is recorded and reported; the file stays where it is. ClamAV cannot scan a file over 2 GB whole; Anton sends it in overlapping parts of 256 MB. So too a smaller file where ClamAV stops at its limit for the amount of data scanned, which counts what it finds embedded in the file as well. ClamAV then sees the bytes of each part, not the format: it finds signatures by byte pattern, not those at the end of the file, tied to the format's structure or by checksum of the whole file. Without a finding the file is recorded as «clean, scanned in parts», not as clean.
  • Checksums: If a delivery brings checksums, they are compared with the files; after storing, Anton recalculates the copy.
  • Format: Every file is checked with the tool for its format (PDF, TIFF, other images, audio and video), text, office documents and packages by their structure, and extension, format and PRONOM ID are compared. For PDF, Anton also records preservation risks: fonts not embedded, encryption, JavaScript, references to programs or other files, and links to the web. Fonts that occur only in the invisible OCR text of a scan are no risk. «Invalid» means damaged or unreadable, for a JPEG also: image data is missing, for instance because the file is cut short. Such a file is taken in and recorded as invalid; on request Anton refuses it. What does not damage the file counts as valid with a warning: an error the tool renders past (such as a faulty colour profile or a reconstructed cross-reference), a PDF that claims to be PDF/A and is not, and a JPEG 2000 image only poppler fails on while MuPDF renders it. media:validate-formats checks the holdings more thoroughly at regular intervals.

Every check is recorded as an event on the medium, since version 0.117.0 with whoever set it off: the person, the command with the account it ran under (<account>@cli:<command>), or the background job (job:<class>). When a medium is deleted, its events stay; the tombstone in the holdings log carries the same medium number (anton#794). Internal users see the events on a record's media page: the latest check of each kind at the top, the full list below. All format validations of the archive, counted day by day and filterable by result, are under Administration → Logs → Format validation for admins.

Format Policy

Before a file is stored as a medium, Anton applies the format rules the archive has switched on. The first rule, tiff_zip, repacks uncompressed TIFF and LZW without a predictor losslessly as Deflate with predictor 2. Before replacing the file, Anton checks that every pixel stayed the same; the checksums of the delivered file are then kept in the event on the medium. SIPs are left alone, because their own checksums prove an untouched delivery. The second rule, pdfx_pdfa2b (PDF/X → PDF/A-2b, PDF), applies only to media in the holdings and runs only through media:migrate; its switch has no effect on new files, and agate does not know it yet.

Which rules hold is set by admins under Admin → Digital preservation → Format Policy, once for the whole archive. No rule can be left out when uploading or importing. The page shows each rule as «if … except … then …»: the formats it applies to, when it leaves a file alone, the steps it runs and what becomes of the original; the commands with tool and version are in a collapsed section below. Where several rules match a file, they run one after the other in the order shown, each on the result of the one before. Every change is recorded with user and time in the security log. agate reads the same settings and applies the rules before it closes a SIP.

What the rules would do to the files of a table import is shown by the inspection before the import. The dry run identifies every linked file, names its technical characteristics and says per rule whether it applies or why not, for instance «exempt for compression Deflate, predictor 2». It changes nothing. On request it computes the size after repacking, file by file.

Storage

For every media file, Anton calculates an MD5 and a SHA-512 checksum in a single read on upload and stores both in the database. SHA-512 is the reference for the integrity check, as OCFL and BagIt recommend; MD5 stays for the DIMAG connection and for recognising duplicate imports. For media from before version 0.98, media:checksum computes SHA-512 afterwards — checking every file against its MD5 as it goes. web and thumb are access copies: Anton makes them and can make them again at any time.

The master is not always the delivered file. What counts is the file at the time it is taken in: what Anton does to it before is preingest, what happens afterwards is a migration. Preingest covers the format rules such as tiff_zip, the processing of a QR upload (HEIC → JPEG, images → PDF, OCR) that the archive switches on, and unpacking archive files. Their result is the file taken in, and so the master. The delivered file stays only where the archive decides so: on the QR upload with «Keep the delivered file» (off by default) as a kept original; the images of «images → PDF» always stay, the delivered file of tiff_zip never does. For HEIC → JPEG, OCR and tiff_zip, an event on the medium records the conversion with tool, version and the checksums of the delivered file, also when it does not stay. Once stored, a master is changed only by media:migrate, with verification and an event; media:repair merely restores the verified content.

Kept originals

Next to some masters lies a kept original in the medium's original/ folder: after a migration with media:migrate the file taken in (unless with --discard-original, which only a rule such as tiff_zip allows), after preingest the delivered file where the archive keeps it: on the QR upload with «Keep the delivered file», and the images of «images → PDF». Since version 0.105 Anton keeps every such file as a version of the medium: with its own MD5 and SHA-512, its format (PRONOM) and an event that records where it came from. Lists, gallery, download and export show the current version. Accounts with the capability Digital preservation and superusers can download a kept version on the record's media page; the holdings log records it.

Originals from before that are registered by media:register-originals. Their checksum is then the one taken on the day of registration, and the event says so: whether the file is unchanged since the upload can no longer be shown.

On tenants with DIMAG, originals can stay local or go to the long-term archive like the masters; operations decide this per installation (see Inge and Dimag).

Format identification and risk

On upload, Anton identifies the format via Siegfried or Fido and records the PRONOM ID; from this it derives the risk assessment according to the NARA Digital Preservation Framework. For existing holdings this can be supplied retrospectively with media:identify. It is evaluated in preservation planning.

When no PRONOM signature matches, Anton records Siegfried's result as UNKNOWN, for instance for a JPEG with zero bytes after its end marker. NARA rates an unknown format «Moderate» with the action «Identify». Such a medium counts as identified and appears in preservation planning as a row of its own, UNKNOWN; «unidentified» there means identification never ran. If a later run with newer signatures recognises the format, it replaces UNKNOWN. A run without a match does not overwrite a known PRONOM ID (anton#849).

media:watch-formats watches the formats for obsolescence (anton#675): when Anton loads a new NARA table (every 28 days), the command assesses every format of the holdings again. Where a rating changes, each medium concerned gets a «risk assessment» event with the old and the new value. Format names follow PRONOM in the release Siegfried identifies with. Preservation planning shows when the holdings were last assessed and with what; the monthly report names risks that went up, new formats, formats that need an action and the unidentified media. It also goes to the contact persons for digital preservation.

Siegfried or Fido on the server

Identification uses Siegfried or Fido installed on the server. If the program is missing, uploads still succeed, only identification stays empty; anton:doctor therefore reports a missing sf as an error. The «unidentified media» tab shows how complete the identification of the holdings is.

Format decisions

Anton migrates when there is a need, not as a precaution. A file stays in the format it was taken in as long as open-source tools render or play it correctly and the access copy is made without errors. A migration is triggered by a failure: in the recurring check whether the file still renders or decodes (media:validate-formats), or when the access copy is made. Whether and into what a file is migrated is decided by the archive. media:migrate knows two rules: tiff_zip repacks TIFF losslessly, in the same format; since version 0.117.0 pdfx_pdfa2b turns PDF/X into PDF/A-2b without rewriting the content (PDF, anton#565). A migration that changes the extension does not exist yet.

For PDF, the format check judges «correctly» by the preservation risks: a PDF without embedded fonts, for example, appears with substitute fonts and looks different without anyone noticing.

Normalisation happens in preingest under the Format Policy and in the holdings by a migration with media:migrate, both verified. In a migration the file taken in stays as a kept original, bit for bit and with a checksum. Anton may discard it only where the rule allows it: tiff_zip verifies every pixel, discards the delivered file and records its checksums in the event, in the holdings with media:migrate --discard-original. Converting is best done before Anton takes the file in, by whoever delivers it; the converted file is then the delivery. Preservation planning is the basis for decisions: it shows which formats are in the holdings, which risk NARA assigns to them and which action NARA recommends.

Which formats are preferred and accepted, and which properties must be preserved, is set out per format category on a page of its own: for images, PDF, text, audio and video, spreadsheets and presentations, and other formats, containers and XML.

Delivery

See Backup or publication? further down.

Checking integrity

Two commands check the holdings against the stored checksums:

Command What it does
media:check --levels=4 Reads every file afresh, calculates the checksum anew (SHA-512 where available, MD5 otherwise) and compares it with the database. With --log-integrity-check, every check is logged as an event — producing a demonstrable history. A missing file is a finding; the check carries on.
media:snapshot --oldest=N --git Writes the reference checksums of all media into lists (SHA-512 and MD5) and commits them to a local Git repository — traceable when a reference changed. In the same pass it checks the N files checked longest ago (or all with --verify) and logs every check.

Both commands also check the local kept originals, each under its own path in the lists. A damaged original does not make its medium a finding.

Operations set the schedule

How often the check runs is determined by a cron job per installation. This allows the schedule to match the size of the holdings: a full run reads every file, and with several terabytes that needs planning.

With Anton as a Service, k & r sets up the check (as of September 2026 for the large archives). On premises, the operating institution sets up the job itself; the commands are available for it.

To be distinguished from this is anton:doctor: it checks the consistency of the database — hierarchy, reference codes, derived fields — and whether the files are present. Checksums are compared by the two commands above.

If the check reports a deviation

If a file deviates from its checksum, media:repair brings it back from the local backup: it goes through the versions from the newest to the oldest (daily, monthly, yearly) and takes the first whose checksum matches the stored reference. The damaged version is not deleted but copied to a quarantine; every step is recorded as an event in the file's history.

Anton only replaces a file with one whose content provably equals the original. And the application only reads the backup, through a read-only mount; if the backup is writable, media:repair refuses to repair. With many deviations at once it repairs nothing and raises an alarm, because the cause then lies elsewhere (disk, mount, malware).

If no local version matches — for instance because the file is newer than the backup — with Anton as a Service we restore it manually from one of the external backup servers; the report of media:repair names everything needed. Tenants with DIMAG are excluded: there the master lies in the long-term archive.

media:repair repairs a local kept original the same way; the threshold for too many deviations counts masters and originals together.

Set up by operations

media:repair needs a local backup that contains the media files, is mounted read-only for the application and is configured (see Installation). Where that is not set up, the command says so and does nothing.

Redundant storage

Redundant storage is a matter for the infrastructure: Anton writes to a local store and optionally to a cloud store, and operations create the copies underneath. With Anton as a Service, the data is held in four copies at three locations, two of them in Zurich; on premises, the operating institution is responsible for this.

Lasting addresses

Records, media, actors, places and keywords carry a UUID that does not change with a move or a merge, unlike the number in a page's address. The address https://<archive>/id/<uuid> leads to the current page: a record's, an authority record's, and for a medium its record's (anton#784). Only what the page itself may show is reached; a locked record answers like an unknown one, with «not found».

The exports and the API name a record by this address (export matrix). The «Permalink» field in a form shows the page address with the number by default (/objects/143); the setting permalink switches it to the lasting address (uuid) or to both (both) (anton#798). For actors, places and keywords it is in the default forms; for records it can be added to the form.

Backup or publication?

The most important distinction, and the one most easily confused:

Artefact Purpose
Native export (anton:export-native) Backup. Lossless and re-importable: metadata in all languages, events, text fields — including private ones —, authority references via UUID and the master media.
SQL dump Backup. The only artefact with users, settings, forms and the NARA assessment of the files (current value only, no history). The events on the media, including every format identification since version 0.107, are also in the PREMIS of the DIP, OCFL and native export. Contains no media files.
DIP (BagIt) and OCFL Delivery packages. Media and metadata bundled, with checksums in the manifest.
RDF/CIDOC, EAD, TEI, Memobase Publication views. Filtered and lossy — Anton cannot be restored from them.
A+ bundle with --include-protected --include-originals Migration package. Graph including blocked data plus original files — for the route into a different system. Not a restore tool for Anton, and not to be hosted publicly because of the personal data it contains.

An RDF or EAD export is not a backup

These formats are built for research and exchange. Among other things they lack the UUIDs and the raw values; in the standard case, private content is filtered out. A backup requires the native export and the SQL dump.

This also applies when exporting with --include-protected --include-originals: that package is intended as a migration route outwards. A way back could be constructed with effort, but there is no tool for it.

Anton keeps the provenance of a file (format identification, checks, format rules, migrations, ingest) in the database. The DIP, OCFL and native export also output it as PREMIS 3, one file per record with media; a receiving long-term archive thus takes over each file's history without Anton's database. A migration appears there as migration with the person who started it, and a replaced file that was not kept, with its checksums, as the medium's source. The NARA assessment is not included; there is no METS.

Which data each format takes with it in detail, which is only in the SQL dump and who can trigger which artefact is shown by the export matrix. In short: the SQL dump and the standard exports go via the interface, the native round trip and the migration package only via the CLI.

For the static publication of a fonds as a standalone website there is a dedicated bundle.

With a connected long-term archive

In installations with a DIMAG connection, Anton hands over every media file to DIMAG on upload via the Inge middleware and keeps a record of whether the handover is verified — see upload status. After that, the bitstream preservation of the master lies with DIMAG.

Fixity then lies with DIMAG

On these installations, the long-term archive checks the masters. media:check therefore skips them with a note that the files are held in DIMAG. On request, Anton deletes the local copy after a verified handover.