Digital archive: overview¶
Anton ingests digital records with verification, identifies their formats and records how well they can be preserved. Anton changes a master only where the archive has set it up or asked for it: at ingest through a format rule or the processing of a QR upload, afterwards through a migration (see Storage). Storage itself and its redundancy are provided by the operating infrastructure; in installations with a connected long-term archive, that archive takes over bitstream preservation. This page shows how the tasks are divided and refers to the detail pages. It also answers the question of which artefact is a backup and which is a publication view.
Division of tasks¶
Digital preservation consists of several layers. Anton covers the archival ones, operations cover storage:
| Task | Anton | Operations | Connected long-term archive |
|---|---|---|---|
| Checksums on transfer | verifies every file of a SIP against the package | ||
| Checksum per file | calculates MD5 and SHA-512 on upload and stores them | ||
| Format identification and risk | PRONOM ID, NARA assessment, preservation planning | provides Siegfried or Fido | |
| Master and access copies | keeps the master, changes it only in preingest (format rules, upload processing) and by a migration with media:migrate, produces web and thumb |
||
| Integrity check | provides media:check, media:snapshot and media:repair, logs the checks |
sets the schedule as a cron job, sets up the backup for media:repair |
takes over fixity of the masters |
| Redundant storage | copies at several locations | bitstream preservation | |
| Format decisions | shows formats that need action |
With Anton as a Service, k & r provides operations. On premises, it is the operating institution.
The chain¶
Transfer¶
During SIP ingest according to eCH-0160, Anton verifies the
checksum declared in the metadata.xml for every file in the package: the
hash is recalculated over the actual file and compared. If it deviates,
validation fails. The algorithm comes from the package itself, so it is not
fixed to MD5. In addition, Anton remembers the checksum of the SIP ZIP and
rejects packages that have already been loaded.
Checks at every ingest¶
Whichever way a file comes in (upload, import, SIP, agate), Anton checks it before it becomes a medium:
- Viruses: Where the virus check is set up, every file is first scanned
with ClamAV. A finding goes to quarantine, the import stops.
media:virus-scanchecks the holdings too, against the latest signatures. A finding there is recorded and reported; the file stays where it is. ClamAV cannot scan a file over 2 GB whole; Anton sends it in overlapping parts of 256 MB. So too a smaller file where ClamAV stops at its limit for the amount of data scanned, which counts what it finds embedded in the file as well. ClamAV then sees the bytes of each part, not the format: it finds signatures by byte pattern, not those at the end of the file, tied to the format's structure or by checksum of the whole file. Without a finding the file is recorded as «clean, scanned in parts», not as clean. - Checksums: If a delivery brings checksums, they are compared with the files; after storing, Anton recalculates the copy.
- Format: Every file is checked with the tool for its format (PDF, TIFF,
other images, audio and video), text, office documents and packages by their
structure, and extension, format and PRONOM ID are
compared. For PDF, Anton also records preservation risks: fonts not embedded,
encryption, JavaScript, references to programs or other files, and links to
the web. Fonts that occur only in the invisible OCR text of a scan are no
risk. «Invalid» means damaged or unreadable, for a JPEG also: image data is
missing, for instance because the file is cut short. Such a file is taken in and
recorded as invalid; on request Anton refuses it. What does not damage the
file counts as valid with a warning: an error the tool renders past (such as
a faulty colour profile or a reconstructed cross-reference), a PDF that
claims to be PDF/A and is not, and a JPEG 2000 image only poppler fails on
while MuPDF renders it.
media:validate-formatschecks the holdings more thoroughly at regular intervals.
Every check is recorded as an event on the medium, since version 0.117.0 with
whoever set it off: the person, the command with the account it ran under
(<account>@cli:<command>), or the background job (job:<class>). When a
medium is deleted, its events stay; the tombstone in the holdings log carries
the same medium number (anton#794). Internal users see the events on a
record's media page: the latest check of each kind at the top, the full list
below. All format validations of the archive, counted day by day and filterable
by result, are under Administration → Logs → Format validation for admins.
Format Policy¶
Before a file is stored as a medium, Anton applies the format rules the
archive has switched on. The first rule, tiff_zip, repacks uncompressed TIFF
and LZW without a predictor losslessly as Deflate with predictor 2. Before
replacing the file, Anton checks that every pixel stayed the same; the
checksums of the delivered file are then kept in the event on the medium.
SIPs are left alone, because their own checksums prove an untouched delivery.
The second rule, pdfx_pdfa2b (PDF/X → PDF/A-2b, PDF),
applies only to media in the holdings and runs only through
media:migrate; its switch has no effect on
new files, and agate does not know it yet.
Which rules hold is set by admins under Admin → Digital preservation → Format Policy, once for the whole archive. No rule can be left out when uploading or importing. The page shows each rule as «if … except … then …»: the formats it applies to, when it leaves a file alone, the steps it runs and what becomes of the original; the commands with tool and version are in a collapsed section below. Where several rules match a file, they run one after the other in the order shown, each on the result of the one before. Every change is recorded with user and time in the security log. agate reads the same settings and applies the rules before it closes a SIP.
What the rules would do to the files of a table import is shown by the inspection before the import. The dry run identifies every linked file, names its technical characteristics and says per rule whether it applies or why not, for instance «exempt for compression Deflate, predictor 2». It changes nothing. On request it computes the size after repacking, file by file.
Storage¶
For every media file, Anton calculates an MD5 and a SHA-512 checksum in
a single read on upload and stores both in the database. SHA-512 is the
reference for the integrity check, as OCFL and BagIt recommend; MD5 stays for
the DIMAG connection and for recognising duplicate imports. For media from
before version 0.98, media:checksum
computes SHA-512 afterwards — checking every file against its MD5 as it goes. web and thumb are access copies: Anton makes them and can make them again at any time.
The master is not always the delivered file. What counts is the file at the time it is taken in: what Anton does to it before is preingest, what happens afterwards is a migration. Preingest covers the format rules such as tiff_zip, the processing of a QR upload (HEIC → JPEG, images → PDF, OCR) that the archive switches on, and unpacking archive files. Their result is the file taken in, and so the master. The delivered file stays only where the archive decides so: on the QR upload with «Keep the delivered file» (off by default) as a kept original; the images of «images → PDF» always stay, the delivered file of tiff_zip never does. For HEIC → JPEG, OCR and tiff_zip, an event on the medium records the conversion with tool, version and the checksums of the delivered file, also when it does not stay. Once stored, a master is changed only by media:migrate, with verification and an event; media:repair merely restores the verified content.
Kept originals¶
Next to some masters lies a kept original in the medium's original/ folder: after a migration with media:migrate the file taken in (unless with --discard-original, which only a rule such as tiff_zip allows), after preingest the delivered file where the archive keeps it: on the QR upload with «Keep the delivered file», and the images of «images → PDF». Since version 0.105 Anton keeps every such file as a version of
the medium: with its own MD5 and SHA-512, its format (PRONOM) and an event that
records where it came from. Lists, gallery, download and export show the
current version. Accounts with the capability Digital preservation and superusers can download a kept version on the record's media page; the holdings log records it.
Originals from before that are registered by
media:register-originals. Their
checksum is then the one taken on the day of registration, and the event says
so: whether the file is unchanged since the upload can no longer be shown.
On tenants with DIMAG, originals can stay local or go to the long-term archive like the masters; operations decide this per installation (see Inge and Dimag).
Format identification and risk¶
On upload, Anton identifies the format via Siegfried or Fido and records
the PRONOM ID; from this it derives the risk assessment according to the
NARA Digital Preservation Framework. For existing holdings this can be
supplied retrospectively with
media:identify. It is evaluated in
preservation planning.
When no PRONOM signature matches, Anton records Siegfried's result as UNKNOWN, for instance for a JPEG with zero bytes after its end marker. NARA rates an unknown format «Moderate» with the action «Identify». Such a medium counts as identified and appears in preservation planning as a row of its own, UNKNOWN; «unidentified» there means identification never ran. If a later run with newer signatures recognises the format, it replaces UNKNOWN. A run without a match does not overwrite a known PRONOM ID (anton#849).
media:watch-formats watches the
formats for obsolescence (anton#675): when Anton loads a new NARA table (every
28 days), the command assesses every format of the holdings again. Where a
rating changes, each medium concerned gets a «risk assessment» event with the
old and the new value. Format names follow PRONOM in the release Siegfried
identifies with. Preservation planning shows when the holdings were last
assessed and with what; the monthly report names risks that
went up, new formats, formats that need an action and the unidentified media. It also goes to the contact persons for digital preservation.
Siegfried or Fido on the server
Identification uses Siegfried or Fido installed on the server. If the
program is missing, uploads still succeed, only identification stays empty;
anton:doctor therefore reports a missing sf as an error.
The «unidentified media» tab shows how complete the identification of the
holdings is.
Format decisions¶
Anton migrates when there is a need, not as a precaution. A file stays in the
format it was taken in as long as open-source tools render or play it
correctly and the access copy is made without errors. A migration is
triggered by a failure: in the recurring check whether the file still renders
or decodes (media:validate-formats),
or when the access copy is made. Whether and into what a file is migrated is
decided by the archive. media:migrate knows two rules: tiff_zip repacks TIFF losslessly, in the same format; since version 0.117.0 pdfx_pdfa2b turns PDF/X into PDF/A-2b without rewriting the content (PDF, anton#565). A migration that changes the extension does not exist yet.
For PDF, the format check judges «correctly» by the preservation risks: a PDF without embedded fonts, for example, appears with substitute fonts and looks different without anyone noticing.
Normalisation happens in preingest under the Format Policy and in the holdings by a migration with media:migrate, both verified. In a migration the file taken in stays as a kept original, bit for bit and with a checksum. Anton may discard it only where the rule allows it: tiff_zip verifies every pixel, discards the delivered file and records its checksums in the event, in the holdings with media:migrate --discard-original. Converting is best done before Anton takes the file in, by whoever delivers it; the converted file is then the delivery. Preservation planning is the basis for decisions: it shows which formats are in the holdings, which risk NARA assigns to them and which action NARA recommends.
Which formats are preferred and accepted, and which properties must be preserved, is set out per format category on a page of its own: for images, PDF, text, audio and video, spreadsheets and presentations, and other formats, containers and XML.
Delivery¶
See Backup or publication? further down.
Checking integrity¶
Two commands check the holdings against the stored checksums:
| Command | What it does |
|---|---|
media:check --levels=4 |
Reads every file afresh, calculates the checksum anew (SHA-512 where available, MD5 otherwise) and compares it with the database. With --log-integrity-check, every check is logged as an event — producing a demonstrable history. A missing file is a finding; the check carries on. |
media:snapshot --oldest=N --git |
Writes the reference checksums of all media into lists (SHA-512 and MD5) and commits them to a local Git repository — traceable when a reference changed. In the same pass it checks the N files checked longest ago (or all with --verify) and logs every check. |
Both commands also check the local kept originals, each under its own path in the lists. A damaged original does not make its medium a finding.
Operations set the schedule
How often the check runs is determined by a cron job per installation. This allows the schedule to match the size of the holdings: a full run reads every file, and with several terabytes that needs planning.
With Anton as a Service, k & r sets up the check (as of September 2026 for the large archives). On premises, the operating institution sets up the job itself; the commands are available for it.
To be distinguished from this is anton:doctor: it
checks the consistency of the database — hierarchy, reference codes, derived
fields — and whether the files are present. Checksums are compared by the two
commands above.
If the check reports a deviation¶
If a file deviates from its checksum, media:repair
brings it back from the local backup: it goes through the versions from the
newest to the oldest (daily, monthly, yearly) and takes the first whose checksum
matches the stored reference. The damaged version is not deleted but copied to a
quarantine; every step is recorded as an event in the file's history.
Anton only replaces a file with one whose content provably equals the original.
And the application only reads the backup, through a read-only mount; if
the backup is writable, media:repair refuses to repair. With many deviations at once it
repairs nothing and raises an alarm, because the cause then lies elsewhere
(disk, mount, malware).
If no local version matches — for instance because the file is newer than the
backup — with Anton as a Service we restore it manually from one of the
external backup servers; the report of media:repair names everything needed.
Tenants with DIMAG are excluded: there the master lies in the long-term archive.
media:repair repairs a local kept original the same
way; the threshold for too many deviations counts masters and originals
together.
Set up by operations
media:repair needs a local backup that contains the media files, is
mounted read-only for the application and is configured (see
Installation).
Where that is not set up, the command says so and does nothing.
Redundant storage¶
Redundant storage is a matter for the infrastructure: Anton writes to a local store and optionally to a cloud store, and operations create the copies underneath. With Anton as a Service, the data is held in four copies at three locations, two of them in Zurich; on premises, the operating institution is responsible for this.
Lasting addresses¶
Records, media, actors, places and keywords carry a UUID that does not change with a move or a merge, unlike the number in a page's address. The address https://<archive>/id/<uuid> leads to the current page: a record's, an authority record's, and for a medium its record's (anton#784). Only what the page itself may show is reached; a locked record answers like an unknown one, with «not found».
The exports and the API name a record by this address (export matrix). The «Permalink» field in a form shows the page address with the number by default (/objects/143); the setting permalink switches it to the lasting address (uuid) or to both (both) (anton#798). For actors, places and keywords it is in the default forms; for records it can be added to the form.
Backup or publication?¶
The most important distinction, and the one most easily confused:
| Artefact | Purpose |
|---|---|
Native export (anton:export-native) |
Backup. Lossless and re-importable: metadata in all languages, events, text fields — including private ones —, authority references via UUID and the master media. |
| SQL dump | Backup. The only artefact with users, settings, forms and the NARA assessment of the files (current value only, no history). The events on the media, including every format identification since version 0.107, are also in the PREMIS of the DIP, OCFL and native export. Contains no media files. |
| DIP (BagIt) and OCFL | Delivery packages. Media and metadata bundled, with checksums in the manifest. |
| RDF/CIDOC, EAD, TEI, Memobase | Publication views. Filtered and lossy — Anton cannot be restored from them. |
A+ bundle with --include-protected --include-originals |
Migration package. Graph including blocked data plus original files — for the route into a different system. Not a restore tool for Anton, and not to be hosted publicly because of the personal data it contains. |
An RDF or EAD export is not a backup
These formats are built for research and exchange. Among other things they lack the UUIDs and the raw values; in the standard case, private content is filtered out. A backup requires the native export and the SQL dump.
This also applies when exporting with --include-protected
--include-originals: that package is intended as a migration route
outwards. A way back could be constructed with effort, but there is no
tool for it.
Anton keeps the provenance of a file (format identification, checks, format rules, migrations, ingest) in the database. The DIP, OCFL and native export also output it as PREMIS 3, one file per record with media; a receiving long-term archive thus takes over each file's history without Anton's database. A migration appears there as migration with the person who started it, and a replaced file that was not kept, with its checksums, as the medium's source. The NARA assessment is not included; there is no METS.
Which data each format takes with it in detail, which is only in the SQL dump and who can trigger which artefact is shown by the export matrix. In short: the SQL dump and the standard exports go via the interface, the native round trip and the migration package only via the CLI.
For the static publication of a fonds as a standalone website there is a dedicated bundle.
With a connected long-term archive¶
In installations with a DIMAG connection, Anton hands over every media file to DIMAG on upload via the Inge middleware and keeps a record of whether the handover is verified — see upload status. After that, the bitstream preservation of the master lies with DIMAG.
Fixity then lies with DIMAG
On these installations, the long-term archive checks the masters.
media:check therefore skips them with a note that the files are held in
DIMAG. On request, Anton deletes the local copy after a verified handover.