Inge and Dimag¶
With Inge it is possible to integrate DIMAG as a repository for the primary data. The original files are then stored not in Anton's local file system but in DIMAG. Only the files optimised for the internet remain in Anton. Where necessary, internal users can download the original files. From the users' perspective there is therefore no difference.
Requirements¶
- Setting
fulltext-from-webpdf: true - Setting
cloud: "inge" - .env INGE_API_TOKEN
- User "Inge" with an email address and
api_tokenfor Inge, holding at least the role editor
Inge needs the editor role
Confirming the ingest and restoring the state before it
(/api/sips/{id}/confirm and /revert) are reserved for editors — a
token alone is not enough. If the service user "Inge" does not hold the
role, Anton answers the closing step of the cycle with a 403 and the
ingest stays unconfirmed.
Course of the SIP ingest¶
Anton¶
- User: SIP upload (zip) (
/sip/uploadsip) - User: SIP validation (
/sip/validation)- Anton can unpack the SIP (unzip) and the metadata file is readable.
- The files from the SIP are present and the checksums are correct.
- Anton can find a parent in Anton for every dossier in the SIP.
- User: Anton ingest (
/sip/ingest)- Backup of the database
- Import SIP (
<dossier>and<dokument>/<datei>)- Entry in the import log, signature
IMPORT-{yyyy}-{NNN} - Import dossiers and documents/files
- Anton creates web versions and thumbnails
- if the SIP ingest takes place with Inge and DIMAG, Anton deletes the master files
- Reference codes and file names are initially based on UUIDs
- Entry in the import log, signature
- Post import (listener
ImportFinished)- Update of the archive hierarchy (
path) - Update of the datings and the full-text index
- Update of the archive hierarchy (
The event MediumAdded triggers the import of the individual media, which is handled asynchronously in each case.
Ingest with Inge into DIMAG¶
The event MediumAdded is triggered with a delay, that is, after the import has been completed and the reference codes have already been corrected. This event triggers the conversion of the media (listener MediumCreateWebVersion). When Inge is used, the original file is copied into the sips path, which Inge can also access. The import into Inge then takes place (Anton\Helpers\Inge::class, import). When Inge reports success, the conversions are carried out and the master medium is deleted.
Inge: - Anton sends one request per file to Inge with the SIP and a list of the Anton media IDs - Inge: ingest of the files into DIMAG - Inge creates a loadXML file - Inge creates an ingest package and stores it on DIMAG's SFTP storage - Inge sends a request to DIMAG: ingest of the SIP - DIMAG: imports the package and sends the result to Inge - Inge: sends the result to Anton - Anton: finalises the SIP ingest - Confirms the ingest (DIMAG report on the run) or restores the state before the ingest from the backup - Sends an email to the user Inge with the result
Retrieving a master file¶
CLI¶
php artisan anton:import --env {slug} --from-sip --no-validation
--create-actors -vv {path/to/sip} --import
Reverting a SIP import or confirming an import with Inge¶
Before a SIP import, Anton backs up the database, so that if anything goes wrong the state before the import can be restored.
The backup name is recorded on the run, in the import log under Admin → Data import → Imports.
The following restores the database from the last/current backup and synchronises the media with the database (namely deletes media that are not registered in the database):
php artisan sip:check-import --env {slug} --id {run_id} -vv --revert
The run_id is the run id from the import log (/import/audit/{run_id}). --list shows the most recent runs, --check inspects the state only.
Only the latest backup
Only this run's backup is restored, and only while it is the latest. An older one discards everything written since — including other people's work.
The following confirms the run and records the DIMAG report on it:
php artisan sip:check-import --env {slug} --id {run_id} -vv --confirm
Checking and repairing media sync (Anton ↔ Inge ↔ Dimag)¶
media:check checks the consistency between the Anton database, the local file system, Inge and Dimag.
# overall picture (counts + verification + orphan check)
php artisan media:check --levels=1,5,6 --env={slug} -vv
# check only one particular SIP (after an interrupted ingest);
# {run_id} is the run id from the import log
php artisan media:check --levels=1,5,6 --sip={run_id} --env={slug} -vv
# repair cloud_status in the DB (when Inge status=20 but the DB is wrong)
php artisan media:check --levels=5 --fix-cloud-status --env={slug} -vv
# delete orphans from Inge/Dimag that are no longer in Anton
php artisan media:check --levels=6 --delete-from-inge --env={slug} -vv
Levels:
| Level | Checks |
|---|---|
| 1 | Count comparison: DB, file system, Inge, Dimag. In case of a discrepancy, a diff table shows the concrete media IDs per system. |
| 2 | DB → file system (skipped with cloud=inge) |
| 3 | File system → DB. With --delete-from-system, orphaned directories are deleted. |
| 4 | Integrity check (checksums, skipped with cloud=inge) |
| 5 | DB → Inge: checks whether all media are present in Inge with status=20. --fix-cloud-status repairs the DB, --delete-local-masters deletes local master files after verification. |
| 6 | Inge/Dimag → DB: finds orphans in Inge or Dimag that are not in Anton. --delete-from-inge deletes them. Also detects media that are stuck in Inge (never reached Dimag). |
At the end, a summary table with all counts and the status of each level is output.
Storage audit (master files and SIP directory)¶
storage:audit checks whether local master files and unpacked SIP directories have been cleaned up.
# overview: how many master files are still held locally? How many SIPs are unpacked?
php artisan storage:audit --env={slug} -vv
# delete unpacked SIP directories (ZIP archives are retained)
php artisan storage:audit --clean-sips --env={slug} -vv
# delete verified local master files (only with cloud=inge, cloud_status=1)
php artisan storage:audit --clean-masters --env={slug} -vv
In Inge installations, local master files should be 0. If not, storage:audit
points this out and --clean-masters cleans up verified files.
Debugging¶
Checking the SIP import data¶
php artisan sip:check --env {slug} --path {path_to_sip} --show-sip_entry
php artisan sip:check --env {slug} --path {path_to_sip} --show-import-array