diff --git a/docs/recensio-exports.md b/docs/recensio-exports.md new file mode 100644 index 00000000..e8520022 --- /dev/null +++ b/docs/recensio-exports.md @@ -0,0 +1,476 @@ +# Recensio export overview + +(generated by AI) + +This document summarizes the export-related code in: + +- `src/recensio/plone/export.py` +- `src/recensio/plone/browser/export.py` +- `src/recensio/plone/browser/templates/export_rm.pt` +- `src/recensio/plone/browser/templates/export_rj.pt` + +## High-level picture + +The export code has three layers: + +1. **Exporter classes** in `export.py` + - collect review data + - render XML or CSV + - store generated files in the portal root + - includes `ContainerExportView`, the minimal BrowserView used to render + per-issue XML (lives here to avoid a circular import with `browser/export.py`) +2. **Browser views** in `browser/export.py` + - trigger export-related actions + - upload an already generated export + - trigger DOI registration + - reset export flags +3. **Registry settings** in the control panel + - SFTP target for Chronicon uploads + - filename prefix for uploaded files + - DOI registration endpoint and credentials + +There is also an important dependency on review-specific XML views elsewhere in +the package: + +- `@@xml` +- `@@xml-lza` +- `@@xml-dara` + +The exporters do not build metadata themselves. They call those views on review +objects and package the results. + +## How the pieces hang together + +### Chronicon flow + +The Chronicon-related workflow is split into two steps: + +1. **Generate** a portal-root file object named `export_metadata_xml.zip` + using `ChroniconExporter` +2. **Upload** that existing ZIP to the remote Chronicon SFTP target using the + `@@chronicon-export` browser view + +So `@@chronicon-export` is an **upload view**, not the generator. + +### Generic bulk export flow + +`MetadataExport` in `browser/export.py` is a generic bulk-export runner. It: + +- finds exporter factories for `IRecensioExporter` +- filters them with `needs_to_run()` +- iterates over published `Issue` and `Volume` objects +- collects published reviews below them +- feeds each review into each exporter via `add_review()` +- calls `export()` on each exporter at the end + +In the **current package snapshot**, `MetadataExport` is exposed as a browser +view named `@@metadata-export` on the Plone site root. There are still no +visible ZCML registrations for the CSV or LZA exporters as `IRecensioExporter` +factories. + +In contrast, the package **does** register a named utility factory for the +Chronicon exporter: + +- `chronicon_exporter` -> `ChroniconExporterFactory` + +So the current wiring is clearest for: + +- the generic export runner `@@metadata-export` +- the Chronicon upload view `@@chronicon-export` +- the DOI update helper `@@dara_update` +- the LZA reset helper `@@reset-lza-export-flag` + +## `export.py` + +## `ContainerExportView` + +A minimal `BrowserView` that wraps the per-issue XML template: + +- template: `browser/templates/export_container_contextless.pt` +- called by `ChroniconExporter.finish_issue()` to render one XML file per issue + or volume +- lives in `export.py` (not in `browser/export.py`) so that `browser/export.py` + can safely import from `export.py` without creating a circular import + +## `BaseExporter` + +`BaseExporter` provides shared helpers: + +- `get_export_obj(portal)` + - looks up the generated export file object in the portal root +- `is_recent(export_xml_obj)` + - treats an existing export file as recent if it was modified less than + 6 days ago + +This is used by several exporters to decide whether a fresh export is needed. + +## `ChroniconExporter` + +Purpose: + +- export review metadata as a ZIP file with **one XML file per issue or volume** + +Important attributes: + +- template: `browser/templates/export_container_contextless.pt` +- stored portal-root file name: `export_metadata_xml.zip` +- per-review XML source: `@@xml` + +How it works: + +- reviews are added in issue/volume order with `add_review(review)` +- each review is rendered through `review.restrictedTraverse("@@xml")()` +- when the current issue/volume changes, `finish_issue()` renders one XML file + for that container and stores it in memory +- `export()` writes all gathered XML files into a ZIP archive and stores it as a + `File` object in the portal root + +Generated internal XML filenames look like: + +- `recensio___.xml` +- or `recensio__.xml` when there is no issue layer + +Concurrency / freshness behavior: + +- `needs_to_run()` returns false if a recent portal-root export already exists +- `running_export()` uses a temporary cache ZIP in the OS temp directory and + treats an export as in progress for up to 1 hour + +Return values from `export()` are status objects such as: + +- file created +- current file already exists +- export already in progress + +## `BVIDExporter` + +Purpose: + +- write a CSV mapping BVID values to review URLs + +Stored portal-root file: + +- `bvid_export.csv` + +Behavior: + +- includes only reviews with a BVID +- writes either the canonical URI or the normal absolute URL depending on the + review configuration + +## `MissingBVIDExporter` + +Purpose: + +- write a CSV listing reviews that still miss a BVID + +Stored portal-root file: + +- `missing_bvid.csv` + +Behavior: + +- stores `(UID, ISBN/ISSN)` pairs for reviews without a BVID + +## `LZAExporter` + +Purpose: + +- export metadata **and full text PDFs** for long-term archive / deposit use + +It subclasses `ChroniconExporter` and changes: + +- stored portal-root file: `export_lza_xml.zip` +- per-review XML source: `@@xml-lza` + +Additional behavior: + +- includes PDFs inside the ZIP next to the XML metadata +- marks each exported review with the annotation key `LZA_EXPORTED` +- skips reviews that were already exported before + +This makes LZA export **incremental** by default. + +## `DaraExporter` + +Purpose: + +- collect `@@xml-dara` output for DOI registration-related export + +Current state: + +- the class exists, but `export()` is still a stub (`pass`) + +So it looks like a scaffold or incomplete path rather than an active export +implementation. + +## `register_doi_raw()` and `register_doi()` + +These functions handle DOI registration / metadata update against the configured +da|ra endpoint. + +They use registry settings for: + +- `doi_registration_url` +- `doi_registration_username` +- `doi_registration_password` + +Behavior: + +- build XML from `@@xml-dara` +- send it via HTTP Basic Auth +- translate HTTP/IO/value errors into user-facing status/message pairs + +Typical successful messages: + +- `DOI successfully registered` +- `Metadata updated` + +## `browser/export.py` + +This module contains only **browser views**. It imports `LZAExporter` and +`register_doi` from `export.py`; the dependency is one-directional. + +## `MetadataExport` + +Purpose: + +- generic bulk-runner for all registered `IRecensioExporter` exporters + +What it does: + +- uses a portal annotation key + `recensio.plone.metadata_export_timestamp` + as a coarse lock +- refuses to start if an export has run within the last 2 hours +- accepts a `force` request parameter to clear that lock first +- walks published issues/volumes and published reviews below them +- calls exporter `add_review()` for each review +- commits the transaction after each issue/volume to reduce memory pressure +- calls `export()` on each exporter and returns a HTML `
`-joined summary + +Registration: + +- ZCML name: `metadata-export` +- context: `Products.CMFPlone.interfaces.IPloneSiteRoot` +- permission: `cmf.ModifyPortalContent` + +Usage URL: + +```text +https://example.org/Plone/@@metadata-export +``` + +Notes: + +- this is the browser entry point that orchestrates the bulk export run +- it supports a `force` request parameter to clear the in-progress timestamp + guard before starting + +## `@@chronicon-export` + +Purpose: + +- upload the current Chronicon ZIP from the portal root to a remote SFTP server + +Registration: + +- ZCML name: `chronicon-export` +- context: `Products.CMFPlone.interfaces.IPloneSiteRoot` +- permission: `cmf.ManagePortal` + +Usage URL: + +```text +https://example.org/Plone/@@chronicon-export +``` + +Configuration used: + +- `xml_export_server` +- `xml_export_username` +- `xml_export_password` +- `xml_export_filename_prefix` + +Behavior: + +1. read SFTP settings from the registry +2. look up `export_metadata_xml.zip` in the portal root +3. open an SFTP connection to the configured host on **port 2222** +4. upload the ZIP +5. name the remote file as: + +```text +[prefix]_[DDMMYY]_all.zip +``` + +Example: + +```text +recensio_050426_all.zip +``` + +Important limitation: + +- this view only uploads an already prepared ZIP +- it does **not** regenerate `export_metadata_xml.zip` + +Typical return values: + +- `no host configured` +- `Could not get export file object: export_metadata_xml.zip` +- `Export successful` +- `Export failed, IOError: ...` +- `Export failed, SSHException: ...` +- `Export failed, / bytes transferred` + +## `DaraUpdate` + +Purpose: + +- register or update DOI metadata for the current object through da|ra + +Behavior: + +- only performs work on `POST` +- calls `register_doi(self.context)` +- stores the result as a Plone status message +- redirects back to the current object afterwards + +Registration: + +- ZCML name: `dara_update` +- context: `recensio.plone.interfaces.IReview` +- permission: `cmf.ModifyPortalContent` + +Usage pattern: + +- intended for review objects +- only does work on `POST` +- redirects back to the review after execution + +## `ResetLZAExportFlag` + +Purpose: + +- recursively clear the `LZA_EXPORTED` annotation so objects can be exported + again by `LZAExporter` + +Behavior: + +- walks the current context recursively +- for any object providing `IReview`, resets the LZA export flag to `False` +- returns a plain text list of affected object paths + +Registration: + +- ZCML name: `reset-lza-export-flag` +- context: `*` +- permission: `cmf.ModifyPortalContent` + +Usage pattern: + +- can be called on a subtree to clear LZA export flags recursively +- returns a plain text list of affected paths + +## Control panel settings involved + +The export code depends on these Recensio settings: + +### Chronicon / SFTP + +- `xml_export_server` +- `xml_export_username` +- `xml_export_password` +- `xml_export_filename_prefix` + +### DOI / da|ra + +- `doi_prefix` +- `doi_registration_url` +- `doi_registration_username` +- `doi_registration_password` + +## Practical summary + +If you want to run the export generation and Chronicon upload path, the +practical flow is: + +1. call `@@metadata-export` on the site root to generate or refresh export + artifacts +2. ensure the Chronicon SFTP settings are configured +3. call `@@chronicon-export` on the site root as a manager + +If you are working on the broader export subsystem, keep in mind: + +- `ChroniconExporter`, `LZAExporter`, BVID exporters, and DOI helpers live in + `export.py` +- `ContainerExportView` also lives in `export.py` to avoid a circular import +- `MetadataExport` is the bulk orchestration layer and is now exposed as + `@@metadata-export` +- `ChroniconExport` handles only the remote SFTP upload step +- `DaraUpdate` and `ResetLZAExportFlag` are also exposed as browser views +- some exporter classes still exist in code without obvious factory + registrations in the current package ZCML + +## Export templates + +### `export_rm.pt` — review of monograph + +Used for reviews of books/monographs. Fields are read directly as Dexterity +attributes (Python 3 / Plone 6 — no AT getter methods): + +| XML element | Context attribute | Behavior | +|---|---|---| +| `isbn` | `context/isbn` | `IBookReview` | +| `language_reviewed_object` | `context.languageReviewedText` | `ITextReview` | +| `pages` | `context/pages` | `ISerial` | +| `place_of_publication` | `context/placeOfPublication` | `IPrintedReview` | +| `publishing_company` | `context/publisher` | `IPrintedReview` | +| `series` | `context/series` | `ISerial` | +| `subtitle` | `context/subtitle` | `IBookReview` | +| `volume_number` | `context/seriesVol` | `ISerial` | +| `year` | `context/yearOfPublication` | `IPrintedReview` | +| `bvid` | `context/bv` | `IBase` | +| `classification_place` | `context.ddcPlace` | `IBase` | +| `classification_subject` | `context.ddcSubject` | `IBase` | +| `classification_time` | `context.ddcTime` | `IBase` | +| `doi` | `context/doi` (condition: `doiRegistrationActive`) | `IBaseReview` / `ISettingsDoi` | +| `language_review` | `context.languageReview` | `IBase` | +| `reviewer` | `context/reviewAuthors` (RelationList) → `author/to_object/lastname` | `IBase` | +| `page_first` | `context/pageStartOfReviewInJournal` | `IPagesOfReviewInJournal` | +| `page_last` | `context/pageEndOfReviewInJournal` | `IPagesOfReviewInJournal` | + +### `export_rj.pt` — review of journal + +Used for reviews of journals / journal articles. Follows the same Dexterity +attribute pattern as `export_rm.pt`. Additional journal-specific fields: + +| XML element | Context attribute | Behavior / field | +|---|---|---| +| `reviewed_journal_editor` | `context/editor` | content type field | +| `reviewed_journal_issn` | `context/issn` | `IJournalReview` | +| `reviewed_journal_issue` | `context/issueNumber` | `IJournalReview` | +| `reviewed_journal_publisher` | `context/publisher` | `IPrintedReview` | +| `reviewed_journal_pubplace` | `context/placeOfPublication` | `IPrintedReview` | +| `reviewed_journal_pubyear_real` | `context/yearOfPublication` | `IPrintedReview` | +| `reviewed_journal_pubyear_fictive` | `context/officialYearOfPublication` | `IJournalReview` | +| `reviewed_journal_title_short` | `context/shortnameJournal` | `IJournalReview` | +| `reviewed_journal_volume` | `context/volumeNumber` | `IJournalReview` | + +The `reviewer` block, `bvid`, `doi`, classification fields, `page_first` / +`page_last`, `filename`, `fulltext`, and `originalurl` are identical to +`export_rm.pt`. + +### Note on `reviewAuthors` + +`reviewAuthors` (defined on `IBase`) is a Dexterity `RelationList` of +`RelationValue` objects pointing to `Person` content items. Templates must go +through `to_object` to reach the actual person: + +```xml + + + + +``` diff --git a/news/3960.feature.md b/news/3960.feature.md new file mode 100644 index 00000000..35e1fe14 --- /dev/null +++ b/news/3960.feature.md @@ -0,0 +1 @@ +Complete the XML export revival work. diff --git a/setup.py b/setup.py index 49cd6087..5bc018b5 100644 --- a/setup.py +++ b/setup.py @@ -55,6 +55,7 @@ "plone.app.discussion", "collective.vdexvocabulary", "collective.z3cform.datagridfield", + "paramiko-ng", "pypdf", "reportlab", "xlrd", diff --git a/src/recensio/plone/browser/configure.zcml b/src/recensio/plone/browser/configure.zcml index 2725f34a..761de6e6 100644 --- a/src/recensio/plone/browser/configure.zcml +++ b/src/recensio/plone/browser/configure.zcml @@ -126,6 +126,114 @@ layer="recensio.plone.interfaces.IRecensioPloneLayer" /> + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + \n".join([name + ": " + str(status) for name, status in statuses]) + + def items(self, portal_type=(), context=None, depth=None): + pc = api.portal.get_tool("portal_catalog") + if context is None: + context = self.context + parent_path = dict(query="/".join(context.getPhysicalPath())) + if depth is not None: + parent_path["depth"] = depth + query = dict( + review_state="published", portal_type=portal_type, path=parent_path + ) + results = pc(**query) + if None in results: + query["b_size"] = len(results) + results = pc(**query) + for item in results: + try: + yield item.getObject() + except AttributeError: + log.warn("Could not getObject: " + item.getPath()) + continue + + def issues_and_volumes(self): + return self.items(portal_type=("Issue", "Volume")) + + def reviews(self, issue): + return self.items(portal_type=REVIEW_TYPES_TO_EXPORT, context=issue, depth=1) + + +class ChroniconExport(BrowserView): + @property + def filename(self): + prefix = api.portal.get_registry_record( + name="recensio.plone.settings.xml_export_filename_prefix" + ) + return "{}_{}_all.zip".format( + prefix, + date.today().strftime("%d%m%y"), + ) + + def __call__(self): + alsoProvides(self.request, IDisableCSRFProtection) + host = api.portal.get_registry_record( + name="recensio.plone.settings.xml_export_server" + ) + username = api.portal.get_registry_record( + name="recensio.plone.settings.xml_export_username" + ) + password = api.portal.get_registry_record( + name="recensio.plone.settings.xml_export_password" + ) + if not host: + return "no host configured" + log.info("Starting XML export to sftp") + + exporter = getUtility(IFactory, name="chronicon_exporter")() + export_xml = exporter.get_export_obj(self.context) + if export_xml is None: + msg = "Could not get export file object: {}".format( + exporter.export_filename + ) + log.error(msg) + return msg + + zipstream = export_xml.file + transport = None + try: + transport = Transport((host, 2222)) + transport.connect(username=username, password=password) + sftp = SFTPClient.from_transport(transport) + attribs = sftp.putfo(zipstream.open(), self.filename) + except (OSError, SSHException) as ioe: + msg = f"Export failed, {ioe.__class__.__name__}: {ioe}" + log.error(msg) + return msg + finally: + if transport is not None: + transport.close() + if attribs.st_size == zipstream.size: + msg = "Export successful" + log.info(msg) + return msg + else: + msg = "Export failed, {}/{} bytes transferred".format( + attribs.st_size, zipstream.size + ) + log.error(msg) + return msg + + +class DaraUpdate(BrowserView): + """Send metadata to da|ra, in effect registering the object's DOI or + updating its metadata if already registered.""" + + def __call__(self): + alsoProvides(self.request, IDisableCSRFProtection) + if self.request.get("REQUEST_METHOD") == "POST": + status, message = register_doi(self.context) + IStatusMessage(self.request).addStatusMessage(message, type=status) + self.request.response.redirect(self.context.absolute_url()) + + +class ResetLZAExportFlag(BrowserView): + def _reset_flag(self, context): + paths = [] + if IReview.providedBy(context): + self.exporter._set_exported(context, value=False) + paths.append("/".join(context.getPhysicalPath())) + for child in getattr(context, "objectValues", lambda: [])(): + paths += self._reset_flag(child) + return paths + + def __call__(self): + alsoProvides(self.request, IDisableCSRFProtection) + self.exporter = LZAExporter() + paths = self._reset_flag(self.context) + return "Reset LZA export flag for the following objects:\n\n" + "\n".join(paths) diff --git a/src/recensio/plone/browser/templates/export_container.pt b/src/recensio/plone/browser/templates/export_container.pt new file mode 100644 index 00000000..2bdb90fe --- /dev/null +++ b/src/recensio/plone/browser/templates/export_container.pt @@ -0,0 +1,21 @@ + + + + + + diff --git a/src/recensio/plone/browser/templates/export_container_contextless.pt b/src/recensio/plone/browser/templates/export_container_contextless.pt new file mode 100644 index 00000000..e03f9780 --- /dev/null +++ b/src/recensio/plone/browser/templates/export_container_contextless.pt @@ -0,0 +1,20 @@ + + + + + + + diff --git a/src/recensio/plone/browser/templates/export_rj.pt b/src/recensio/plone/browser/templates/export_rj.pt new file mode 100644 index 00000000..a6473927 --- /dev/null +++ b/src/recensio/plone/browser/templates/export_rj.pt @@ -0,0 +1,60 @@ + + + Blažek + David + + + Institut der Akademie der Wissenschaften der Tschechischen Republik + 1210-9029! + 2! + !Slovanský ústav AV ČR + !Praha + !2009 + 2010 + !Germanoslavica. Zeitschrift für germano-slawische Studien + Germanoslavica + !20 + !English, Deutsch + + + Slowenien, Polen, Lettland, Deutschland, Tschechien + Sprachgeschichte, Wirtschaftsgeschichte, Literaturgeschichte, Siedlungs-, Stadt- und Ortsgeschichte, Bildungs-, Wissenschafts-, Schul- und Universitätsgeschichte, Sozial- und Kulturgeschichte + 17. Jahrundert, 19. Jahrhundert, 20. Jahrhundert + 10.15463/rec.xxxxxxx + Deutsch + Slowenisch-deutscher Sprachvergleich, Riga / Handel, Tschechisch-deutscher Sprachkontakt, Arnim, Bettina von + http://recensio.net/r/8578c6f5e0304309991c1b12abeccc30 + francia-recensio/2010-4/mittelalter-moyen-age-500-1500/die-inszenierung-der-vergangenheit-im-mittelalter.pdf + http://www.recensio.net/rezensionen/zeitschriften/francia-recensio/2010-4/mittelalter-moyen-age-500-1500/die-inszenierung-der-vergangenheit-im-mittelalter + 2 + 3 + diff --git a/src/recensio/plone/browser/templates/export_rj_dara.pt b/src/recensio/plone/browser/templates/export_rj_dara.pt new file mode 100644 index 00000000..37c3ed24 --- /dev/null +++ b/src/recensio/plone/browser/templates/export_rj_dara.pt @@ -0,0 +1,115 @@ + + + Text + + + de + Rezension + + + en + Review + + + + ed891fc49fdbbb562278aa88e9ffb40d + 1 + + + + <language>de</language> + <titlename>Rezension: + <tal:title replace="title">Autor: Titel. Untertitel</tal:title></titlename> + + + + + + Hans + + Testperson + + + + + http://www.recensio.net/ + + 10.15463/rec.1517192636 + + 1970 + + + Download + + + + de + + Creative-Commons-Lizenz Namensnennung-Keine kommerzielle Nutzung-Keine Bearbeitung (CC-BY-NC-ND) + http://creativecommons.org/licenses/by-nc-nd/2.0/deed.de + + + + + + de + + Aufsatzsammlung + + + + + + 978-3-406-68822-5 + ISSN + Reviews + + + http://www.sehepunkte.de/2016/06/28546.html + URL + IsVariantFormOf + + + + + + Journal + + + + Herausgeber + + + + Titel der Zeitschrift + 1970 + + + + + ISSN + 978-3-406-68822-5 + ISSN + + + + + + diff --git a/src/recensio/plone/browser/templates/export_rm.pt b/src/recensio/plone/browser/templates/export_rm.pt new file mode 100644 index 00000000..4a0c40b0 --- /dev/null +++ b/src/recensio/plone/browser/templates/export_rm.pt @@ -0,0 +1,61 @@ + + + + + 978-5-91022-116-5 + Русский + 896 + Moskva + AIRO-XXI + Quellen und Darstellungen zur Zeitgeschichte + 2: Burnye proryvy, rasbitye nadeždy. Russkie i nemcy v mežvoennye gody + Rossija i Germanija v XX veke + 2 + 2010 + + + Slowenien, Polen, Lettland, Deutschland, Tschechien + Sprachgeschichte, Wirtschaftsgeschichte, Literaturgeschichte, Siedlungs-, Stadt- und Ortsgeschichte, Bildungs-, Wissenschafts-, Schul- und Universitätsgeschichte, Sozial- und Kulturgeschichte + 17. Jahrundert, 19. Jahrhundert, 20. Jahrhundert + 10.15463/rec.xxxxxxx + Deutsch + + Blažek + David + + Slowenisch-deutscher Sprachvergleich, Riga / Handel, Tschechisch-deutscher Sprachkontakt, Arnim, Bettina von + http://recensio.net/r/8578c6f5e0304309991c1b12abeccc30 + francia-recensio/2010-4/mittelalter-moyen-age-500-1500/die-inszenierung-der-vergangenheit-im-mittelalter.pdf + http://www.recensio.net/rezensionen/zeitschriften/francia-recensio/2010-4/mittelalter-moyen-age-500-1500/die-inszenierung-der-vergangenheit-im-mittelalter + 2 + 3 + diff --git a/src/recensio/plone/browser/templates/export_rm_dara.pt b/src/recensio/plone/browser/templates/export_rm_dara.pt new file mode 100644 index 00000000..3e67cab1 --- /dev/null +++ b/src/recensio/plone/browser/templates/export_rm_dara.pt @@ -0,0 +1,142 @@ + + + Text + + + de + Rezension + + + en + Review + + + + ed891fc49fdbbb562278aa88e9ffb40d + 1 + + + + <language>de</language> + <titlename>Rezension: + <tal:title replace="title">Autor: Titel. Untertitel</tal:title></titlename> + + + + + + Hans + + Testperson + + + + + http://www.recensio.net/ + + 10.15463/rec.1517192636 + + 1970 + + + Download + + + + de + + Creative-Commons-Lizenz Namensnennung-Keine kommerzielle Nutzung-Keine Bearbeitung (CC-BY-NC-ND) + http://creativecommons.org/licenses/by-nc-nd/2.0/deed.de + + + + + + de + + Aufsatzsammlung + + + + + + 978-3-406-68822-5 + ISBN + Reviews + + + http://www.sehepunkte.de/2016/06/28546.html + URL + IsVariantFormOf + + + + + + BookOrMonograph + + + It seems to happen quite often that there is an editor/author entry where both first + name and last name are the empty string. This causes an error on the dara server if + we don't filter it out. + + + + + Hans + + Testautor + + + + + + + + Testautor + , + Hans + + + + + + Titel der Monographie + 1970 + + + + + + + + ISBN + 978-3-406-68822-5 + ISBN + + + + + + diff --git a/src/recensio/plone/browser/xml.py b/src/recensio/plone/browser/xml.py new file mode 100644 index 00000000..12ed6297 --- /dev/null +++ b/src/recensio/plone/browser/xml.py @@ -0,0 +1,174 @@ +from html import escape +from io import BytesIO +from plone.i18n.locales.languages import _languagelist +from Products.Five.browser import BrowserView +from recensio.plone.adapter.parentgetter import IParentGetter +from recensio.plone.config import REVIEW_TYPES +from zipfile import ZipFile +from zope.component import getUtility +from zope.schema.interfaces import IVocabularyFactory + + +AUTHOR_TMPL = """ %(firstname)s + %(lastname)s +""" +EDITOR_TMPL = """ %(firstname)s + %(lastname)s +""" + + +class XMLRepresentation(BrowserView): + filename = "recensio_exportx.xml" + include_fulltext = False + + def get_lang_name(self, code): + if not code: + return "" + return _languagelist.get(code, {"native": code})["native"] + + def get_parent(self, meta_type): + return IParentGetter(self.context).get_parent_object_of_type(meta_type) + + def get_publication_shortname(self): + return self.get_parent("Publication").getId() + + def get_publication_title(self): + return self.get_parent("Publication").Title() + + def get_package_journal_volume(self): + return self.get_parent("Volume").getId() + + def get_package_journal_volume_title(self): + return self.get_parent("Volume").Title() + + _voc_names = { + "ddcPlace": "recensio.plone.vocabularies.region_values", + "ddcTime": "recensio.plone.vocabularies.epoch_values", + "ddcSubject": "recensio.plone.vocabularies.topic_values", + } + + def get_voc_title(self, typ, term): + factory = getUtility(IVocabularyFactory, self._voc_names[typ]) + vocab = factory(self.context) + try: + return vocab.getTerm(term).title or "" + except LookupError: + return "" + + def list_authors(self): + out = "" + num = 1 + for rel in self.context.authors or []: + obj = rel.to_object + if obj: + out += AUTHOR_TMPL % dict( + num=num, + firstname=escape(obj.firstname or ""), + lastname=escape(obj.lastname or ""), + ) + num += 1 + return out + + def list_editors(self): + out = "" + num = 1 + for rel in self.context.editorial or []: + obj = rel.to_object + if obj: + out += EDITOR_TMPL % dict( + num=num, + firstname=escape(obj.firstname or ""), + lastname=escape(obj.lastname or ""), + ) + num += 1 + return out + + +class XMLRepresentationLZA(XMLRepresentation): + include_fulltext = True + + +class XMLRepresentationContainer(XMLRepresentation): + def __call__(self): + self.request.response.setHeader("Content-type", "application/zip") + self.request.response.setHeader( + "Content-disposition", "inline;filename=%s" % self.filename + ) + zipdata = self.get_zipdata() + self.request.response.setHeader("content-length", str(len(zipdata))) + return zipdata + + def issues(self): + pc = self.context.portal_catalog + parent_path = dict(query="/".join(self.context.getPhysicalPath())) + results = pc(review_state="published", portal_type=("Issue"), path=parent_path) + for item in results: + yield item.getObject() + + def write_zipfile(self, zipfile): + for issue in self.issues(): + xmlview = issue.restrictedTraverse("@@xml") + xml = xmlview.template(xmlview) + filename = xmlview.filename + zipfile.writestr(filename, bytes(xml.encode("utf-8"))) + + def get_zipdata(self): + stream = BytesIO() + zipfile = ZipFile(stream, "w") + self.write_zipfile(zipfile) + zipfile.close() + zipdata = stream.getvalue() + stream.close() + return zipdata + + +class XMLRepresentationPublication(XMLRepresentationContainer): + @property + def filename(self): + return "recensio_%s.zip" % (self.get_publication_shortname()) + + +class XMLRepresentationVolume(XMLRepresentationContainer): + @property + def filename(self): + return "recensio_{}_{}.zip".format( + self.get_publication_shortname(), + self.get_package_journal_volume(), + ) + + +class XMLRepresentationIssue(XMLRepresentation): + + def __call__(self): + self.request.response.setHeader("Content-type", "application/xml") + self.request.response.setHeader( + "Content-disposition", "inline;filename=%s" % self.filename + ) + return self.template(self) + + def get_package_journal_pubyear(self): + volume = self.get_parent("Volume") + return getattr(volume, "year_of_publication", None) or None + + def get_package_journal_issue(self): + return self.get_parent("Issue").getId() + + def get_package_journal_issue_title(self): + return self.get_parent("Issue").Title() + + @property + def filename(self): + return "recensio_{}_{}_{}.xml".format( + self.get_publication_shortname(), + self.get_package_journal_volume(), + self.get_package_journal_issue(), + ) + + def reviews(self): + pc = self.context.portal_catalog + parent_path = dict(query="/".join(self.context.getPhysicalPath()), depth=3) + results = pc( + review_state="published", portal_type=REVIEW_TYPES, path=parent_path + ) + for item in results: + yield item.getObject() diff --git a/src/recensio/plone/config.py b/src/recensio/plone/config.py index f5744429..0df24ef9 100644 --- a/src/recensio/plone/config.py +++ b/src/recensio/plone/config.py @@ -5,3 +5,9 @@ "Review Article Collection", "Review Exhibition", ] +REVIEW_TYPES_TO_EXPORT = [ + "Review Journal", + "Review Monograph", + "Review Article Journal", + "Review Article Collection", +] diff --git a/src/recensio/plone/configure.zcml b/src/recensio/plone/configure.zcml index 7cb50009..0db3d794 100644 --- a/src/recensio/plone/configure.zcml +++ b/src/recensio/plone/configure.zcml @@ -45,6 +45,12 @@ + + ") + exporter = ChroniconExporter() + with patch.object( + review, "restrictedTraverse", return_value=xml_view + ) as mock_rt: + exporter.add_review(review) + mock_rt.assert_called_once_with(exporter.xml_view_name) + xml_view.assert_called_once() + self.assertEqual(exporter.reviews_xml, [""]) + self.assertEqual(exporter.current_issue, self.issue) + + def test_add_review_finishes_previous_issue_on_change(self): + review1 = _make_review(self.issue, id="r1") + # create a second issue + with api.env.adopt_user(SITE_OWNER_NAME): + issue2 = api.content.create( + type="Issue", id="issue2", title="Issue 2", container=self.vol + ) + review2 = _make_review(issue2, id="r2") + + exporter = ChroniconExporter() + xml1 = MagicMock(return_value="") + xml2 = MagicMock(return_value="") + + # Patch finish_issue so we can record when it fires + finished = [] + + def capturing_finish(): + finished.append(exporter.current_issue) + # minimal stub: just clear state without rendering template + exporter.issues_xml[exporter.get_issue_filename()] = "stub" + exporter.reviews_xml = [] + exporter.current_issue = None + + exporter.finish_issue = capturing_finish + + with patch.object(review1, "restrictedTraverse", return_value=xml1): + exporter.add_review(review1) + with patch.object(review2, "restrictedTraverse", return_value=xml2): + exporter.add_review(review2) + + self.assertEqual(len(finished), 1) + self.assertEqual(finished[0], self.issue) + self.assertEqual(exporter.current_issue, issue2) + + def test_running_export_returns_none_when_no_cache(self): + exporter = ChroniconExporter() + with patch("recensio.plone.export.path.exists", return_value=False): + self.assertIsNone(exporter.running_export()) + + def test_export_produces_zip_with_xml_entries(self): + exporter = ChroniconExporter() + exporter.issues_xml = {"recensio_test-pub_test-vol_test-issue.xml": ""} + zipdata = exporter.get_zipdata() + zf = ZipFile(BytesIO(zipdata)) + self.assertIn("recensio_test-pub_test-vol_test-issue.xml", zf.namelist()) + + def test_export_returns_in_progress_when_cache_exists(self): + from datetime import datetime + + exporter = ChroniconExporter() + exporter.issues_xml = {} + with patch.object(exporter, "running_export", return_value=datetime.now()): + result = exporter.export() + self.assertIsInstance(result, StatusFailureAlreadyInProgress) + + +# --------------------------------------------------------------------------- +# LZAExporter +# --------------------------------------------------------------------------- + + +class TestLZAExporter(unittest.TestCase): + + layer = RECENSIO_PLONE_INTEGRATION_TESTING + + def setUp(self): + self.portal = self.layer["portal"] + setRoles(self.portal, TEST_USER_ID, ["Manager"]) + self.pub, self.vol, self.issue = _make_container(self.portal) + + def tearDown(self): + with api.env.adopt_user(SITE_OWNER_NAME): + api.content.delete(self.portal["test-reviews"]) + + def test_export_filename(self): + self.assertEqual(LZAExporter.export_filename, "export_lza_xml.zip") + + def test_xml_view_name(self): + self.assertEqual(LZAExporter.xml_view_name, "@@xml-lza") + + def test_set_and_check_exported_flag(self): + review = _make_review(self.issue) + exporter = LZAExporter() + self.assertFalse(exporter._is_exported(review)) + exporter._set_exported(review) + self.assertTrue(exporter._is_exported(review)) + + def test_set_exported_false_clears_flag(self): + review = _make_review(self.issue) + exporter = LZAExporter() + exporter._set_exported(review, True) + exporter._set_exported(review, False) + self.assertFalse(exporter._is_exported(review)) + + def test_already_exported_review_is_skipped(self): + review = _make_review(self.issue) + exporter = LZAExporter() + exporter._set_exported(review) + xml_view = MagicMock(return_value="") + with patch.object(review, "restrictedTraverse", return_value=xml_view): + exporter.add_review(review) + xml_view.assert_not_called() + self.assertEqual(exporter.reviews_xml, []) + + def test_new_review_is_added_and_marked(self): + review = _make_review(self.issue) + exporter = LZAExporter() + xml_view = MagicMock(return_value="") + review_view = MagicMock() + review_view.get_review_pdf.return_value = None + with patch.object(review, "restrictedTraverse", return_value=xml_view): + with patch( + "recensio.plone.export.api.content.get_view", return_value=review_view + ): + exporter.add_review(review) + self.assertTrue(exporter._is_exported(review)) + self.assertEqual(exporter.reviews_xml, [""]) + + def test_pdf_included_in_zip_when_present(self): + review = _make_review(self.issue) + exporter = LZAExporter() + xml_view = MagicMock(return_value="") + fake_pdf = MagicMock() + fake_pdf.open.return_value.__enter__ = lambda s: s + fake_pdf.open.return_value.__exit__ = MagicMock(return_value=False) + fake_pdf_blob = MagicMock() + fake_pdf_blob.read.side_effect = [b"%PDF-fake", b""] + fake_pdf.open.return_value = fake_pdf_blob + review_view = MagicMock() + review_view.get_review_pdf.return_value = fake_pdf + + with patch.object(review, "restrictedTraverse", return_value=xml_view): + with patch( + "recensio.plone.export.api.content.get_view", return_value=review_view + ): + exporter.add_review(review) + + exporter.issues_xml["dummy.xml"] = "" + buf = BytesIO() + zf = ZipFile(buf, "w") + exporter.write_zipfile(zf) + zf.close() + buf.seek(0) + names = ZipFile(buf).namelist() + pdf_entries = [n for n in names if n.endswith(".pdf")] + self.assertEqual(len(pdf_entries), 1) + + +# --------------------------------------------------------------------------- +# register_doi / register_doi_raw +# --------------------------------------------------------------------------- + + +class TestRegisterDoi(unittest.TestCase): + + layer = RECENSIO_PLONE_INTEGRATION_TESTING + + def setUp(self): + self.portal = self.layer["portal"] + self.request = self.layer["request"] + setRoles(self.portal, TEST_USER_ID, ["Manager"]) + pub, vol, self.issue = _make_container(self.portal) + self.review = _make_review(self.issue) + + def tearDown(self): + with api.env.adopt_user(SITE_OWNER_NAME): + api.content.delete(self.portal["test-reviews"]) + + def _mock_settings(self): + settings = MagicMock() + settings.doi_registration_username = "user" + settings.doi_registration_password = "pass" + settings.doi_registration_url = "https://dara.example.org/api" + registry = MagicMock() + registry.forInterface.return_value = settings + return registry + + def _mock_response(self, status_code): + response = MagicMock() + response.status_code = status_code + response.text = "" + response.raise_for_status.return_value = None + return response + + def _mock_http_error(self, status_code, body=""): + response = MagicMock() + response.status_code = status_code + response.text = body + err = requests.exceptions.HTTPError(response=response) + return err + + def test_register_doi_raw_posts_with_basic_auth(self): + registry = self._mock_settings() + xml_view = MagicMock(return_value="") + fake_response = self._mock_response(201) + + with patch("recensio.plone.export.queryUtility", return_value=registry): + with patch.object(self.review, "restrictedTraverse", return_value=xml_view): + with patch( + "recensio.plone.export.requests.post", return_value=fake_response + ) as mock_post: + code = register_doi_raw(self.review) + + self.assertEqual(code, 201) + _args, kwargs = mock_post.call_args + self.assertEqual(kwargs["auth"], ("user", "pass")) + self.assertIn("application/xml", kwargs["headers"]["Content-Type"]) + + def test_register_doi_returns_success_on_201(self): + registry = self._mock_settings() + xml_view = MagicMock(return_value="") + fake_response = self._mock_response(201) + + with patch("recensio.plone.export.queryUtility", return_value=registry): + with patch.object(self.review, "restrictedTraverse", return_value=xml_view): + with patch( + "recensio.plone.export.requests.post", return_value=fake_response + ): + status, message = register_doi(self.review) + + self.assertEqual(status, "success") + self.assertIn("registered", message) + + def test_register_doi_returns_success_on_200(self): + registry = self._mock_settings() + xml_view = MagicMock(return_value="") + fake_response = self._mock_response(200) + + with patch("recensio.plone.export.queryUtility", return_value=registry): + with patch.object(self.review, "restrictedTraverse", return_value=xml_view): + with patch( + "recensio.plone.export.requests.post", return_value=fake_response + ): + status, message = register_doi(self.review) + + self.assertEqual(status, "success") + self.assertIn("updated", message.lower()) + + def test_register_doi_handles_401(self): + registry = self._mock_settings() + xml_view = MagicMock(return_value="") + err = self._mock_http_error(401) + + with patch("recensio.plone.export.queryUtility", return_value=registry): + with patch.object(self.review, "restrictedTraverse", return_value=xml_view): + with patch("recensio.plone.export.requests.post", side_effect=err): + status, message = register_doi(self.review) + + self.assertEqual(status, "error") + self.assertIn("login", message.lower()) + + def test_register_doi_handles_400(self): + registry = self._mock_settings() + xml_view = MagicMock(return_value="") + err = self._mock_http_error(400) + + with patch("recensio.plone.export.queryUtility", return_value=registry): + with patch.object(self.review, "restrictedTraverse", return_value=xml_view): + with patch("recensio.plone.export.requests.post", side_effect=err): + status, message = register_doi(self.review) + + self.assertEqual(status, "error") + self.assertIn("xml", message.lower()) + + def test_register_doi_handles_connection_error(self): + registry = self._mock_settings() + xml_view = MagicMock(return_value="") + + with patch("recensio.plone.export.queryUtility", return_value=registry): + with patch.object(self.review, "restrictedTraverse", return_value=xml_view): + with patch( + "recensio.plone.export.requests.post", + side_effect=OSError("connection refused"), + ): + status, message = register_doi(self.review) + + self.assertEqual(status, "error") + self.assertIn("contacting", message.lower()) + + def test_register_doi_raw_rejects_non_http_scheme(self): + registry = self._mock_settings() + registry.forInterface.return_value.doi_registration_url = "file:///etc/passwd" + xml_view = MagicMock(return_value="") + + with patch("recensio.plone.export.queryUtility", return_value=registry): + with patch.object(self.review, "restrictedTraverse", return_value=xml_view): + with self.assertRaises(ValueError): + register_doi_raw(self.review) + + def test_register_doi_handles_valueerror(self): + registry = self._mock_settings() + xml_view = MagicMock(return_value="") + + with patch("recensio.plone.export.queryUtility", return_value=registry): + with patch.object(self.review, "restrictedTraverse", return_value=xml_view): + with patch( + "recensio.plone.export.requests.post", + side_effect=ValueError("bad url"), + ): + status, message = register_doi(self.review) + + self.assertEqual(status, "error") + self.assertIn("updating", message.lower())