Execution, products and traceability
Describes version 2.0Table 34 and Table 35 describe the processing exchange and automation responsibilities.
The PyAPS runner turns a request and its configuration into a MOS or IFU workflow. It creates the module commands, groups work that can run in parallel and defines the dependencies between stages. Preparation, scientific fitting and L2 assembly remain separate, so intermediate products can be inspected or used by contributed software. SPA services execute and monitor the resulting jobs.
Relationship to the WEAVE SPA
PyAPS implements the APS part of the wider WEAVE SPA. CPS and upstream transfer services deliver calibrated Level-1 FITS products; PyAPS classifies and normalises those products, executes selected scientific modules, and assembles APS Level-2 products; archive and release services then register and distribute the outputs. PyAPS interfaces with the surrounding scheduler, Operational Repository and archive services.
| Stage | Product or service boundary | PyAPS responsibility |
|---|---|---|
| Ingest | CPS Level-1 FITS | Read mode, arm, resolution, binning and source metadata. |
| Normalise | APSOB/APSTARG/APSSPEC | Convert MOS, LIFU, MIFU, MOSLIFU and MOSMIFU layouts to the common object model. |
| Prepare | APS intermediate products | Mask, correct, select, bin and attach wavelength-dependent resolution. |
| Analyse | Redrock, RVSpecFit (RVS), FERRE, pPXF and line modules | Translate the common representation to engine-specific inputs and interpret outputs. |
| Assemble | APS Level-2 FITS | Merge module products, quality fields, checksums and processing history. |
| Release | Operational repository/archive | Expose generated products to surrounding SPA services. |
PyAPS implements WEAVE preparation, adapters, orchestration and product assembly. CPS, scheduler and archive services, and the scientific algorithms inside external engines, remain separate components. The operational record links this flow to run identifiers, queue events and archive inventories.
In addition to each product’s scientific workflow, PyAPS has an operational control layer. It contains the workflow generator, the APS data management system (DMS), and the WDAP-facing controller. The generator writes the executable module scripts and Slurm dependency graph; the DMS discovers or updates L1/L2 processing records, submits or regenerates jobs, polls cluster state and supports retraction; the WDAP-facing layer exchanges processing states with the Operational Repository and retries transient connectivity failures. The Slurm service, database services and repository remain external systems connected through the control interfaces and state transitions shown here.
For MOS, Redrock is submitted first; RVS follows Redrock, FERRE follows RVS, and pPXF can run concurrently after classification. Emission-line fitting follows pPXF, line-strength measurement waits for the pPXF and emission stages, and the L2 merger waits for all six analysis branches. For IFU, preparation is the common predecessor and the Galactic and extragalactic workers are independent branches. The generated dependencies use Slurm afterany semantics: a downstream job is released when its predecessor terminates, while module-level warning and empty-output handling determines whether a useful partial product can still be merged. Each generated task propagates its command exit state; IFU workers isolate exceptions by target but fail the job when all applicable targets fail; and the merger writes missing optional component groups as empty extensions with route-specific APSQFLAG bits. A partial product is accepted only above route-minimum content checks. For example, MOS requires classification plus at least one stellar or galaxy table, whereas IFU ExGal requires both patch and binned-spectrum tables.
The DMS separately classifies timeout, node and out-of-memory states as retry-worthy. The WDAP-facing controller reports aggregate FAILED or PROCESSED state through the Operational Repository boundary. Resubmission follows the configurable WEAVE processing policy. Retries are bounded by a specified attempt limit; once this limit is reached, automatic resubmission stops and the system is notified for intervention.
| Component | Responsibility |
|---|---|
| Workflow generator | Resolve inputs/configuration; emit module scripts, resource requests and MOS/IFU/CS dependencies. |
| DMS | Discover or update processing records; generate, regenerate, submit and retract L2/CS jobs; synchronise Slurm state. |
| WDAP controller | Exchange L1/L2/CS processing states with the Operational Repository and apply bounded retry handling. |
| Slurm and databases | External execution and state services queried by PyAPS. |
| Night monitor | Combine database records with live Slurm state and expose per-night status and log summaries. |
What is in the public package. The workflow generator (the runner) and the scientific modules are part of the public PyAPS package. The database and job management (DMS), the transfer controller and the night monitor named in this section belong to the operations layer that runs the official system for the survey; they are maintained separately and are not part of the public package.
APS Explorer as the inspection and access layer
APS Explorer links these processing records to interactive L1 and L2 inspection. W9 describes the common loaders, headers-based view selection, standalone and server deployment, session isolation and large-file controls. Figure 10 of the paper illustrates the spatial, spectral and tabular views of one LIFU product.
The Level-2 merger maps module outputs into WEAVE FITS extensions and populates available processing identifiers, quality fields and provenance references. It writes FITS checksums and data sums for integrity checking.