Modular scientific-software integration
Describes version 2.0PyAPS provides a managed execution layer around the scientific engines that estimate redshifts, velocities, stellar parameters, emission-line properties and stellar-population quantities. Core adapters validate the common APSOB representation, translate it into engine-specific arrays, resolution objects, configuration files or intermediate FITS products, allocate resources, monitor execution and validate the returned products. Other large survey pipelines use a similar modular separation (Guy et al., 2023; Gilmore et al., 2022; Westfall et al., 2019). In PyAPS, the same approach is adapted to WEAVE and reused across all input forms described in W3.5.
| Element | Established comparator practice | PyAPS-specific system content | Scope of comparison |
|---|---|---|---|
| Common spectral/data interface | DESI and Gaia-ESO use stable exchange products; MUSE joins 24 IFUs through pixel tables; APOGEE supplies a common multi-extension loader; and MaNGA treats spaxel and binned spectra through a shared analysis interface (Guy et al., 2023; Weilbacher et al., 2020; Nidever et al., 2015; Gilmore et al., 2022; Westfall et al., 2019) | APSOB normalises MOS, LIFU, mIFU and their fibre-level forms across arm, resolution, binning and stacking variants before target-level analysis | Unified WEAVE fibre-level/reconstructed-IFU contract spanning Galactic and extragalactic analysis |
| WEAVE-specific scientific processing | Survey systems implement instrument-specific quality, target-detection and calibration-propagation stages | PyAPS implements quality-aware preparation, ranked classification safeguards, seg2d/seg3d IFU detection, patch/bin construction, map reconstruction and fibre-to-spaxel/bin LSF propagation |
Scientific methods designed for the combined WEAVE fibre and reconstructed-IFU product space |
| External scientific engines | Survey pipelines routinely combine independently developed classifiers, fitters and template libraries | PyAPS routes one common object through Redrock, RVSpecFit, FERRE, pPXF and LS; EMI is its WEAVE emission-line implementation of pPXF. All paths include validation, resources, monitoring, failure handling and output quality control (QC) | Managed scientific-execution and quality-control layer across all principal methods |
| Resolution handling | IFU analysis pipelines propagate instrumental resolution into fitting and binning products (Westfall et al., 2019) | PyAPS imports WEAVE on-sky per-fibre calibration products and maps them to fibres, spaxels, bins and bounded resolution groups | End-to-end WEAVE LSF propagation across physical and reconstructed spatial elements |
| Automation, jobs and scale | DESI and other survey systems document parallel workflow execution, provenance and release processing (Guy et al., 2023; Gilmore et al., 2022) | PyAPS combines dependency-aware job generation, mode-aware resource policy, APSOB memory reuse, target/arm/patch concurrency, monitoring, retry control and L1-L2 provenance | Unified local/Slurm automation, monitoring and release-processing control |
The runner also defines contributed-software adapters selected by observing mode, resolution, binning and target class. An adapter may receive normalised Level-1 inputs, an intermediate classifier or stellar-analysis product, or an assembled APS Level-2 product. These exchange points let contributed software reuse intermediate processing without moving its algorithm into the PyAPS core. The release documentation records which adapters are active and who maintains them.
Integration mechanisms differ substantially by engine. Python packages can be called through imported APIs, while FERRE is a separately compiled Fortran program controlled through namelists and spectral files, subprocess execution, polling, parsing and result merging (Allende Prieto & the APOGEE Team, 2015). PyAPS therefore contains engine-specific process counts, memory requests, grid policies, failure interpretation and product conversion. The wrapper controls the complete run, while each scientific analysis program remains focused on its numerical method and receives properly prepared WEAVE data.
System boundary
Within the WEAVE Science Processing and Analysis system, the Core Processing System performs observatory reduction and supplies Level-1 spectra, PyAPS is the official Python implementation of the Advanced Processing System for higher-level scientific analysis, and the WEAVE Archive System distributes the resulting products (Jin et al., 2024). Transfer services, repositories, schedulers and monitoring connect these components into the operational dataflow. Figure 3 of the paper and Table 33 identify the information exchanged at each interface and distinguish the PyAPS software from the surrounding services.
| Element | Primary responsibility | Information crossing the boundary | Relation to PyAPS |
|---|---|---|---|
| CPS | Observatory reduction, calibration, quick-look and Level-1 production | L1 FITS products containing count-level spectra, inverse variance, masks, the wavelength-dependent sensitivity function and calibration/provenance metadata | Upstream L1 provider |
| APS/PyAPS | WEAVE-specific scientific processing, automated job/HPC management, orchestration and Level-2 assembly | APSOB/APSTARG/APSSPEC states, scientific intermediate products, LSF inputs, job/resource state and WEAVE-format L2 products | Scientific and automation system described here |
| Operational Repository | Processing state, product records and workflow metadata | Job state, identifiers, paths, timestamps and operational provenance | External infrastructure service |
| Data transfer service | Defined transfer/data-state interface between processing services and repositories | File/state transfer requests and status information | External infrastructure interface |
| WASP | Proposal, observing planning and execution context | Target/configuration information retrieved by CPS and associated observing identifiers | Upstream planning context |
| WAS | Archive, distribution and user access to released products | Validated L2 products, catalogues, metadata and access records | Downstream consumer and distributor |
| Contributed software | Independent scientific or intermediate-product analysis | L1, intermediate or L2 exchange products under managed contracts | PyAPS supplies preparation, execution control, QC, monitoring, provenance and product integration |
PyAPS integrates Redrock, RVSpecFit, FERRE, pPXF, Voronoi binning and PowerBin, whose numerical methods are developed by their respective projects (Guy et al., 2023; Anand et al., 2024; Koposov, 2019; Allende Prieto et al., 2006; Allende Prieto & the APOGEE Team, 2015; Allende Prieto, 2017; Cappellari & Emsellem, 2004; Cappellari, 2017; Cappellari & Copin, 2003; Cappellari, 2025). PyAPS supplies the complete WEAVE scientific and operational contract around them: input and output validation, configuration, resolution matching, process and memory control, parallelisation, monitoring, diagnostics, failure interpretation, quality control, product mapping and orchestration. In addition, PyAPS owns the WEAVE-specific decisions that connect calibrated products to those engines: quality-aware data preparation, target routing, IFU segmentation and patching, spatial aggregation and LSF propagation. Contributed software uses the same controlled L1, intermediate or L2 exchange contracts.
The on-sky LSF package is a team-developed supplementary component of the WEAVE dataflow between L1 and L2, rather than external survey software. Ardern-Arentsen et al. (in preparation) describe the Instrument Consortium calibration derived from twilight spectra using ULySS-based relative-LSF fitting (Koleva, Prugniel, Bouchard & Wu, 2009). PyAPS consumes and propagates the resulting spline products.