Static and dynamic data model
Describes version 2.0Table 2, Table 3 and Table 4 describe collection membership, construction order and field-level contracts.
The APSOB data model is intentionally dynamic at the collection level. The same Python object can hold a MOS target, an IFU spaxel or a spatially binned representation, while the shape and metadata of its arrays remain tied to the input product and selected processing options. This page records the fields that are stable enough to define the scientific contract and the fields that must be read from the object at runtime.
Object hierarchy
| Layer | Runtime content |
|---|---|
| APSOB | Input mode, arm collection, target list, global metadata, calibration and resolution state. |
| APSTARG | Target/fibre/spaxel identity, coordinates, class/routing metadata, provenance and one or more spectra. |
| APSSPEC | Wavelength grid, flux-calibrated spectrum, propagated inverse variance (IVAR), retained L1 sensitivity function, SNR, arm/setup identifiers and callable or evaluated LSF. |
| Spatial/bin record | Spaxel membership, bin ID, coordinates, bin SNR, flux weights and links to patch tables. |
| Analysis product | Module-specific parameters, model spectra, warnings, quality flags and provenance keys. |
For MOS, an APSTARG normally corresponds to one physical fibre. For a cube, it corresponds to one reconstructed spaxel during preparation; after spatial binning, the analysis record refers to a set of spaxels and a derived bin spectrum. The object hierarchy therefore preserves both the uniform spectral access contract and the geometry needed to interpret a result.
Stable fields and runtime fields
The stable part of the contract consists of wavelength, flux, inverse variance, target identifier and provenance. Runtime fields depend on the chosen flags: joined versus separate arms, rest-frame versus observed-frame wavelengths, global versus fibre/spaxel/bin LSF, and linear versus logarithmic sampling. A module must therefore request the fields it supports and cannot assume that every APSSPEC has the same array length or wavelength frame.
APSOB construction sequence
The common object is not created by a single format conversion. The loader applies an ordered preparation sequence; individual flags can disable optional steps, but the resulting processing state is retained in the object and in the product provenance. The sequence below makes explicit which operations are shared before an engine-specific adapter receives an APSSPEC.
| Stage | Operation | Contract produced for downstream modules |
|---|---|---|
| 0 | Validate file, mode, camera, resolution, binning and target metadata | Accepted input form and diagnostic state; invalid layouts stop before scientific fitting. |
| 1 | Build or select target/fibre/spaxel records | APSTARG identities, coordinates, masks and catalogue/provenance fields. |
| 2 | Load count-level arm arrays, the L1 SENSFUNC calibration and the wavelength grid |
APSSPEC count-level spectrum, IVAR, retained sensitivity function and arm/setup metadata. |
| 3 | Clip wavelength limits and mark NaN/invalid values | Consistent valid-pixel mask and finite-array contract. |
| 4 | Apply gap, safe-gap, sky-residual/OH and telluric masks | IVAR/mask state that prevents known unusable pixels entering a fit. |
| 5 | Optionally fill gaps and remove cosmic rays | Explicitly flagged repaired values; unrepaired gaps remain distinguishable. |
| 6 | Multiply the count-level spectrum by SENSFUNC, propagate IVAR by dividing by SENSFUNC, and apply arm-wise IVAR normalisation |
Flux-calibrated spectrum and uncertainty weights on the selected flux scale, with calibration and normalisation metadata. |
| 7 | Compute S/N and optional legacy arm-ratio correction | Target/spaxel selection diagnostics and recorded processing flags. |
| 8 | Optionally convert air wavelengths to vacuum wavelengths | Wavelength frame required by the selected classifier or scientific engine. |
| 9 | Import and interpolate LSF/FWHM information | Global, fibre, spaxel or bin resolution access and its fallback state. |
| 10 | Join or retain separate arms; optionally collapse/co-add spectra | Engine-ready one- or multi-arm APSSPEC objects with constituent provenance. |
The order is scientifically consequential. For example, masking and IVAR normalisation affect Redrock ranking, wavelength-frame selection affects rest-frame galaxy analysis, and resolution import must precede template/data matching. The sequence is therefore part of the PyAPS interface contract, not merely an implementation detail; exact flag values and optional branches remain run-specific.
| Field family | Produced during | Used by |
|---|---|---|
| Mode/arm/setup | Header inspection and configuration resolution | Target selection, wavelength construction, template/grid choice and product naming. |
| Flux/IVAR | APSOB masking, sensitivity-function calibration with uncertainty propagation, and arm normalisation | All scientific engines; IVAR zero or large error communicates unusable pixels. |
| Wavelength | Header WCS and clipping; optional vacuum or redshift transform | Redrock, RVSpecFit (RVS), FERRE, pPXF, emission and line-strength modules. |
| SNR | APSOB broadband/reduction calculation; IFU adaptive thresholds | Target/spaxel filtering and binning diagnostics. |
| LSF/FWHM | Calibration import and fibre/spaxel/bin aggregation | Redrock resolution matrices, RVS/FERRE convolution and pPXF template matching. |
| Class/redshift | Redrock and patch editing | Branch selection, rest-frame preparation, template choice and quality reporting. |
| Spatial provenance | Cube reconstruction and patch/bin preparation | Spaxel-weighted LSF, bin membership, masks and L2 linkage. |
| Warnings/quality control (QC) | Each adapter and merger | Fallback interpretation, user inspection and release-level quality flags. |
Memory and sharing semantics
The dynamic model is also a memory model. Target spectra in one arm refer to rows of shared processed arrays and share wavelength and invariant resolution objects where possible. An adapter may request a serialisable or multiprocessing shared-memory representation, but the ownership and lifetime of that view depend on the worker boundary. The measured memory and serialisation performance of these sharing strategies is reported in the accompanying paper.
Common data and configuration architecture
Input classification and normalisation
PyAPS begins by interpreting the observing mode, camera, disperser, resolution and detector binning recorded in a Level-1 FITS product. The l1_fileinfo path distinguishes five normalised input forms: MOS, LIFU, MIFU, MOSLIFU and MOSMIFU. Two-dimensional fibre products and three-dimensional reconstructed cubes are converted to a common target and spectrum representation while retaining mode-specific coordinates, physical fibre identifiers and input history where these exist.
The object hierarchy and shared-array representation are defined above in W3. The following tables add the setup selectors and spectral fields used during ingestion.
Configuration space
Here ExGal denotes the extragalactic branch (galaxies and QSOs), while Gal denotes the Galactic branch (stars and white dwarfs). The APS v2.0 source contains 30 base input-form-resolution-binning selectors and 75 accepted arm-specific resolution keys before stacking, target selection and science-module routing. These counts describe the configurations recognised by PyAPS v2.0. They do not describe how often each configuration is observed.
| Input form | Typical geometry | Resolution mode (nominal ) |
Detector binning |
Common representation |
|---|---|---|---|---|
| MOS | Fibre spectra | LR ( ); HR ( ) |
11, 21, 41 | APSTARG per selected fibre |
| LIFU | Reconstructed cube/spaxels | LR ( ); HR ( ) |
11, 21, 41 | APSTARG per spaxel |
| MIFU | Reconstructed mini-IFU | LR ( ); HR ( ) |
11, 21, 41 | APSTARG per spaxel |
| MOSLIFU | LIFU fibre-level L1 layout analogous to MOS | LR ( ); HR ( ) |
11, 21, 41 | APSTARG per selected fibre |
| MOSMIFU | mIFU fibre-level L1 layout analogous to MOS | LR ( ); HR ( ) |
11, 21, 41 | APSTARG per selected fibre |
Every row in Table 5 contributes six base selectors and 15 arm-specific keys: three LR binning cases each accept BLUE and RED, while three HR cases each accept the internal PyAPS keys BLUE, GREEN and RED. The published WEAVE nomenclature is Blue-1 and Blue-2 (Jin et al., 2024); at HR, the internal BLUE key maps to Blue-1 and the internal GREEN key maps to Blue-2. The resulting
and
counts are reproduced in the machine-readable configuration crosswalk. The v2.0 interface therefore comprises 30 base selectors and 75 arm-specific keys. Stacking depth, target selection and scientific-module routing are independent dimensions and are not folded into these counts.
Table 5 summarises the mode-specific input forms, while Table 6 lists the common information passed to later modules. The input form, resolution and binning are identified during ingestion, before the scientific modules receive APSTARG and APSSPEC objects. Product-specific details are therefore handled once in APSOB, while later modules use the same internal structure.
| Object/field | Role in processing |
|---|---|
| APSOB | Collection, mode metadata, arm products and shared resolution state |
| APSTARG | Target, fibre or spaxel identity, coordinates and selection metadata |
| APSSPEC | Wavelength, flux, inverse variance, the retained L1 sensitivity-function vector and LSF access |
| Processing record | Input identifiers, calibration links and reconstruction ancestry |
Selection, correction and metadata
During construction, APSOB selects targets or spatial regions, builds the wavelength representation, normalises inverse variance, applies the requested L1 sensitivity-function calibration and spectral corrections, loads wavelength-dependent resolution, and optionally joins or splits arms. The resulting target metadata preserves identifiers needed for catalogue matching, module exchange and product assembly. Exact options are configuration- and workflow-dependent and are stored in the run configuration and processing record.
Object and memory model
Within one processed arm, target spectra share the wavelength object and refer to rows of processed two-dimensional flux and inverse-variance arrays, while retaining the corresponding L1 sensitivity-function calibration. This avoids allocating a full independent arm array for every target. Invariant global-resolution objects are also shared. The current load path uses in-place application of the count-to-flux sensitivity function, reduction-based signal-to-noise prefiltering and hoisted per-arm metadata to reduce temporary arrays and Python object work. Representative timing and peak-memory measurements are reported in the accompanying paper.
| Field(s) | Contents | Origin |
|---|---|---|
| APSOB: one per observation (one set of input files) | ||
mode, setups, camera,resolution, res_mode, xbin, ybin |
Normalised observing mode and per-arm resolution/binning setup | L1 header, normalised by l1_fileinfo |
obid, obsdate, fpmode, caldate |
Exposure/observation identifiers and calibration date | L1 header |
wlranges, wlranges_def |
Working and maximum available wavelength ranges | L1 data; working range may be user-restricted |
origin, wcs |
Field-centre coordinates; cube WCS (IFU) | L1 header |
calfiles, lsffiles,lsf_template |
Calibration and LSF file references | Team-developed supplementary WEAVE LSF product, resolved from L1 caldate |
aps_ids, idfx, idxf |
Target/spaxel identifier list and index maps | APS bookkeeping, not physical data |
join_arms, split_arms,collapse, skysub, funit,arms_ratio |
Processing configuration | APS configuration choice |
| APSTARG: one per selected target, fibre or spaxel ( per APSOB) |
||
aps_id |
Target/fibre/spaxel identifier | L1 fibre table (MOS) or APS-synthesised (IFU spaxel) |
cname, targid, targsrvy,targclass, fib_status,targuse, targprog |
Catalogue identity, survey and class metadata | L1 header/catalogue |
targra, targdec |
Target coordinates | L1 catalogue |
spectra |
List of |
Container (see below) |
meta |
List of per-arm metadata dictionaries |
Mixed (see below) |
split_arms,mask_bad_overlap |
Per-target processing flags | APS processing choice |
| APSSPEC: one per arm, per target ( per APSTARG; once arms are joined) |
||
wave |
Wavelength grid for this arm | L1 header/data; common grid built (and joined) by APS |
flux, ivar |
Flux and inverse-variance arrays | L1 count-level data after the selected APS operations (sensitivity-function calibration, gaps, tellurics, cosmic rays, sky-subtraction choice and arm join) |
sens |
Wavelength-dependent count-to-flux sensitivity function | L1 SENSFUNC, retained unmodified after its application |
R, Rcsr |
Resolution matrix (dense/sparse) | APS-computed (resolution deconvolution) |
meta[i]: SETUP, mode,AIRVAC, TELLUR, SKYSUB |
Arm identity and processing-mode flags | APS-derived from L1 header and configuration |
meta[i]: GAP_FILL,SENS_CORR, OFFSET_APPLIED,IVAR_NORMALIZED,IVAR_WEIGHT,ARMS_RATIO_APPLIED |
Record of which corrections were applied, and with what parameters | APS processing record |
meta[i]: fwhm, gfwhm |
Per-fibre/global instrumental resolution (FWHM/LSF) | Supplementary WEAVE LSF product, ingested and interpolated by APS |
meta[i]: APS_ID, NSPEC,TARGNAME, CNAME, TARGID,TARGSRVY, TARGCLASS,TARGRA(_0), TARGDEC(_0),FIB_STATUS, TARGUSE,TARGPROG |
Per-arm duplicate copy of the target’s identity fields | L1 catalogue, repeated from the APSTARG level |
meta[i]: GAPS_MASKED,CR_CLEAN |
Per-target quality flags for this arm | APS-computed |
meta[i]: SNR |
Per-target signal-to-noise for this arm | APS-computed from the L1 flux/ivar arrays |