Turning data into science
The Cambridge Centre of Excellence for Astronomical Data (CamCEAD) is based at the Institute of Astronomy, University of Cambridge. We build the data processing systems, scientific software and computing infrastructure behind some of the largest astronomical surveys and space missions in the world.
Our job is to turn very large amounts of complicated observational data into reliable products that scientists can use straight away. That involves automated pipelines, cloud processing, artificial intelligence, high-performance computing and data-heavy scientific workflows, all of which connect today's instruments to the science that comes out of them.
CamCEAD's roots go back to the Automatic Plate Measuring (APM) machine, which in the 1970s and 80s digitised photographic plates from sky surveys and showed how automation and computing could handle astronomical data on a large scale. We still work where instruments and computing meet.
From telescope to results
In modern astronomy, collecting the light is often not the hardest part. The difficulty is processing, analysing and making sense of the data. Telescopes and space observatories produce terabytes of observations every clear night, and every published catalogue, calibrated spectrum and new result depends on a long chain of software, algorithms and computers.
We build the whole of that chain: automated reduction pipelines, quality control, survey databases, cloud processing centres, web services, visualisation tools and science archives. These systems run for the full lifetime of large international survey programmes and supply data to thousands of astronomers around the world.
We also run our own platforms: CamCEADHub, a shared computing environment for analysing survey data, and Darkroom, a processing and visualisation platform for large imaging datasets.
Surveys and space missions we work on
CamCEAD is a partner in several of the largest ground-based surveys and space missions, and provides the science pipelines, operational data systems and computing that they rely on every day.
On the ground, we work on the major spectroscopic survey facilities:
WEAVE, the wide-field multi-object spectrograph on the William Herschel Telescope. CamCEAD built the whole central pipeline system and runs the data flow for every WEAVE survey.
A fully configured WEAVE field positioner plate
© WEAVE Consortium
4MOST, the 4-metre Multi-Object Spectroscopic Telescope on ESO's VISTA telescope, which will take spectra of tens of millions of objects across the southern sky.
4MOST first light
© ESO / 4MOST Consortium
MOONS, the Multi-Object Optical and Near-infrared Spectrograph for ESO's Very Large Telescope, which will study how galaxies evolve and the structure of the Milky Way in the infrared.
The MOONS instrument at the VLT
© ESO / MOONS Consortium
WST, the Wide-field Spectroscopic Telescope, a proposed facility that would take spectra of far more objects at once than any current instrument. CamCEAD is contributing to the design of its data systems.
In space, we contribute to missions in stellar and planetary science:
- Gaia, ESA's astrometric mission, which has mapped more than a billion stars. CamCEAD runs photometric processing for the Gaia consortium and has also worked on ground-based follow-up and data validation, including the Gaia-ESO Survey.
- PLATO, ESA's forthcoming planet-hunting mission, where CamCEAD is developing data infrastructure and pipelines.
We draw on many years of experience from earlier programmes such as APM, the Isaac Newton Telescope Wide Field Survey, UKIDSS, VISTA and 2dF. That experience, and the technical skills that came with it, is what lets us work at the scale the new facilities need.
Beyond astronomy
IMAXT researchers use virtual reality to move through 3D maps of tumour tissue.
© IMAXT / Cancer Research UK Cambridge Institute
The problems we solve for astronomy turn up in other fields too. What we have learned about handling large datasets, artificial intelligence, machine learning, image analysis and distributed computing is now being used in other areas of research that deal with a lot of data.
In particular, we work on joint programmes in biomedical imaging and computational medicine:
- IMAXT (Imaging and Molecular Annotation of Xenografts and Tumours), where our high-throughput imaging pipelines and AI methods are used to build three-dimensional maps of tumours, giving researchers a far more detailed view of cancer.
- SPACE, which takes spatial omics and our computational methods to a wider range of biological and clinical research.
The idea behind this work is simple: good data systems help science wherever the data is large and complicated, whether it comes from a telescope or a microscope.
Looking ahead
Telescopes are getting bigger, but the bigger change is in the amount of data and the complexity of the questions scientists want to answer with it.
The next generation of facilities will produce more data than ever before and will need new approaches to processing, artificial intelligence and scientific computing:
- The Extremely Large Telescope (ELT), the largest optical and near-infrared telescope in the world, due to start operating in the late 2020s.
- The Square Kilometre Array (SKA), the world's largest radio telescope, which will produce exabytes of data a year.
- The Vera C. Rubin Observatory, which will image the whole southern sky every three nights.
- The Wide-field Spectroscopic Telescope (WST), the next big step in multi-object spectroscopy.
- New space missions that will follow Gaia and PLATO.
CamCEAD is preparing for these by building data systems that scale, run automatically and can make use of AI. We combine a long history in astronomical surveys with current experience in cloud computing, artificial intelligence and high-performance data processing, so that scientists everywhere can get the most out of the observations still to come.