HARPSpol Reduction Pipeline

Reducing all HARPS polarimetry data from the ESO archive using PyReduce. The goal is a consistent, polarimetry-focussed reduction of all public data for publication on PolarBase.

Requirements

• uv (manages Python 3.13+ and all dependencies)
• GNU parallel (for batch processing)
• An ESO archive account (username set in config.py)

All Python dependencies are managed automatically by uv via pyproject.toml.

Important version notes:

• astroquery >= 0.4.11 is required. Versions <= 0.4.7 are broken due to ESO archive API changes (July 2025). The login API also changed to keyword-only arguments in 0.4.8+.
• pyreduce-astro >= 0.8a4 with native dual-beam support.

Tested with: astroquery 0.4.11, pyreduce 0.8a4, barycorrpy 0.4.4, numpy 2.4, scipy 1.17, astropy 7.2.

Setup

git clone ssh://git@codeberg.org/verveine/harpspol.git
cd harpspol
uv sync

PyReduce is expected at ../PyReduce (editable install via pyproject.toml). Adjust the path in [tool.uv.sources] if your checkout is elsewhere.

Pipeline Overview

Script	Purpose
01_query_archive.py	Query ESO archive for HARPSpol science observations
02_download.py	Download science and calibration FITS files
03_reduce.py	Run PyReduce (calibrations and/or science extraction)
04_demodulate.py	Polarimetric demodulation (4-exposure Stokes V/I)
05_combine_channels.py	Merge blue + red demodulated spectra
config.py	Shared configuration (paths, channels, constants)
run_all.sh	Batch runner with GNU parallel

Quick Start

1. Query the ESO archive

# All HARPSpol data from 2012
uv run python 01_query_archive.py --start 2012-01-01 --end 2013-01-01

# Filter by ESO program ID
uv run python 01_query_archive.py --start 2012-01-01 --end 2013-01-01 --program 187.D-0917

Outputs datasets.ecsv (QC-approved science files) and calib_nights.ecsv (calibration nights).

2. Download data

uv run python 02_download.py --all                # science + calibrations
uv run python 02_download.py --science             # science only
uv run python 02_download.py --calibs              # calibrations only
uv run python 02_download.py --calibs --night 2012-07-14  # single night

Files are organized as:

• data/science/{TARGET}_{TPL_START}/raw/ — 4 raw science FITS per observation
• data/calibs/{YYYY-MM-DD}/raw/ — bias, flat, ThAr for each night

3. Reduce calibrations

uv run python 03_reduce.py calibs --night 2012-07-14

Runs: bias, flat, trace, curvature, scatter, norm_flat, wavecal_master, wavecal. Output goes to data/calibs/{night}/reduced/.

4. Extract science spectra

uv run python 03_reduce.py science --obs HD-96446_2012-07-14T22-55-44

Automatically finds the nearest calibration night, symlinks products, and runs optimal extraction. Output goes to data/science/{obs}/reduced/.

5. Demodulate

uv run python 04_demodulate.py --obs HD-96446_2012-07-14T22-55-44
uv run python 04_demodulate.py --obs HD-96446_2012-07-14T22-55-44 --plot  # with diagnostics
uv run python 04_demodulate.py --obs HD-96446_2012-07-14T22-55-44 --channel BLUE  # single channel

Computes Stokes V/I (or Q/I for linear pol), null spectrum, continuum normalization, and barycentric correction. Output: data/science/{obs}/demodulated/demodulated_{blue,red}.fits.

6. Combine channels

uv run python 05_combine_channels.py --obs HD-96446_2012-07-14T22-55-44

Concatenates blue and red into data/science/{obs}/demodulated/demodulated.fits.

Batch Processing

Run everything

./run_all.sh                    # all steps, 4 parallel jobs
./run_all.sh -j 8               # 8 parallel jobs
./run_all.sh --program 187.D-0917  # filter by program

Run specific steps

./run_all.sh --step calibs                  # calibrations only
./run_all.sh --step science                 # science extraction only
./run_all.sh --step science,demod,combine   # skip calibs
./run_all.sh --step demod -j 12             # re-run demodulation, 12 jobs

Available steps: calibs, science, demod, combine (or all).

Process all observations for a program

# Demodulate + combine all observations from program 187.D-0917
./run_all.sh --step demod,combine --program 187.D-0917 -j 8

Process a batch with the individual scripts

# All observations, parallel science extraction
ls -d data/science/*/raw | xargs -I{} dirname {} | xargs -n1 basename | \
    parallel -j4 uv run python 03_reduce.py science --obs {}

# Demodulate all reduced observations
uv run python 04_demodulate.py --all
uv run python 04_demodulate.py --all --program 187.D-0917

# Combine all demodulated observations
uv run python 05_combine_channels.py --all
uv run python 05_combine_channels.py --all --program 187.D-0917

Program ID Filtering

All scripts support --program to filter by ESO program ID (substring match):

uv run python 01_query_archive.py --program 187.D-0917   # filter archive query
uv run python 02_download.py --all --program 187.D-0917   # download only this program
uv run python 04_demodulate.py --all --program 187.D-0917 # demodulate only this program
./run_all.sh --program 187.D-0917                         # full pipeline for one program

For 03_reduce.py, the --program flag works with the all subcommand:

uv run python 03_reduce.py all --program 187.D-0917

Directory Structure

harpspol/
├── config.py                    # shared configuration
├── 01_query_archive.py          # ESO archive query
├── 02_download.py               # data download
├── 03_reduce.py                 # PyReduce calibration + science
├── 04_demodulate.py             # polarimetric demodulation
├── 05_combine_channels.py       # merge blue + red channels
├── run_all.sh                   # batch runner
├── pyproject.toml               # uv project (dependencies)
├── datasets.ecsv                # (generated) science file list
├── calib_nights.ecsv            # (generated) calibration nights
│
├── data/
│   ├── calibs/
│   │   └── {YYYY-MM-DD}/
│   │       ├── raw/             # raw calibration FITS
│   │       └── reduced/         # PyReduce output (bias, flat, traces, wavecal)
│   │
│   └── science/
│       └── {TARGET}_{TPL_START}/
│           ├── raw/             # 4 raw science FITS
│           ├── reduced/         # symlinked calibs + extracted science spectra
│           └── demodulated/     # demodulated Stokes spectra
│
├── doc/                         # paper catalog scripts & data
└── paper/                       # manuscript

Demodulated Output Format

The output FITS files contain a binary table with these columns:

Column	Unit	Description
WAVE	nm	Wavelength (barycentric-corrected)
I	ADU	Stokes I (total intensity)
I/Ic		Continuum-normalized intensity
ERR_I	ADU	Error on I
ERR_I/Ic		Error on I/Ic
Ic	ADU	Continuum
STOKES		Stokes V/I (or Q/I, U/I)
ERR_STOKES		Error on Stokes parameter
STOKES/Ic		Stokes parameter scaled by normalized intensity
ERR_STOKES/Ic		Error on scaled Stokes
NULL		Null spectrum (diagnostic)
ERR_NULL		Error on null
NULL/Ic		Null scaled by normalized intensity
ERR_NULL/Ic		Error on scaled null
ORDER		Order label (e.g., blue22, red05)

How It Works

This pipeline replaces the previous 11-step workflow that required dual-path calibration (1-beam for wavecal, 2-beam for traces) and manual copying of products between directories. PyReduce now handles dual-beam instruments natively via fibers_per_order: 2 in the instrument config, which auto-pairs the two beams from the Wollaston prism.

The demodulation algorithm (ratio method for 4 circular polarimetry exposures):

1. Extract upper/lower beam spectra from each exposure using GROUP and M columns
2. Estimate and remove blaze function (spline fit to median lower-beam spectrum)
3. Cross-correlate beams to find scale and wavelength offset
4. Interpolate upper beam onto lower beam wavelength grid
5. Compute R = (u1/d1)(u3/d3) / (u2/d2)(u4/d4), then V/I = (R^{1/4} - 1) / (R^{1/4} + 1)
6. Compute null spectrum: RN = (u1/d1)(u2/d2) / (u3/d3)(u4/d4)
7. Fit continuum, normalize, apply barycentric correction

Contact

Alexis Lavail