1. Overview
  2. Deployment
  3. Monitoring
  4. Maintaining
  5. Common Errors and Known Solutions
  6. Features
  7. Extinction Model and Burn-in
  8. Scientific Results
  9. Low-Latency Injections
  10. References
  11. Template Banks
  12. Pastro
  13. Offline Analysis
  14. Offline Filtering Rota
  15. Offline Rota
  16. Dropped data filtering
  17. Offline SNR Optimizer/Coinc generation
  18. MDC Documentation
  19. How to contribute to this documentation
  20. Diagrams

  1. Overview
  2. Deployment
  3. Monitoring
  4. Maintaining
  5. Common Errors and Known Solutions
  6. Features
  7. Extinction Model and Burn-in
  8. Scientific Results
  9. Low-Latency Injections
  10. References
  11. Template Banks
  12. Pastro
  13. Offline Analysis
  14. Offline Filtering Rota
  15. Offline Rota
  16. Dropped data filtering
  17. Offline SNR Optimizer/Coinc generation
  18. MDC Documentation
  19. How to contribute to this documentation
  20. Diagrams

Offline Operations Manual

Table of Contents


Offline Configuration

  • Configuration
  • Philosophy on managing config files / best practices.
  • Documentation of options set in the configuration file.
  • Describe where to find pre-set configuration files.

The instructions below describe the procedures for running an offline analysis.

Workflow to run an offline analysis

Setting up an analysis

This page is a tutorial for deploying a GstLAL offline analysis.

1. Clusters and accounts

To launch your own analysis you can use your albert.einstein account at any LDG compatible cluster.

Note: This tutorial is specific for an analysis on the Penn State cluster at ICDS. The production GstLAL analyses are run using shared accounts. Use gstlalcbc.offline accounts. As of March 2024, do not launch offline analyses on ICDS.


On CIT and ICDS the singularity bind paths are different. Bind paths are paths on the host filesystem that you want to still be available for reading and writing from within the container.

On ICDS we always bind /ligo eg,

singularity run -B /ligo <build-dir>

On CIT you don’t need a bind path, but we often bind $TMPDIR, eg

singularity run -B $TMPDIR <build-dir>

Additionally on CIT you will need a different profile than what is included in the o4a-offline-containers:main container used below. FIXME: currently we don’t have a dedicated profile for people to launch test analyses on CIT.


2. Build

The environments needed to run GstLAL O4a analyses are defined by docker containers managed in the various branches at https://git.ligo.org/gstlal/o4a-containers. These containers include an installation of GstLAL and all the dependencies needed for running the O4a analysis. Each branch contains a configuration file specific to a particular search type (eg AllSky search, etc.) The docker containers are defined by the Dockerfile in each branch and are regularly published by the gitlab CI to stay up to date with the branch of GstLAL they are based on.

To build the writable docker image, run the following:

singularity build --fix-perms --sandbox <build-dir> docker://containers.ligo.org/gstlal/o4a-containers:<branch>

This will pull a writable container of the <branch> into the directory specified by <build-dir>.

Note: we use singularity to pull and run containers instead of using docker. They are mostly interchangeable, but docker requires root priveleges which the average user won’t have on LDG clusters. By making the image writable you will be able to make changes to the code installed in the image. This is useful if you plan to do any dev work with the image.

3. Set up

Now that you have a build ready to use, it’s time to set up a run directory for an offline analysis. We will refer to this directory path as <run-dir>. All of the necessary files to get started are included in the build. Copy them over using this command:

cp -r <build-dir>/offline-analysis/* <run-dir>

4. Configuration

1. In config.yml:

  • Set the start and stop time to the gps start and end time for the specific chunk. For O4a chunk definitions of AllSky, see here.
  • Set appropriate directories under data and mass-model. The paths under data should be the paths to the appropriate directories in the <run-dir>. For the data section, an example is:
    • data-dir=/path/to/run-dir/input_data
    • injection-dir=/path/to/run-dir/injection_data
    • filter-dir=/path/to/run-dir/filter
    • rank-dir=/path/to/run-dir/rank
  • If you want to run injections, make sure that the inj-frame-cache corresponds to the injections.file that was used to generate the injection frames.
  • Set appropriate summary options. For example:
    • For summary: webdir: /home/albert.einstein/public_html/gstlal/observing/4/ssm/o4a_offline
    • For open-box: webdir: /home/albert.einstein/public_html/gstlal/observing/4/ssm/o4a_offline/OPEN-BOX
  • Under source, provide frame-cache and remove datafind-server if you want to use a frame-cache for data.
  • Change the profile to match the cluster you are on.
  • Change accounting-group-user to the correct albert.einstein credentials.
  • Change singularity-image appropriately.
  • Change directives to the path to your x509_proxy

2. Create Makefile:

  • Run singularity exec <image-name> gstlal_inspiral_workflow init -c config.yml.
  • This generates a makefile with all workflows, and all aprrpriate options read from config.yml.

3. x509 proxy

  • Run X509_USER_PROXY=x509_proxy ecp-get-cert -i LIGO -u albert.einstein to create a x509 proxy.
  • Add the path to the created x509 proxy under condor > directives > environment.

5. Run a workflow:

There are 6 workflows you can generate:

  • psd
  • svd
  • filter
  • injection-setup
  • injection-filter
  • rank

To launch:

  • Run singularity exec <image-name> make <workflow> to generate the dag.
  • condor_submit_dag <dag-name>

FIXME FIXME FIXME

Below needs to be updated and fixed.

reank

Workflow to run a rerank dag

1. Get familiar with the workflow:

Refer to offline configuration tutorial for how to run the offline analysis.

2. Modify the configuration file

Copy the configuration file and env.sh from the offline analysis chunk directory to its rerank directory. In the configuration file (config.yml), add the following:

data product options

  • analysis-dir: Set this to the path to the analysis directory of the chunk one is running a rerank for.
  • rerank-dir: Set this to the path to the rerank directory

summary options

  • webdir: Set this to the public_html path for offline analysis rerank closed and open box pages. The name of the directory should be <chunk{N}_{gps_start_time}_{gps_end_time}>

condor options

  • profile: Set this to the profile matching the cluster that the analysis will be launched on.
  • accounting-group: FIX ME
  • accounting-group-user: albert.einstein (change to your name)
  • singularity-image: Point to the singularity image

Note that other parts in the configuration file should not change unless necessary.

3. Run the analysis

Reference offline configuration tutorial. Below are commands specfic for running a rerank dag. To make a Makefile from config.yml, run:


  singularity exec <image> gstlal_inspiral_workflow init -c config.yml --workflow rerank

Follow the instructions in offline configuration tutorial to make a dag, launch an analysis, monitor an analysis, and make a summary page.

Analsyis specfic instructions

  • For O4a offline SSM, see here for details and here for ROTA information.