1. Overview
  2. Deployment
  3. Monitoring
  4. Maintaining
  5. Common Errors and Known Solutions
  6. Features
  7. Extinction Model and Burn-in
  8. Scientific Results
  9. Low-Latency Injections
  10. References
  11. Template Banks
  12. Pastro
  13. Offline Analysis
  14. Offline Filtering Rota
  15. Offline Rota
  16. Dropped data filtering
  17. Offline SNR Optimizer/Coinc generation
  18. MDC Documentation
  19. How to contribute to this documentation
  20. Diagrams

  1. Overview
  2. Deployment
  3. Monitoring
  4. Maintaining
  5. Common Errors and Known Solutions
  6. Features
  7. Extinction Model and Burn-in
  8. Scientific Results
  9. Low-Latency Injections
  10. References
  11. Template Banks
  12. Pastro
  13. Offline Analysis
  14. Offline Filtering Rota
  15. Offline Rota
  16. Dropped data filtering
  17. Offline SNR Optimizer/Coinc generation
  18. MDC Documentation
  19. How to contribute to this documentation
  20. Diagrams

Offline Rota

Table of Contents


Overview

After a fixed interval during O4, we will create create an Open Box of the events we’ve seen so far. This fixed interval is one month (two weeks) in O4b (O4a). The Open Box consists of a rerank of the online data products (triggers and background). We use the branch offline_new-workflow, since it contains the code to find the required online data products, rerank them, and also implements the new extinction model. The ROTA starts on Friday, and involves performing reranks on Bob (Edward) and Rick (Jacob) from the start of ER16 (ER15) till the midnight of the previous day (Thursday, EST), combining the two reranks, uploading the results to GraceDB, creating an open box presentation, and presenting it at the next CBC call.

Tagging the branch

We will use version control of the branch to ensure that our results our reproducible. Follow instructions here to tag the latest version of the branch. The tagged bracnhes should start with the name o4b-open-box. Make a new container where the build name is followed by the tag. Eg: gstlal-o4b-open-box-v1.1.0

Setting up the reranks

The reranks are set up in these directories: -

Bob

Login Node: gstlalcbc.offline@submit.ligo.uwm.edu
Build: /home/gstlalcbc.offline/observing/4/b/builds/<new-build>
Run: /home/gstlalcbc.offline/observing/4/b/runs/open_boxes/trigs.bob_yymmdd

Rick

Login Node: gstlalcbc.offline@ligo-hd-02.gwave.ics.psu.edu
Build: /ligo/home/ligo.org/gstlalcbc.offline/observing/4/b/builds/<new-build>
Run: /ligo/home/ligo.org/gstlalcbc.offline/observing/4/b/runs/open_boxes/trigs.rick_yymmdd

where yymmdd is the date on Friday.
Update the container with the latest tagged branch in the relevant build dirs.
You can download the required config from the O4a-offline repo (or copy it over from a previous rerank).

Making the injection file

Since late O4a, we have been processing live injections. We will rerank these live injections in our O4b open boxes. Each month’s injection files are stored in
/ligo/home/ligo.org/gstlalcbc.offline/observing/4/b/runs/injections/monthly.
Combine these injections by

singularity exec -B <base-dir> <build-dir> ligolw_add <list of monthly injection files upto current month> --output injections-yymm-YYMM.xml.gz

where yymm and YYMM are the first and last year-month of injections being processed, respectively. 

# NEMO
/home

# ICDS
/ligo

Copy over the newly created injection file to the relevant run directories. Due to restrictions on NEMO, the copying over has to be done through the user’s personal account. Run the command below in Bob’s rerank directory

scp albert.einstein@ligo-hd-02.gwave.ics.psu.edu:/ligo/home/ligo.org/gstlalcbc.offline/observing/4/b/runs/injections/injections-mm-MM.xml.gz .

Creating and submitting the DAG

In the config, you should change all the necessary parameters. If you’re copying it from a previous rerank, you’ll only have to change stop (config and chunk), rank-dir, injections-file and the summary section. Copy or scp over the injection-file to the run directories. You can create a Makefile by doing

singularity exec -B <base-dir> <build-dir> gstlal_inspiral_workflow init -c <config-filename> -w rank

Once that is done, you can set up the dag by doing. (This command might take a while, run on screen or tmux)

export OPENBLAS_NUM_THREADS=1
singularity exec -B <base-dir> <build-dir> make all -j<N>

When this command is run on the head node where online dags are running, N should be a maximum of

5 #CIT
10 #ICDS
32 #NEMO

Important

Make sure you have a valid scitoken or x509.
Before submitting, check the DAG once to ensure that no output files remap or edit any online files.
Submit the dag after the make command finishes.
Create the summary page once the dag finishes - Run make summary in the appropriate singularity environment. (This command might take a while, run on screen or tmux)
If you run into errors, look at Known Errors and Fixes

Combining the reranks

A bit of details about the new extinction model first. It performs two rounds of extinction, the first on every bin’s individual dist_stat_pdf. These get marginalized and saved to a file like rank/dist_stat_pdfs/13678/H1L1-GSTLAL_DIST_STAT_PDFS-1367800000-7356818.xml.gz. Then, the zerolag is added to this PDF, the second round of extinction with this zerolag is performed, and the result gets saved to rank/post_dist_stat_pdfs/13678/H1L1-GSTLAL_POST_DIST_STAT_PDFS-1367800000-7356818.xml.gz. When combining analyses, we don’t want to use the analysis-specific zerolag, we want to use the clustered-across-analyses zerolag. We also want to perform the second round of extinction not with the analysis-specific zerolag, but with the clustered-across-analyses zerolag. Consequently, we need to take the zerolag of the two analyses, cluster them, and then take the singly-extincted PDFs from both analyses, marginalize them and perform the second round of extinction with the clustered-across-analyses zerolag. After that, we can compute FARs for the new zerolag with the new doubly-extincted PDF. Fortunately, there’s a script which does all this for you.

All-sky open box

  1. In the IMBH rerank dir, 
    mkdir combined
cd combined
  2. Get the combine_analyses_far script from the the O4a-offline repo (or copy it over from a previous rerank). You’ll need to change the ANALYSIS_DATE and STOP parameters. Just run that script, and the combined summary page should be ready.

Fortunately, there’s a script which does all this for you. In the Bob rerank dir, create a dir called combined. In this dir, obtain the combine_analyses script from the the O4a-offline repo (or copy it over from a previous rerank). You’ll need to change the ANALYSIS_DATE and STOP parameters. Just run that script, and the combined summary page should be ready.

Uploading the results to GraceDB

We intend to upload every week’s results to GraceDB with the offline tag. Since the results are cumulative, there will be a version number for the results, so people can look at the latest ones. This is still a work in progress.

Creating an Open Box presentation

Get the print_open_box_tables script from the O4a-offline repo, and run it like this:

./print_open_box_tables --start 1395500000 --stop <stop> --database H1L1V1-0000_1839_GSTLAL_TRIGGERS-1395500000-<duration>.sqlite --far-threshold 3.8e-7 --previous-database ../../trigs.ebob_241215/combined/H1L1V1-0000_1839_GSTLAL_TRIGGERS-1395500000-22774018.sqlite

This will print out a bunch of tables compatible with markdown. Create a git issue for the week’s offline ROTA in the the O4a-offline repo issues page, and copy and paste the tables in the issue description. This will be the open box presentation for that week.

Presenting at the CBC call

If there are significant changes or new events that are intersting and should be discussed, you can sign up to give a presentation at the CBC call.

Old directories

The old O4a reranks are set up in these directories:
Edward: /home/gstlalcbc.offline/observing/4/a/runs/trigs.edward_yymmdd
Jacob: /ligo/home/ligo.org/gstlalcbc/observing/4/a/offline/runs/trigs.jacob_yymmdd \

Known Errors and Fixes

  1. If you encounter
cp: cannot open '/ligo/home/ligo.org/gstlalcbc/observing/4/b/runs/trigs.rick_o4b/mass_model/inj_fgmc_model_O4.h5' for reading: Permission denied

Login through the online account and run chmod a+r on the relevant directory

  1. lalinspiral_injfind might get held. Increase memory to 19G and release
condor_qedit --constraint "JobStatus == 5" RequestMemory 19000 && condor_release -all