1. Overview
  2. Deployment
  3. Monitoring
  4. Maintaining
  5. Common Errors and Known Solutions
  6. Features
  7. Extinction Model and Burn-in
  8. Scientific Results
  9. Low-Latency Injections
  10. References
  11. Template Banks
  12. Pastro
  13. Offline Analysis
  14. Offline Filtering Rota
  15. Offline Rota
  16. Dropped data filtering
  17. Offline SNR Optimizer/Coinc generation
  18. MDC Documentation
  19. How to contribute to this documentation
  20. Diagrams

  1. Overview
  2. Deployment
  3. Monitoring
  4. Maintaining
  5. Common Errors and Known Solutions
  6. Features
  7. Extinction Model and Burn-in
  8. Scientific Results
  9. Low-Latency Injections
  10. References
  11. Template Banks
  12. Pastro
  13. Offline Analysis
  14. Offline Filtering Rota
  15. Offline Rota
  16. Dropped data filtering
  17. Offline SNR Optimizer/Coinc generation
  18. MDC Documentation
  19. How to contribute to this documentation
  20. Diagrams

Dropped Data Filtering

Table of Contents


Overview

The offline results are reranked using online results, and some data that were dropped in online analysis for some reason need to be refiltered for completeness. We will filter the dropped data using the same whitened & decomposed templates that were used online, hereafter referred to simply as SVDs. The non-injection triggers output by this filtering will be included in the final combined reranks.

Instructions

  1. Locate the latest container from dropped_data_refiltering branch of the o4b-containers repository. The builds on the shared account are located at

    • Bob: /home/gstlalcbc.offline/observing/4/a(b)/builds/
    • Rick: /ligo/home/ligo.org/gstlalcbc/observing/4/a(b)/offline/builds/ and follow the naming convention offline_abcdefgh where abcdefgh is the first 8 characters of the git commit hash.

  2. (This should be done only once for each cluster across all the chunks.) Generate reconstruction segments files for every SVD bin.
    Filtering the dropped data segments is performed without changing --start/--stop options in gstlal_inspiral job but by giving a tailored segment file to --recontruction-segment option. This option restricts the segments where the job reconstruct a SNR timeseries from the SVD basis to the physical template basis, which is the most computationally expensive part. Therefore, it allows us to do effectively the same thing as changing --start/--stop options without the potential concerns, e.g., PSD measurement robustness in the case of analyzed segment being too short. Since a filtering dag depends on the reconstruction segment file, it needs to be generated in a bin-dependent way before launching the dag.

    First, create an analysis directory.

    • Bob: /home/gstlalcbc.offline/observing/4/a(b)/refiltering_dropped_data/
    • Rick: /ligo/home/ligo.org/gstlalcbc/observing/4/a(b)/offline/refiltering_dropped_data Copy Makefile.recseg from the build (with sandbox mode).
    cp <singularity-build-path>/offline_analysis/Makefile.recseg <analysis-directory>

    Modify SINGULARITY_IMAGE, BIND, ONLINE_DIR, RERANK_DIR variables appropriate for each analysis. Then run

    make all -f Makefile.recseg

    This will create segment file for every SVD bin under reconstruction_segment/ together with other by-products like those in misc_files,diff, etc…

  3. There is a script that automates setting a filtering dag for each chunk. You need to copy create_dropped_data_refilter_dag.sh from the build and modify the following variables accordingly:

    refilter_dir= # ADDME
analysis= # "rick" or "bob"
inj_dirs=$(ls -d /ligo/home/ligo.org/gstlalcbc/observing/4/b/offline/injections/trigs.rick_*)
analysisdir= # ADD online analysis directory
singularitybuild= # ADDME
bind= # ADDME, for example "/ligo" in ICDS
marie_curie=  # ADDME

    Then, run with stdout piped into a separate txt file

    ./create_dropped_data_refilter_dag.sh > create_dropped_data_refilter_dag.output

    While it is running, you can also keep track of the output by

    tail -f create_dropped_data_refilter_dag.output

    in a separate terminal. Once this finishes, there will be run directories, e.g., trigs.rick_1397923218_604800, in each of which filter_inspiral_dag.dag is ready to be submitted.

  4. Submit and monitor the dag and keep track of their progress in this wiki.