Detecting Burst Gravitational Waves with Wavelets and Neural Networks

layout: true
<div class="my-header"></div>
<div class="my-footer"><img src="../img/ucf.png" /><p>David C. Wright III</p></div>
---
class: center, middle

# Detecting Burst Gravitational Waves with Wavelets and Neural Networks
<figure>
    <img src="https://www.ligo.org/science/GW-Overview/images/burst_tn.jpg" alt="Burst" height=200>
    <font size="2">
      <figcaption>An example signal from an burst gravitational wave source.
        Image Credit: A. Stuver/LIGO using data from C. Ott, D. Burrows, et al.</figcaption>
    </fontfigcaption>
</figure>

---

## Introduction
- Gravitational waves are ripples in space-time created by accelerating masses
<figure>
  <center><img src="https://www.ligo.org/science/GW-Overview/images/binary-wave_tn.jpg" alt="Burst" height=200>
    <font size="2">
      <figcaption>Artist's impression of gravitational waves from two orbiting black holes.
        Image Credit: T. Carnahan (NASA GSFC)</figcaption>
    </fontfigcaption>
    </font>
  </center>
</figure>

- A **strong** gravitational wave produces disturbances on the order of \\(10^{-18}\\) meters,
or \\(\tfrac{1}{1000}\\) the width of a proton
- We need extremely sensitive equipment to measure these disturbances
---

## LIGO
- **L**aser **I**nterferometer **G**ravitational-Wave **O**bservatory
- Two detectors, one in Louisiana and another in Washington
  - They act as noise filters for each other, and the time delay between them can help locate the source of the gravitational wave
<figure>
  <center><img src="https://cdn.britannica.com/20/188520-050-B6F32AB7/Laser-Interferometer-Gravitational-Wave-Observatory-Hanford-Washington-installations.jpg" alt="Burst" height=200>
    <font size="2">
      <figcaption>Hanford, Washington LIGO Observatory. Image Credit: Caltech/MIT/LIGO Lab</figcaption>
    </fontfigcaption>
    </font>
  </center>
</figure>
- Can detect disturbances \\(\tfrac{1}{10000}\\) the width of a proton

---

## LIGO, How does it Work?
- LIGO is a large interferometer: a device which merges two or more light 
sources together to form an interference pattern

- Interference is the superposition of two or more waves to form a new wave
.pull-left[
<figure>
  <center><img src="https://www.ligo.caltech.edu/system/media_files/binaries/172/small/waveinterference.gif?1431722556" alt="Burst" height=200>
  </center>
</figure>
]
<figure>
  <center><img src="https://www.ligo.caltech.edu/system/media_files/binaries/237/original/Basic_michelson_labeled.jpg?1435862648" alt="Burst" height=200>
    <font size="2">
      <figcaption>Hanford, Washington LIGO Observatory. Image Credit: Caltech/MIT/LIGO Lab</figcaption>
    </fontfigcaption>
    </font>
  </center>
</figure>
- At rest, LIGO is designed for totally destructive interference
- Signal will be detected when length changes and interference is no longer totally destructive
---

## LIGO Animation
<center>
<iframe width="560" height="315" src="https://www.youtube-nocookie.com/embed/tQ_teIUb3tE" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture" allowfullscreen></iframe>
</center>

---

## Burst Gravitational Waves
- The least is known about the origins of these gravitational waves
- Short, "chirp" signals
- Hypotheses that systems such as supernovae or gamma ray bursts may produce these waves, but too little is known about the details of these systems to predict the form of these waves
<figure>
<center>
    <img src="https://www.ligo.org/science/GW-Overview/images/burst_tn.jpg" alt="Burst" height=200>
    <font size="2">
      <figcaption>An example signal from an burst gravitational wave source.
        Image Credit: A. Stuver/LIGO using data from C. Ott, D. Burrows, et al.</figcaption>
    </font>
</center>
</figure>
---

## Wavelets and Wavelet Transforms
- Wavelets are "mathematical functions that cut up data into different frequency components and then study each component with a resolution matched to its scale" (Graps, 1995)
- Wavelet transform is akin to a short time Fourier transform (STFT), yielding time-frequency localization
- Differences between the two include localization in space of the basis functions, varying "windows", and size of basis set
.pull-left[
<figure>
<center>
    <img src="https://upload.wikimedia.org/wikipedia/commons/thumb/0/0a/MorletWaveletMathematica.svg/1920px-MorletWaveletMathematica.svg.png" alt="Burst" width=200>
    <font size="2">
      <figcaption>Morlet wavelet. Image Credit: Wikipedia</figcaption>
    </font>
</center>
</figure>
]
<figure>
<center>
    <img align="right" src="https://upload.wikimedia.org/wikipedia/commons/thumb/c/c4/STFT_and_WT.jpg/500px-STFT_and_WT.jpg" alt="Burst" width=330>
</center>
</figure>
---

## Scaleograms
- Analogous to a Fourier transform spectrogram
- Visualization of signal with time-frequency localization
<figure>
<center>
    <img src="https://upload.wikimedia.org/wikipedia/commons/2/2b/VariableFrequency.jpg" alt="Burst" width=400>
    <font size="2">
      <figcaption>Spectrogram of an FM signal. In this case the signal frequency is modulated with a sinusoidal frequency vs. time profile. Image Credit: Wikipedia</figcaption>
    </font>
</center>
</figure>
---

## Convolutional Neural Networks (CNN)
<figure >
<center>
    <img src="https://miro.medium.com/max/2400/1*vkQ0hXDaQv57sALXAJquxA.jpeg" alt="Burst" width=400>
    <font size="2">
      <figcaption>Overview of a convolutional neural network's architecture. Image Credit: Sumit Saha towardsdatascience.com</figcaption>
    </font>
</center>
</figure>

- Designed for 2D image data, but can be used for 1D and 3D data
- The main feature of CNNs is convolutional layers
- Convolutional layers reduce the size of input data while retaining the features of the data 
---

## Convolution and Filters
- Convolution is an operation which maps a matrix to a scalar 
  - Equivalent to sum of Hadamard (element-wise) product between two matrices
- A **convolution kernel** or **filter** is a matrix that slides along the data and extracts specific features such as edges or color
- The CNN will learn these filters during training
<figure >
<center>
    <img src="https://miro.medium.com/max/500/1*GcI7G-JLAQiEoCON7xFbhg.gif" alt="Burst" width=400>
    <font size="2">
      <figcaption>Convolution of a 5x5x1 image with a 3x3x1 kernel to get a 3x3x1 convolved feature. Image Credit: Sumit Saha towardsdatascience.com</figcaption>
    </font>
</center>
</figure>
---

## General Steps in a CNN
1. Initialize all filters and weights with random values
2. Pass data through convolution (+ activation) and pooling layers
3. Feed feature maps to fully connected layers for classification
4. Update weights in network as usual

<figure >
<center>
    <img src="https://ujwlkarn.files.wordpress.com/2016/08/screen-shot-2016-08-10-at-3-38-39-am.png?w=494" alt="Burst" width=200>
    <font size="2">
      <figcaption>Max Pooling. Image Credit: Stanford</figcaption>
    </font>
</center>
</figure>
<figure >
<center>
    <img src="https://ujwlkarn.files.wordpress.com/2016/08/screen-shot-2016-08-07-at-9-15-21-pm.png?w=748" alt="Burst" width=400>
    <font size="2">
      <figcaption>Training a CNN. Image Credit: Ujjwal Karn</figcaption>
    </font>
</center>
</figure>
---

## Training a CNN to Detect Burst Gravitational Waves
- A collection of these waves of sufficient size does not exist, so it must be simulated 
- Use **LALSimulation** tool provided by the LIGO Scientific Collaboration
  - For this project, simulate linearly polarized sine-Gaussian bursts with randomized parameters
- Simulate both burst gravitational waves and pure noise and compute the scaleogram of each simulation

.pull-left[![:scale 90%](../img/signal.svg)]
.right[![:scale 42%](../img/signal+noise.svg)]
---

## Training Data
<figure >
<center>
    <img src="../img/train-data.svg" alt="Burst" width=450>
    <font size="2">
      <figcaption>Scaleograms of training data and their classifications.</figcaption>
    </font>
</center>
</figure>
---

## Network Architecture and Training Methods
- Simulating the images and computing the scaleograms takes some time, so it was not feasible to generate a sufficiently large dataset on a desktop computer
  - 2000 images took over 10 hours on a Ryzen 2600X
- Transfer learning was utilized to fill the gaps
  - Exploit networks which have already been trained to detect a wide variety of features
  - In this project, GoogLeNet (InceptionV3) was used
- The [Keras](https://keras.io/) deep-learning API was used to download the pre-trained model and modify it for the needs of this project 
- The data was split roughly 70/20/10 for training/validation/testing
- The base model was frozen, and the top layer was removed to allow training of new classifiers (signal or noise)
- Used Keras's early-stopping callback and a dropout layer to avoid overfitting
- Binary Cross-Entropy loss function
- It took around 1.5 hours to train for 100 epochs on an AMD Ryzen 2600X 6-core processor (GPU was not available)
---

<figure >
<center>
    <img src="../img/100-epochs-stats.svg" alt="Burst" height=550>
    <font size="2">
      <figcaption>Overview of model training.</figcaption>
    </font>
</center>
</figure>
---

<figure>
<center>
    <img src="../img/predict-labels.svg" alt="Burst" height=580>
    <font size="2">
      <figcaption>Model prediction of class labels on test set with 98% accuracy.</figcaption>
    </font>
</center>
</figure>
---

## Future Plans 
- Simulate and classify other types of burst gravitational waves
- Speed up simulation of data
- Fine tune the model by training more than just the top layer

## References

- Graps A. 1995. IEEE Comp. Sci. Engi. 2:50–61
- LIGO Scientific Collaboration. 2018. LIGO Algorithm Library - LALSuite
- Markowitz J, Zanolin M, Cadonati L, Katsavounidis E. 2008. arXiv:0810.2264 [gr-qc]
- The LIGO Scientific Collaboration, the Virgo Collaboration. 2010. Phys. Rev. D. 81(10):102001

Check [my GitHub](https://github.com/davecwright3/ligo-scaleogram-cnn) for the code