Download a test dataset

If you don’t have your own data to use right away, that is ok. You can download test data to try out the tool by clicking on the test data button. A .csv file of HDPE Raman spectrum will download on your computer. This file can also be used as a template for formatting .csv data into an Open Specy accepted format. The following line of code does the same:

data("raman_hdpe")

Choose whether to share your uploaded data or not

Before uploading, indicate if you would like to share the uploaded data or not using the slider. If selected, any data uploaded to the tool will automatically be shared under CC-BY 4.0 license and will be available for researchers and other ventures to use to improve spectral analysis, build machine learning tools, etc. Some users may choose not to share if they need to keep their data private. If switched off, none of the uploaded data will be stored or shared in Open Specy.

Upload/Read Data

Open Specy allows for upload of .csv, .asp, .jdx, .0, .spc, and .spa files. .csv files should always load correctly but the other file types are still in beta development, though most of the time these files work perfectly. It is best practice to cross check files in the proprietary software they came from and Open Specy before use in Open Specy. Due to the complexity of these file types, we haven’t been able to make them fully compatible yet. If your file is not working, please contact the administrator and share the file so that we can get it to work.

For the most consistent results, files should be converted to .csv format before uploading to Open Specy. The specific steps to converting your instrument’s native files to .csv can be found in its software manual or you can check out Spectragryph, which supports many spectral file conversions (see Mini Tutorial section: File conversion in Spectragryph to Open Specy accepted format).

If uploading a .csv file, label the column with the wavenumbers wavenumber and name the column with the intensities intensity.

Wavenumber units must be cm^-1. Any other columns are not used by the software. Always keep a copy of the original file before alteration to preserve metadata and raw data for your records.

To upload data, click Browse and choose one of your files to upload, or drag and drop your file into the gray box. At this time you can only upload one file at a time.

Upon upload and throughout the analysis, intensity values are min-max normalized (Equation 1).

Equation 1: Max-Min Normalization

The following R functions from the Open Specy package will also read in spectral data accordingly:

read_text(".csv")
read_asp(".asp")
read_0(".0")

Viewing Spectra Plot

After spectral data are uploaded, it will appear in the main window. This plot is selectable, zoomable, and provides information on hover. You can also save a .png file of the plot view using the camera icon at the top right when you hover over the plot. This plot will change the view based on updates from the Intensity Adjustment selection.

Intensity Adjustment

Open Specy assumes that intensity units are in absorbance units but Open Specy can adjust reflectance or transmittance spectra to absorbance units using this selection in the upload file tab. The transmittance adjustment uses the \(\log_{10} 1/T\) calculation which does not correct for system or particle characteristics. The reflectance adjustment use the Kubelka-Munk equation \(\frac{(1-R)^2}{2R}\). If none is selected, Open Specy assumes that the uploaded data is an absorbance spectrum and does not apply an adjustment.

This is the respective R code:

library(dplyr)
#> 
#> Attaching package: 'dplyr'
#> The following objects are masked from 'package:stats':
#> 
#>     filter, lag
#> The following objects are masked from 'package:base':
#> 
#>     intersect, setdiff, setequal, union

raman_adj <- raman_hdpe %>%
  adj_intens()

head(raman_adj)
#>   wavenumber  intensity
#> 1    301.040 0.00000000
#> 2    304.632 0.03037975
#> 3    308.221 0.02784810
#> 4    311.810 0.02405063
#> 5    315.398 0.02531646
#> 6    318.983 0.02025316

Share metadata on known spectra

To share metadata about your spectrum, click the metadata input button. When sharing data, please provide as much metadata as you can. Metadata helps make shared data as useful as possible. Metadata inputs each have examples provided in the input. The examples disappear when the box is clicked and will not be saved if nothing is input by the user. Mandatory inputs are marked with a red asterisk. If these inputs are not filled, the data will be considered uninterpretable and will be discarded. Inputs left blank will be left blank in the metadata sheet and interpreted as “unknown” or “not applicable”. To share metadata, click the share data button at the bottom of the metadata inputs.

When the user clicks the Share Data button their current uploaded data and metadata is sent to an open-access online repository.

All inputs from the metadata (described below) are input to a metadata sheet. The metadata sheet is given the same unique name as the data, but it ends with “_form”. The exact same data is saved as would be downloaded using the download data button (described below). All high quality uploaded data with metadata will eventually be reviewed by spectroscopy experts and added to the internal library if it passes review. If multiple files are going to be uploaded with metadata, the cells will also stay filled with the last input after the share data button is clicked. Just upload the next dataset and change the metadata inputs that are different. If more than 50 files will be shared at once, you can contact the website administrator to get a bulk upload sheet for more rapid upload.

Type

share_spec(raman_hdpe,
           metadata = c(user_name = "Win Cowger",
                        contact_info = "wincowger@gmail.com",
                        spectrum_type = "Raman",
                        spectrum_identity = "HDPE")
           )

to share your spectral data from the R console.

Preprocess Spectra Plot

The preprocess spectra plot shows the uploaded spectra in comparison to the processed spectra that has been processed using the processing inputs on the page. It will automatically update with any new slider inputs. This allows the user to tune the inputs to optimize the signal to noise ratio. The goal with preprocessing is to make peak regions have high intensities and non-peak regions should have low intensities.

Preprocessing Tools

When the slider is green for the tool type, that means that that tool is being used to preprocess the spectrum. If the slider is clicked blank, the cog button to the right will disappear to indicate that the tool is no longer being used.

If the cog button is clicked, any functions associated with that tool will be displayed and can be manipulated to process the spectrum.

Smoothing Polynomial

The first step of the Open Specy preprocessing routing is spectral smoothing. The goal of this function is to increase the signal to noise ratio (S/N) without distorting the shape or relative size of the peaks. The value on the slider is the polynomial order of the Savitzky-Golay (SG) filter. The SG filter is fit to a moving window of 11 data points where the center point in the window is replaced with the polynomial estimate. The number of data points in the window is not user adjustable. Higher numbers lead to more wiggly fits and thus less smooth, lower numbers lead to more smooth fits, a 7th order polynomial will make the spectrum have almost no smoothing. If smoothing is set to 0 then no smoothing is conducted on the spectrum. When smoothing is done well, peak shapes and relative heights should not change. Typically a 3rd order polynomial (3 on the slider) works to increase the signal to noise without distortion, but if the spectrum is noisy, decrease polynomial order and if it is already smooth, increase the polynomial order to the maximum (7). Examples of smoothing below:

Sample raman_hdpe spectrum with different smoothing polynomials (p) from Cowger et al. (2020).

The different degrees of smoothing were achieved with the following R commands:

smooth_intens(raman_hdpe, p = 1)
smooth_intens(raman_hdpe, p = 4)

The intensity-adjusted sample spectrum raman_adj is smoothed accordingly:

raman_smooth <- raman_adj %>% 
  smooth_intens()

head(raman_smooth)
#>   wavenumber  intensity
#> 1    301.040 0.00000000
#> 2    304.632 0.01568318
#> 3    308.221 0.02461353
#> 4    311.810 0.02828915
#> 5    315.398 0.02820811
#> 6    318.983 0.02586852

Baseline Correction Polynomial

The second step of Open Specy’s preprocessing routine is baseline correction. The goal of baseline correction is to get all non-peak regions of the spectra to zero absorbance. The higher the polynomial order, the more wiggly the fit to the baseline. If the baseline is not very wiggly, a more wiggly fit could remove peaks which is not desired. The baseline correction algorithm used in Open Specy is called “iModPolyfit” (Zhao et al. 2007). This algorithm iteratively fits polynomial equations of the specified order to the whole spectrum. During the first fit iteration, peak regions will often be above the baseline fit. The data in the peak region is removed from the fit to make sure that the baseline is less likely to fit to the peaks. The iterative fitting terminates once the difference between the new and previous fit is small. An example of a good baseline fit below.

Sample raman_hdpe spectrum with different degrees of background subtraction (Cowger et al., 2020).

The smoothed sample spectrum raman_smooth is background-corrected as follows:

raman_bgc <- raman_smooth %>% 
  subtr_bg()

head(raman_bgc)
#>   wavenumber   intensity
#> 1    301.040 0.000000000
#> 2    304.632 0.000000000
#> 3    308.221 0.006298355
#> 4    311.810 0.008146146
#> 5    315.398 0.007025667
#> 6    318.983 0.004412447

Spectral Range

The final step of preprocessing is restricting the spectral range. Sometimes the instrument operates with high noise at the ends of the spectrum and sometimes the baseline fit can produce distortions at the ends of the spectrum, both can be removed using this routine. You should look into the signal to noise ratio of your specific instrument by wavelength to determine what wavelength ranges to use. Distortions due to baseline fit can be assessed from looking at the preprocess spectra plot. Additionally, you can restrict the range to examine a single peak or a subset of peaks of interests. This function allows users to isolate peaks of interest for matching, while removing noise and influence from less relevant spectral data.

Download Data

After you have the preprocessing parameters set, we recommend that you download the preprocessed data for your records. The download data button will append the uploaded data to three columns created by the preprocessing parameters. “Wavelength” and “Absorbance” are columns from the data uploaded by the user. “NormalizedIntensity” is the max-min normalized value (Equation 1) of the “Absorbance”. “Smoothed” is the Savitzky-Golay filter specified by the slider explained above. “BaselineRemoved” is the smoothed and baseline corrected value that is visible on the center plot.

Spectrum Type

The spectra type input on the “Match spectra” tab specifies the type of spectra (Raman or FTIR) that the user has uploaded and wants to match to. This input will tell the website whether to use the FTIR library or the Raman library to make the match.

Spectrum To Analyze

The spectra to analyze input specifies if the tool will match the Uploaded spectra (unaltered by the inputs on the Preprocess Spectra tab) or the Processed Spectra (manipulated by the inputs in the Preprocess Spectra Tab).

Region To Match

The region to match input specifies if the “Full Spectrum” will match the entire range of the spectra (including non peak regions) in the reference database. This is the most intuitive match. Or should the Peaks Only match just the peak regions in the reference database. This is an advanced feature proposed in Renner et al. (2017). This can be a less intuitive approach but in cases where there are few peaks and high baseline interference, it could be the best option. In cases where non-peak regions are important for the interpretation of the match, this is not the best approach.

Match Table

The selectable table shows the top material matches returned by the tool, their Pearson’s r value, and the organization they were provided by. When rows are selected their spectra are added to the match plot. The spectrum being matched and reference library are determined by the previously mentioned parameters. During the matching process, one final cleaning step happens using a simple minimum subtraction algorithm (Equation 2) which in many cases will allow unprocessed spectra to remove subtle baseline, but will not harm the spectra which has no baseline. Then, these aligned data are tested for correlation using the Pearson’s r. The Pearson’s r is used as a match quality indicator and the spectra from the top 1000 best matches are returned from the library. You can restrict the libraries which are displayed in the table by clicking the box that says All under the Organization column.

Similarly you can restrict the range of Pearson's r values or search for specific material types.

Equation 2: Minimum Subtraction

The same table can be returned using the Open Specy library commands in the R console.

# Fetch current spectral library from https://osf.io/x7dpz/
get_lib()
# Load library into global environment
spec_lib <- load_lib()
# Match spectrum with library and retrieve meta data
match_spec(raman_bgc, library = spec_lib, which = "raman")

Selection Metadata

Whatever match is selected from the match table may have additional metadata about it. That metadata will be displayed below the plot. Some of this metadata may assist you in interpreting the spectra. For example, if the spectra has metadata which says it is a liquid and you are analyzing a solid particle, that spectrum may not be the best match.

The R command for manual metadata selection using sample_name == 5381 as example is:

find_spec(sample_name == 5381, library = spec_lib, which = "raman")

Match Plot

This plot is dynamically updated by selecting matches from the match table. The red spectrum is the spectrum that you selected from the reference library and the white spectrum is the spectrum that you are trying to identify. Whenever a new dataset is uploaded, the plot and data table in this tab will be updated. These plots can be saved as a .png by clicking the camera button at the top of the plot.

How to interpret the reported matches

There are several important things to consider when interpreting a spectral match including the library source, the Pearson’s r, and other metrics.

File Conversion in SpectraGryph to Open Specy Accepted Format

1. Download Spectragryph from https://www.effemm2.de/spectragryph/down.html

2. Open Spectragryph and upload your file by dragging and dropping it into the console.

3. Click File, Save/export data, save data as, and save it as an spc file. ¸

4. Then upload that .spc file to Open Specy.

Conceptual diagram of data flow through Open Specy