moseq2_extract.helpers package

Helpers - Data Module

Contains helper functions for handling/storing data during extraction.

moseq2_extract.helpers.data.build_index_dict(files_to_use)

Create a dictionary for the index file from a list of files and respective metadatas.

Args: files_to_use (list): list of paths to extracted h5 files.

Returns: output_dict (dict): index-file dictionary containing all aggregated extractions.

moseq2_extract.helpers.data.build_manifest(loaded, format, snake_case=True)

Build a manifest file used to contain extraction result metadata from h5 and yaml files.

Args: loaded (list of dicts): list of dicts containing loaded h5 data. format (str): filename format indicating the new name for the metadata files in the aggregate_results dir. snake_case (bool): whether to save the files using snake_case

Returns: manifest (dict): dictionary of extraction metadata.

moseq2_extract.helpers.data.check_completion_status(status_filename)

Read a results_00.yaml (status file) and checks whether the session has been fully extracted.

Args: status_filename (str): path to results_00.yaml

Returns: complete (bool): If True, data has been extracted to completion.

moseq2_extract.helpers.data.copy_manifest_results(manifest, output_dir)

Copy all consolidated manifest results to their respective output files.

Args: manifest (dict): manifest dictionary containing all extraction h5 metadata to save output_dir (str): path to directory where extraction results will be aggregated.

moseq2_extract.helpers.data.create_extract_h5(h5_file, acquisition_metadata, config_data, status_dict, scalars_attrs, nframes, roi, bground_im, first_frame, first_frame_idx, last_frame_idx, **kwargs)

write acquisition metadata, extraction metadata, computed scalars, timestamps, and original frames/frames_mask to extracted h5.

Args: h5_file (h5py.File object): opened h5 file object to write to. acquisition_metadata (dict): Dictionary containing extracted session acquisition metadata. config_data (dict): dictionary object containing all required extraction parameters. (auto generated) status_dict (dict): dictionary that helps indicate if the session has been extracted fully. scalars_attrs (dict): dict of computed scalar attributes and descriptions to save. nframes (int): number of frames being recorded roi (np.ndarray): Computed 2D ROI Image. bground_im (np.ndarray): Computed 2D Background Image. first_frame (np.ndarray): Computed 2D First Frame Image. timestamps (numpy.array): Array of session timestamps. kwargs (dict): additional keyword arguments.

moseq2_extract.helpers.data.handle_extract_metadata(input_file, dirname)

Extract metadata and timestamp in the extraction.

Args: input_file (str): path to input file to extract dirname (str): path to directory where extraction files reside.

Returns: acquisition_metadata (dict): key-value pairs of JSON contents timestamps (1D array): list of loaded timestamps tar (bool): indicator for whether the file is compressed.

moseq2_extract.helpers.data.load_extraction_meta_from_h5s(to_load, snake_case=True)

Load extraction metadata from h5 files.

Args: to_load (list): list of paths to h5 files. snake_case (bool): whether to save the files using snake_case

Returns: loaded (list): list of loaded h5 dicts.

Helpers - Extract Module

Extraction-helper utility functions.

moseq2_extract.helpers.extract.make_output_movie(results, config_data, offset=0)

Create an array for output movie with filtered video and cropped mouse on the top left

Args: results (dict): dict of extracted depth frames, and original raw chunk to create an output movie. config_data (dict): dict of extraction parameters containing the crop sizes used in the extraction. offset (int): current offset being used, automatically set if chunk_overlap > 0

Returns: output_movie (numpy.ndarray): output movie to write to mp4 file.

moseq2_extract.helpers.extract.process_extract_batches(input_file, config_data, bground_im, roi, frame_batches, str_els, output_mov_path, scalars=None, h5_file=None, video_pipe=None, **kwargs)

Compute extracted frames and save them to h5 files and avi files.

Args: input_file (str): path to depth file config_data (dict): dictionary containing extraction parameters (autogenerated) bground_im (numpy.ndarray): background image roi (numpy.ndarray): roi image frame_batches (list): list of batches of frames to serially process. str_els (dict): dictionary containing OpenCV StructuringElements output_mov_path (str): path and filename of the output movie generated by the extraction scalars (list): list of keys to scalar attribute values h5file (h5py.File): opened h5 file to write extracted batches to video_pipe (subprocess.PIPE): open pipe to location where preview extraction is being written. kwargs (dict): Extra keyword arguments.

Returns:

moseq2_extract.helpers.extract.run_local_extract(to_extract, config_file, skip_extracted=False)

Run the extract command on given list of sessions to extract on a local platform.

Args: to_extract (list): list of paths to files to extract config_file (str): path to configuration file containing pre-configured extract and ROI skip_extracted (bool): Whether to skip already extracted session.

moseq2_extract.helpers.extract.run_slurm_extract(input_dir, to_extract, config_data, skip_extracted=False)

moseq2_extract.helpers.extract.set_tracking_model_parameters(results, min_height, tracking_model_ll_clip, chunk_overlap, **kwargs)

Threshold and clip the masked frame data if use_tracking_model = True and update results.

Args: results (dict): dict of extracted depth frames and mask frames to threshold to update; output of extract_chunk(). min_height (int): distance from floor to threshold out of extracted image. tracking_model_ll_clip (numpy.ndarray): clipped frame regions based on EM loglikelihoods computed in extract_chunk(). chunk_overlap (int): number of frames each extracted chunk is overlapping with the next.

Returns: results (dict): updated results dict with thresholded and clipped mask_frames. tracking_init_mean (float): mean value for EM Tracking tracking_init_cov (float): covariance value for EM Tracking

moseq2_extract.helpers.extract.write_extracted_chunk_to_h5(h5_file, results, config_data, scalars, frame_range, offset)

Write extracted frames, frame masks, and scalars to an open h5 file.

Args: h5_file (H5py.File): open results_00 h5 file to save data in. results (dict): extraction results dict. config_data (dict): dictionary containing extraction parameters (autogenerated) scalars (list): list of keys to scalar attribute values frame_range (range object): current chunk frame range offset (int): frame offset

Returns:

Helpers - Wrappers Module

Wrapper functions for data processing in extraction.

moseq2_extract.helpers.wrappers.aggregate_extract_results_wrapper(input_dir, format, output_dir, mouse_threshold=0.0)

Aggregate results to one folder and generate index file (moseq2-index.yaml).

Args: input_dir (str): path to base directory containing all session folders format (str): string format for metadata to use as the new aggregated filename output_dir (str): name of the directory to create and store all results in mouse_threshold (float): threshold value of mean frame depth to include session frames

Returns: indexpath (str): path to generated index file including all aggregated session information.

moseq2_extract.helpers.wrappers.convert_raw_to_avi_wrapper(input_file, output_file, chunk_size, fps, delete, threads, mapping)

compress a raw depth file into an avi file (with depth values) that is 8x smaller.

Args: input_file (str): Path to depth file to convert output_file (str): Path to output avi file chunk_size (int): Size of frame chunks to iteratively process fps (int): frame rate. delete (bool): Delete the original depth file if True. threads (int): Number of threads used to encode video. mapping (str or int): Indicate which video stream to from the inputted file

Returns:

moseq2_extract.helpers.wrappers.copy_h5_metadata_to_yaml_wrapper(input_dir, h5_metadata_path)

Copy user specified metadata from h5path to a yaml file.

Args: input_dir (str): path to directory containing h5 files h5_metadata_path (str): path within h5 to desired metadata to copy to yaml.

moseq2_extract.helpers.wrappers.copy_slice_wrapper(input_file, output_file, copy_slice, chunk_size, fps, delete, threads, mapping)

Copy a segment of an input depth recording into a new video file.

Args: input_file (str): Path to depth file to read segment from output_file (str): Path to outputted video file with copied slice. copy_slice (2-tuple): Frame range to copy from input file. chunk_size (int): Size of frame chunks to iteratively process fps (int): Frames per second. delete (bool): Delete the original depth file if True. threads (int): Number of threads used to encode video. mapping (str or int): Indicate which video stream to from the inputted file

Returns:

moseq2_extract.helpers.wrappers.extract_wrapper(input_file, output_dir, config_data, num_frames=None, skip=False)

Extract depth videos.

Args: input_file (str): path to depth file output_dir (str): path to directory to save results in. config_data (dict): dictionary containing extraction parameters. num_frames (int): number of frames to extract. skip (bool): indicates whether to skip file if already extracted extract (function): extraction function state

Returns: output_dir (str): path to directory containing extraction

moseq2_extract.helpers.wrappers.generate_index_from_agg_res_wrapper(input_dir)

Generate index file from aggregated results folder.

Args: input_dir (str): path to aggregated results folder

moseq2_extract.helpers.wrappers.get_roi_wrapper(input_file, config_data, output_dir=None)

Compute ROI given depth file.

Args: input_file (str): path to depth file. config_data (dict): dictionary of ROI extraction parameters. output_dir (str): path to desired directory to save results in.

Returns: roi (numpy.ndarray): ROI image to plot in GUI bground_im (numpy.ndarray): Background image to plot in GUI first_frame (numpy.ndarray): First frame image to plot in GUI