moseq2_extract.extract package

Extract - Extract Module

Extraction helper utility for computing scalar feature values performing cleaning, cropping and rotating operations.

moseq2_extract.extract.extract.extract_chunk(chunk, use_tracking_model=False, spatial_filter_size=(3,), temporal_filter_size=None, tail_filter_iters=1, iters_min=0, strel_tail=array([[0, 0, 0, 0, 1, 0, 0, 0, 0], [0, 1, 1, 1, 1, 1, 1, 1, 0], [0, 1, 1, 1, 1, 1, 1, 1, 0], [1, 1, 1, 1, 1, 1, 1, 1, 1], [1, 1, 1, 1, 1, 1, 1, 1, 1], [1, 1, 1, 1, 1, 1, 1, 1, 1], [0, 1, 1, 1, 1, 1, 1, 1, 0], [0, 1, 1, 1, 1, 1, 1, 1, 0], [0, 0, 0, 0, 1, 0, 0, 0, 0]], dtype=uint8), strel_min=array([[1, 1, 1, 1, 1], [1, 1, 1, 1, 1], [1, 1, 1, 1, 1], [1, 1, 1, 1, 1], [1, 1, 1, 1, 1]], dtype=uint8), min_height=10, max_height=100, mask_threshold=-20, use_cc=False, bground=None, roi=None, rho_mean=0, rho_cov=0, tracking_ll_threshold=-100, tracking_model_segment=True, tracking_init_mean=None, tracking_init_cov=None, tracking_init_strel=array([[0, 0, 0, 0, 1, 0, 0, 0, 0], [0, 1, 1, 1, 1, 1, 1, 1, 0], [0, 1, 1, 1, 1, 1, 1, 1, 0], [1, 1, 1, 1, 1, 1, 1, 1, 1], [1, 1, 1, 1, 1, 1, 1, 1, 1], [1, 1, 1, 1, 1, 1, 1, 1, 1], [0, 1, 1, 1, 1, 1, 1, 1, 0], [0, 1, 1, 1, 1, 1, 1, 1, 0], [0, 0, 0, 0, 1, 0, 0, 0, 0]], dtype=uint8), flip_classifier=None, flip_classifier_smoothing=51, frame_dtype='uint8', progress_bar=True, crop_size=(80, 80), true_depth=673.1, centroid_hampel_span=5, centroid_hampel_sig=3, angle_hampel_span=5, angle_hampel_sig=3, model_smoothing_clips=(-300, -150), tracking_model_init='raw', compute_raw_scalars=False, **kwargs)

Extract mouse from the depth videos.

Args: chunk (np.ndarray): chunk to extract - (chunksize, height, width) use_tracking_model (bool): The EM tracker uses expectation-maximization to fit improve mouse detection. spatial_filter_size (tuple): spatial kernel size used in median filtering. temporal_filter_size (tuple): temporal kernel size used in median filtering. tail_filter_iters (int): number of filtering iterations on mouse tail iters_min (int): minimum tail filtering filter kernel size strel_tail (cv2::StructuringElement): filtering kernel size to filter out mouse tail. strel_min (cv2::StructuringElement): filtering kernel size to filter mouse body in cable recording cases. min_height (int): minimum (mm) distance of mouse to floor. max_height (int): maximum (mm) distance of mouse to floor. mask_threshold (int): Threshold on log-likelihood to include pixels for centroid and angle calculation use_cc (bool): boolean to use connected components in cv2 structuring elements bground (np.ndarray): 2D numpy array representing previously computed median background image of entire extracted recording. roi (np.ndarray): 2D numpy array representing previously computed roi (area of bucket floor) to search for mouse within. rho_mean (int): smoothing parameter for the mean rho_cov (int): smoothing parameter for the covariance tracking_ll_threshold (float): threshold for calling pixels a cable vs a mouse (usually between -16 to -12). tracking_model_segment (bool): boolean for whether to use only the largest blob for EM updates. tracking_init_mean (float): Initialized mean value for EM Tracking tracking_init_cov (float): Initialized covariance value for EM Tracking tracking_init_strel (cv2::StructuringElement - Ellipse): initial structuring element to use in EM tracking model. flip_classifier (str): path to pre-selected flip classifier. flip_classifier_smoothing (int): amount of smoothing to use for flip classifier. frame_dtype (str): Data type for processed frames save_path: (str): Path to save extracted results progress_bar (bool): Display progress bar crop_size (tuple): size of the cropped mouse image. true_depth (float): the computed detected true depth value for the middle of the arena centroid_hampel_span (int): Hampel filter span kernel size centroid_hampel_sig (int): Hampel filter standard deviation angle_hampel_span (int): Angle filter span kernel size angle_hampel_sig (int): Angle filter standard deviation model_smoothing_clips (tuple): Model smoothing clips tracking_model_init (str): Method for tracking model initialization compute_raw_scalars (bool): Compute scalars from unfiltered crop-rotated data.

Returns: results (dict): dict object containing the following keys: chunk (numpy.ndarray): bg subtracted and applied ROI version of original video chunk depth_frames(numpy.ndarray): cropped and oriented mouse video chunk mask_frames (numpy.ndarray): cropped and oriented mouse video chunk scalars (dict): computed scalars (str) mapped to 1d numpy arrays of length=nframes. flips(1d array): list of frame indices where the mouse orientation was flipped. parameters (dict): mean and covariance estimates for each frame (if em_tracking=True), otherwise None.

Extract - Proc Module

Video pre-processing utilities for detecting ROIs and extracting raw data.

moseq2_extract.extract.proc.apply_roi(frames, roi)

Apply ROI to data.

Args: frames (np.ndarray): input frames to apply ROI. roi (np.ndarray): selected ROI to extract from input images.

Returns: cropped_frames (np.ndarray): Frames cropped around ROI Bounding Box.

moseq2_extract.extract.proc.clean_frames(frames, prefilter_space=(3,), prefilter_time=None, strel_tail=array([[0, 0, 0, 1, 0, 0, 0], [0, 1, 1, 1, 1, 1, 0], [1, 1, 1, 1, 1, 1, 1], [1, 1, 1, 1, 1, 1, 1], [1, 1, 1, 1, 1, 1, 1], [0, 1, 1, 1, 1, 1, 0], [0, 0, 0, 1, 0, 0, 0]], dtype=uint8), iters_tail=None, frame_dtype='uint8', strel_min=array([[1, 1, 1, 1, 1], [1, 1, 1, 1, 1], [1, 1, 1, 1, 1], [1, 1, 1, 1, 1], [1, 1, 1, 1, 1]], dtype=uint8), iters_min=None, progress_bar=False)

Simple temporal and/or spatial filtering, median filter and morphological opening.

Args: frames (np.ndarray): Frames (frames x rows x columns) to filter. prefilter_space (tuple): kernel size for spatial filtering prefilter_time (tuple): kernel size for temporal filtering strel_tail (cv2.StructuringElement): Element for tail filtering. iters_tail (int): number of iterations to run opening frame_dtype (str): frame encodings strel_min (int): minimum kernel size iters_min (int): minimum number of filtering iterations progress_bar (bool): display progress bar

Returns: filtered_frames (numpy.ndarray): frames x rows x columns

moseq2_extract.extract.proc.compute_scalars(frames, track_features, min_height=10, max_height=100, true_depth=673.1)

Compute extracted scalars.

Args: frames (np.ndarray): frames x r x c, uncropped mouse track_features (dict): dictionary with tracking variables (centroid and orientation) min_height (float): minimum height of the mouse max_height (float): maximum height of the mouse true_depth (float): detected true depth

Returns: features (dict): dictionary of scalars

moseq2_extract.extract.proc.crop_and_rotate_frames(frames, features, crop_size=(80, 80), progress_bar=False)

Crop mouse from image and orients it such that the head is pointing right

Args: frames (3d np.ndarray): frames to crop and rotate features (dict): dict of extracted features, found in result_00.h5 files. crop_size (tuple): size of cropped image. progress_bar (bool): Display progress bar.

Returns: cropped_frames (3d np.ndarray): Crop and rotated frames.

moseq2_extract.extract.proc.feature_hampel_filter(features, centroid_hampel_span=None, centroid_hampel_sig=3, angle_hampel_span=None, angle_hampel_sig=3)

Filter computed extraction features using Hampel Filtering.

Args: features (dict): dictionary of video features centroid_hampel_span (int): Centroid Hampel Span Filtering Kernel Size centroid_hampel_sig (int): Centroid Hampel Signal Filtering Kernel Size angle_hampel_span (int): Angle Hampel Span Filtering Kernel Size angle_hampel_sig (int): Angle Hampel Span Filtering Kernel Size

Returns: features (dict): filtered version of input dict.

moseq2_extract.extract.proc.get_bbox(roi)

return an array with the x and y boundaries given ROI.

Args: roi (np.ndarray): ROI boolean mask to calculate bounding box.

Returns: bbox (np.ndarray): Bounding Box around ROI

moseq2_extract.extract.proc.get_bground_im_file(frames_file, frame_stride=500, med_scale=5, output_dir=None, **kwargs)

Load or compute background from file.

Args: frames_file (str): path to the depth video frame_stride (int): stride size between frames for median bground calculation med_scale (int): kernel size for median blur for background images. kwargs (dict): extra keyword arguments

Returns: bground (numpy.ndarray): background image

moseq2_extract.extract.proc.get_flips(frames, flip_file=None, smoothing=None)

Predict frames where mouse orientation is flipped to later correct.

Args: frames (numpy.ndarray): frames x rows x columns, cropped mouse flip_file (str): path to pre-trained scipy random forest classifier smoothing (int): kernel size for median filter smoothing of random forest probabilities

Returns: flips (numpy.array): array for flips

moseq2_extract.extract.proc.get_frame_features(frames, frame_threshold=10, mask=array([], dtype=float64), mask_threshold=-30, use_cc=False, progress_bar=False)

Use image moments to compute features of the largest object in the frame

Args: frames (3d np.ndarray): input frames frame_threshold (int): threshold in mm separating floor from mouse mask (3d np.ndarray): input frame mask for parts not to filter. mask_threshold (int): threshold to include regions into mask. use_cc (bool): Use connected components. progress_bar (bool): Display progress bar.

Returns: features (dict of lists): dictionary with simple image features mask (3d np.ndarray): input frame mask.

moseq2_extract.extract.proc.get_largest_cc(frames, progress_bar=False)

Returns largest connected component blob in image

Args: frames (numpy.ndarray): frames x rows x columns, uncropped mouse progress_bar (bool): display progress bar

Returns: foreground_obj (numpy.ndarray): frames x rows x columns, true where blob was found

moseq2_extract.extract.proc.get_roi(depth_image, strel_dilate=array([[1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1], [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1], [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1], [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1], [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1], [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1], [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1], [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1], [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1], [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1], [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1], [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1], [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1], [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1], [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1]], dtype=uint8), dilate_iterations=0, erode_iterations=0, strel_erode=None, noise_tolerance=30, bg_roi_weights=(1, 0.1, 1), overlap_roi=None, bg_roi_gradient_filter=False, bg_roi_gradient_kernel=7, bg_roi_gradient_threshold=3000, bg_roi_fill_holes=True, get_all_data=False, **kwargs)

Compute an ROI using RANSAC plane fitting and simple blob features.

Args: depth_image (np.ndarray): Singular depth image frame. strel_dilate (cv2.StructuringElement): dilation shape to use. dilate_iterations (int): number of dilation iterations. erode_iterations (int): number of erosion iterations. strel_erode (int): image erosion kernel size. noise_tolerance (int): threshold to use for noise filtering. bg_roi_weights (tuple): weights describing threshold to accept ROI. overlap_roi (np.ndarray): list of ROI boolean arrays to possibly combine. bg_roi_gradient_filter (bool): Boolean for whether to use a gradient filter. bg_roi_gradient_kernel (tuple): Kernel size of length 2, e.g. (1, 1.5) bg_roi_gradient_threshold (int): Threshold for noise gradient filtering bg_roi_fill_holes (bool): Boolean to fill any missing regions within the ROI. get_all_data (bool): If True, returns all ROI data, else, only return ROIs and computed Planes kwargs (dict) Dictionary containing bg_roi_depth_range parameter for plane_ransac()

Returns: rois (list): list of detected roi images. roi_plane (np.ndarray): computed ROI Plane using RANSAC. bboxes (list): list of computed bounding boxes for each respective ROI. label_im (list): list of scikit-image image properties ranks (list): list of ROI ranks. shape_index (list): list of rank means.

moseq2_extract.extract.proc.im_moment_features(IM)

Use the method of moments and centralized moments to get image properties.

Args: IM (numpy.ndarray): depth image

Returns: features (dict): returns a dictionary with orientation, centroid, and ellipse axis length

moseq2_extract.extract.proc.model_smoother(features, ll=None, clips=(-300, -125))

Apply spatial feature filtering.

Args: features (dict): dictionary of extraction scalar features ll (numpy.array): array of loglikelihoods of pixels in frame clips (tuple): tuple to ensure video is indexed properly

Returns: features (dict): smoothed version of input features

moseq2_extract.extract.proc.threshold_chunk(chunk, min_height, max_height)

Threshold out depth values that are less than min_height and larger than max_height.

Args: chunk (np.ndarray): Chunk of frames to threshold (nframes, width, height) min_height (int): Minimum depth values to include after thresholding. max_height (int): Maximum depth values to include after thresholding. dilate_iterations (int): Number of iterations the ROI was dilated.

Returns: chunk (3D np.ndarray): Updated frame chunk.

Extract - ROI Module

ROI detection pre-processing utilities for fitting a plane to an input depth image.

moseq2_extract.extract.roi.plane_fit3(points)

Fit a plane to 3 points (min number of points for fitting a plane)

Args: points (numpy.ndarray): each row is a group of points, columns correspond to x,y,z.

Returns: plane (numpy.array): linear plane fit–>a*x+b*y+c*z+d

moseq2_extract.extract.roi.plane_ransac(depth_image, bg_roi_depth_range=(650, 750), iters=1000, noise_tolerance=30, in_ratio=0.1, progress_bar=False, mask=None, **kwargs)

Fit a plane using a naive RANSAC implementation

Args: depth_image (numpy.ndarray): background image to fit plane to bg_roi_depth_range (tuple): min/max depth (mm) to consider pixels for plane iters (int): number of RANSAC iterations noise_tolerance (float): distance from plane to consider a point an inlier in_ratio (float): fraction of points required to consider a plane fit good progress_bar (bool): display progress bar mask (numpy.ndarray): boolean mask to find region to use kwargs (dict): dictionary containing extra keyword arguments from moseq2_extract.proc.get_roi()

Returns: best_plane (numpy.array): plane fit to data dist (numpy.array): distance of the calculated coordinates and “best plane”

Extract - Track Module

Expectation-Maximization mouse tracking utilities.

moseq2_extract.extract.track.em_get_ll(frames, mean, cov, progress_bar=False)

Return mouse tracking log-likelihoods for each frame given tracking parameters.

Args: frames (numpy.ndarray): depth frames mean (numpy.array): frames x d, mean estimates cov (numpy.array): frames x d x d, covariance estimates progress_bar (bool): use a progress bar

Returns: ll (numpy.ndarray): frames x rows x columns, log likelihood of all pixels in each frame

moseq2_extract.extract.track.em_init(depth_frame, depth_floor, depth_ceiling, init_strel=array([[0, 0, 0, 0, 1, 0, 0, 0, 0], [0, 1, 1, 1, 1, 1, 1, 1, 0], [0, 1, 1, 1, 1, 1, 1, 1, 0], [1, 1, 1, 1, 1, 1, 1, 1, 1], [1, 1, 1, 1, 1, 1, 1, 1, 1], [1, 1, 1, 1, 1, 1, 1, 1, 1], [0, 1, 1, 1, 1, 1, 1, 1, 0], [0, 1, 1, 1, 1, 1, 1, 1, 0], [0, 0, 0, 0, 1, 0, 0, 0, 0]], dtype=uint8), strel_iters=1)

Estimate depth frame contours using OpenCV, and select the largest chosen contour to initialize a mask for EM tracking.

Args: depth_frame (numpy.ndarray): depth frame to initialize mask with. depth_floor (float): distance from camera to bucket floor. depth_ceiling (float): max depth value. init_strel (cv2.structuringElement): structuring Element to compute mask. strel_iters (int): number of morphological iterations.

Returns: mouse_mask (numpy.ndarray): mask of depth frame.

moseq2_extract.extract.track.em_iter(data, mean, cov, lamd=0.1, epsilon=0.1, max_iter=25)

Use EM tracker to iteratively update the mean and covariance variables using Expectation Maximization up to the max inputted number of iterations.

Args: data (numpy.ndarray): frame, x, y, z coordinates to use mean (numpy.array): dx1, current mean estimate cov (numpy.array): current covariance estimate lambd (float): constant to add to diagonal of covariance matrix epsilon (float): tolerance on change in likelihood to terminate iteration max_iter (int): maximum number of EM iterations

Returns: mean (numpy.array): updated mean cov (numpy.array): updated covariance

moseq2_extract.extract.track.em_tracking(frames, raw_frames, segment=True, ll_threshold=-30, rho_mean=0, rho_cov=0, depth_floor=10, depth_ceiling=100, progress_bar=True, init_mean=None, init_cov=None, init_frames=10, init_method='raw', init_strel=array([[0, 0, 0, 0, 1, 0, 0, 0, 0], [0, 1, 1, 1, 1, 1, 1, 1, 0], [0, 1, 1, 1, 1, 1, 1, 1, 0], [1, 1, 1, 1, 1, 1, 1, 1, 1], [1, 1, 1, 1, 1, 1, 1, 1, 1], [1, 1, 1, 1, 1, 1, 1, 1, 1], [0, 1, 1, 1, 1, 1, 1, 1, 0], [0, 1, 1, 1, 1, 1, 1, 1, 0], [0, 0, 0, 0, 1, 0, 0, 0, 0]], dtype=uint8))

Naive tracker, use EM update rules to follow a 3D Gaussian around the room.

Args: frames (numpy.ndarray): filtered frames. raw_frames (numpy.ndarray): chunk to track mouse in. segment (bool): use only the largest blob for em updates ll_threshold (float): threshold on log likelihood for segmentation rho_mean (float): smoothing parameter for the mean rho_cov (float): smoothing parameter for the covariance depth_floor (float): height in mm for separating mouse from floor depth_ceiling (float): max height in mm for mouse from floor. progress_bar (bool): display progress bar. init_mean (np.ndarray): array of inital frame pixel means. init_cov (np.ndarray): array of inital frame pixel covariances. init_frames (int): number of frames to include in the init calulation init_method (str): mode in which to process inputs init_strel (cv2.structuringElement): structuring Element to compute mask.

Returns: model_parameters (dict): mean and covariance estimates for each frame