Data Module

FlyEye Analysis is built upon quantitative expression measurements read from .silhouette files using the flyeye.data.silhouette submodule. Each labeled cell measurement becomes a record in a Pandas dataframe.

Cell measurement data are subject to three levels of organization:

1. Cells objects contain one or more expression level measurements

2. Disc objects contain all expression level measurements from a single .silhouette file

3. Experiment objects contain multiple Disc instances collected under similar conditions

The images contained in .silhouette files are managed separarely by the flyeye.data.image submodule.

Silhouette

Interface for managing the .silhouette filetype.

class flyeye.data.silhouette.Silhouette(path)

Interface to a FlyEye Silhouette file.

Attributes:

path (str) - path to Silhouette file

feed (dict) - feed file containing layer IDs

feud (dict) - feud file containing cell type labels

Properties:

is_flipped_about_yz (bool) - if True, invert about YZ plane

is_flipped_about_xy (bool) - if True, invert about XY plane

flip_about_xy()

Flip disc vertically.

flip_about_yz()

Flip disc horizontally.

property is_flipped_about_xy

Boolean flag for flipping disc vertically.

property is_flipped_about_yz

Boolean flag for flipping disc horizontally.

load_orientation()

Ensure that feed file includes disc orientation. If it does not, as in older silhouette filetypes, the default orientation is preserved.

read_json(filename)

Read contents of specified JSON file.

Args:

filename (str) - filename

Returns:

out (dict) - file contents

write_feed()

Writes contents of feed dictionary to silhouette file.

write_feud()

Writes contents of feud dictionary to silhouette file.

write_json(filename, data)

Write contents of <data> to specified JSON file.

Args:

filename (str) - filename

data (dict) - data contents

class flyeye.data.silhouette.SilhouetteData(path, recompile=False)

Interface to data within a FlyEye Silhouette file.

Upon instantiation, individual cell measurements are aggregated into a data.cells.Cells compatible DataFrame.

Measurement data must be read on a layer-by-layer basis the first time a Silhouette object is instantiated. Following this initial reading, the aggregated measurement data are serialized and stored within the silhouette file. These serialized measurements may then be accessed directly during future use. The recompile flag indicates whether the serialized measurements should be ignored upon instantiation.

Attributes:

data (pd.DataFrame) - cell measurement data

Inherited attributes:

path (str) - path to Silhouette file

feed (dict) - feed file containing layer IDs

feud (dict) - feud file containing cell type labels

is_flipped_about_yz (bool) - if True, invert about YZ plane

is_flipped_about_xy (bool) - if True, invert about XY plane

compile_measurements()

Compile measurements from all layers (slow access).

property labels

pd.Series of labels keyed by (layer_id, segment_id).

load(recompile=False)

Read all contour and orientation data from silhouette file.

Args:

recompile (bool) - if True, recompile measurements from all layers

load_measurements()

Load serialized measurements (fast access).

static parse_contour(contour)

Convert contour to list format.

Args:

contour (dict) - contour from silhouette file

Returns:

ctr_list (list) - values in data.cells.Cells compatible list format

read_contours(all_labels={}, include_unlabeled=False)

Read contours from silhouette file.

Args:

all_labels (dict) - {layer_id: {contour_id: label}} for each layer

include_unlabeled (bool) - if True, include unlabeled segments

Returns:

data (pd.DataFrame) - data.cells.Cells compatible dataframe of contours

read_labels()

Load segment labels from silhouette file.

Returns:

labels (dict) - {layer_id: {contour_id: label}} entries for each layer

save_measurements()

Save serialized measurements for fast access.

Cells

Cells are a collection of one or more labeled expression measurements. Cells may be a subsample of a single Disc, or a combination of several discs.

class flyeye.data.cells.CellProperties

Properties of Cells object.

property cell_types

Unique cell types.

property channels

List of unique fluorescence channels.

property normalized_channels

List of normalized channel names.

property num_channels

Number of unique fluorescence channels.

class flyeye.data.cells.Cells(data=None, normalization=None)

Object represents a population of cells. Each cell is completely described by a single record in a DataFrame of cell measurements. These measurements include cell positions, expression levels, and cell type annotations. Object may contain cells of one or more cell types.

Attributes:

data (pd.DataFrame) - cell measurement data

normalization (str or int) - channel used to normalize intensities

static annotate_fraction(ax, ratio, p=2.5)

Add fraction of cells above midline.

Args:

ax (mpl.axes.AxesSubplot)

ratio (array like) - vector of ratios

p (float) - text position relative to center line

apply_lag(lag)

Shift cells in time.

Args:

lag (float) - shift (NOTE: x-positions are unaffected)

static get_binned_mean(x, values, bins=None, bin_width=1)

Bin cells and compute mean for each bin.

Args:

x (pd.Series) - coordinate on which to bin values

values (pd.Series) - values to be aggregated

bins (np array) - edges for specified bins

bin_width (float) - width of bins used if no bins specified

Returns:

bin_centers (np.ndarray) - bin centers

means (np array) - mean value within each bin

get_nuclear_diameter()

Returns median nuclear diameter. Diameters are approximated as that of a circle with equivalent area to each nuclear contour.

Returns:

nuclear_diameter (float) - median diameter

plot_dynamics(channel, ax=None, scatter=False, average=True, interval=False, marker_kw={}, line_kw={}, interval_kw={}, ma_kw={})

Plot expression dynamics for specified channel.

Args:

channel (str) - expression channel

ax (mpl.axes.AxesSubplot) - if None, create axes

scatter (bool) - if True, add markers for each measurement

average (bool) - if True, add moving average

interval - if True, add confidence interval for moving average

marker_kw (dict) - keyword arguments for marker formatting

line_kw (dict) - keyword arguments for line formatting

interval_kw (dict) - keyword arguments for interval formatting

ma_kw (dict) - keyword arguments for interval construction

Returns:

ax (mpl.axes.AxesSubplot)

plot_resampled_dynamics(channel, ax=None, average=True, interval=False, marker_kw={}, line_kw={}, interval_kw={}, resampling_kw={})

Plot expression dynamics for specified channel, resampling from discrete subpopulations of cells.

Args:

channel (str) - expression channel

ax (mpl.axes.AxesSubplot) - if None, create axes

average (bool) - if True, add moving average

interval - if True, add confidence interval for moving average

line_kw (dict) - keyword arguments for line formatting

interval_kw (dict) - keyword arguments for interval formatting

resampling_kw (dict) - keyword arguments for disc resampler

Returns:

ax (mpl.axes.AxesSubplot)

plot_spectrogram(channel, periods=None, ymax=None, ax=None, **kwargs)

Plot Lomb Scargle periodogram.

Args:

channel (str or int) - expression channel

periods (array like) - spectral frequencies to be tested

ymax (float) - max spectral power

ax (mpl.axes.AxesSubplot)

kwargs: spectrogram visualization keywords

Returns:

ax (mpl.axes.AxesSubplot)

scatterplot(x, y, color='grey', s=5, alpha=0.5, fraction=False, ax=None)

Create XY scatterplot of two fluorescence channels.

Args:

x, y (str or int) - channels used for x and y axes

color (str) - marker color

s (float) - marker size

alpha (float) - transparency of markers

fraction (bool) - if True, annotate fraction above midline

ax (mpl.axes.AxesSubplot) - if None, create figure

Returns:

ax (mpl.axes.AxesSubplot)

select_by_position(xmin=- inf, xmax=inf, ymin=- inf, ymax=inf, zmin=- inf, zmax=inf, tmin=- inf, tmax=inf)

Select subset of cells within specified spatial bounds.

Args:

xmin, xmax (float) - x-coordinate bounds

ymin, ymax (float) - y-coordinate bounds

zmin, zmax (float) - z-coordinate (layer number) bounds

tmin, tmax (float) - time interval bounds

Returns:

cells (data.cells.Cells) - copied subset of cells

select_cell_type(cell_types)

Select subset of cells corresponding to a specified label.

Args:

cell_types (str or list) - type of cells to be selected (e.g. pre, r8)

Returns:

cells (data.cells.Cells)

sort(by='centroid_x')

Sort cell measurements in place.

Args:

by (str) - key on which measurements are sorted

Discs

Discs are the set of cells that comprise an individual eye disc. This submodule contains all of the methods required to convert an individual disc’s cell measurements into developmental timepoints. Measurements are imported from .silhouette files at this level.

class flyeye.data.discs.Disc(silhouette, normalization=None, furrow_velocity=2.0, offset=None, bit_depth=None)

Object representing all cells in a single eye disc.

Attributes:

silhouette (flyeye.SilhouetteData) - data from silhouette file

furrow_velocity (float) - furrow inverse-velocity (hours per column)

offset (float) - time by which disc is shifted from first R8

bit_depth (int) - fluorescence intensity bit depth, log2 scaled

triangulation (processing.triangulation.Triangulation)

Inherited Attributes:

normalization (str or int) - channel used to normalize intensities

data (pd.DataFrame) - cell measurement data

Properties:

path (str) - unique silhouette filepath

is_flipped_about_yz (bool) - if True, disc is inverted about YZ plane

is_flipped_about_xy (bool) - if True, disc is inverted about XY plane

apply_lag(offset=None)

Shift disc in time.

Args:

offset (str or float) - position of new origin, if ‘first_r8’ shift such that first R8 occurs at time zero

apply_time_scaling()

Apply distance-to-time scaling to generate time vector.

detrend(channels, order=1)

Add detrended fluctuations for each fluorescence channel within each cell type to disc dataframe.

Args:

channels (iterable) - channels to be detrended

order (int) - polyorder for local trendataitting

flip_about_xy(save=False)

Flip disc bottom to top.

Args:

save (bool) - if True, save new orientation to silhouette file

flip_about_yz(save=False)

Flip disc left to right.

Args:

save (bool) - if True, save new orientation to silhouette file

static from_silhouette(path, normalization=None, furrow_velocity=2.0, recompile=False, **kwargs)

Instantiate disc from silhouette file.

Args:

path (str) - silhouette filepath

normalization (str or int) - normalization channel

furrow_velocity (float) - furrow inverse-velocity (hours per column)

recompile (bool) - if True, recompile measurements from all layers

kwargs: keyword arguments for disc instantiation

Returns:

disc (data.discs.Disc)

get_multipotent_layers(q=0.9)

Determine which layers span multipotent cells. Bounds correspond to lower and upper quantiles taken from a normal fit to all progenitor layers.

Args:

q (float) - width parameter (interquantile range), 0 to 1

Returns:

bounds (np.ndarray, length 2) - lower and upper bounds

load_imagestack()

Load image stack from silhouette file.

Returns:

stack (data.image.ImageStack)

normalize_by_reference(reference)

Normalize expression levels by the reference channel.

Args:

reference (str or int) - channel used for normalization

normalize_expression(max_value)

Convert each channel’s fluorescence to a 0-1 scale.

Args:

max_value (float) - maximum fluorescence intensity

set_ratio(num, den)

Add fluorescence ratio to dataframe, defined by <num>/<den> channels.

standardize_labels()

Convert all alternate precursor labels to ‘pre’

class flyeye.data.discs.DiscProperties

Properties for Disc class.

property is_flipped_about_xy

If True, disc is inverted about XY plane.

property is_flipped_about_yz

If True, disc is inverted about YZ plane.

property path

Path to silhouette file.

Experiments

An Experiment is a set of one or more Disc objects collected under the same experimental conditions. Measurements are often aggregated between discs and compared at this level.

class flyeye.data.experiments.Experiment(dirpath, normalization, auto_alignment=True, align_by='ch1_normalized', **kwargs)

Object representing multiple eye discs obtained under a single set of conditions.

Attributes:

discs (dict) - {disc ID: data.discs.Disc} pairs

num_discs (int) - number of discs within experiment

align_discs(channel)

Align all discs within experiment.

Args:

channel (str) - expression channel by which discs are aligned

align_to_first_r8(disc_id=0)

Shift all discs s.t. t=0 is the first R8 in the reference disc.

Args:

disc_id (int) - index of disc used as reference

apply_lag(lag=0)

Apply time shift to all discs in experiment.

Args:

lag (float) - time shift applied to each disc

get_cells(cell_type='pre', **selection_kw)

Return Cells object for all specified cells.

Args:

cell_type (str or list) - type of cells to select

selection_kw: keyword arguments for cell position selection

Returns:

cells (data.cells.Cells)

get_early_neuron_data(N=10, lower_slip=0, upper_slip=1)

Compile Dataframe of early R cells and concurrent progenitors.

Args:

N (int) - number of reference cells defining time window

lower_slip (float) - extension before first reference cell, hours

upper_slip (int) - reference cells skipped (excludes outliers)

Returns:

data (DataFrame) - measurement data for early R cells and concurrent progenitors

get_pairwise_alignment(window_size=10, **kw)

Compute pairwise quality of alignment between each disc.

Args:

window_size (int) - number of cells for smoothing

kw: keyword arguments for DiscAlignment

Returns:

scores (np.ndarray) - mean quality of alignment for each disc

static load(dirpath, normalization, **kwargs)

Load discs from silhouette files.

Args:

dirpath (str) - path to directory containing silhouette files

normalization (str or int) - normalization channel

kwargs: keyword arguments for disc instantiation

Returns:

discs (dict) - {disc_id: data.discs.Disc} pairs

property num_discs

Number of discs in experiment.

property num_progenitors

Number of progenitor measurements in experiment.

select_by_concurrency(reference_types, N=10, lower_slip=0, upper_slip=0)

Select cells concurrent with first N identified cells of reference cell type.

Args:

reference_types (array like) - reference cell type(s)

N (int) - number of reference cells defining time window

lower_slip (float) - extension before first reference cell, hours

upper_slip (int) - reference cells skipped (excludes outliers)

Returns:

data (DataFrame) - cells concurrent with reference cell type

set_ratio(num, den)

Add fluorescence ratio to each disc’s dataframe, defined by <num>/<den> channels.

Image

A submodule for managing, processing, and visualizing images contained in a .silhouette file.

class flyeye.data.image.Image(im)

Object contains a floating point three channel image.

Attributes: im (np.ndarray[float32]) - RGB pixel values, (X,Y,3)

build_colorfilter(scheme='rgb', channels='rgb', reference='r')

Build filter for converting between RGB and MG colorschemes.

Args:

scheme (str) - colorscheme, ‘rgb’ or ‘mg’

channels (str) - included channels, one or more of ‘rgbmg’

reference (str) - reference channel (used for rgb to mg conversion)

Returns:

colorfilter (np.ndarray[np.float32]) - multiplicative colorfilter

static from_tiff(path, **kwargs)

Read image file using matplotlib.image.imread.

All formats except PNG require pillow or PIL. PNG images are converted from 8-bit to floating point format by default, while other image formats must be manually converted.

Args:

path (str) - image path

kwargs: keyword arguments for Image instantiation

Returns:

im (data.image.Image)

static load_image(path)

Load image from <path>.

render(ax=None, size=(2, 2), scheme='rgb', channels='rgb', reference='r')

Render image.

Args:

ax (matplotlib.axes.AxesSubplot) - if None, create figure

size (tuple) - image size, used if no axis provided

scheme (str) - colorscheme, ‘rgb’ or ‘mg’

channels (str) - included channels, one or more of ‘rgbmg’

reference (str) - reference channel (used for rgb to mg conversion)

Returns:

ax (matplotlib.axes.AxesSubplot)

class flyeye.data.image.ImageStack(im)

Object representing a 3D stack of RGB image layers. Stack is compiled from sequentially numbered images of each layer within a silhouette file.

Attributes:

im (np.ndarray[float32]) - RGB pixel values. Array is shaped (N,X,Y,3) where N is the number of layers and (X,Y) are the dimensions of a single cross section.

static from_silhouette(path, fmt='png')

Load imagestack from sequentially numbered images of each layer.

Args:

path (str) - path to silhouette file

fmt (str) - image format for all layers

Returns:

stack (data.image.ImageStack)

get_layer(layer_id)

Return RGB image of individual layer.

Args:

layer_id (int) - layer number

Returns:

image (data.image.Image)

project_max(min_layer=0, max_layer=- 1)

Return maximum intensity projection across specified layers.

Args:

min_layer (int) - lower layer bound

max_layer (int) - upper layer bound

Returns:

image (data.image.Image)

project_mean(min_layer=0, max_layer=- 1)

Return mean intensity projection across specified layers.

Args:

min_layer (int) - lower layer bound

max_layer (int) - upper layer bound

Returns:

image (data.image.Image)

class flyeye.data.image.ScalarField(im)

Object representing a floating-point 2D scalar field that can take on positive or negative values.

Attributes:

im (np.ndarray[float32]) - scalar values, (X,Y)

fig (matplotlib.figure.Figure) - image rendering

property abs_max

Maximum absolute pixel value.

apply_colormap(cmap, vmin=0, vmax=2)

Render image to RGB format using provided colormap.

Args:

cmap (matplotlib.colors.ColorMap)

vmin, vmax (int) - color map scale

Returns:

im (np.ndarray[float]) - RGB image

apply_filter(func)

Apply filter to image.

Args:

func (function) - function operating on image array

fliplr()

Flip image about Y axis.

static from_8bit(im)

Instantiate from 8-bit image.

get_crop_indices(**kwargs)

Compute image index boundaries from fractional bounds.

property max

Maximum pixel value.

property min

Minimum pixel value.

render(size=(2, 2), cmap=None, vmin=0, vmax=1, ax=None)

Render image.

Args:

size (tuple) - figure size

cmap (matplotlib.cm.ColorMap) - colormap, defaults to Greys

vmin, vmax (float) - colorscale bounds

ax (matplotlib.axes.AxesSubplot) - if None, create figure

rotate(angle)

Apply rotation to image.

save(name, directory='./', dpi=300, fmt='png', transparent=True)

Save rendered image.

Args:

name (str) - file name

directory (str) - target directory

dpi (int) - resolution

fmt (str) - image format

transparent (bool) - if True, remove background

smooth(**kwargs)

Smooth image.

kwargs: keyword arguments for smoothing operation

static uint8_to_float32(im_uint8)

Convert 8-bit array to floating point values on a 0-1 interval.

Args:

im_uint8 (np.ndarray[uint8])

Returns:

im_float32 (np.ndarray[float32])

class flyeye.data.image.Smoothing(niters=3, size=3)

Smoothing operation applied to images.

Default behavior entails three iterations of grey closing followed by a 3-pixel wide median filter.

Attributes:

niters (int) - number of grey closing iterations

size (int) - pixel size of median filter in XY dimensions

static grey_closing(im, niters=3)

Apply grey closing operation to image.

Args:

im (np.ndarray(np.float32)) - pixel values. Shape may be (X,Y), (X,Y,3), or (N,X,Y,3) depending on image type.

niters (int) - number of grey closing iterations

Returns

im (np.ndarray(np.float32))

static median_filter(im, size=3)

Apply grey closing operation to image.

Args:

im (np.ndarray(np.float32)) - pixel values. Shape may be (X,Y), (X,Y,3), or (N,X,Y,3) depending on image type.

size (int) - pixel size of median filter in XY dimensions

Returns

im (np.ndarray(np.float32))