--save-progress flag added for training. Saves training progress to a .csv file (
During training, the model will be saved after each epoch by default. Use
--no-save-checkpoints to supress this behaviour and save disk space. Each model file can be large, and if you don't have much training data, they can be generated quickly.
Bug in the standalone tool
New combined cell detection and registration viewer (see Visualisation)
There is a new and improved GUI for manually segmenting linear tracks and volumes in standard space (see Manually segment in standard space).
Registration can now deal with data in a non-standard orientation, see Image orientation
Detected cell positions can now be manually curated and saved (see Visualisation).
Bug causing data to occasionally be loaded in the incorrect order for registration is fixed
amap_vis has been replaced with
cellfinder_view_cells has been replaced with
Support is now officially for Windows and Linux. macOS should generally work, but there may be issues.
Bugs left in
cellfinder_cell_standard after refactoring main code are fixed.
Compatible with napari v0.3.0
cellfinder_view_3D has been removed
Windows bug fix thanks to @EmanPaoli
Prebuilt wheels now released for Linux, Windows and macOS (untested)
Update dependencies & refactor usage of amap api
Training can now be saved every N epochs using
Bug fixed in
Pandas dependencies updated thanks to @amedyukhina
Fix memory leak in training
Update curation interface
New all python training data generation interface,
cellfinder_curate. Details here.
Heatmap and converting cells to brainrender format moved to neuro
Lots of utility functions moved to imlib
Automatic deployment with Travis
Very basic Windows support (allthough some issues remain.)
Pixel sizes are now defined in um (rather than mm). Users specifying a metadata file will not be affected. Those specifying the pixel sizes manually will need to change their usage. Flags such as
--x-pixel-mm are now
All cell detection parameters (e.g.
--soma-diameter) are now defined in um or um3 (rather than mm)
Registration parameters are now defined via command-line arguments (e.g.
--freeform-use-n-steps) rather than a config file.
Installation is simplified (no need to install tensorflow separately)
Updated default parameters to better work on noisy data.
Memory consumption during inference reduced
Registration code is now in amap
Compatibility with tensorflow 2.1.0
Cube extraction is now carried out on the fly during classification (and not saved to disk). This prevents file system issues due to the generation of (potentially) hundreds of thousands of cells.
Likelihood during training of each individual augmentation transformation reduced to 10% (from 50%)
Bug fix in
cellfinder_view_cells to deal with Napari API change
New viewer to inspect batches of generated cubes
Removed some old, unnecessary tests
Fixed a bug in
Removed NiftyNet dependency. Classification is now implemented in
THIS MEANS YOU NEED TO TRAIN A NEW MODEL. OLD MODELS ARE INCOMPATIBLE!
cellfinder_view_cells command which uses napari to view cells overlaid on raw data.
cellfinder_gen_region_vol command line tool to generate images of specific brain regions (for figures etc)
roi_transform now supports ROIs generated in the original, full-size images
cellfinder_cells_to_brainrender to allow classified cells to be viewed in BrainRender
cellfinder_view_cells2D function is removed, and replaced with
Cube scaling is now 3D (theoretically leading to better classification
cellfinder_view_3D now uses napari.
Brain image IO moved to brainio
Reading a slurm environment (if present) moved to
Logging moved to fancylog
roi_transform command line tool to transform ImageJ ROI sets into images in standard space.
Fixes a bug causing heatmaps to be scaled incorrectly
Only minor changes to documentation and testing.
--max-ram to specify a maximum amount of RAM to use (in GB). Currently only affects processes that will scale to use as much RAM as is available (such as cube extraction).
cellfinder_xml_scale command line tool to rescale the cell positions within an XML file. For compatibility with other software, or if your data has been scaled after cell detection.
THIS VERSION HAS NOT BEEN FULLY TESTED
Instructions for installing CUDA 9 and cuDNN via conda added.
cellfinder_region_summary now has a
--sum-regions flag combine child regions.
Bug which caused heatmaps to be calculated incorrectly is fixed.
Bug which causes
--regions-list input to
cellfinder_region_summary to be parsed incorrectly is fixed.
Bug which crashed summary csv generation if a cell is transformed incorrectly due to rounding errors is fixed.
Bug which crashed volume calculation in registration if a region is missing is fixed.
Bug causing registration to crash if a custom registration file without a
base_folder entry is fixed.
urllib3 version is specified to prevent atlas downloading from failing.
A bug which caused the atlas not be downloaded when using
cellfinder_register is fixed.
Registration now respects the
Python 3.5 is no longer supported.
Detected cell positions can now be transformed into standard space. By default, if cell classification and registration has been performed, an additional
cells_in_standard_space.xml file will be generated. Use
--no-standard_space to prevent this behaviour. This functionality can also be run using the standalone script
Pixel sizes must still be specified, but instead of using the
-z flags, a
--metadata flag can be used instead. Supported formats include BakingTray recipe files, mesoSPIM metadata files or cellfinder custom metadata files (see below). If both pixel sizes and metadata are provided, the command line arguments will take priority.
An installation bug (due to new requirements of gputools is fixed.)
The main command-line interface to cellfinder is now
cellfinder_run has been removed (and may call an old version of the software). Please use
cellfinder from now on.
cellfinder_gen_cubes has been fixed
The main command-line interface to cellfinder will be
cellfinder_run will stop working soon (and may call an old version of the software). Please use
cellfinder from now on.
THIS VERSION HAS NOT BEEN FULLY TESTED Use Version 0.2.3a if you don't need the new features
Pixel sizes must now be specified with
cellfinder.IO.cells.get_cells() can now read the
.yml files output from the cell counting plugin of MaSIV.
Registration will now calculate a
volumes.csv file which contains the volumes of each brain area in the atlas, in the sample brain.
--summarise flag will now save more information, including:
Cell counts combined across hemispheres
Cell densities in cells per mm3.
Registration will also calculate the reverse transforms (i.e. sample -> atlas)
--figures flag will create a subdirectory of 3D images in the coordinate space of the downsampled raw data (and registered atlas) that can be used to generate figures e.g. in FIJI. Currently three files will be generated by default.
heatmap.nii - This is a 3D histogram, showing cell densities
throughout the brain. Likely to be more useful than visualising cellpositions for dense areas. By default, this is smoothed, and masked (so that the smoothing doesn't "bleed" outside of the brain).
glass_brain.nii This is a binary image, just showing the brain surface. Plotting this in 3D along with heatmap.nii gives a nice sense of cell density throughout the brain. ClearVolume works well for this.
boundaries.nii Another binary image, but showing the divisions between each region in the atlas. More useful for smaller, 3D figures to delineate region boundaries. Also useful to assess registration by overlaying on the raw, downsampled data.
Bug on certain systems where cube extraction could use too much RAM, and crash the system has been fixed.
Output directory structure has been reorganised, and simplified slightly.
Code coverage increased to 58%.
A lot of the code has been refactored, and (hopefully simplified).
Log files will now print more information (e.g. information about the git repository, and internal parameters).
Registration can now handle data in any orientation. The default is coronal, and in the NifTI standard orientation (origin is the most ventral, posterior, left voxel). If your data does not match this, use the
There is now a standalone registration command,
cellfinder_register. This will be the maintained python version of aMAP going forward.
There is now an option
--remove-intermediate to remove all intermediate files generated by cellfinder. Use with caution.
All downsampled data, registered atlases and cell locations are in the same coordinate space as the raw data (although possibly scaled). This fixes issues #22 and #23, and allows overlay of the cell positions on the downsampled data from the registration step.
Cube extraction will now use as many CPU cores as available, slightly speeding up this step. However, if you don't have much RAM (or a lot of CPU cores) see #36.
Input data paths no longer need to end in the channel number. For channel referencing purposes, this will be set automatically, or can be overridden with
By default, registration will now save downsampled versions of all channels. if you don't want these files, use
Cube extraction will now default to extracting cubes of 50um x 50um x 100um. This can be overridden with
--y_pixel_mm_network. Currently there is no rescaling in z.
Cube extraction now supports the output of ROI sorter, rather than just xml.
Code coverage increased to 32%.
Logging now includes diagnostic information.
Cellfinder can now be installed with a single command:
pip install git+https://github.com/adamltyson/cellfinder
The atlas will now no longer be downloaded automatically upon installation of cellfinder, but on the first use of the atlas. To download in advance,
cellfinder_download can be used.
New function to load
.nrrd files (in
Data can now be passed as a directory of 2D .tif files (rather than just a text file of file paths).
If a network was trained with a cubes of a different pixel size to that of the input data, then the pixel spacing of the output cubes can be changed (e.g. using:
Cell (and candidate) positions can be saved as
.csv (as well as
.xml), by using
cellfinder_xml_crop has been added. This curates an input
.xml file and outputs a file with only those positions within given anatomical areas. Currently, the cells, and the atlas need to be in the same anatomical space.
Bug which caused cellfinder to hang when using part of a node managed by
SLURM is fixed.
If a cube cannot be extracted for any reason (such as proximity to the edge of an image), the default is now to not save it (rather than save an empty cube in it's place). This can be overridden with
Error messages when a cube cannot be extracted (due to proximity to the edge of the image) have been clarified.
Code is now automatically formatted with Black, which requires Python >= 3.6 to run.
Testing has been added (although currently there is only one test).
All commits & pull requests will be build by Travis.
For development, cellfinder should be installed using
pip install -e .[dev]in the repository.
Simple summary information can be generated with the
--summarise flag. This will run registration (if it wasn't run already) and associate each cell output from the cell detection step with a brain region. This info is saved as a csv file.
The mouse brain atlas can be downloaded to any directory when installing cellfinder using the
--atlas-install-path flag. This flag can also be used to point to an existing atlas installation (e.g. from aMAP).
This will update the default config file so the correct atlas is sourced (and a second isn't downloaded.)
cellfinder should know what parts have already been run, so if you re-run a
cellfinder_run command, it won't repeat any parts of the pipeline.
cellfinder_gen_cubes function to generate tiff cubes from an xml file independent of the rest of cellfinder. Useful for generating training data.
cellfinder_count_summary function to combine a brain registered to the allen atlas with cell counts. Generates a csv file of cells per region.
cellfinder_region_summary function to organise (align by brain area) csv files of summary cell counts from multiple brains.
cellfinder_view_cells2D function to view cells overlaid on raw data. Uses hardlyany RAM, but I recommend ROI sorterinstead.
cellfinder_view_3D function. A very rudimentary 3D viewer, but can load any file type in use by cellfinder.
Some API docs
Logging that interfered with multiprocessing when the
--verbose tag was used has been emoved. This was an issue in case cellfinder was stopped in the middle of cell classification as it could cause GPU memory to not be released (and not show up in
Documentation moved from pydocmd to sphinx.
Installation requires configobj (as well as Cython).
--dev if you want to build the documentation yourself.
Can run cell detection on N-channels
Streamlined training of cell classification network via
Includes (rudimentary) integration of aMAP for registration to the Allen brain atlas
Further options to only run parts of the analysis
--tensorboard to launch tensorboard automatically during training
cellfinder_view to launch the (very simple) image/cell viewer
Docs generated via pydocmd
Output of cell candidate detection step
cells.xml can now be opened in Roi Sorter
Docs generated by pydocmd means that installation info is currently here
First version compatible with NiftyNet 0.5.0, TensorFlow 1.12.0 and therefore CUDA 9/10.
Option to specify external configuration file for cell candidate classification
All results saved to a single directory
Option to keep or discard bright spots classified as artifact during cell candidate detection
Z positions of cell candidates when analysing a z-subset of the data fixed in the resulting
Cell candidate detection .xml file only provides positive candidate types
<Type>1</Type> for compatibility with Roi Sorter
No longer relies on NiftyNet model zoo (although this may return)
Much simplified installation (no manual cp/mv)
Moving towards a single API, calls to n_count_python etc. broken
Multiprocessing compatible with shared resources (e.g. HPC job schedulers)
Developed with Python 3.5, but should work with 3.6/3.7. no Python 2 compatibility
HPC compatibility only tested with SLURM, but should work with LSF & SGE
Final version compatible with niftynet 0.2.2 and tensorflow 1.4.1 (and therefore CUDA 8).