GitLab

  - Fixed bug where the 'Other sequences' could be treated as the maximum
    against which the '*_mp_max' columns are calculated.

  - Removed dependency on external tssv package (it is no longer compatible).
  - Greatly increased performance by deduplicating the input reads.
  - Removed the -q/--is-fastq option in favour of automatic detection.
  - Changed the default value for -m/--mismatches from 0.08 to 0.1.
  - Changed the default value for -n/--indel-score from 1 to 2.
  - Added the -X/--no-deduplicate option to disable deduplication.
  - Fixed potential crash that could occur under very specific circumstances.

Fixing a bug (introduced in v1.1.0) in the TSSV report output file.

* General changes in v1.1.0.dev3:
  * Allele name heuristics: don't produce insertions at the end of the
    prefix or at the beginning of the suffix; just include extra STR
    blocks.
  * FDSTools will no longer crash with a 'column not found' error when
    an input file is empty. This situation is now treated as if the
    expected columns existed, but no lines of actual data were present.
    This greatly helps in tracking down issues in pipelines involving
    multiple tools, as tools will now shutdown gracefully if an upstream
    tool fails to write output.
* Allelefinder v1.0.1:
  * Fixed crash that occurred when converting sequences to allele names
    format while no library file was provided.
  * Don't crash when output pipe is closed.
* BGAnalyse v1.0.1:
  * Don't crash when output pipe is closed.
* BGCorrect v1.0.2:
  * Don't crash on empty input files.
  * Don't crash when output pipe is closed.
* BGEstimate v1.1.2:
  * Don't crash whe...

* TSSV v1.1.0:
  * Added option -T/--num-threads (default: 1), which controls the
    number of worker threads TSSV may spawn to run the analysis.
    

* Samplevis v2.2.0:
  * Fixed incorrect calculation of 'percentage of highest' if the 'sequence'
    with the highest read count within a marker is the aggregated 'Other
    sequences' data. In exceptional cases, this could have resulted in the
    erroneous omission of an allele in the visualisation (graphs and/or
    tables).
* Stuttermodelvis v2.0.3:
  * Fixed bug that caused HTML visualisations with embedded data to fail
    while loading.
  * Fixed glitch where, in HTML visualisations with embedded data and a custom
    title, the custom title was truncated to the last '.' as if it were a file
    name.
* Pipeline v1.0.3:
  * Fixed glitch that caused the 'bgprofiles.html' output file of the
    reference-database analysis to lack a proper title.
* BGRawVis v2.0.1:
  * Fixed glitch where, in HTML visualisations with embedded data and a custom
    title, the custom title was truncated to the last '.' as if it were a file
    name.
  * Changed default save filename in HTML visualisations to 'bgprofiles-raw'.
* Profilevis v2.0.1:
  * Fixed glitch where, in HTML visualisations with embedded data and a custom
    title, the custom title was truncated to the last '.' as if it were a file
    name.
  * Changed default save filename in HTML visualisations to 'bgprofiles'.

* General changes in v1.0.1:
  * Fixed crash that occurred when using the -i option to run the same command
    on multiple input files.
  * The usage string now always starts with 'fdstools', even if FDSTools was
    invoked using some other command (e.g. on Windows, FDSTools gets invoked
    through a file called 'fdstools-script.py').
  * Fixed bug with the -d/--debug option being ignored if placed before the
    tool name on systems running Python 2.7.9 or later.
  * FDSTools library files may now contain IUPAC ambiguous bases in the
    prefix and suffix sequences of STR markers (except the first sequence,
    as it is used as the reference).  Additionally, optional bases may be
    represented by lowercase letters.
  * If no explicit prefix/suffix is given for an alias, the prefix/suffix of
    the corresponding marker is assumed instead. This situation was not
    handled correctly when converting from raw sequences to TSSV or allelename
    format, which resulted in the alias ...

* Samplevis v2.1.2 (additionally):
  * The net effect of the allele calling thresholds (table filtering options)
    is now visualised in the graphs as a dashed vertical red line.
  * Fixed issue with allele calling thresholds not working anymore after having
    used the 'Save page' link in HTML visualisations.

* Samplevis v2.1.2:
  * Added 'Save page' link to HTML visualisations, which offers for download a
    copy of the entire HTML visualisation including the user's changes.
  * Added automatic allele calling to static visualisations.
* Pipeline v1.0.2:
  * Added -A/--in-allelelist option to the pipeline tool to provide an existing
    allele list file when running the ref-db analysis, bypassing Allelefinder.
* Vis v1.0.3:
  * The -n/--min-abs and -s/--min-per-strand options now accept non-integer
    values as well.
  * Added six options to control the Table Filtering Options of Samplevis.
  * The Display Options now have a separate option group on the command line.

There is no 'Release Candidate' classifier, so this is the next best thing.

* General changes in v1.0.0rc1:
  * Fixed bug that caused variant descriptions in allele names of
    non-STR markers to be prepended with plus signs similar to suffix variants
    in STR markers.  When attempting to convert these allele names back to raw
    sequences, FDSTools would crash with an 'Invalid allele name' error.
* Allelevis v2.0.1 (additionally):
  * In the tooltip in HTML visualisations, a line break may now only be
    inserted in allele names after an underscore character (_) or after a
    repeat block in STR allele names.  If the input file contains raw
    sequences, line breaks may now be introduced anywhere in the sequence.
* Samplevis v2.1.1:
  * Added tooltip support to HTML visualisations.  Moving the mouse pointer
    over one of the alleles in the graph now displays a tooltip giving
    per-strand read counts of that allele.  The tooltip may include a
    'new allele' note if the input sample was analysed with FindNewAlleles.
  * The allele tables in HTML visualisations will now grow much wider than
    before if the screen (or window) is very narrow.
  * In the tables in HTML visualisations, a line break may now only be inserted
    in allele names after an underscore character (_) or after a repeat block
    in STR allele names.  If the input file contains raw sequences, line breaks
    may now be introduced anywhere in the sequence.
  * Improved determination of column widths of the allele tables when printing
    an HTML visualisation.
  * When printing an HTML visualisation, the graph and the corresponding table
    of a marker will be kept on the same page in all browsers now.
  * Fixed glitch that caused 'Infinity%' or 'NaN%' to be written in some cells
    in the allele tables in HTML visualisations for sequences that had zero
    reads (before or after correction).  These cells will remain empty now.
* Pipeline v1.0.1 (additionally):
  * The Pipeline tool will now only write the command lines of the tools it
    runs if the -d/--debug option was specified.
* Library v1.0.1 (additionally):
  * Added proper examples for non-STR markers and aliases.
* Stuttermodel v1.1.1:
  * Minor change to internal variant representation.

* General changes in v0.0.6.dev1:
  * Tools that take a list of files as their argument (through the -i option or
    as positionals) now explicitly support glob patterns.  This means they will
    interpret '*' and '?' characters as wildcards for 'zero or more characters'
    and 'any one character', respectively.  On Unix-like systems this is
    generally done by the shell, but on Windows one had to specify every file
    name completely.
* BGEstimate v1.1.1:
  * Added option -p/--profiles which can be used to provide a previously
    created background noise profiles file.  BGEstimate will read starting
    values from this file instead of assuming zero noise.
* BGMerge v1.0.2:
  * Small code changes to facilitate explicit glob pattern matching support.
* Pipeline v1.0.1:
  * The Pipeline tool will no longer check the existence of the files specified
    for the -S/--in-samples option; instead, this is left to the downstream
    tools to find out, consistent with how this works with the other input file
    options.
* Allelevis v2.0.1:
  * Added tooltip support to HTML visualisations.  Moving the mouse pointer
    over a node or edge in the graph now displays a tooltip giving allele names
    and sample counts.
* Stuttermodelvis v2.0.1:
  * Changed the unit in the horizontal axis title from 'bp' to 'nt'.
* Library v1.0.1:
  * Updated some of the comments describing the sections.

* General changes in v0.0.5:
  * The TSSV tool now depends on version 0.4.0 of TSSV.
  * Added new Pipeline tool that runs one of three default analysis pipelines
    automatically given a configuration file with tool options and input/output
    file names. The three available pipeline options are 'reference-sample',
    analysing a single reference sample with TSSV and Stuttermark;
    'reference-database', analysing a collection of reference samples with
    BGEstimate and Stuttermodel; and 'case-sample', analysing a single case
    sample with TSSV, BGPredict, BGMerge, BGCorrect, and Samplestats.
  * Added new Library tool that creates an empty FDSTools library file. Users
    may optionally specify the intented use of the library (STR markers,
    non-STR-markers, or both). Only the sections that apply to the given types
    of markers will be included in the output. The [aliases] section is not
    included by default, but an option is available to add it.
  * Added new tool BGAnalyse which can be used to analyse the remaining amount
    of noise in reference samples after correction.  This tool is a more
    sensitive successor of the 'Blame' tool.
  * Added new visualisation BGAnalysevis for visualising data obtained from
    BGAnalyse. This visualisation allows for identifying unclean or otherwise
    suboptimal samples by comparing the lowest, highest, and/or total remaining
    noise after correction for each marker in each sample.
  * The Blame tool was removed in favour of BGAnalyse.
* Libconvert v1.1.0:
  * When converting to FDSTools format, Libconvert automatically creates an
    empty FDSTools library file with the same contents as what would be
    obtained from the new Library tool without arguments.
  * The -a/--aliases option was modified such that it has the same effect as
    the -a/--aliases option of the new Library tool. This means that without
    this option specified, the [aliases] section will not be present in the
    output anymore.
  * The ability of the Libconvert tool to produce an empty FDSTools library
    file if no input file was given has been removed from the documentation
    (but not from the tool itself).
* TSSV v1.0.2:
  * Added new option -n/--indel-score which can be used to increase the
    penalty given to insertions and deletions in the flanking sequences w.r.t.
    the penalty given to mismatches.
  * NOTE: Requires TSSV v0.4.0 or newer to be installed.
* Vis v1.0.2:
  * Changed default value of -n/--min-abs from 15 to 5.
  * Added -I/--input2 option, which allows for specifying a file with raw data
    points for Stuttermodelvis and Profilevis.
  * Added support for creating BGAnalysevis visualisations.
* Profilevis v2.0.0:
  * Replaced the simple Options overlay with responsive design options panels
    in HTML visualisations.
  * Alleles and sequences are now sorted by CE allele length when applicable.
  * Added option to plot BGHomRaw data on top of the profiles.
  * Added marker selection menu for easier filtering.
* BGRawvis v2.0.0:
  * Replaced the simple Options overlay with responsive design options panels
    in HTML visualisations.
  * Sequences are now sorted by CE allele length when applicable.
  * Changed default minimum number of reads from 15 to 5.
  * Added marker selection menu for easier filtering.
* Stuttermodelvis v2.0.0:
  * Replaced the simple Options overlay with responsive design options panels
    in HTML visualisations.
  * Fixed glitch that caused the graphs to be re-rendered twice when loading
    a file by drag-and-drop in HTML visualisations.
  * Fixed glitch that made it possible to replace the data that was embedded
    in an HTML visualisation through drag-and-drop.
  * Added repeat unit selection menu for easier filtering.
* Allelevis v2.0.0:
  * Replaced the simple Options overlay with responsive design options panels
    in HTML visualisations.
  * Reduced Vega graph spec complexity by using the new Rank transform to
    position the subgraphs.
  * Fixed glitch that caused unnecessary padding around the graph.
* Samplestats v1.1.0:
  * Changed default allele calling option thresholds:
    * Changed default value of -m/--min-pct-of-max from 5.0 to 2.0.
    * Changed default value of -p/--min-pct-of-sum from 3.0 to 1.5.
  * Mentioned allele calling in the tool descriptions.
* Samplevis v2.1.0:
  * Changed default minimum number of reads for graph filtering from 15 to 5.
  * Changed default table filtering options:
    * Percentage of highest allele per marker changed from 5% to 2%.
    * Percentage of the marker's total reads changed from 3% to 1.5%.
    * Minimum number of reads in both orientations changed from 0 to 1.

* Global changes in v0.0.4:
  * FDSTools will now print profiling information to stdout when the -d/--debug
    option was specified.
  * Fixed bug where specifying '-' as the output filename would be taken
    literally, while it should have been interpreted as 'write to standard out'
    (Affected tools: BGCorrect, Samplestats, Seqconvert, Stuttermark).
  * Added more detailed license information to FDSTools.
* BGEstimate v1.1.0:
  * Added a new option -g/--min-genotypes (default: 3). Only alleles that occur
    in at least this number of unique heterozygous genotypes will be
    considered. This is to avoid 'contamination' of the noise profile of one
    allele with the noise of another. If homozygous samples are available for
    an allele, this filter is not applied to that allele. Setting this option
    to 1 effectively disables it. This option has the same cascading effect as
    the -s/--min-samples option, that is, if one allele does not meet...

* Added Noise column to the allele tables.
* Added number of reads before correction to the allele tables.
* Added raw numbers of reads to the Correction and Recovery columns of
  the allele tables.
* Fixed issue with Samplevis HTML visualisations in Firefox and Internet
  Explorer that caused an uncessesary horizontal scroll bar in the
  options panel.

Changed:
* The PctRecovery as used for automatic allele selection in Samplevis
  HTML visualisations as well as Samplestats is now computed w.r.t. the
  number of reads after correction, instead of the number of reads
  before correction.

Added:
* Added X_recovery columns to the output of the Samplestats tool. The
  value is equal to X_add / X_corrected * 100.

Fixed:
* Fixed a glitch in Samplevis HTML visualisations, where it would fail
  to correctly maintain the user-(de)selected alleles when switching
  Split Markers on or off.

Removed:
* Removed -L/--check-length option from the TSSV tool, because it had no
  effect. Instead, the TSSV tool will always enforce the expected allele
  lengths specified in the library file. Behaviour has not changed.

Fixed:
* The -c/--stuttermark-column option of Allelefinder was not filtering
  out the non-ALLELE sequences as it was supposed to. (This issue was
  introduced in ce7f34fb, in which a bug was fixed that caused this
  option to filter all sequences, including the ones marked with ALLELE.
  So it turns out this option has been broken since 732e83ba.)

Improved:
* Allelefinder will no longer reject a marker based on the number of
  reads of 'Other sequences'.
* Adjusted sequence alignment parameters for mtDNA sequences to produce
  allele names that more closely follow historical mtDNA mutation
  nomenclature.

* Turned all '0.1dev' tool version numbers to '1.0.0'.
* Changed Stuttermark's version number from '1.5' to '1.5.0'.
* Added version numbers to the visualisations.
* Updated README.rst to include all tools, but removed the usage details
  of Stuttermark because it is highly impractical to include usage
  details for all tools in the README file. I'll leave that to the
  -h/--help option and the yet-to-write FDSTools User's Handbook.

FDSTools would sometimes produce suboptimal alignments. Most notably, it
it would produce multiple smaller insertions/deletions when the
difference between two sequences could be described by one larger
insertion/deletion in combination with a base substitution. The latter
description is often more biologically sound and also usually results in
a shorter allele name.

* Fixed a bug that sometimes caused FDSTools to choose an incorrect path
  through the alignment matrix, producing a suboptimal alignment.
* Tweaked the alignment parameters to produce more meaningful results.

This was missing from commit 29fcc171

Added:
* Added a new section expected_allele_length to the FDSTools library
  format. In this section, the minimum and (optionally) maximum allele
  length of each marker can be specified.
* Added -L/--check-length option to the TSSV tool. If specified, the
  tool will use the expected_allele_length values to filter the results.
* Samplevis can now truncate long allele names to a given number of
  characters (defaulting to 70).
* Added an option to Samplestats to keep negatives when filtering (abs
  filter).

Changed:
* Renamed the --aggregate-below-minimum option of the TSSV tool to
  --aggregate-filtered.

Improved:
* Added an option to read_sample_data_file such that other code can
  request or require that the X_corrected columns are used.
* Samplestats will now round to 4 or 5 significant digits if a value is
  above 1000 or 10000, respectively.
* BGHomRaw will no longer round the forward, reverse, and total columns.
* When generating mtDNA allele names, FDSTools will now try to avoid
  creating gaps in the alignment of the sequences against the reference.
* Grouped the filtering options of the TSSV tool in its help text.
* Cleaned up some leftover code for special sequence value handling
  (more specifically: code that expected ensure_sequence_format to
  return False for special sequence values, which it no longer does).
* Cleaned up some dead legacy code in reduce_read_counts.

New tool findnewalleles:
* Given a list of known sequences, this tool can go through sample data
  files to mark all sequences that are not on the list.

Fixed:
* BGHomRaw, BGEstimate, BGHomStats, Stuttermodel, and Blame did not
  ignore the 'Other sequences' and 'No data' values that may occur in
  the place of a sequence as they were supposed to.

Improved:
* BGHomRaw will now include the sample tag in the "Missing allele X of
  marker Y" error message.
  
Changed:
* The -F/--sequence-format argument from BGHomRaw now defaults to "raw".

Visualisations:
* Updated Vega to version 2.5.0.
* The new version of Vega allowed the sorting to be fixed in Samplevis,
  Profilevis, BGRawvis, and Stuttermodelvis.
* Samplevis:
  * The 'Other sequences' bars are now drawn with an outline only.
  * STR alleles are now sorted by allele length by default (this can be
    toggled with a checkbox in HTML visualisations, and with an option
    in the Vis tool).
  * Fixed the clipping of the start of long allele names when printing
    SVGs from Google Chrome.
  * Added a note (as '?' help tooltip) to the Common axis range option
    in the HTML visualisation, to inform the user of the fact that the
    Split markers option needs to be off for it to work.

Fixed:
* Fixed crash in Samplestats. It would crash if BGCorrect columns were
  present.
* Fixed glitch in Samplevis that allowed clicking the 'Other sequences'
  bars if the input data already contained the 'Other sequences' entry.

Improved:
* The TSSV tool will now drop any sequences that contain anything other
  than A, C, T, and G. If the -A option is given, these sequences will
  still be added to the marker aggregates. Many other tools will fail
  when confronted with such invalid sequences, especially when allele
  names need to be generated.
* In Samplevis, the sequences are now consistently sorted (except for
  some inconsistency caused by a bug in Vega). The sorting is based on
  read counts and is the same as used for the allele tables in Samplevis
  HTML visualisations.
* Added a comment line that mentions genome build GRCh38 and rCRS to the
  genome_position block in the libconvert output. This is mainly for
  documentation purposes; users are free to change this line if they use
  a different reference.
* Minor styling changes to Samplevis HTML visualisations.