1. 26 Jul, 2016 1 commit
    • Hoogenboom, Jerry's avatar
      Various bug fixes and refinements throughout FDSTools · 08cf6ddd
      Hoogenboom, Jerry authored
      * Global changes in v0.0.4:
        * FDSTools will now print profiling information to stdout when the -d/--debug
          option was specified.
        * Fixed bug where specifying '-' as the output filename would be taken
          literally, while it should have been interpreted as 'write to standard out'
          (Affected tools: BGCorrect, Samplestats, Seqconvert, Stuttermark).
        * Added more detailed license information to FDSTools.
      * BGEstimate v1.1.0:
        * Added a new option -g/--min-genotypes (default: 3). Only alleles that occur
          in at least this number of unique heterozygous genotypes will be
          considered. This is to avoid 'contamination' of the noise profile of one
          allele with the noise of another. If homozygous samples are available for
          an allele, this filter is not applied to that allele. Setting this option
          to 1 effectively disables it. This option has the same cascading effect as
          the -s/--min-samples option, that is, if one allele does not meet the
          threshold, the samples with this allele are excluded which may cause some
          of the other alleles of these samples to fall below the threshold as well.
      * Stuttermodel v1.1.0:
        * Stuttermodel will now only output a fit for one strand if it could also
          obtain a fit for the other strand (for the same marker, unit, and stutter
          depth). This new behaviour can be disabled with a new -O/--orphans option.
        * Fixed bug that caused Stuttermodel to output only the raw data points for
          -1 and +1 stutter when normal output was supressed.
      * BGCorrect v1.0.1:
        * Added new column 'weight' to the output. The value in this column expresses
          the number of times that the noise profile of that allele fitted in the
          sample.
      * Samplestats v1.0.1:
        * Samplestats will now round to 4 or 5 significant digits if a value is
          above 1000 or 10000, respectively. Previously, this was only done for the
          combined 'Other sequences' values.
        * The 'Other sequences' lines will now also include values for
          total_recovery, forward_recovery, and reverse_recovery.
        * The total_recovery, forward_recovery, and reverse_recovery columns are no
          longer placed to the left of all the other columns generated by
          Samplestats.
        * The help text for Samplestats erroneously listed the X_recovery_pct instead
          of X_recovery.
        * Added support for the new 'weight' column produced by BGCorrect when the
          -a/--filter-action option is set to 'combine'.
      * BGPredict v1.0.1:
        * Greatly reduced memory usage.
        * BGPredict will now output nonzero values below the threshold set by
          -n/--min-pct if the predicted noise ratio of the same stutter on the other
          strand is above the threshold. Previously, values below the threshold were
          clipped to zero, which may cause unnecessarily high strand bias in the
          predicted profile.
      * BGMerge v1.0.1:
        * Reduced memory usage.
      * TSSV v1.0.1:
        * Renamed the '--is_fastq' option to '--is-fastq'. It was the only option
          with an underscore instead of a hyphen in FDSTools.
        * Fixed crash that would occur if -F/--sequence-format was set to anything
          other than 'raw'.
      * Libconvert v1.0.1:
        * Specifying '-' as the first positional argument to libconvert will now
          correctly interpret this as "read from stdin" instead of throwing a "file
          not found" error (or reading from a file named "-" if it exists).
      * Seqconvert v1.0.1:
        * Internal naming of the first positional argument was changed from 'format'
          to 'sequence-format'. This was done for consistency with the
          -F/--sequence-format option in other tools, giving it the same name in
          Pipeline configuration files.
      * Vis v1.0.1:
        * Added -j/--jitter option for Stuttermodelvis (default: 0.25).
        * Vis would not allow the -n/--min-abs and the -s/--min-per-strand options to
          be set to 0.
      * Stuttermodelvis v1.0.0beta2:
        * HTML visualisations now support drawing raw data points on top of the fit
          functions. The points can be drawn with an adjustable jitter to reduce
          overlap.
        * Fixed a JavaScript crash that would occur in HTML visualisations if the
          Repeat unit or Marker name filter resulted in an invalid regular expression
          (e.g., when the entered value ends with a backslash).
        * Reduced Vega graph spec complexity by using the new Rank transform to
          position the subgraphs.
        * HTML visualisations made with the -O/--online option of the Vis tool will
          now contain https URLs instead of http.
      * Samplevis v1.0.1:
        * Fixed a JavaScript crash that would occur in HTML visualisations if the
          Marker name filter resulted in an invalid regular expression (e.g., when
          the entered value ends with a backslash).
        * Reduced Vega graph spec complexity by using the new Rank transform to
          position the subgraphs.
        * Fixed a glitch where clicking the 'Truncate sequences to' label would
          select the marker spacing input.
        * The 'Notes' table cells with 'BGPredict' in them now get a light orange
          background to warn the user that their background profile was computed.
          If a sequence was explicitly 'not corrected', 'not in ref db', or
          'corrected as background only', the same colour is used.
        * The message bar at the bottom of Samplevis HTML visualisations will now
          grow no larger than 3 lines. A scroll bar will appear as needed.
        * HTML visualisations made with the -O/--online option of the Vis tool will
          now contain https URLs instead of http.
      * BGRawVis v1.0.1:
        * Fixed a JavaScript crash that would occur in HTML visualisations if the
          Marker name filter resulted in an invalid regular expression (e.g., when
          the entered value ends with a backslash).
        * Reduced Vega graph spec complexity by using the new Rank transform to
          position the subgraphs.
        * HTML visualisations made with the -O/--online option of the Vis tool will
          now contain https URLs instead of http.
      * Profilevis v1.0.1:
        * Fixed a JavaScript crash that would occur in HTML visualisations if the
          Marker name filter resulted in an invalid regular expression (e.g., when
          the entered value ends with a backslash).
        * Reduced Vega graph spec complexity by using the new Rank transform to
          position the subgraphs.
        * HTML visualisations made with the -O/--online option of the Vis tool will
          now contain https URLs instead of http.
      * Allelevis v1.0.0beta2:
        * Fixed potential crash/corruption that could occur with very unfortunate
          combinations of sample names and marker names.
        * HTML visualisations made with the -O/--online option of the Vis tool will
          now contain https URLs instead of http.
        * Added two more colours to the legend, such that a maximum of 22 markers is
          now supported without re-using colours.
      * Updated bundled D3 to v3.5.17.
      * Updated bundled Vega to v2.6.0.
      08cf6ddd
  2. 10 Mar, 2016 1 commit
    • Hoogenboom, Jerry's avatar
      Allelefinder fixes and mtDNA allele naming · e30c0408
      Hoogenboom, Jerry authored
      Fixed:
      * The -c/--stuttermark-column option of Allelefinder was not filtering
        out the non-ALLELE sequences as it was supposed to. (This issue was
        introduced in ce7f34fb, in which a bug was fixed that caused this
        option to filter all sequences, including the ones marked with ALLELE.
        So it turns out this option has been broken since 732e83ba.)
      
      Improved:
      * Allelefinder will no longer reject a marker based on the number of
        reads of 'Other sequences'.
      * Adjusted sequence alignment parameters for mtDNA sequences to produce
        allele names that more closely follow historical mtDNA mutation
        nomenclature.
      e30c0408
  3. 09 Mar, 2016 1 commit
  4. 08 Mar, 2016 1 commit
    • Hoogenboom, Jerry's avatar
      Version numbering for everything · da8d1d1b
      Hoogenboom, Jerry authored
      * Turned all '0.1dev' tool version numbers to '1.0.0'.
      * Changed Stuttermark's version number from '1.5' to '1.5.0'.
      * Added version numbers to the visualisations.
      * Updated README.rst to include all tools, but removed the usage details
        of Stuttermark because it is highly impractical to include usage
        details for all tools in the README file. I'll leave that to the
        -h/--help option and the yet-to-write FDSTools User's Handbook.
      da8d1d1b
  5. 04 Feb, 2016 1 commit
    • Hoogenboom, Jerry's avatar
      Updated handling of 'No data' and 'Other sequences' · 9ed2f3d1
      Hoogenboom, Jerry authored
      Improved:
      * Added -A/--aggregate-below-minimum option to the TSSV tool. This will
        add a line with 'Other sequences' to the output summing all sequences
        that were not reported because they had less reads than was specified
        with the -a/--minimum option.
      * Clarified the help text for the -D/--dir option of the TSSV tool.
      
      Fixed:
      * Updated all tools to consistently handle cases where 'No data' or
        'Other sequences' occurs in place of a sequence.
      9ed2f3d1
  6. 04 Nov, 2015 1 commit
    • Hoogenboom, Jerry's avatar
      Implemented support for non-STR markers, improved file handling and more · 1083919c
      Hoogenboom, Jerry authored
      Additions and improvements to the FDSTools library file format:
      * New [genome_position] section in FDSTools-style library files allows
      for specifying the chromosome and position of each marker.
      * New [no_repeat] section in FDSTools-style library files allows for
      including non-STR markers.
      * Comma/semicolon/space-separated values in FDSTools-style library files
      can now also be separated by tab characters and multiple consecutive
      separators are no longer collapsed (with the exception of whitespace).
      * If no prefix and/or suffix has been specified for an alias, the
      prefix/suffix of the marker itself is used.
      * Implemented support for non-STR markers (e.g. SNP clusters) and mtDNA
      markers. Allele names of the latter follow mtDNA nomenclature.
      * Improved the logic of generating STR allele names for sequences that
      have a prefix or suffix sequence that was not included in the library
      file.
      * Updated and clarified various explanatory texts in generated FDSTools
      library files.
      
      Fixed:
      * Fixed a bug that caused prefix/suffix variants in aliases to go
      missing in allele names.
      
      Improved file handling:
      * Library files are now closed immediately after parsing them.
      * Sample data input files are opened one at a time now.
      
      Visualisations:
      * Updated Vega to version 2.3.1.
      * Worked around a bug in Google Chrome that caused the 'Save image' link
      to stop working after having been used once.
      1083919c
  7. 01 Sep, 2015 2 commits
    • jhoogenboom's avatar
      Cleanup and minor enhancements · 03fc3d49
      jhoogenboom authored
      * BGCorrect and Stuttermark will now exit with an error message if
        more than one input file for the same sample is specified and no
        separate output files are given. Previously these tools would
        just overwrite the output file repeatedly, discarding the output
        of all but the last data file of the sample.
      * Removed to main() functions and related stubs from the tools
        because they are not actually runnable directly anyway.
      * Added some more help text to some of the tools.
      * Doubled the size of the marker name filter input element on the
        HTML visualisations.
      03fc3d49
    • jhoogenboom's avatar
      Various bug fixes and additions · ce7f34fb
      jhoogenboom authored
      Fixed:
      * Fixed crash that would occur when an empty sequence (primer dimer) is converted from raw to TSSV-style (or allelename) format.
      * Fixed bug in BGHomRaw that caused incorrect sample tags in the output.
      * Fixed bug that caused allele names with negative CE numbers and names of primer dimers to be regarded as 'invalid allele names' even though FDSTools generated those names itself.
      * Fixed crash when reading sample data while looking for an annotation column.
      * Fixed bug in Allelefinder resulting in the complete absence of output that occurred when a column name with Stuttermark output was specified.
      
      Changed:
      * Restyled the Options box on HTML visualisations. It is now less transparent and oriented more vertically to reduce overlap with the visualisation. Options are now presented in groups.
      * Updated Vega to version 2.2.1.
      
      New:
      * Added *_corrected columns to BGCorrect output for convenience. E.g., the total_corrected column contains the value of total-total_noise+total_add.
      * Added -L/--log-scale option to the Vis tool.
      ce7f34fb
  8. 21 Aug, 2015 1 commit
    • jhoogenboom's avatar
      Introducing Profilevis, and various bug fixes · b7d64a4f
      jhoogenboom authored
      * New visualisation Profilevis added to the package, but not yet to
        the Vis tool.
      * The Vis tool now prints a helpful error message if no output file
        was specified, instead of printing half a megabyte of HTML and
        minified JavaScript to the terminal.
      * Fixed crash that occurred when attempting to convert the sequence
        of an alias to its allele name.
      * Fixed various bugs in the functions that convert sequences to
        TSSV-style and allele names. Only the conversion of non-matching
        sequences was affected.
      * Added "max_expected_copies" section to the FDSTools library
        format. The default value is 2. Allelefinder will now use these
        as the maximum number of alleles per marker if the
        -a/--max-alleles option is not specified.
      * The section headers in the FDSTools library format are now case
        insensitive.
      b7d64a4f
  9. 14 Aug, 2015 1 commit
    • jhoogenboom's avatar
      Introducing Samplevis · 300549e0
      jhoogenboom authored
      * New visualisation Samplevis visualises sample data files.
        (Note: visualisations are currently stored in the package, but
        are not available via FDSTools commands yet. A new tool is going
        to be introduced later, which will copy the visualisation files
        to a user-selected folder.)
      * Including the current versions of Vega and D3 for completeness.
      * Fixed missing numpy dependency in setup.py.
      * Clarified some option help texts in Allelefinder based on
        feedback by Rick and Kris.
      300549e0
  10. 07 Aug, 2015 1 commit
    • jhoogenboom's avatar
      Reworked input/output file arguments · 7f23c2e0
      jhoogenboom authored
      * All tools now write to stdout by default. Tools that support
        writing report files write those to stderr by default. The
        -o/--output and -r/--report options can be used to override
        these.
      * Tools that operated on one sample at a time (bgcorrect,
        seqconvert, stuttermark) now support batch processing. The new
        -i/--input argument takes a list of files. In batch mode,
        the -o/--output argument can be used to specify a list of
        corresponding output files (which must be the same length). It
        is also possible to specify a format string to automatically
        generate file names. -o/--output defaults to "\1-\2.out" which is
        automatically expanded to "sampletag-toolname.out". The old
        positional arguments [IN] and [OUT] are maintained and allow for
        conveniently running the tools on a single sample file.
        [IN] is mutually exclusive with -i/--input and [OUT] is mutually
        exclusive with -o/--output. [OUT] now also accepts the filename
        format, but when not in batch mode, it still defaults to stdout.
        Note that by default, the sample tag is extracted from the input
        filenames by simply stripping the extension. This means a minimal
        batch processing command like "fdstools stuttermark -i *.csv"
        automatically creates a "...-stuttermark.out" file next to each
        CSV file in the current working directory.
      * Libconvert now also supports only specifying an output file.
        This makes it easier to write the default FDSTools library to a
        new file. E.g., "fdstools libconvert mynewfile.txt" now creates
        "mynewfile.txt" if it does not exist, and writes the default
        library to it. Most helpful.
      7f23c2e0
  11. 06 Aug, 2015 1 commit
    • jhoogenboom's avatar
      Greatly increased argument help · b30bdbbc
      jhoogenboom authored
      * All tools now have a longer description in the tool-specific help
        page.
      * Arguments are now presented in groups and the order is the same
        across tools.
      
      Furthermore:
      * Fixed bug that rendered BGHomStats and BGEstimate with the -H
        option useless.
      * The report of Allelefinder and BGEstimate is now written to
        sys.stderr by default. This means the report is now always
        generated (but it may be sent directly to /dev/null explicitly by
        the user). The big plus is that the progress of the tools is
        visible in the terminal when the tools are run by hand.
      b30bdbbc
  12. 05 Aug, 2015 1 commit
  13. 04 Aug, 2015 1 commit
    • jhoogenboom's avatar
      Introducing Blame · 8685a304
      jhoogenboom authored
      * New tool Blame can be used to find particularly dirty samples and
        to construct a DNA profile of the contaminator.
      * Fixed bug BGCorrect that resulted in incorrect values in the
        *_add columns.
      * BGEstimate and BGHomStats no longer crash if a library file is
        provided.
      * SeqConvert can now use a different library file for the output,
        thereby offering some possibilities to update allele names when a
        library file gets updated.
      * Replaced various uses of map() by generator expressions and
        listcomps for increased readability speed (although slightly).
      8685a304
  14. 30 Jul, 2015 1 commit
  15. 27 Jul, 2015 1 commit
    • jhoogenboom's avatar
      Updates to allelefinder · f9543ed9
      jhoogenboom authored
      * Allelefinder can now combine data from multiple files into a
        single sample (this happens when the same sample tag was
        extracted from their names).
      * Allelefinder can now automatically convert sequences to a given
        format (this is optional though). This is particularly useful
        when combining the knownalleles.csv and newalleles.csv files of
        a sample. (Note that allelefinder still assumes that the files
        contain different alleles; no attempt is made to check whether
        the same allele was represented in multiple files.)
      f9543ed9
  16. 24 Jul, 2015 1 commit
  17. 23 Jul, 2015 1 commit
    • jhoogenboom's avatar
      Laying foundations · 160594c5
      jhoogenboom authored
      * Introducing a new, extended library file format to support
        allele name generation.  The new libconvert tool can convert
        TSSV libraries to the new format and vice versa.
      * Added functions for converting between raw sequences, TSSV-style
        sequences, and allele names.
      * Added global -d/--debug option.
      
      Stuttermark updates:
      * Stuttermark now automatically converts input sequences to
        TSSV-style if a library is provided.
      * Stuttermark will no longer crash if there is no 'name' column.
        Instead, all sequences are taken to belong to the same marker.
      
      New tools:
      * libconvert converts between FDSTools and TSSV library formats.
      * seqconvert converts between raw sequences, TSSV-style sequences,
        and allele names.
      * allelefinder detects the true alleles in reference samples.
      160594c5