1. 29 Feb, 2016 1 commit
    • Hoogenboom, Jerry's avatar
      Expected allele lengths, and more · 29fcc171
      Hoogenboom, Jerry authored
      Added:
      * Added a new section expected_allele_length to the FDSTools library
        format. In this section, the minimum and (optionally) maximum allele
        length of each marker can be specified.
      * Added -L/--check-length option to the TSSV tool. If specified, the
        tool will use the expected_allele_length values to filter the results.
      * Samplevis can now truncate long allele names to a given number of
        characters (defaulting to 70).
      * Added an option to Samplestats to keep negatives when filtering (abs
        filter).
      
      Changed:
      * Renamed the --aggregate-below-minimum option of the TSSV tool to
        --aggregate-filtered.
      
      Improved:
      * Added an option to read_sample_data_file such that other code can
        request or require that the X_corrected columns are used.
      * Samplestats will now round to 4 or 5 significant digits if a value is
        above 1000 or 10000, respectively.
      * BGHomRaw will no longer round the forward, reverse, and total columns.
      * When generating mtDNA allele names, FDSTools will now try to avoid
        creating gaps in the alignment of the sequences against the reference.
      * Grouped the filtering options of the TSSV tool in its help text.
      * Cleaned up some leftover code for special sequence value handling
        (more specifically: code that expected ensure_sequence_format to
        return False for special sequence values, which it no longer does).
      * Cleaned up some dead legacy code in reduce_read_counts.
      29fcc171
  2. 22 Feb, 2016 1 commit
    • Hoogenboom, Jerry's avatar
      Fixed a crash in Samplestats, and minor improvements · 13e0d781
      Hoogenboom, Jerry authored
      Fixed:
      * Fixed crash in Samplestats. It would crash if BGCorrect columns were
        present.
      * Fixed glitch in Samplevis that allowed clicking the 'Other sequences'
        bars if the input data already contained the 'Other sequences' entry.
      
      Improved:
      * The TSSV tool will now drop any sequences that contain anything other
        than A, C, T, and G. If the -A option is given, these sequences will
        still be added to the marker aggregates. Many other tools will fail
        when confronted with such invalid sequences, especially when allele
        names need to be generated.
      * In Samplevis, the sequences are now consistently sorted (except for
        some inconsistency caused by a bug in Vega). The sorting is based on
        read counts and is the same as used for the allele tables in Samplevis
        HTML visualisations.
      * Added a comment line that mentions genome build GRCh38 and rCRS to the
        genome_position block in the libconvert output. This is mainly for
        documentation purposes; users are free to change this line if they use
        a different reference.
      * Minor styling changes to Samplevis HTML visualisations.
      13e0d781
  3. 02 Feb, 2016 1 commit
    • Hoogenboom, Jerry's avatar
      Big update: Bumped version to v0.0.3 · ebf700a7
      Hoogenboom, Jerry authored
      Updated Stuttermark to v1.5. WARNING: This version of Stuttermark is
      INCOMPATIBLE with output from previous versions of FDSTools and TSSV.
      
      Introducing TSSV-Lite
      * New tool tssv acts as a wrapper around TSSV-Lite (tssvl). Its primary
        purpose is to allow running TSSV-Lite without having to convert the
        FDSTools library to TSSV format, and to offer allelename output. Like
        all other tools in FDSTools, it also works with TSSV library files but
        its allele name generation capabilities are limited in that case.
      
      Changed:
      * TSSV-Lite and the new TSSV tool in FDSTools have two columns renamed
        w.r.t. the original TSSV program: 'name' has been changed to 'marker',
        and 'allele' has been changed to 'sequence'. All tools in FDSTools
        have been updated to use the new column names. This change affects
        Allelefinder, BGCorrect, BGEstimate, BGHomRaw, BGHomStats, BGPredict,
        Blame, Samplestats, Samplevis, Stuttermark, Stuttermodel, and
        Seqconvert. Note that this change will BREAK COMPATIBILITY of these
        tools with old data files.
      
      Fixed:
      * In Samplevis HTML visualisations, the "percentage recovery" table
        filtering option used the absolute number of recovered reads instead.
      * Added PctRecovery to the tables in Samplevis HTML visualisations.
      * BGPredict will now print a nice error message if the -n/--min-pct
        option is set to zero or a negative number, to avoid division by zero.
      * Samplestats would crash if the input file contained the flags column.
      * FDSTools would crash when trying to convert sequences to allele names
        using a TSSV library.
      
      Improved:
      * Libconvert will no longer include duplicate sequences in the STR
        defenition when converting to TSSV format and the reference sequence
        of one of the markers is the same as one of its aliases, or when
        aliases of one marker share one or more prefix or suffix sequences.
      * Updated add_input_output_args() such that the output file is a
        positional argument (instead of -o) for tools that have a single input
        file and no support for batches.
      * Updated add_sequence_format_args() such that the library file can be
        made a required argument.
      * Refined the FDSTools package description, since FDSTools does more
        than just noise filteirng.
      * FDSTools will now do a marginally better job at producing allele names
        for sequences that do not exactly match the provided STR pattern. When
        seeking the longest matching portion of the sequence, it will now also
        test the reversed sequence with a reversed pattern, which sometimes
        yields a longer match. It is still not optimal, though, but some
        refactoring has been done to move away from regular expressions.
      * BGCorrect will now also fill in correction_flags for newly added
        sequences.
      * Adjusted the help text of Samplestats to include the fact that the -c
        and -y options have an OR relation instead of an AND relation.
      * BGCorrect, BGEstimate, BGHomRaw, BGHomStats, BGPredict, and
        Stuttermodel will now ignore special values that may appear in the
        place of a sequence (currently: 'Other sequences' and 'No data').
      
      Removed:
      * The -m/--marker-column and -a/--allele-column arguments of BGPredict
        had no effect and have been removed.
      
      Visualisations:
      * Updated bundled D3 to v3.5.12.
      * In HTML visualisations, if the page is scrolled to the right edge when
        an option is changed that causes the graphs to become wider, the page
        now remains scrolled to the right.
      * Samplevis HTML visualisations:
        * Added 'Clear manually added/removed' link to the table filtering.
        * Reduced flicker of the mouse cursor in Internet Explorer.
        * Added 'Common axis range' checkbox (only available when 'Split
          markers' is off).
        * Added 'Save table' link to save the table of selected alleles to a
          tab-separated file.
        * Added 'PctRecovery' column to the tables of selected alleles.
        * An alert box is now shown when a data file is loaded that contains
          markers that have 'No data'.
        * Added 'Percentage of total reads' to the graph filtering options.
        * Added a note to the table filtering options to explain that the
          minimum percentage correction and recovery have an OR relation.
      ebf700a7
  4. 18 Jan, 2016 1 commit
    • Hoogenboom, Jerry's avatar
      Various fixes and improvements · 4f9286e4
      Hoogenboom, Jerry authored
      Fixed:
      * Fixed a crash in BGMerge.
      * Fixed bug in BGCorrect that resulted in incorrect values in the
        *_add and *_corrected columns (yes, you, 8685a304).
      * Fixed a glitch in BGCorrect that prevented it from ever writing
        corrected_bgestimate in the correction_flags column.
      
      Improved:
      * BGEstimate will now include the sample tag in the error messages for
        missing alleles and alleles with 0 reads.
      * Strand bias lines in Samplevis are now clamped to the 0-100% range.
        BGCorrect may cause forward read percentages outside this range.
      
      Visualisations:
      * Updated Vega to version 2.4.2.
      * Fixed drag-'n-drop behaviour for HTML visualisations in Internet
        Explorer and Firefox.
      * Fixed the Save Image link when viewing HTML visualisations in
        Internet Explorer 10 and above.
      * Added http-equiv="X-UA-Compatible" content="IE=edge" meta-tag to all
        visualisations to prevent Internet Explorer from entering quirks mode.
      * Samplevis:
        * Fixed glitch that would sometimes cause a second horizontal scroll
          bar to appear.
        * Graphs now render much more quickly when 'Split markers' is on, and
          Chrome no longer crashes on large sample files with this option set.
      4f9286e4
  5. 16 Nov, 2015 1 commit
    • Hoogenboom, Jerry's avatar
      Various fixes and improvements · 313867bc
      Hoogenboom, Jerry authored
      Fixed:
      * The 'to' base in variants called on mtDNA was incorrect. This bug could also cause FDSTools to crash.
      * FDSTools would crash if you tried to generate an allele name for a primer dimer of an mtDNA marker. (Now, you get an insane but entirely accurate allele name instead.)
      * Fixed bug that caused some perfectly valid mtDNA allele names to be rejected when attempting to convert them back to raw sequences.
      
      Improved:
      * You can now also specify the ending position of the markers in the FDSTools library. If you do, you may also additionally specify a second start position (and optionally also a second end position, and so on). FDSTools will interpret this as that the marker is the concatenation of each of these fragments. This was primarily introduced to support mtDNA fragments that contain (somewhere in the middle) the origin of mtDNA base numbering.
      * More helpful error message when format violations are detected while parsing the library file.
      * More helpful error message when the -e/--tag-expr regular expression could not be compiled.
      * Added a paragraph about sequence alignment caching to the help text of Seqconvert.
      * Added a 'flags' column to BGCorrect output, which gives information about the data that was used to do the correction.
      
      Background noise profiles:
      * Removed -C/--cross-tabular option from BGEstimate, BGPredict, and BGMerge and also removed the ability to read files in this format.
      * BGEstimate, BGHomStats, and BGPredict now add a column 'tool' with their name to the output.
      313867bc
  6. 04 Nov, 2015 1 commit
    • Hoogenboom, Jerry's avatar
      Implemented support for non-STR markers, improved file handling and more · 1083919c
      Hoogenboom, Jerry authored
      Additions and improvements to the FDSTools library file format:
      * New [genome_position] section in FDSTools-style library files allows
      for specifying the chromosome and position of each marker.
      * New [no_repeat] section in FDSTools-style library files allows for
      including non-STR markers.
      * Comma/semicolon/space-separated values in FDSTools-style library files
      can now also be separated by tab characters and multiple consecutive
      separators are no longer collapsed (with the exception of whitespace).
      * If no prefix and/or suffix has been specified for an alias, the
      prefix/suffix of the marker itself is used.
      * Implemented support for non-STR markers (e.g. SNP clusters) and mtDNA
      markers. Allele names of the latter follow mtDNA nomenclature.
      * Improved the logic of generating STR allele names for sequences that
      have a prefix or suffix sequence that was not included in the library
      file.
      * Updated and clarified various explanatory texts in...
      1083919c
  7. 01 Sep, 2015 2 commits
    • jhoogenboom's avatar
      Cleanup and minor enhancements · 03fc3d49
      jhoogenboom authored
      * BGCorrect and Stuttermark will now exit with an error message if
        more than one input file for the same sample is specified and no
        separate output files are given. Previously these tools would
        just overwrite the output file repeatedly, discarding the output
        of all but the last data file of the sample.
      * Removed to main() functions and related stubs from the tools
        because they are not actually runnable directly anyway.
      * Added some more help text to some of the tools.
      * Doubled the size of the marker name filter input element on the
        HTML visualisations.
      03fc3d49
    • jhoogenboom's avatar
      Various bug fixes and additions · ce7f34fb
      jhoogenboom authored
      Fixed:
      * Fixed crash that would occur when an empty sequence (primer dimer) is converted from raw to TSSV-style (or allelename) format.
      * Fixed bug in BGHomRaw that caused incorrect sample tags in the output.
      * Fixed bug that caused allele names with negative CE numbers and names of primer dimers to be regarded as 'invalid allele names' even though FDSTools generated those names itself.
      * Fixed crash when reading sample data while looking for an annotation column.
      * Fixed bug in Allelefinder resulting in the complete absence of output that occurred when a column name with Stuttermark output was specified.
      
      Changed:
      * Restyled the Options box on HTML visualisations. It is now less transparent and oriented more vertically to reduce overlap with the visualisation. Options are now presented in groups.
      * Updated Vega to version 2.2.1.
      
      New:
      * Added *_corrected columns to BGCorrect output for convenience. E.g., the total_corrected column contains the value of total-total_noise+tot...
      ce7f34fb
  8. 12 Aug, 2015 1 commit
    • jhoogenboom's avatar
      Introducing BGMerge · 6207d485
      jhoogenboom authored
      * New tool BGMerge can be used to merge background noise profiles
        (e.g., merge BGPredict output with a database previously
        obtained from BGEstimate).
      * Fixed two major bugs in BGPredict that resulted in incorrect fit
        functions being used.
      * BGEstimate, BGPredict, BGHomStats, Blame, and StutterModel no
        longer crash if a library file is specified.
      * Added reverse strand profile estimation to BGPredict.
      6207d485
  9. 11 Aug, 2015 1 commit
    • jhoogenboom's avatar
      Introducing BGPredict · 276a0439
      jhoogenboom authored
      * New tool BGPredict predicts background noise profiles (containing
        only stutter products) for user-supplied alleles/sequences using
        a trained stutter model obtained from Stuttermodel. Currently
        only the amounts of the forward strand are predicted.
      * New option -L/--min-lengths for Stuttermodel allows to set a
        minimum required number of unique repeat lengths to base the
        fits on (default: 5).
      * Updated formatting of output of Stuttermodel: added '+' sign to
        positive stutter, limited r2 scores to 3 decimal places, and now
        all coefficients are written in scientific notation with 3
        decimal places.
      * The --output-column option of SeqConvert now defaults to using
        the value of --allele-column.
      276a0439
  10. 07 Aug, 2015 1 commit
    • jhoogenboom's avatar
      Reworked input/output file arguments · 7f23c2e0
      jhoogenboom authored
      * All tools now write to stdout by default. Tools that support
        writing report files write those to stderr by default. The
        -o/--output and -r/--report options can be used to override
        these.
      * Tools that operated on one sample at a time (bgcorrect,
        seqconvert, stuttermark) now support batch processing. The new
        -i/--input argument takes a list of files. In batch mode,
        the -o/--output argument can be used to specify a list of
        corresponding output files (which must be the same length). It
        is also possible to specify a format string to automatically
        generate file names. -o/--output defaults to "\1-\2.out" which is
        automatically expanded to "sampletag-toolname.out". The old
        positional arguments [IN] and [OUT] are maintained and allow for
        conveniently running the tools on a single sample file.
        [IN] is mutually exclusive with -i/--input and [OUT] is mutually
        exclusive with -o/--output. [OUT] now also accepts the filename
        format, but when not in batch mode, it still defaults to stdout.
        Note that by default, the sample tag is extracted from the input
        filenames by simply stripping the extension. This means a minimal
        batch processing command like "fdstools stuttermark -i *.csv"
        automatically creates a "...-stuttermark.out" file next to each
        CSV file in the current working directory.
      * Libconvert now also supports only specifying an output file.
        This makes it easier to write the default FDSTools library to a
        new file. E.g., "fdstools libconvert mynewfile.txt" now creates
        "mynewfile.txt" if it does not exist, and writes the default
        library to it. Most helpful.
      7f23c2e0
  11. 06 Aug, 2015 1 commit
    • jhoogenboom's avatar
      Greatly increased argument help · b30bdbbc
      jhoogenboom authored
      * All tools now have a longer description in the tool-specific help
        page.
      * Arguments are now presented in groups and the order is the same
        across tools.
      
      Furthermore:
      * Fixed bug that rendered BGHomStats and BGEstimate with the -H
        option useless.
      * The report of Allelefinder and BGEstimate is now written to
        sys.stderr by default. This means the report is now always
        generated (but it may be sent directly to /dev/null explicitly by
        the user). The big plus is that the progress of the tools is
        visible in the terminal when the tools are run by hand.
      b30bdbbc
  12. 05 Aug, 2015 1 commit
  13. 04 Aug, 2015 1 commit
    • jhoogenboom's avatar
      Introducing Blame · 8685a304
      jhoogenboom authored
      * New tool Blame can be used to find particularly dirty samples and
        to construct a DNA profile of the contaminator.
      * Fixed bug BGCorrect that resulted in incorrect values in the
        *_add columns.
      * BGEstimate and BGHomStats no longer crash if a library file is
        provided.
      * SeqConvert can now use a different library file for the output,
        thereby offering some possibilities to update allele names when a
        library file gets updated.
      * Replaced various uses of map() by generator expressions and
        listcomps for increased readability speed (although slightly).
      8685a304
  14. 03 Aug, 2015 1 commit
    • jhoogenboom's avatar
      Introducing BGHomStats · a09131d9
      jhoogenboom authored
      * New tool BGHomStats computes statistics (minimum, maximum, mean,
        and sample variance) of noise ratios in homozygous samples.
      * The default BGEstimate output format has been changed to be
        compatible with that of BGHomStats. The cross-tabular output
        format is still available as an option because it easily uses 90%
        less disk space. BGCorrect (and other future tools that use noise
        profiles) will work with both formats.
      * Fixed bug in the --min-samples option of BGEstimate that could
        cause some alleles with less than the specified number of samples
        to be included if --drop-samples is used at the same time.
      * The user now receives an error message if there are unknown
        arguments. The error message lists the usage string of the
        requested tool. (Argparse's default was to print the general
        FDSTools usage string, which is not helpful.)
      a09131d9
  15. 30 Jul, 2015 1 commit
  16. 29 Jul, 2015 1 commit
    • jhoogenboom's avatar
      Introducing bgestimate · be745e64
      jhoogenboom authored
      I could write about all its features here, but instead I will point
      out some future plans to highlight the things that are possibly not
      optimal in their current implementation.
      
      There are a number of things I plan to change in the future:
      * The output format is currently JSON, perhaps a carefully designed
        tabular format is a better choice. The benefit of switching to a
        tabluar format is that the data can be loaded into e.g. Excel as
        well.
      * The profiles are currently produced separately for forward and
        reverse reads. I would prefer to integrate these into a single
        computation that estimates allele balance in the heterozygotes
        using both strands as well.
      * I would like to add information about strand bias of the alleles
        as well. The most straightforward way to do this is to set only
        the forward reads of the true allele to 100 and treat the reverse
        reads the same as all background products. You will then obtain a
        number of reverse reads observed for ever 100 forward reads of
        the true allele.
      * I think it would be appropriate to make sure the values in the
        allele balance matrices of each sample ('Ax' in the source code)
        should add up to 1. For homozygotes, it is currently a scalar 1,
        the sum of the elements tend to be more than 1. This means that a
        heterozygous sample has a stronger influence on the profiles than
        a homozygous sample.
      be745e64