We use a patched magic package to work with the version of magiclib that
is packaged with CentOS 6.7.

https://stackoverflow.com/questions/34541129/mutalyzer-py-test-fails-with-python-magic-error

Mutalyzer used the `reverse_complement` function from the description
extractor, which was a unicode string-only implementation.

With this update, that implementation was removed and the BioPython
reverse complement function used instead in the description
extractor. Unfortunately this cannot work with unicode strings, so we
now use our own `reverse_complement` which works on unicode strings.

This speeds up lookup of transcript mappings by genomic position
a lot. By filtering on bin index, such a query now uses the index
on the bin column, where previously this would involve a
sequential table scan.

http://interval-binning.readthedocs.org/

Issue #50 showed a problem in our file encoding detection, caused
by our cut-off for the confidence as reported by the cchardet [1]
library:

    >>> import cchardet
    >>> s = u'NM_000052.4:c.2407\u20132A>G'
    >>> b = s.encode('WINDOWS-1252')
    >>> cchardet.detect(b)
    {'confidence': 0.5, 'encoding': u'WINDOWS-1252'}

We require a confidence stictly greater than 0.5 and default to
UTF8 otherwise.

If, however, we try the same thing using the chardet [2] library,
we get a higher confidence for the same string:

    >>> import chardet
    >>> chardet.detect(b)
    {'confidence': 0.73, 'encoding': 'windows-1252'}

So the two obvious ways to solve this are:

1. Lower the confidence threshold.
2. Use chardet instead of cchardet.

We implement the second solution here, since it also removes a C
library dependency and we are not worried by performance.

Of course the detected encoding remains a guess which can still
be wrong!

[1] https://github.com/PyYoshi/cChardet
[2] https://github.com/chardet/chardet

Fixes #50

This is a work in progress as there still seem to be some bugs. For example,
some unit tests fail due to incorrect descriptions generated and others fail
due to a crash.

This fixes Spyne to not crash on POST requests to the HTTP/RPC+JSON
webservice.

Note that all return values still use byte strings. Changing those will
touch a larger part of the codebase, and will be done in another commit.

As per [1]:

> Unlike the Python str, the Spyne String is not for arbitrary byte
> streams. You should not use it unless you are absolutely, positively
> sure that you need to deal with text data with an unknown encoding. In
> all other cases, you should just use the Unicode type. They actually
> look the same from outside, this distinction is made just to properly
> deal with the quirks surrounding Python-2’s unicode type.
>
> Remember that you have the ByteArray and File types at your disposal
> when you need to deal with arbitrary byte streams.
>
> The String type will be just an alias for Unicode once Spyne gets
> ported to Python 3. It might even be deprecated and removed in the
> future, so make sure you are using either Unicode or ByteArray in your
> interface definitions.

[1] http://spyne.io/docs/2.10/manual/03_types.html#strings

This fixes uploading base64 encoded data to the JSON webservice. For
example:

    echo "NM_003002.2:c.274delT\nXXX:g.1del" | base64 > test.base64
    curl \
      -d 'process=SyntaxChecker' \
      -d 'argument=hg19' \
      --data-urlencode 'data@test.base64' \
      'http://127.0.0.1:8082/submitBatchJob'

Upstream Spyne crashes on POST requests to the HTTP/RPC+JSON webservice.
We patched it in a rather hacky way.

This was a regression from the old codebase, where we installed Spyne
separately from our LUMC GitHub mirror. This is now also referenced in
the requirements.txt file.

Thanks to Ken Doig for reporting the issue.

See http://pytest.org/

This is quite a large commit, touching many things related to developer
documentation. It is all focussed on getting as much of this as possible
into the new Sphinx-based documentation.

Some highlights:

- Start Sphinx-based developer documentation, including fairly complete
  instructions for installation and configuration.
- Remove epydoc API docs.
- Rework some docstrings to conform to reStructuredText, so they can be
  used in the API docs generated by Sphinx.
- Move all of the top-level text files to reStructuredText so they can
  linked from the Sphinx-based docs and for consistency.
- Remove many obsolete things from the extras/ directory, including old
  installation scripts and migrations.

Many of the installation related documentation and scripts are removed
or adapted in light of the new automated deployment using Ansible.

This is The Good Stuff. The entire test suite can now be run without
having to setup a database, running the batch checker, any of the web
services or the website. It even passes without an internet connection.
In, like, 30 seconds! Awesome!

This means tests don't randomly fail after some reference sequence
changes on the NCBI server and it doesn't take an entire configured
server with mapping database setup to run the tests. Those are things
of the past! No more frustrations, Mutalyzer is testable!

Going down now...

The mountain screamed three times today
I guess it thought it'd like to play
How much does one have to pay
To fry a peak and melt away
Launching titan's breath on mine
The sweating measure lands on time

And the old man, down by the river
Well he walks up and he walks on down
To the spaceship that's parked at your doorstep
And it's waiting to take you away now

Goin' down now
Goin' down now

Looking for the rate that crowed
He's hooked up down in Mexico
Slap my nerve now give me more
It's my disaster friend, not yours

And the old man, down by the river
Well he walks up and he walks on down
To the spaceship that's parked at your doorstep
And it's waiting to take you away now

And the last one, it's down by the river
Where he gets up and he walks on down
To the spaceship that's parked at your doorstep
And it's waiting to take you away now

It's down by the river, it's always this way now
It's down by the river, it's always this way now

Going down now
Going down now
now, now, now

down, down, down

This includes changing a lot of routes and parameter names to be more
consistent. We try to remain backwards compatible as much as possible
by providing redirects from old routes and parameter names.

Port the entire batch job infrastructure, including scheduler, to use
the SQLAlchemy ORM instead of the old Db module.

Remove the dependency on configobj and have default values for all
configuration settings. User settings are defined in a Python module
pointed to by the MUTALYZER_SETTINGS environment variable.

We also clean up many configuration settings and remove some that
are no longer used.

Do not try to implement a full UNIX daemon suitable for init.d with the
batch daemon, but rather keep it simple and depend on something like
supervisord to implement process control.

We now also correctly handle SIGINT (Ctrl+C) and SIGTERM signals.

See http://stackoverflow.com/a/12297929