Don't fix what ain't broken. Unfortunately, string handling in Mutalyzer
really is broken. So we fix it.

Internally, all strings should be represented by unicode strings as much as
possible. The main exception are large reference sequence strings. These can
often better be BioPython sequence objects, since that is how we usually get
them in the first place.

These changes will hopefully make Mutalyzer more reliable in working with
incoming data. As a bonus, they're a first (small) step towards Python 3
compatibility [1].

Our strategy is as follows:

1. We use `from __future__ import unicode_literals` at the top of every file.
2. All incoming strings are decoded to unicode (if necessary) as soon as
   possible.
3. Outgoing strings are encoded to UTF8 (if necessary) as late as possible.
4. BioPython sequence objects can be based on byte strings as well as unicode
   strings.
5. In the database, everything is UTF8.
6. We worry about uploaded and downloaded reference files and batch jobs in a
   later commit.

Point 1 will ensure that all string literals in our source code will be
unicode strings [2].

As for point 4, sometimes this may even change under our eyes (e.g., calling
`.reverse_complement()` will change it to a byte string). We don't care as
long as they're BioPython objects, only when we get the sequence out we must
have it as unicode string. Their contents are always in the ASCII range
anyway.

Although `Bio.Seq.reverse_complement` works fine on Python byte strings (and
we used to rely on that), it crashes on a Python unicode string. So we take
care to only use it on BioPython sequence objects and wrote our own reverse
complement function for unicode strings (`mutalyzer.util.reverse_complement`).

As for point 5, SQLAlchemy already does a very good job at presenting decoding
from and encoding to UTF8 for us.

The Spyne documentation has the following to say about their `String` and
`Unicode` types [3]:

> There are two string types in Spyne: `spyne.model.primitive.Unicode` and
> `spyne.model.primitive.String` whose native types are `unicode` and `str`
> respectively.
>
> Unlike the Python `str`, the Spyne `String` is not for arbitrary byte
> streams. You should not use it unless you are absolutely, positively sure
> that you need to deal with text data with an unknown encoding. In all other
> cases, you should just use the `Unicode` type. They actually look the same
> from outside, this distinction is made just to properly deal with the quirks
> surrounding Python-2's `unicode` type.
>
> Remember that you have the `ByteArray` and `File` types at your disposal
> when you need to deal with arbitrary byte streams.
>
> The `String` type will be just an alias for `Unicode` once Spyne gets ported
> to Python 3. It might even be deprecated and removed in the future, so make
> sure you are using either `Unicode` or `ByteArray` in your interface
> definitions.

So let's not ignore that and never use `String` anymore in our webservice
interface.

For the command line interface it's a bit more complicated, since there seems
to be no reliable way to get the encoding of command line arguments. We use
`sys.stdin.encoding` as a best guess.

For us to interpret a sequence of bytes as text, it's key to be aware of their
encoding. Once decoded, a text string can be safely used without having to
worry about bytes. Without unicode we're nothing, and nothing will help
us. Maybe we're lying, then you better not stay. But we could be safer, just
for one day. Oh-oh-oh-ohh, oh-oh-oh-ohh, just for one day.

[1] https://docs.python.org/2.7/howto/pyporting.html
[2] http://python-future.org/unicode_literals.html
[3] http://spyne.io/docs/2.10/manual/03_types.html#strings

See http://pytest.org/

This is The Good Stuff. The entire test suite can now be run without
having to setup a database, running the batch checker, any of the web
services or the website. It even passes without an internet connection.
In, like, 30 seconds! Awesome!

This means tests don't randomly fail after some reference sequence
changes on the NCBI server and it doesn't take an entire configured
server with mapping database setup to run the tests. Those are things
of the past! No more frustrations, Mutalyzer is testable!

Going down now...

The mountain screamed three times today
I guess it thought it'd like to play
How much does one have to pay
To fry a peak and melt away
Launching titan's breath on mine
The sweating measure lands on time

And the old man, down by the river
Well he walks up and he walks on down
To the spaceship that's parked at your doorstep
And it's waiting to take you away now

Goin' down now
Goin' down now

Looking for the rate that crowed
He's hooked up down in Mexico
Slap my nerve now give me more
It's my disaster friend, not yours

And the old man, down by the river
Well he walks up and he walks on down
To the spaceship that's parked at your doorstep
And it's waiting to take you away now

And the last one, it's down by the river
Where he gets up and he walks on down
To the spaceship that's parked at your doorstep
And it's waiting to take you away now

It's down by the river, it's always this way now
It's down by the river, it's always this way now

Going down now
Going down now
now, now, now

down, down, down

This involves making the SQLAlchemy session reconfigurable at run-time,
which is done automatically on updating the Mutalyzer configuration using
configuration update callbacks.

git-svn-id: https://humgenprojects.lumc.nl/svn/mutalyzer/trunk@528 eb6bd6ab-9ccd-42b9-aceb-e2899b4a52f1

git-svn-id: https://humgenprojects.lumc.nl/svn/mutalyzer/trunk@527 eb6bd6ab-9ccd-42b9-aceb-e2899b4a52f1

The old way of using the configuration file was by instantiating a Config
object which read the file. This instance was passed to every function and
object that might need it.

The new way is by simply calling config.get('name') to get the configuration
value for 'name'. This lazily reads the configuration file and the contents
are cached for future calls.


git-svn-id: https://humgenprojects.lumc.nl/svn/mutalyzer/branches/implicit-config-branch@408 eb6bd6ab-9ccd-42b9-aceb-e2899b4a52f1

git-svn-id: https://humgenprojects.lumc.nl/svn/mutalyzer/branches/refactor-mutalyzer-branch@334 eb6bd6ab-9ccd-42b9-aceb-e2899b4a52f1

git-svn-id: https://humgenprojects.lumc.nl/svn/mutalyzer/branches/refactor-mutalyzer-branch@254 eb6bd6ab-9ccd-42b9-aceb-e2899b4a52f1

git-svn-id: https://humgenprojects.lumc.nl/svn/mutalyzer/branches/refactor-mutalyzer-branch@253 eb6bd6ab-9ccd-42b9-aceb-e2899b4a52f1

Locations and python paths are correct with respect to the setup.py file.
For development, use
  svn up && python setup.py develop
and for deployment use
  svn up && sudo python setup.py install



git-svn-id: https://humgenprojects.lumc.nl/svn/mutalyzer/branches/refactor-mutalyzer-branch@250 eb6bd6ab-9ccd-42b9-aceb-e2899b4a52f1

- Added error recovery to batch daemon.
- Added example init script for batch daemon.


git-svn-id: https://humgenprojects.lumc.nl/svn/mutalyzer/branches/refactor-mutalyzer-branch@244 eb6bd6ab-9ccd-42b9-aceb-e2899b4a52f1

git-svn-id: https://humgenprojects.lumc.nl/svn/mutalyzer/branches/refactor-mutalyzer-branch@225 eb6bd6ab-9ccd-42b9-aceb-e2899b4a52f1

git-svn-id: https://humgenprojects.lumc.nl/svn/mutalyzer/branches/refactor-mutalyzer-branch@224 eb6bd6ab-9ccd-42b9-aceb-e2899b4a52f1

- Ditched use of web.session, just use web.cookies.
- Added detection of configuration file location.
- Removed ./var directory, everything is now in standard locations, but also
  configurable.
- Added more standard version management.



git-svn-id: https://humgenprojects.lumc.nl/svn/mutalyzer/branches/refactor-mutalyzer-branch@222 eb6bd6ab-9ccd-42b9-aceb-e2899b4a52f1

git-svn-id: https://humgenprojects.lumc.nl/svn/mutalyzer/branches/refactor-mutalyzer-branch@218 eb6bd6ab-9ccd-42b9-aceb-e2899b4a52f1

Mostly variant checker, now using exceptions on some places.



git-svn-id: https://humgenprojects.lumc.nl/svn/mutalyzer/branches/refactor-mutalyzer-branch@216 eb6bd6ab-9ccd-42b9-aceb-e2899b4a52f1

git-svn-id: https://humgenprojects.lumc.nl/svn/mutalyzer/branches/refactor-mutalyzer-branch@214 eb6bd6ab-9ccd-42b9-aceb-e2899b4a52f1

git-svn-id: https://humgenprojects.lumc.nl/svn/mutalyzer/branches/refactor-mutalyzer-branch@211 eb6bd6ab-9ccd-42b9-aceb-e2899b4a52f1

This commmit brings in some bugs that take some time to fix. It will be
recommited in a new branch 'exon-deletions-branch' in a few moments.



git-svn-id: https://humgenprojects.lumc.nl/svn/mutalyzer/trunk@197 eb6bd6ab-9ccd-42b9-aceb-e2899b4a52f1

src/Modules/Mutator.py:
- Added method (add_removed_sites) to give splice sites that should be
  ignored. Now newSplice() does not process these splice sites.

src/Mutalyzer.py:
- Handle whole-exon deletions and exon-fusions by removing splice sites
  where we can. A notice is printed in this case. Otherwise, if splice
  sites are hit in some other way, no protein product is predicted. This
  fixes Trac issues #35 and #36.



git-svn-id: https://humgenprojects.lumc.nl/svn/mutalyzer/trunk@195 eb6bd6ab-9ccd-42b9-aceb-e2899b4a52f1

src/Modules/Mutator.py:
- The newSplice() method no longer includes insertions in exons for the
  first and last sites in the given list of splice sites. The result is
  that this inclusion in exons does not apply to translation start/end
  and CDS start/end. This fixes issue #39.



git-svn-id: https://humgenprojects.lumc.nl/svn/mutalyzer/trunk@190 eb6bd6ab-9ccd-42b9-aceb-e2899b4a52f1

src/Modules/Mutator.py:
- For deletions, position shifts are now active from the first position
  following the deletion. Previous behaviour was from (but not including)
  the first position of the deletion itself.
- Added method to check if the resulting shift gets smaller at some
  specific position: Mutator.shift_minus_at(position).
- The positions in the shift list are now interpreted as a shift from
  (and including) the listed position. Previous behaviour was following
  (and not including) the listed position.
- The Mutator.newSplice(sites) method has some additional logic. This
  fixes some bugs, including Trac bug #30.

src/tests/test_mutator.py:
- Written a lot of unit tests for the changes in Mutator.py.



git-svn-id: https://humgenprojects.lumc.nl/svn/mutalyzer/trunk@188 eb6bd6ab-9ccd-42b9-aceb-e2899b4a52f1