|
|
|
# Welcome to Wiki Shark cluster
|
|
|
|
## New
|
|
|
|
# Welcome to the old Wiki Shark cluster
|
|
|
|
## New cluster is operational
|
|
|
|
Shark will be renewed this year with a new scheduler SLURM in a new network environment.
|
|
|
|
People who would like to take a look and try the new scheduler please read the new [SLURM wiki](https://git.lumc.nl/shark/SHARK/wikis/slurm/slurm_home).
|
|
|
|
This new SLURM cluster is a test cluster this means we cannot guarantee 100% uptime.
|
|
|
|
|
|
|
|
## Introduction
|
|
|
|
In this wiki, we will introduce the Shark cluster. We assume you know your way around in the terminal and that you will use the cluster in a responsible way and respect the available resources. Remember that a cluster runs many different processes submitted by many different users. If you are not sure what you are doing, ask your administrator before submitting very intensive jobs! For more examples and useful commands you can use our [gitlab repository](git@git.lumc.nl:shark/SHARK.git).
|
|
|
|
|
|
|
|
Shark cluster runs with [Open Grid Scheduler/Grid Engine](http://gridscheduler.sourceforge.net/) this is a free and open-source batch-queuing system for distributed resource management. OGS/GE is based on Sun Grid Engine, and maintained by the same group of external (i.e. non-Sun) developers who started contributing code to Sun Grid Engine since 2001.
|
|
|
|
|
|
|
|
**Current release of Open Grid Scheduler** : *Grid Engine 2011.11p1*
|
|
|
|
|
|
|
|
## Introduction and interesting starting points
|
|
|
|
The following links contain the minimal information you need to get started on the shark cluster.
|
|
|
|
When using the cluster you acknowledge that you have read and understand the following articles.
|
|
|
|
|
|
|
|
* [Guidelines](Guidelines) -- what you can and can't do on this cluster
|
|
|
|
* [Function of the Shark cluster](Primairy) -- Primary usage Shark cluster
|
|
|
|
* [Network shares](Network_shares) -- where to put your data
|
|
|
|
* [Queues](Queues) -- which queue is for what
|
|
|
|
* [Batch queuing commands](qsubCommands) -- explanation of the qsub command
|
|
|
|
* [Examples](Examples) -- examples how to qsub your jobs
|
|
|
|
* [Makefile](Makefile) -- Howto use qmake with a Makefile
|
|
|
|
* [SGE state symbol codes](StateCode) -- Sun Grid Engine SGE state letter symbol codes meanings
|
|
|
|
* [FAQ](FAQ) -- Frequently asked questions and answers
|
|
|
|
* [Presentations](Presentations) -- Interesting presentations
|
|
|
|
* [Testsets](testsets) -- Featured test data
|
|
|
|
|
|
|
|
### For [FSW users](https://git.lumc.nl/mwjhhuijberts/Shark4FSW/) there is a dedicated Wiki page.
|
|
|
|
|
|
|
|
## Getting started
|
|
|
|
|
|
|
|
##### Accounting
|
|
|
|
* [How to get a user account on the Shark Cluster](ResearchAccount)
|
|
|
|
* [Short intro course for new Shark users](Intro_course) -- 2hr. intro course mandatory for new Users.
|
|
|
|
* [Shark Introduction course schedule 2020](Intro_schedule_2020)
|
|
|
|
* [How to configure public key authentication](PublicKeyAuth)
|
|
|
|
* [How to change your researchlumc password](ChangePasswd)
|
|
|
|
With a new cluster comes a new WIKI : [SLURM wiki](https://git.lumc.nl/shark/shark-centos-slurm-user-guide/-/wikis/home).
|
|
|
|
|
|
|
|
Please start working on the new cluster instead of using the old shark environment.
|
|
|
|
|
|
|
|
##### Connecting
|
|
|
|
* [How to connect to the Shark Cluster](Howtoconnect)
|
|
|
|
* [How to access grid storage](AccessingGridStorage)
|
|
|
|
* [Graphical access SHARK cluster](Graphical)
|
|
|
|
|
|
|
|
##### Usage
|
|
|
|
* [How to use the Evironment Modules](EnvironmentModules)
|
|
|
|
* [Forking an input stream to several programs](StreamForking)
|
|
|
|
* [How to submit a R-script with qsub](qsub_R-script)
|
|
|
|
* [How to submit a MPICH2 job](MPICH2)
|
|
|
|
* [How to start an IPython Notebook session](IPythonNotebook)
|
|
|
|
* [How to correct long PacBio reads using pacBioToCA](pacBioToCA)
|
|
|
|
* [Using the fsl analysis tools for FMRI, MRI and DTI brain imaging data](UsingFSL)
|
|
|
|
|
|
|
|
## Configuration Shark cluster
|
|
|
|
* [Configuration Type of headnode and blade servers](Configuration)
|
|
|
|
* [Graphical representation of the shark cluster](Graphicalview)
|
|
|
|
* [Description of the execute nodes](DescriptionExenode)
|
|
|
|
* [Hardware ownership](HardwareOwnership)
|
|
|
|
New login node : res-hpc-lo01.researchlumc.nl
|
|
|
|
|
|
|
|
## Limitations
|
|
|
|
* [How to use more then the default memory allocation for my job](MemoryLimitations)
|
|
|
|
|
|
|
|
## Support
|
|
|
|
* [Contact Information](Contact_info_shark)
|
|
|
|
* [Ganglia Shark cluster monitor](http://shark.lumcnet.prod.intern/ganglia/?c=Shark%20Cluster&m=load_one&r=hour&s=descending&hc=4&mc=2) -- view the load, disk space etc. on the Shark cluster
|
|
|
|
* [Grid Engine HOWTOs](http://arc.liv.ac.uk/SGE/howto/).
|
|
|
|
* [Gitlab repository (for course material and example scripts).](git@git.lumc.nl:shark/SHARK.git)
|
|
|
|
|
|
|
|
## Usage Statistics
|
|
|
|
* [Shark Usage 2012](http://shark.lumcnet.prod.intern/stats/SharkStats2012.html)
|
|
|
|
|
|
|
|
## Usefull Links
|
|
|
|
* [Wiki gridengine](http://wiki.gridengine.info/wiki/index.php/Main_Page)
|
|
|
|
* [This is an archive of Stephan Grell's blog posts.](http://wiki.gridengine.info/wiki/index.php/StephansBlog)
|
|
|
|
* [sge_intro(1) - Linux man page](http://linux.die.net/man/1/sge_intro)
|
|
|
|
|
|
|
|
## Documentation
|
|
|
|
* Setup and install the [Shark](Shark_setup) head node with Ubuntu 10.04.04 LTS and Open Grid Scheduler
|
|
|
|
* Setup [LDAP/Kerberos authentication](Ubuntu) within a Microsoft Windows 2008 Domain
|
|
|
|
* Open Grid Scheduler [FineTune and solutions](FineTune/Solutions)
|
|
|
|
* [Shark Ascii logos](SharkLogos) |
