Contributing

These document pages can be downloaded by anyone with access to the MPS Git repository. If you do not already have access to this please read this short section: Getting Access

All of the documentation is written in a form of plain text markup, that is then converted into HTML (or latex, epub, man pages...) using a project called Sphinx.

The main advantage of this over something like a wiki is:

  • Offline editing in whatever text editor you wish, with all the comfort and convenience your personal writing environment may bring. No more losing hours of writing due to a browser page refresh.
  • Better for long writing sessions. You write your edits in your local git checkout of the documentation, so you don’t ‘lock’ a page preventing someone else form editing it, leaving you free to work on a page of documentation for as long as you like. This is very advantageous for when documentating a piece of work as you do it, so you can leave your editor open and update as you progress through the task.
  • The structure of the entire documentation is implicit in the way it is constructed, so finding a page in the hierarchy is more obvious. WIth a wiki, any structure to the pages has to be imposed and regularly maintained through internal hyperlinks, which rarely happens or is maintained, leaving people to have to bookmark any pages of use to ensure they can find them consistently.

Getting the Docs

Note

The only prerequisite to contributing is to install the version control system ‘git’ on your computer. For linux this is packaged for all distributions, but for Mac and Linux, you likely need to download this from here: http://git-scm.com/downloads

You need access to the MPS git repo, see here if you don’t already have this: Getting Access

Downloading the docs is simple, just checkout the documentation repo from the git server,

## This is the link to the HPC User Guide documentation (info.hpc.susx.ac.uk/guide)
$ git clone git@git.hpc.susx.ac.uk:sites/hpc-guide

## There are two further documentation sites at present

## HPC Sysadmin notes (info.hpc.susx.ac.uk/admin)
$ git clone git@git.hpc.susx.ac.uk:sites/hpc-admin
## MPS specific User documentation notes (info.hpc.susx.ac.uk/mps)
$ git clone git@git.hpc.susx.ac.uk:sites/mps

Document Structure

One of the main benefits of the Sphinx style documentation generators is that documentation structure is an inherent property, not something that has to be tacked on afterwards. The structure of the site is reflected in the filesystem, with every page having it’s own file.

After cloning the docs to your machine, you can browse around the project. As of writing the top level directory likes like this:

$ ls hpc-guide
best-practices.rst  faq.rst             help.rst         Makefile           requirements.txt  storage.rst
_build              getting-access.rst  index.rst        parallel.rst       running-jobs.rst  structure.rst
conf.py             git.rst             interactive.rst  queues.rst         software.rst      _templates
contributing.rst    gpu.rst             lustre.rst       remote-access.rst  _static

All the pages of the site are in files with the extension .rst, which stands for reStructuredText, which is a type of plaintext markup language, similar to Markdown.

The top-level layout of the site is contained in index.rst which contains a list of the first-level sections which appear in the left sidebar of the site. This is also the page you land on when you go to the root of the site.

The other files (eg: Makefile, requirements.txt conf.py etc... are all relevant only if you want to build the documentation locally – this is useful once you get into using this system, as you can work completely offline and see how your edits look in html on your computer without having to push anything live yet.

Making an edit

Before starting work you want to make sure you have an up to date copy of the documents from the git server, so run:

$ cd ~/path/to/site
$ git pull

If you just want to edit a page, simply open the .rst file for the corresponding page and edit it in your favorite editor. There is a Page Template you can use that covers the most common syntax - you can click on the View page source link at the top right of every page to see the raw text too.

Then push that contribution back to the site,

$ git commit -a -m "Add content"
$ git push

And that’s it. The push to the server will trigger a rebuild of the site and your changes will be pushed live immediately.

To simplify this further, you can wrap all of this up into one single command, by adding the following line to your ~/.gitconfig,

$ vim ~/.gitconfig

# Add the following lines
  [alias]
          cc = !git pull && git add -u && git commit -m 'Adding content' && git push

Then you can just run git cc from within the project to checkout and push back your changes.

Note

If this is the first time you have used git, run the following commands to set up your identity for git:

$ git config --global user.name "John Doe"
$ git config --global user.email johndoe@example.com

Merging Changes

If you have used git before then skip the following. This section is intended as a quick git primer for people who haven’t used it to handle merging.

Very rarely, you may need to use a bit more of git to handle a merge conflict. This would only happen if you were changing the same line of text as someone else around the same time. However git, being designed for source control management, is designed for this.

Here’s an example:

You have a file in the project called lustre.rst that you want to work on while you are travelling on the train to a conference. So you have the site checked out on your laptop already, but since you don’t have any network connection at the moment, you can’t checkout the latest revision of the site.

Still you can open the file and write you content and commit it to git like outlined above, just you can’t fulfill the git pull and git push at the moment.

$ vim lustre.rst  ## Fixing a URL
$ git commit -a -m 'Adding content'
$ git show HEAD
  commit e0ed254bc0499f9768cdef8c5621e11f25b0faf6
  Author: Matt Raso-Barnett <matthewrasobarnett@gmail.com>
  Date:   Fri Aug 15 08:38:27 2014 +0100

      Adding content

  diff --git a/lustre.rst b/lustre.rst
  index ba34252..263f950 100644
  --- a/lustre.rst
  +++ b/lustre.rst
  @@ -10,7 +10,7 @@ For lustre mds/oss installations a patched kernel is required before
   installing lustre, however for a lustre client, this is not required.

   There are some pre-built RPMs from here:
  -`http://build.whamcloud.com/`_
  +http://build.whamcloud.com/

   but for the 1.8 series of lustre, these are not built against the
   current SL 6.5 kernel.

You then work on something else for the next 6 hours while you are travelling. The next day when you have network again, you go back to this project and try to push your changes up to the server:

$ git cc ## or just git push if you haven't set up the alias like above
remote: Counting objects: 5, done.
remote: Compressing objects: 100% (3/3), done.
remote: Total 3 (delta 2), reused 0 (delta 0)
Unpacking objects: 100% (3/3), done.
From git.hpc.susx.ac.uk:sites/cluster-notes
   af49925..4f588e1  master     -> origin/master
Auto-merging lustre.rst
CONFLICT (content): Merge conflict in lustre.rst
Automatic merge failed; fix conflicts and then commit the result.

however what we find here is that someone else has made a conflicting change already to the server. This might happen if someone changed the URL we edited above to something else,

We can see this by running,

$ git status
On branch master
Your branch and 'origin/master' have diverged,
and have 1 and 1 different commit each, respectively.
  (use "git pull" to merge the remote branch into yours)

You have unmerged paths.
  (fix conflicts and run "git commit")

Unmerged paths:
  (use "git add <file>..." to mark resolution)

  both modified:      lustre.rst

no changes added to commit (use "git add" and/or "git commit -a")

$ git diff lustre.rst
diff --cc lustre.rst
index 263f950,951920e..0000000
--- i/lustre.rst
+++ w/lustre.rst
@@@ -10,7 -10,7 +10,11 @@@ For lustre mds/oss installations a patc
  installing lustre, however for a lustre client, this is not required.

  There are some pre-built RPMs from here:
++<<<<<<< HEAD
 +http://build.whamcloud.com/
++=======
+ https://build.hpdd.intel.com/
++>>>>>>> b9d7072f0237b78d0dd047285516a8240bef4f38

  but for the 1.8 series of lustre, these are not built against the
  current SL 6.5 kernel.

So we can see that someone made the same fix as ourselves in removing the backticks from the URL, but they also changed the URL. If you open lustre.rst in your text editor you will have a block at that line that looks like the diff output above.

To resolve the merge conflict it is as simple as editing this file to how it should be after the merge. If your remote colleague was correct to change the URL as well as removing the backticks, then you want to keep their change:

$ vim lustre.rst  ## Delete everything except for the line you want to keep
$ cat lustre.rst
... snip ...
There are some pre-built RPMs from here:
https://build.hpdd.intel.com/
... snip ...

$ git add lustre.rst
$ git commit
## This will open a window in your EDITOR (eg: vim) with the following content.
## Just save and close this window to complete the commit.

Merge branch 'master' of git.hpc.susx.ac.uk:sites/cluster-notes

Conflicts:
        lustre.rst
#
# It looks like you may be committing a merge.
# If this is not correct, please remove the file
#       .git/MERGE_HEAD
# and try again.


# Please enter the commit message for your changes. Lines starting
# with '#' will be ignored, and an empty message aborts the commit.
# On branch master
# Your branch and 'origin/master' have diverged,
# and have 1 and 1 different commit each, respectively.
#   (use "git pull" to merge the remote branch into yours)
#
# All conflicts fixed but you are still merging.
#
# Changes to be committed:
#       modified:   lustre.rst
#

and you have completed the merge. The conflict has been resolved, you have told git which edit to choose from, and all it good. All that is left is to push this back to the server (all this has been done locally at the moment so you can still go back and change things if you decide you made the wrong decision),

$ git push

and you are done.

This is a very verbose description of the process, but for this kind of project, you will only have to do this very rarely, if all. But git is a very powerful tool, so it’s useful to show to how to easily it handles the situation.