These document pages can be downloaded by anyone with access to the MPS Git repository. If you do not already have access to this please read this short section: Getting Access
All of the documentation is written in a form of plain text markup, that is then converted into HTML (or latex, epub, man pages...) using a project called Sphinx.
The main advantage of this over something like a wiki is:
- Offline editing in whatever text editor you wish, with all the comfort and convenience your personal writing environment may bring. No more losing hours of writing due to a browser page refresh.
- Better for long writing sessions. You write your edits in your local git checkout of the documentation, so you don’t ‘lock’ a page preventing someone else form editing it, leaving you free to work on a page of documentation for as long as you like. This is very advantageous for when documentating a piece of work as you do it, so you can leave your editor open and update as you progress through the task.
- The structure of the entire documentation is implicit in the way it is constructed, so finding a page in the hierarchy is more obvious. WIth a wiki, any structure to the pages has to be imposed and regularly maintained through internal hyperlinks, which rarely happens or is maintained, leaving people to have to bookmark any pages of use to ensure they can find them consistently.
Getting the Docs¶
The only prerequisite to contributing is to install the version control system ‘git’ on your computer. For linux this is packaged for all distributions, but for Mac and Linux, you likely need to download this from here: http://git-scm.com/downloads
You need access to the MPS git repo, see here if you don’t already have this: Getting Access
Downloading the docs is simple, just checkout the documentation repo from the git server,
## This is the link to the HPC User Guide documentation (info.hpc.susx.ac.uk/guide) $ git clone email@example.com:sites/hpc-guide ## There are two further documentation sites at present ## HPC Sysadmin notes (info.hpc.susx.ac.uk/admin) $ git clone firstname.lastname@example.org:sites/hpc-admin ## MPS specific User documentation notes (info.hpc.susx.ac.uk/mps) $ git clone email@example.com:sites/mps
One of the main benefits of the Sphinx style documentation generators is that documentation structure is an inherent property, not something that has to be tacked on afterwards. The structure of the site is reflected in the filesystem, with every page having it’s own file.
After cloning the docs to your machine, you can browse around the project. As of writing the top level directory likes like this:
$ ls hpc-guide best-practices.rst faq.rst help.rst Makefile requirements.txt storage.rst _build getting-access.rst index.rst parallel.rst running-jobs.rst structure.rst conf.py git.rst interactive.rst queues.rst software.rst _templates contributing.rst gpu.rst lustre.rst remote-access.rst _static
The top-level layout of the site is contained in index.rst which contains a list of the first-level sections which appear in the left sidebar of the site. This is also the page you land on when you go to the root of the site.
The other files (eg: Makefile, requirements.txt conf.py etc... are all relevant only if you want to build the documentation locally – this is useful once you get into using this system, as you can work completely offline and see how your edits look in html on your computer without having to push anything live yet.
Making an edit¶
Before starting work you want to make sure you have an up to date copy of the documents from the git server, so run:
$ cd ~/path/to/site $ git pull
If you just want to edit a page, simply open the .rst file for the corresponding page and edit it in your favorite editor. There is a Page Template you can use that covers the most common syntax - you can click on the View page source link at the top right of every page to see the raw text too.
Then push that contribution back to the site,
$ git commit -a -m "Add content" $ git push
And that’s it. The push to the server will trigger a rebuild of the site and your changes will be pushed live immediately.
To simplify this further, you can wrap all of this up into one single command, by adding the following line to your ~/.gitconfig,
$ vim ~/.gitconfig # Add the following lines [alias] cc = !git pull && git add -u && git commit -m 'Adding content' && git push
Then you can just run git cc from within the project to checkout and push back your changes.
If this is the first time you have used git, run the following commands to set up your identity for git:
$ git config --global user.name "John Doe" $ git config --global user.email firstname.lastname@example.org
If you have used git before then skip the following. This section is intended as a quick git primer for people who haven’t used it to handle merging.
Very rarely, you may need to use a bit more of git to handle a merge conflict. This would only happen if you were changing the same line of text as someone else around the same time. However git, being designed for source control management, is designed for this.
Here’s an example:
You have a file in the project called lustre.rst that you want to work on while you are travelling on the train to a conference. So you have the site checked out on your laptop already, but since you don’t have any network connection at the moment, you can’t checkout the latest revision of the site.
Still you can open the file and write you content and commit it to git like outlined above, just you can’t fulfill the git pull and git push at the moment.
$ vim lustre.rst ## Fixing a URL $ git commit -a -m 'Adding content' $ git show HEAD commit e0ed254bc0499f9768cdef8c5621e11f25b0faf6 Author: Matt Raso-Barnett <email@example.com> Date: Fri Aug 15 08:38:27 2014 +0100 Adding content diff --git a/lustre.rst b/lustre.rst index ba34252..263f950 100644 --- a/lustre.rst +++ b/lustre.rst @@ -10,7 +10,7 @@ For lustre mds/oss installations a patched kernel is required before installing lustre, however for a lustre client, this is not required. There are some pre-built RPMs from here: -`http://build.whamcloud.com/`_ +http://build.whamcloud.com/ but for the 1.8 series of lustre, these are not built against the current SL 6.5 kernel.
You then work on something else for the next 6 hours while you are travelling. The next day when you have network again, you go back to this project and try to push your changes up to the server:
$ git cc ## or just git push if you haven't set up the alias like above remote: Counting objects: 5, done. remote: Compressing objects: 100% (3/3), done. remote: Total 3 (delta 2), reused 0 (delta 0) Unpacking objects: 100% (3/3), done. From git.hpc.susx.ac.uk:sites/cluster-notes af49925..4f588e1 master -> origin/master Auto-merging lustre.rst CONFLICT (content): Merge conflict in lustre.rst Automatic merge failed; fix conflicts and then commit the result.
however what we find here is that someone else has made a conflicting change already to the server. This might happen if someone changed the URL we edited above to something else,
We can see this by running,
$ git status On branch master Your branch and 'origin/master' have diverged, and have 1 and 1 different commit each, respectively. (use "git pull" to merge the remote branch into yours) You have unmerged paths. (fix conflicts and run "git commit") Unmerged paths: (use "git add <file>..." to mark resolution) both modified: lustre.rst no changes added to commit (use "git add" and/or "git commit -a") $ git diff lustre.rst diff --cc lustre.rst index 263f950,951920e..0000000 --- i/lustre.rst +++ w/lustre.rst @@@ -10,7 -10,7 +10,11 @@@ For lustre mds/oss installations a patc installing lustre, however for a lustre client, this is not required. There are some pre-built RPMs from here: ++<<<<<<< HEAD +http://build.whamcloud.com/ ++======= + https://build.hpdd.intel.com/ ++>>>>>>> b9d7072f0237b78d0dd047285516a8240bef4f38 but for the 1.8 series of lustre, these are not built against the current SL 6.5 kernel.
So we can see that someone made the same fix as ourselves in removing the backticks from the URL, but they also changed the URL. If you open lustre.rst in your text editor you will have a block at that line that looks like the diff output above.
To resolve the merge conflict it is as simple as editing this file to how it should be after the merge. If your remote colleague was correct to change the URL as well as removing the backticks, then you want to keep their change:
$ vim lustre.rst ## Delete everything except for the line you want to keep $ cat lustre.rst ... snip ... There are some pre-built RPMs from here: https://build.hpdd.intel.com/ ... snip ... $ git add lustre.rst $ git commit ## This will open a window in your EDITOR (eg: vim) with the following content. ## Just save and close this window to complete the commit. Merge branch 'master' of git.hpc.susx.ac.uk:sites/cluster-notes Conflicts: lustre.rst # # It looks like you may be committing a merge. # If this is not correct, please remove the file # .git/MERGE_HEAD # and try again. # Please enter the commit message for your changes. Lines starting # with '#' will be ignored, and an empty message aborts the commit. # On branch master # Your branch and 'origin/master' have diverged, # and have 1 and 1 different commit each, respectively. # (use "git pull" to merge the remote branch into yours) # # All conflicts fixed but you are still merging. # # Changes to be committed: # modified: lustre.rst #
and you have completed the merge. The conflict has been resolved, you have told git which edit to choose from, and all it good. All that is left is to push this back to the server (all this has been done locally at the moment so you can still go back and change things if you decide you made the wrong decision),
$ git push
and you are done.
This is a very verbose description of the process, but for this kind of project, you will only have to do this very rarely, if all. But git is a very powerful tool, so it’s useful to show to how to easily it handles the situation.