Hack images, store outputs in git again for now

Easier to track what each script change does if we keep the before and
after versions of the markdown in git too.  Clean this up eventually,
but simplifies development.

1 files changed, 229 insertions, 0 deletions

diff --git a/markdown/BuildingFromSource.md b/markdown/BuildingFromSource.md
new file mode 100644
index 0000000..8f2a4fd
--- /dev/null
+++ b/markdown/BuildingFromSource.md
@@ -0,0 +1,229 @@
+[[PageOutline]]
+
+# Building Cryptech Software/Firmware/Bitstream From Source
+
+Everything you need to build our software, firmware, and FPGA
+bitstreams from source yourself is publicly available, but the process
+is a bit complicated.  Overall, there are two methods, one of which
+our developers use while writing this stuff, the other of which we use
+for the automated reproducible builds which go into our binary
+distributions.  Both methods eventually boil down to "get the source
+code then run make", but the details differ.
+
+## What developers do
+
+We check out copies of all the several dozen separate repositories and
+carefully arrange them in a tree structure which matches the official
+naming scheme.  Yes, really.  It's tedious, but we have
+[export:/user/sra/build-tools/https-sync-repos.py a script to automate this].
+Be warned that this script is a kludge which relies on parsing
+XML from this Wiki; this is nasty, but reasonably stable, because the
+XML itself is generated by another script.
+
+Once you have this tree, you can hop around within it, building
+whichever bits are of interest to you.  So if you want to rebuild just
+the HSM firmware (the C code that runs on the ARM), you would go to
+`sw/stm32` and run `make` there.
+
+## What we do for reproducible builds
+
+Reproducible builds use the same tree structure (as they must for the
+various Makefiles to work properly), but the entire tree is embedded
+in a git "superrepository" which also contains the release engineering
+goo necessary to make the whole thing work.  Do `git help submodule`
+for an introduction to git's submodule mechanism.
+
+With this model, one just checks out a copy of
+[source:/releng/alpha the superrepository],
+runs `make` in its top directory, and eventually
+the complete package pops out the other side.
+
+```
+git clone https://git.cryptech.is/releng/alpha.git
+cd alpha
+make
+```
+
+That's the good news.  The bad news is that this process has higher
+demands on its build environment: it expects to find the a complete
+tool set, including the XiLinx synthesis tools, the several different
+cross compilers for the firmware, and the `pbuilder` system for
+building clean room packages for Ubuntu and Debian.
+
+As a compromise, one can use this source tree as if it were the
+development source tree described above: just use the supermodule to
+pull down everything else, but then ignore the supermodule and build
+individual pieces as if you'd checked out all the repositories by
+hand.
+
+## Skip all this git mess and just download a tarball
+
+There's another alternative, which is simpler than any of the above:
+just download the source tarball.  Since the only build environments
+we support at the moment are Debian Jessie and Ubuntu Xenial, which
+also happen to be environments for which we build binary packages, you
+can just use APT:
+
+```
+apt-get source cryptech-alpha
+```
+
+Which will give you the same tree structure, but without all the git fun.
+
+## Build environment
+
+Our software and firmware developers use the Debian and Ubuntu Linux
+distributions.  Our current build box for binary packages runs Debian
+Jessie.
+
+Our Verilog developers use various environments and have been known to
+use graphical tools, but synthesis of the bitstreams that go in our
+binary packages is done via the XiLinx command line tools on the same
+Debian Jessie machine as the software and firmware builds.
+
+Which tools you need will of course depend on exactly what you're
+trying to do.
+
+Most of the tools work on either 32-bit or 64-bit machines, but if you
+intend to run the full binary package build script, you'll need a
+64-bit machine (or VM) because the tools won't build 64-bit binaries
+on a 32-bit machine.
+
+Basic tool set (not all required for every purpose, but they're all
+supported Debian packages so it's usually easier just to install them
+all and not worry about it):
+
+```
+  apt-get install git pbuilder ubuntu-dev-tools rsync sudo
+  apt-get install python-yaml python-serial python-crypto python-ecdsa
+  apt-get install gcc-arm-none-eabi gdb-arm-none-eabi
+  apt-get install gcc-avr binutils-avr avr-libc
+  apt-get -t jessie-backports install debootstrap distro-info-data
+  apt-get install reprepro ubuntu-archive-keyring
+```
+
+This is not an exhaustive list, because some of the other packages we
+use are pulled in by these as dependencies.
+
+You will also need a copy of the XiLinx tools, which is tedious enough
+that it's described in a separate section, below.
+
+Once you have all the tools installed, you'll need a copy of the
+source tree, as explained in the preceeding sections.
+
+pbuilder requires a bit of setup (you can skip this if you're not
+trying to do the full binary package build):
+
+```
+for code in jessie xenial; do for arch in i386 amd64; do pbuilder-dist $code $arch create; done; done
+ln -s jessie_result ~/pbuilder/jessie-amd64_result
+ln -s xenial_result ~/pbuilder/xenial-amd64_result
+```
+
+## Installing the XiLinx tools
+
+XiLinx tools setup is a bit involved.  You can skip this section if
+you don't intend to build FPGA bitstreams.
+
+We use the command line versions of the XiLinx tools, but installing
+them requires a graphical environment, because the XiLinx installer
+and license manager are GUI tools.  If you're running this on a server
+and don't already have a graphical environment installed, you can get
+away with something fairly minimal.  For example, if you have a VNC
+viewer such as "Chicken of the VNC" on your laptop, you can get away
+with a fairly minimal X11 toolset:
+
+```
+apt-get install tightvncserver xterm icewm
+```
+
+If you're already running X11 on your laptop and are comfortable with
+extruding that to the build machine, eg, via `ssh -Y`, you can just
+use that (not recommended for long-haul use, eg, if the laptop is in
+Boston and the server is in Reykjavik).
+
+You'll need to start by using a web browser to download the
+[Xilinx ​ISE Design Suite](http://www.xilinx.com/support/download/index.html/content/xilinx/en/downloadNav/design-tools.html).
+
+XiLinx only supports specific versions of Red Hat and Suse Linux, but
+their tools do run on Debian and Ubuntu.  A few  caveats:
+
+
+* Debian and Ubuntu symlink `/bin/sh` to `/bin/dash`, which can't handle
+
+  some of the syntax used in XiLinx's shell scripts, so you'll need to
+  change that symlink to point to `/bin/bash`.
+
+
+* Although the XiLinx software can be installed as user or root, by
+
+  default it wants to install into /opt/Xilinx, so you need to install
+  as root if you want to do that.
+
+
+* The XiLinx tools are disk hogs, so if you're building a VM for this,
+
+  you'll probably want to give it at least 30-40 GB of disk space.
+
+Step-by-step installation:
+
+1. Unpack `Xilinx_ISE_DS_Lin_14.7_1015_1.tar` (or whatever version you have).
+2. In an X11 environment, cd to `Xilinx_ISE_DS_Lin_14.7_1015_1`, and run `sudo ./xsetup`
+3. Click through two screens of license agreements.
+4. Select `ISE WebPACK`.
+5. Unselect (or leave unselected) Install Cable Drivers.
+6. Go!
+
+Well, not quite.  You'll need to convince the ISE tools that you have
+a valid license to use the ISE tools.  Go to
+http://www.xilinx.com/products/design-tools/ise-design-suite/ise-webpack.htm,
+click the `Licensing Solutions` link.  On the page to which that takes
+you, expand the section `Obtain a license for Free or Evaluation
+product`.  To download the ISE Webpack, you should have created an
+account, so now you can go to the Licensing Site and use that account
+to create a Certificate Based License.
+
+You do not need to go through the HostID dance, just say Do It. You
+will then receive a certificate in email (not an X.509 certificate)
+which you will be able to use. Then start the ISE Webpack by issuing
+the command `ise`.  Go to the Help menu and Manage Licenses. Use the
+resulting new License Manager window to install the `.lic` file.  This
+process is complex and flakey.
+
+Here's
+[another description of installing ISE on Ubuntu](http://www.armadeus.com/wiki/index.php?title=ISE_WebPack_installation_on_Linux).
+
+The `ise` binary referred to above is in `/opt/Xilinx/14.7/ISE_DS/ISE/bin/lin64/ise`
+(or in `.../lin/ise`, but the pbuilder setup requires a 64-bit build machine).
+
+When running this remotely under tightvncserver, setup looks something like this:
+
+```
+vncserver :0 -geometry 1280x768 -depth 16 -localhost
+export DISPLAY=:0 XAUTHORITY=~/.Xauthority
+icewm&
+```
+
+Then, either in the same shell as the above or in an xterm in the new display
+
+```
+cd Xilinx_ISE_DS_Lin_14.7_1015_1
+sudo ./xsetup
+
+cd
+/opt/Xilinx/14.7/ISE_DS/ISE/bin/lin64/ise
+```
+
+It turns out you don't really need to run the whole `ise` tool to
+get to the license manager, you can just run
+
+```
+/opt/Xilinx/14.7/ISE_DS/common/bin/lin64/xlcm -manage
+```
+
+But you do have to source the appropriate settings file first, none of
+the XiLinx tools work properly without that:
+
+```
+. /opt/Xilinx/14.7/ISE_DS/settings64.sh
+```