<?xml version="1.0" encoding="utf-8"?>
<feed xmlns="http://www.w3.org/2005/Atom">
  <title>Pointed Set blog</title>
  <link href="https://www.pointedset.ca/blog/index.html"/>
  <link rel="self" href="https://www.pointedset.ca/blog/atom.xml"/>
  <id>tag:pointedset.ca,2018:blog/feed</id>
  <updated>2021-03-07T00:32:43.537131+00:00</updated>
  <author><name>Mathieu Guay-Paquet</name></author>
  <entry>
    <title>Does this count as a self-hosting compiler?</title>
    <link href="https://www.pointedset.ca/blog/2021/03/06/byter.html"/>
    <id>tag:pointedset.ca,2021-03-06:is-this-a-compiler</id>
    <updated>2021-03-06T10:35:22+00:00</updated>
    <content><![CDATA[<p>Today's goal <i>[edit: this took a couple of days actually]</i> is to write a text-to-binary converter, in x86-64 assembly, as an ELF executable file.
<h2 id="input-and-output-format">Input and output format</h2>
<p>Following the sage advice of Leslie Lamport, let's <a href="https://www.microsoft.com/en-us/research/publication/state-problem-describing-solution/">State the Problem Before Describing the Solution</a>, by describing the input format and the expected output of this program, before we write any code.
<ul>
<li>An input file is a sequence of zero or more newline-terminated lines.
<li>A line contains either a comment or numbers.
<li>A comment line starts with '#' and ends at the first newline character after that. In between, all bytes are allowed, including NUL.
<li>A numbers line contains a sequence of zero or more numbers, separated by spaces, with possible leading spaces. But no trailing spaces, we're not savages.
<li>A number is a pair of hex digits <code>[0-9A-Fa-f]</code>.
</ul>
<pre><code>input   = line*
line    = (empty | comment | numbers) [\n]
empty   =
comment = [#] [^\n]*
numbers = [ ]* number ([ ]+ number)*
number  = digit digit
digit   = [0-9A-Fa-f]
</code></pre>
<p>The output is simply a sequence of bytes, one for each number which appears on a non-comment line, in order.
<p>The input is taken from <code>stdin</code>, and output is sent to <code>stdout</code>. The program exits with status 0 on success, and a nonzero status on failure (in which case some bytes may have been written to <code>stdout</code>).
<h2 id="finite-state-machine">Finite state machine</h2>
<p>The input format is very simple, so we can parse it with a (very) finite state machine. If we encode the states as powers of two (so, 1, 2, 4, 8, 16), then we can test whether the current state is one of an arbitrary set of states with a single instruction, so let's do that. The possible states are:
<ol>
<li value=1>Beginning of line. This happens at the start of input, and also after every newline. This state accepts the characters <code>[\n# 0-9A-Fa-f]</code> and end-of-file.
<li value=2>Inside a comment. This state accepts all characters, but <code>[\n]</code> is treated specially.
<li value=4>Waiting for a number. This happens after a space. This state accepts the characters <code>[ 0-9A-Fa-f]</code>.
<li value=8>Inside a number. This happens after the first digit. This state accepts the characters <code>[0-9A-Fa-f]</code>.
<li value=16>After a number. This happens after the second digit. This state accepts the characters <code>[\n ]</code>.
</ol>
<h2 id="main-loop">Main loop</h2>
<p>The main loop of the program is very simple:
<ol>
<li>Read some input characters into a buffer. If there are none, exit.
<li>Feed each input character into the finite state machine, potentially producing an output character.
<li>Write out the output buffer, then go back to step 1.
</ol>
<h2 id="memory-layout">Memory layout</h2>
<p>We'll be using a fixed size buffer (one page, 4KB) to read the input, to avoid doing a syscall for every single character of input.
<p>Since every byte of input produces at most one byte of output, and no backtracking is necessary, we can just use the same buffer to gather output bytes before doing a syscall to write them out. This means that our memory allocation state is always captured by just a few pointers, which keep the same relative ordering throughout program execution:
<ol>
<li>Start of the 4KB buffer (constant).
<li>End of buffered output bytes (variable).
<li>Start of buffered input bytes (variable).
<li>End of buffered input bytes (variable).
<li>End of the 4KB buffer (constant).
</ol>
<p>And if we keep all addresses within the first 2GB of virtual memory (those addresses which can be represented as positive signed 32-bit pointers), we can manipulate our 64-bit pointers using 32-bit registers, which will save us from using a bunch of REX instruction prefixes.
<h2 id="how-to-read">How to read</h2>
<p>The simplest way to read bytes from <code>stdin</code> is with the <code>read</code> syscall, but it's important to pay attention to the return value:
<ul>
<li>return value 0 means end-of-file;
<li>return value -4 means <code>EINTR</code>, aka "try again";
<li>other negative return values mean various errors; and
<li>positive return values mean "this many bytes got read".
</ul>
<p>In particular, the existence of "try again" means that we need a loop, in addition to all the conditionals for dealing with end-of-file and IO errors.
<pre><code>rdi = 0(stdin)
rsi = (buffer)
rdx = (count)
do:
  rax = 0(read)
  syscall
  cmp rax, -4(eintr)
  while equal
test rax, rax
if negative goto panic
if zero goto end-of-file
(count) in rax
(garbage) in rcx
(garbage) in r11
</code></pre>
<h2 id="how-to-write">How to write</h2>
<p>Writing is similar to reading, with one extra complication: we want to make sure all output bytes are written before moving on, but the <code>write</code> syscall may not all consume all the bytes we give it before returning. So, we need to do it in a nested loop:
<ul>
<li>an inner loop to deal with EINTR; and
<li>an outer loop to deal with partial writes.
</ul>
<pre><code>rdi = 1(stdout)
rsi = (buffer)
rdx = (count)
test rdx, rdx
if nonzero:
  do:
    do:
      rax = 1(write)
      syscall
      cmp rax, -4(eintr)
      while equal
    test rax, rax
    if negative goto panic
    rsi += rax
    rdx -= rax
    while nonzero
(garbage) in rcx
(garbage) in r11
</code></pre>
<h2 id="register-assignments">Register assignments</h2>
<p>Our entire program state fits within a few bytes, so we have an embarassment of riches when it comes to register assignments. To cut down the choices, let's make up some
arbitrary constraints:
<ul>
<li>If we avoid registers <code>r8</code> through <code>r15</code>, we can avoid a bunch of REX instruction prefixes, so let's do that.
<li>It's probably best to avoid messing with the stack pointer <code>rsp</code>, in case the kernel or interrupts assume that it's always valid.
<li><code>rax</code>, <code>rcx</code>, <code>rdx</code>, <code>rsi</code>, <code>rdi</code>, <code>r11</code> are all used by syscalls, so they're not automatically preserved when we read or write bytes (or exit the program). We'd better avoid them for the state of the finite state machine.
<li>That leaves <code>rbx</code> and <code>rbp</code> completely free, so far. <code>rbp</code> feels like a pointer, so it can hold a pointer to the constants-and-code area, as a base for offsets. <code>rbx</code> can hold the finite state machine state.
<li>And for fun, let's use all of the weirdly specific x86 instructions, like <code>lodsb</code> and <code>stosb</code> and <code>loop</code>. Then it's natural to use:
  <ul>
  <li><code>rax</code> to hold the byte being processed;
  <li><code>rdi</code> to point to the end of the write buffer;
  <li><code>rsi</code> to point to the start of the read buffer; and
  <li><code>rcx</code> to hold the remaining number of input bytes.
  </ul>
</ul>
<h2 id="an-unexpected-abstraction">An unexpected abstraction</h2>
<p>While writing the finite state machine code, I found that I was repeating the same sequence of instructions a few times, just with different constants. This sequence already contained a bunch of conditional jumps, so I started wondering whether I could move the code around to de-duplicate it. The answer was yes!
<p>The reason is that for three of the possible input characters, namely newline, '#', and space, the handling is the same:
<ul>
<li>there's a fixed set of states which accept the character;
<li>the following state only depends on the character (after newline we're at the start of a line, after '#' we're inside a comment, and after space we're waiting for a number); and
<li>for states which don't accept the character, we panic.
</ul>
<p>We can put the set of accepting states in <code>dl</code> and the following state in <code>dh</code> unconditionally, and branch to the "check transition" code if the character matches.
<p>What I like about this abstraction is that I noticed it because of the code repetition, but then after thinking about it I found a conceptual reason.
<h2 id="code-layout">Code layout</h2>
<p>For no particular reason, I initially wrote the code in the following order (with jump offsets missing, to be filled in later):
<ol>
<li>read input
<li>finite state machine
<li>found digit
<li>second digit
<li>check transition
<li>next iteration
<li>write output
<li>panic
<li>exit
</ol>
<p>I optimistically wrote all my branches using the short form of the jump instructions, which only accept a jump distance in the range [-128, 127], and of course this didn't work right away. Surprisingly, though, only a few jump targets were too far away, so I was able to reorder the labels to accomodate this:
<ol>
<li>read input
<li>finite state machine
<li>found digit
<li>panic
<li>exit
<li>check transition
<li>next iteration
<li>write output
<li>second digit
</ol>
<p>Basically, "panic" and "exit" moved up closer to the center, to be reachable from all the code, and "second digit" moved out of the way, to the end. Almost all of the code blocks actually fall through to the next block, so this was basically the only way to move the blocks around without introducing extra jump instructions. Thankfully all of the jump distances were fine afterwards (including, just barely, the jump from "write output" to "read input", with a distance of -127 bytes!).
<h2 id="debugging">Debugging</h2>
<p>I "compiled" <a href="byter.bytes">the source code below</a> using <a href="byter.py">a tiny python3 script</a>, and ran in step by step in gdb on a few small input files. While doing this, I found the following bugs:
<ul>
<li>I accidentally called syscall 76 (truncate) instead of syscall 60 (exit) at first. Hopefully it didn't do any damage when called with a pointer to a very low address as the path argument?
<li>I miscalculated 1 jump distance.
<li>I wrote the wrong ModRM byte in 3 instructions.
<li>I had the opposite jump condition in 1 place.
<li>I forgot to check the state of the finite state machine before exiting at end of file. Fixing this will require adding more code, so more bytes, somewhere in the middle. Which means that I have to recalculate jump targets and probably switch to a long-form jump somewhere :( I think I'll just fix that in version 2, right now the program seems to compile <em>correct</em> input files correctly, at least.
</ul>
<h2 id="version-1-success">Version 1 success!</h2>
<p>The program has one known bug, but it's able to "compile" its own source code into an identical copy of itself, yay!
<pre><code>$ ./byter.py &lt;byter.bytes &gt;byter &amp;&amp; chmod +x byter
$ ./byter &lt;byter.bytes &gt;self_byter &amp;&amp; \
&gt;     cmp byter self_byter &amp;&amp; \
&gt;     echo 'success!'
success!
</code></pre>
<h2 id="the-source-code">The source code</h2>
<pre><code># -------------------------------------------------
# byter.bytes -- a text-to-binary converter
# -------------------------------------------------

# -------------------------------------------------
# ELF header
# -------------------------------------------------
# magic number &lt;DEL&gt;"ELF"
7f 45 4c 46
# 64-bit architecture
             02
# little-endian
                01
# ELF version, the one and only
                   01
# no ELF extensions
                      00
# reserved
                           00 00 00 00  00 00 00 00
# executable file
02 00
# x86-64 architecture
      3e 00
# file version, the one and only
             01 00 00 00
# address of first instruction to execute
                           b0 00 40 00  00 00 00 00
# file offset of the program header table
40 00 00 00  00 00 00 00
# file offset of the section header table (absent)
                           00 00 00 00  00 00 00 00
# architecture-specific flags
00 00 00 00
# size of this ELF header
             40 00
# size of a program header
                   38 00
# count of program headers
                           02 00
# size of a section header
                                 40 00
# count of section headers
                                        00 00
# index of the section name section header (absent)
                                              00 00

# -------------------------------------------------
# 1st program header (code)
# -------------------------------------------------

# loadable segment
01 00 00 00
# permissions: read+execute
             05 00 00 00
# file offset
                           00 00 00 00  00 00 00 00
# memory address
00 00 40 00  00 00 00 00
# reserved
                           00 00 00 00  00 00 00 00
# size in file
5a 01 00 00  00 00 00 00
# size in memory
                           5a 01 00 00  00 00 00 00
# 4KB alignment
00 10 00 00  00 00 00 00

# -------------------------------------------------
# 2nd program header (read/write buffer)
# -------------------------------------------------

# loadable segment
                           01 00 00 00
# permissions: read+write
                                        06 00 00 00
# file offset
00 00 00 00  00 00 00 00
# memory address
                           00 00 41 00  00 00 00 00
# reserved
00 00 00 00  00 00 00 00
# size in file
                           00 00 00 00  00 00 00 00
# size in memory
00 10 00 00  00 00 00 00
# 4KB alignment
                           00 10 00 00  00 00 00 00

# -------------------------------------------------
# code
# -------------------------------------------------

# lea rbp, [rip-06] (start of code, end of headers)
8d 2d fa ff  ff ff
# mov bl, 01 (state: start of line)
                   b3 01

## read input
# xor edi, edi (stdin)
                           33 ff
# mov esi, [rbp-28] (start of buffer)
                                 8b 75  d8
# mov edx, [rbp-10] (size of buffer)
                                           8b 55 f0
# do: mov eax, edi (read syscall)
8b c7
# syscall
      0f 05
# cmp eax, -4 (eintr)
             3d fc ff ff   ff
# je [rip-0b] (retry)
                              74 f5
# test eax, eax
                                    85  c0
# js [rip+46] (panic)
                                           78 46
# jz [rip+49] (exit)
                                                 74
49
# mov ecx, eax (size of input)
   8b c8
# mov edi, esi (start of buffer)
         8b  fe

## finite state machine
# lodsb
                ac

# cmp al, 0a (newline)
                   3c 0a
# mov dl, 13 (states which accept newline)
                           b2 13
# mov dh, 01 (state: start of line)
                                 b6 01
# je [rip+43] (check transition)
                                        74 43

# test bl, 02 (state: inside comment)
                                              f6 c3
02
# jnz [rip+44] (next iteration)
   75 44

# cmp al, 20 (space)
         3c  20
# mov dl, 15 (states which accept space)
                b2 15
# mov dh, 04 (state: waiting for number)
                      b6   04
# je [rip+36] (check transition)
                              74 36

# cmp al, 23 ('#')
                                    3c  23
# mov dl, 01 (states which accept '#')
                                           b2 01
# mov dh, 02 (state: inside comment)
                                                 b6
02
# je [rip+2e] (check transition)
   74 2e

# cmp al, 30 ('0')
         3c  30
# jb [rip+1e] (panic)
                72 1e
# cmp al, 39 ('9')
                      3c   39
# jbe [rip+0c] (found digit)
                              76 0c
# or al, 20 (smash case)
                                    0c  20
# cmp al, 61 ('a')
                                           3c 61
# jb [rip+14] (panic)
                                                 72
14
# cmp al, 66 ('f')
   3c 66
# ja [rip+10] (panic)
         77  10
# add al, 09
                04 09

## found digit
# test bl, 05 (states which accept a first digit)
                      f6   c3 05
# jz [rip+40] (second digit)
                                 74 40
# mov bl, 08 (state: middle of number)
                                        b3 08
# shl al, 4
                                              c0 e0
04
# mov bh, al
   8a f8
# jmp [rip+12] (next iteration)
         eb  12

## panic
# mov edi, 1 (nonzero exit status)
                bf 01 00   00 00
## exit
# mov eax, 3c (exit syscall)
                                 b8 3c  00 00 00
# syscall
                                                 0f
05

## check transition
# test bl, dl (acceptable states)
   84 da
# jz [rip-10] (panic)
         74  f0
# mov bl, dh
                8a de
## next iteration
# loop [rip-54] (finite state machine)
                      e2   ac

## write output
# mov edx, edi (end of output buffer)
                              8b d7
# mov edi, 1 (stdout)
                                    bf  01 00 00 00
# mov esi, [rbp-28] (start of buffer)
8b 75 d8
# sub edx, esi (get size of output buffer)
         2b  d6
# jz [rip-7f] (read input)
                74 81
# do: mov eax, edi (write syscall)
                      8b   c7
# syscall
                              0f 05
# cmp eax, -4 (eintr)
                                    3d  fc ff ff ff
# je [rip-0b] (retry)
74 f5
# test eax, eax
      85 c0
# js [rip-31] (panic)
             78 cf
# add esi, eax
                   03 f0
# sub edx, eax
                           2b d0
# jmp [rip-17] (maybe write more)
                                 eb e9

## second digit
# test bl, 08 (states which accept a second digit)
                                       f6 c3 08
# jz [rip-3c] (panic)
                                                74
c4
# mov bl, 10 (state: end of number)
   b3 10
# and al, 0f
         24  0f
# or al, bh
                0a c7
# stosb
                      aa
# jmp [rip-33] (next iteration)
                           eb cd
</code></pre>
]]></content>
  </entry>
  <entry>
    <title>A practical difference between Props and Types in Lean</title>
    <link href="https://www.pointedset.ca/blog/2020/02/06/proptype.html"/>
    <id>tag:pointedset.ca,2020-02-06:props-and-types-in-lean</id>
    <updated>2020-02-06T15:10:03+00:00</updated>
    <content><![CDATA[<p>
Recently, I've been playing around with <a href="https://leanprover.github.io/">Lean</a>, a proof assistant (and technically a programming language) that got a lot of publicity in the past year from a few mathematicians like <a href="https://wwwf.imperial.ac.uk/~buzzard/">Kevin Buzzard</a> and <a href="https://www.math.sciences.univ-nantes.fr/~gouezel/">Sébastien Gouëzel</a>.
<p>
In Lean, there are lots of definitions that treat <code>Prop</code>s and <code>Type</code>s uniformly (that is, the definition takes an argument whose type is <code>Sort l</code> for an arbitrary <code>l</code>), but there are places where something can only be a <code>Prop</code>, and places where something can only be a <code>Type</code>. And in those cases, there are often two parallel definitions which do essentially the same thing, except that one is for <code>Prop</code>s and one is for <code>Type</code>s. So what's different between the cases where we can have a uniform definition for both, and the cases where we need two separate definitions?
<p>
In this post, I want to give a small concrete example of a case where <code>Prop</code>s and <code>Type</code>s don't behave the same in Lean. You can follow along using this link to <a href="https://leanprover.github.io/live/3.4.1/#code=#check%20@proof_irrel%0A--%20proof_irrel%20:%20%E2%88%80%20%7Ba%20:%20Prop%7D%20(h%E2%82%81%20h%E2%82%82%20:%20a),%20h%E2%82%81%20=%20h%E2%82%82%0A%0A#check%20@congr_arg%0A--%20congr_arg%20:%20%E2%88%80%20%7B%CE%B1%20:%20Sort%20u_1%7D%20%7B%CE%B2%20:%20Sort%20u_2%7D%20%7Ba%E2%82%81%20a%E2%82%82%20:%20%CE%B1%7D%20(f%20:%20%CE%B1%20%E2%86%92%20%CE%B2),%20a%E2%82%81%20=%20a%E2%82%82%20%E2%86%92%20f%20a%E2%82%81%20=%20f%20a%E2%82%82%0A%0Ainductive%20flower%20:%20Type%0A%7C%20grow%20:%20forall%20(name%20:%20string),%20flower%0A%0Ainductive%20pretty%20:%20Prop%0A%7C%20grow%20:%20forall%20(name%20:%20string),%20pretty%0A%0Adef%20rose%20:%20flower%20:=%20flower.grow%20%22rose%22%0Adef%20violet%20:%20flower%20:=%20flower.grow%20%22violet%22%0A%0Atheorem%20not_equal_strings%20:%20%22rose%22%20%E2%89%A0%20%22violet%22%20:=%0A%20%20dec_trivial%0Atheorem%20not_equal_flowers%20:%20rose%20%E2%89%A0%20violet%20:=%0A%20%20not_equal_strings%20%E2%88%98%20flower.grow.inj%0A%0Adef%20pretty_rose%20:%20pretty%20:=%20pretty.grow%20%22rose%22%0Adef%20pretty_violet%20:%20pretty%20:=%20pretty.grow%20%22violet%22%0A%0Atheorem%20equal_proofs%20:%20pretty_rose%20=%20pretty_violet%20:=%0A%20%20proof_irrel%20_%20_%0A%0Adef%20forget_difference%20:%20flower%20-%3E%20pretty%0A%7C%20(flower.grow%20name)%20:=%20pretty.grow%20name%0A%0Adef%20cant_remember_difference%20:%20pretty%20-%3E%20flower%0A%7C%20(pretty.grow%20name)%20:=%20flower.grow%20name%0A--%20error:%20recursor%20'pretty.dcases_on'%20can%20only%20eliminate%20into%20Prop%0A%0Adef%20pick_one%20:%20pretty%20-%3E%20flower%20:=%0A%20%20fun%20_,%20flower.grow%20%22thistle%22%0A%0A#print%20nonempty%0A/-%0A@%5Bclass%5D%0Ainductive%20nonempty%20:%20Sort%20u%20%E2%86%92%20Prop%0Aconstructors:%0Anonempty.intro%20:%20%E2%88%80%20%7B%CE%B1%20:%20Sort%20u%7D,%20%CE%B1%20%E2%86%92%20nonempty%20%CE%B1%0A-/%0A%0Adef%20look_at_this_one%20:%20pretty%20-%3E%20(nonempty%20flower)%0A%7C%20(pretty.grow%20name)%20:=%20nonempty.intro%20(flower.grow%20name)%0A%0A#print%20prefix%20flower%0A/-%0Aflower%20:%20Type%0Aflower.cases_on%20:%20%CE%A0%20%7BC%20:%20flower%20%E2%86%92%20Sort%20l%7D%20(n%20:%20flower),%20(%CE%A0%20(name%20:%20string),%20C%20(flower.grow%20name))%20%E2%86%92%20C%20n%0Aflower.grow%20:%20string%20%E2%86%92%20flower%0Aflower.grow.inj%20:%20%E2%88%80%20%7Bname%20name_1%20:%20string%7D,%20flower.grow%20name%20=%20flower.grow%20name_1%20%E2%86%92%20name%20=%20name_1%0Aflower.grow.inj_arrow%20:%20%CE%A0%20%7Bname%20name_1%20:%20string%7D,%20flower.grow%20name%20=%20flower.grow%20name_1%20%E2%86%92%20%CE%A0%20%E2%A6%83P%20:%20Sort%20l%E2%A6%84,%20(name%20=%20name_1%20%E2%86%92%20P)%20%E2%86%92%20P%0Aflower.grow.inj_eq%20:%20%E2%88%80%20%7Bname%20name_1%20:%20string%7D,%20flower.grow%20name%20=%20flower.grow%20name_1%20=%20(name%20=%20name_1)%0Aflower.grow.sizeof_spec%20:%20%E2%88%80%20(name%20:%20string),%20flower.sizeof%20(flower.grow%20name)%20=%201%20+%20sizeof%20name%0Aflower.has_sizeof_inst%20:%20has_sizeof%20flower%0Aflower.no_confusion%20:%20%CE%A0%20%7BP%20:%20Sort%20l%7D%20%7Bv1%20v2%20:%20flower%7D,%20v1%20=%20v2%20%E2%86%92%20flower.no_confusion_type%20P%20v1%20v2%0Aflower.no_confusion_type%20:%20Sort%20l%20%E2%86%92%20flower%20%E2%86%92%20flower%20%E2%86%92%20Sort%20l%0Aflower.rec%20:%20%CE%A0%20%7BC%20:%20flower%20%E2%86%92%20Sort%20l%7D,%20(%CE%A0%20(name%20:%20string),%20C%20(flower.grow%20name))%20%E2%86%92%20%CE%A0%20(n%20:%20flower),%20C%20n%0Aflower.rec_on%20:%20%CE%A0%20%7BC%20:%20flower%20%E2%86%92%20Sort%20l%7D%20(n%20:%20flower),%20(%CE%A0%20(name%20:%20string),%20C%20(flower.grow%20name))%20%E2%86%92%20C%20n%0Aflower.sizeof%20:%20flower%20%E2%86%92%20%E2%84%95%0A-/%0A%0A#print%20prefix%20pretty%0A/-%0Apretty%20:%20Prop%0Apretty.cases_on%20:%20%E2%88%80%20%7BC%20:%20Prop%7D,%20pretty%20%E2%86%92%20(string%20%E2%86%92%20C)%20%E2%86%92%20C%0Apretty.dcases_on%20:%20%E2%88%80%20%7BC%20:%20pretty%20%E2%86%92%20Prop%7D%20(n%20:%20pretty),%20(%E2%88%80%20(name%20:%20string),%20C%20_)%20%E2%86%92%20C%20n%0Apretty.drec%20:%20%E2%88%80%20%7BC%20:%20pretty%20%E2%86%92%20Prop%7D,%20(%E2%88%80%20(name%20:%20string),%20C%20_)%20%E2%86%92%20%E2%88%80%20(n%20:%20pretty),%20C%20n%0Apretty.drec_on%20:%20%E2%88%80%20%7BC%20:%20pretty%20%E2%86%92%20Prop%7D%20(n%20:%20pretty),%20(%E2%88%80%20(name%20:%20string),%20C%20_)%20%E2%86%92%20C%20n%0Apretty.grow%20:%20string%20%E2%86%92%20pretty%0Apretty.rec%20:%20%E2%88%80%20%7BC%20:%20Prop%7D,%20(string%20%E2%86%92%20C)%20%E2%86%92%20pretty%20%E2%86%92%20C%0Apretty.rec_on%20:%20%E2%88%80%20%7BC%20:%20Prop%7D,%20pretty%20%E2%86%92%20(string%20%E2%86%92%20C)%20%E2%86%92%20C%0A-/%0A%0Adef%20cant_remember%20:%20pretty%20-%3E%20flower%20:=%0A%20%20@pretty.rec%0A%20%20%20%20flower%20%20--%20The%20target%20type%0A%20%20%20%20(fun%20name,%20flower.grow%20name)%20%20--%20The%20callback%20for%20%60pretty.grow%60%0A--%20error:%20type%20mismatch%0A%0Adef%20can_forget%20:%20flower%20-%3E%20pretty%20:=%0A%20%20@flower.rec%0A%20%20%20%20(fun%20_,%20pretty)%20%20--%20The%20target%20type%20is%20%60pretty%60%20for%20all%20flowers%0A%20%20%20%20(fun%20name,%20pretty.grow%20name)%20%20--%20The%20callback%20for%20%60flower.grow%60">all the code in this post in the Lean playground</a> if you want.
<h2 id="proof-irrelevance">
Proof irrelevance
</h2>
<p>
One of the few fundamental differences between <code>Prop</code>s and <code>Type</code>s is that we have <em>proof irrelevance</em> for <code>Prop</code>s. Concretely, this shows up in Lean as the following theorem, which is available from the core library:
<pre><code>#check @proof_irrel
-- proof_irrel : ∀ {a : Prop} (h₁ h₂ : a), h₁ = h₂
</code></pre>
<p>
This theorem says that, given a proposition <code>a</code> and two proofs <code>h₁</code> and <code>h₂</code> of <code>a</code>, the two proofs are always equal in the eyes of Lean.
<p>
This might seem innocuous, but let's combine it with another theorem from the core library:
<pre><code>#check @congr_arg
-- congr_arg : ∀ {α : Sort u_1} {β : Sort u_2} {a₁ a₂ : α} (f : α → β), a₁ = a₂ → f a₁ = f a₂
</code></pre>
<p>
This theorem (which is uniform between <code>Prop</code>s and <code>Type</code>s, because its parameters are <code>Sort</code>s), says that when we apply a function <code>f</code> to equal arguments (<code>a₁</code> and <code>a₂</code>, in this case), the results are equal. This is a totally reasonable theorem. If it wasn't true, <code>f</code> would not be a well-defined function.
<p>
But together, these two theorems say that whenever we have a function from a <code>Prop</code> to a <code>Type</code>, that function has to be a constant function. The result can't depend on the structure of the proof that we feed to the function. It can only depend on the <em>existence</em> of a proof, and it must always be the same result.
<p>
The confusing thing is that when we have a function from a <code>Prop</code> to another <code>Prop</code>, we <em>are</em> allowed to pattern-match on the structure of the input proof, so it looks like the function's result can depend on the details of the proof. But as long as we produce a proof of the target <code>Prop</code> in all code branches of the function, the result is always equal, by proof irrelevance! So the function is still a constant function, even though it doesn't look like one.
<h2 id="example-flowers-are-pretty">
Example: flowers are pretty
</h2>
<p>
Ok, I said I would give a concrete example, but so far it's been pretty theoretical.
<p>
Let's say we want to talk about a world with flowers in it. And in this world, the defining characteristic of a flower is its name, so we might write the following type definition to capture it:
<pre><code>inductive flower : Type
| grow : forall (name : string), flower
</code></pre>
<p>
There's just one constructor for <code>flower</code>s, <code>flower.grow</code>, which takes an arbitrary <code>string</code> for the name of the flower, and returns the corresponding <code>flower</code>.
<p>
We also want to talk about whether this is a pretty world, and let's say it's pretty iff there exists a flower. We can capture this property by having a constructor with the same type signature as the constructor for <code>flower</code>s:
<pre><code>inductive pretty : Prop
| grow : forall (name : string), pretty
</code></pre>
<p>
Apart from having a different type name (<code>flower</code> vs <code>pretty</code>), the only difference is that <code>flower</code> is a <code>Type</code>, and <code>pretty</code> is a <code>Prop</code>. We could justify this by saying that <em>flower</em> is a noun, a thing that the world can <em>have</em>, whereas <em>pretty</em> is an adjective, a thing that the world can <em>be</em>.
<p>
Let's prove that there exist two different flowers:
<pre><code>def rose : flower := flower.grow "rose"
def violet : flower := flower.grow "violet"

theorem not_equal_strings : "rose" ≠ "violet" :=
  dec_trivial
theorem not_equal_flowers : rose ≠ violet :=
  not_equal_strings ∘ flower.grow.inj
</code></pre>
<p>
On the other hand, any two proofs that the world is pretty are equal. Lean believes that roses and violets are pretty in exactly the same way:
<pre><code>def pretty_rose : pretty := pretty.grow "rose"
def pretty_violet : pretty := pretty.grow "violet"

theorem equal_proofs : pretty_rose = pretty_violet :=
  proof_irrel _ _
</code></pre>
<p>
So, we can easily write a function from the <code>Type</code> to the <code>Prop</code>:
<pre><code>def forget_difference : flower → pretty
| (flower.grow name) := pretty.grow name
</code></pre>
<p>
But Lean will complain if we try to go in the other direction:
<pre><code>def cant_remember_difference : pretty → flower
| (pretty.grow name) := flower.grow name
-- error: recursor 'pretty.dcases_on' can only eliminate into Prop
</code></pre>
<p>
Does this mean that the world of <code>Prop</code>s is a black hole and that we can never escape back into the world of <code>Type</code>s? No! There are a few allowed ways to write a function which takes a proof of a <code>Prop</code> and returns a value of a <code>Type</code>, such as:
<ul>
<li>Not using the proof in the function body. This seems pretty silly, but it's a useful escape hatch. And it can still act as a contract imposed on the function callers: we're only allowed to call the function in contexts where a proof exists.
<li>Using the proof for control flow. If we have a code branch that's unreachable in our function, <code>false.elim</code> will let us provide a proof of unreachability instead of writing actual code for that branch, even if the function is supposed to return a value of a <code>Type</code>.
<li>Using the proof for recursion. Lean allows us to write recursive functions, but only when it can prove that the recursion will terminate, instead of entering an infinite loop. Most of the time it can do that automatically, but sometimes we have to provide (part of) the proof.
</ul>
<p>
As an example of the first point, we could write the following function, which is totally fine:
<pre><code>def pick_one : pretty → flower :=
  fun _, flower.grow "thistle"
</code></pre>
<p>
This function just ignores the pretty flower that we give it, and instead gives us back a pretty thistle, always. Another workaround is to turn <code>flower</code> into a <code>Prop</code>, by using <code>nonempty</code>:
<pre><code>#print nonempty
/-
@[class]
inductive nonempty : Sort u → Prop
constructors:
nonempty.intro : ∀ {α : Sort u}, α → nonempty α
-/
</code></pre>
<p>
This turns any <code>Type</code> <code>α</code> into the proposition that "there exists a value of type <code>α</code>". So, we can write the following function, which doesn't completely ignore the pretty flower we give it:
<pre><code>def look_at_this_one : pretty → (nonempty flower)
| (pretty.grow name) := nonempty.intro (flower.grow name)
</code></pre>
<p>
In effect, wrapping a <code>Type</code> in <code>nonempty</code> gives us a version of the <code>Type</code> where we promise to not actually depend on the value, only on its existence, and the rest of the type system enforces that promise. Once we make that promise, we're allowed to depend on the structure of the input proof.
<h2 id="under-the-hood">
Under the hood
</h2>
<p>
So, concretely, how does the type system enforce the promise that we can only depend on the structure of a proof when we're producing a proof of a <code>Prop</code>? To answer this, we have to look at the fundamental, low-level building blocks that Lean provides for us when we write an inductive definition. For the example above, we can see that Lean automatically defines a whole bunch of stuff just from the definitions of <code>flower</code> and <code>pretty</code>:
<pre style="white-space: pre; overflow-x: auto;"><code>
#print prefix flower
/-
flower : Type
flower.cases_on : Π {C : flower → Sort l} (n : flower), (Π (name : string), C (flower.grow name)) → C n
flower.grow : string → flower
flower.grow.inj : ∀ {name name_1 : string}, flower.grow name = flower.grow name_1 → name = name_1
flower.grow.inj_arrow : Π {name name_1 : string}, flower.grow name = flower.grow name_1 → Π ⦃P : Sort l⦄, (name = name_1 → P) → P
flower.grow.inj_eq : ∀ {name name_1 : string}, flower.grow name = flower.grow name_1 = (name = name_1)
flower.grow.sizeof_spec : ∀ (name : string), flower.sizeof (flower.grow name) = 1 + sizeof name
flower.has_sizeof_inst : has_sizeof flower
flower.no_confusion : Π {P : Sort l} {v1 v2 : flower}, v1 = v2 → flower.no_confusion_type P v1 v2
flower.no_confusion_type : Sort l → flower → flower → Sort l
flower.rec : Π {C : flower → Sort l}, (Π (name : string), C (flower.grow name)) → Π (n : flower), C n
flower.rec_on : Π {C : flower → Sort l} (n : flower), (Π (name : string), C (flower.grow name)) → C n
flower.sizeof : flower → ℕ
-/

#print prefix pretty
/-
pretty : Prop
pretty.cases_on : ∀ {C : Prop}, pretty → (string → C) → C
pretty.dcases_on : ∀ {C : pretty → Prop} (n : pretty), (∀ (name : string), C _) → C n
pretty.drec : ∀ {C : pretty → Prop}, (∀ (name : string), C _) → ∀ (n : pretty), C n
pretty.drec_on : ∀ {C : pretty → Prop} (n : pretty), (∀ (name : string), C _) → C n
pretty.grow : string → pretty
pretty.rec : ∀ {C : Prop}, (string → C) → pretty → C
pretty.rec_on : ∀ {C : Prop}, pretty → (string → C) → C
-/
</code></pre>
<p>
In both cases, Lean generates lots of stuff, but it all boils down to just three things, and everything else is defined from these three things:
<ul>
<li>The type name (in this case, <code>flower</code> and <code>pretty</code>), so that we can write function signatures using the new type.
<li>The type constructors (<code>flower.grow</code> and <code>pretty.grow</code>), so that we can produce values of the new type.
<li>The eliminator for the type (<code>flower.rec</code> and <code>pretty.rec</code>), so that we can write functions which consume a value of the new type and whose result depends on the internals of the value. This is where the difference between <code>Prop</code>s and <code>Type</code>s happens.
</ul>
<p>
The type signatures for the eliminators look a bit weird, so let's break them down, starting with <code>pretty.rec</code> because it's simpler:
<pre><code>pretty.rec : ∀ {C : Prop}, (string → C) → pretty → C
</code></pre>
<ul>
<li>The return type is <code>(pretty → C)</code>, that is, a function from proofs of prettiness to values of some target type <code>C</code>.
<li>The second parameter has type <code>(string → C)</code>, that is, it's a function from <em>names</em> of pretty flowers to values of the target type <code>C</code>. The whole information content of a proof of prettiness is the name of a pretty flower (that is, the argument to the constructor <code>pretty.grow</code>), so that's what we have to work with to produce a value of the target type <code>C</code>. If we had more constructors for <code>pretty</code>, the eliminator <code>pretty.rec</code> would have more parameters like this, which take the <em>arguments</em> of a constructor and produce a value of the target type <code>C</code>. Each one of these parameters is like a callback that knows how to handle values produced by a particular constructor. And if we have callbacks for all possible constructors, the eliminator just dispatches to the appropriate callback.
<li>The first parameter is <code>{C : Prop}</code>, a target type for the function that <code>pretty.rec</code> returns. And sure enough, this is forced to be a <code>Prop</code>, not a <code>Type</code> nor an unrestricted <code>Sort</code>.
</ul>
<p>
For example, we could use <code>pretty.rec</code> to try (and fail) again to write the "obvious" function <code>pretty → flower</code>:
<pre><code>def cant_remember : pretty → flower :=
  @pretty.rec
    flower  -- The target type
    (fun name, flower.grow name)  -- The callback for `pretty.grow`
-- error: type mismatch
</code></pre>
<p>
It doesn't look like it, but the eliminator <code>flower.rec</code> follows the same pattern:
<pre><code>flower.rec : Π {C : flower → Sort l}, (Π (name : string), C (flower.grow name)) → Π (n : flower), C n
</code></pre>
<ul>
<li>The first parameter is <code>{C : flower → Sort l}</code>, a <em>family</em> of target types for the function that <code>flower.rec</code> returns. That's because Lean uses dependent types, so each <code>flower</code> value can have its own target type. Also, the target types are in <code>Sort l</code>, so they can be <code>Prop</code>s or <code>Type</code>s.
<li>The second parameter has type <code>(Π (name : string), C (flower.grow name))</code>, so it's a callback that uses the argument <code>(name : string)</code> of the <code>flower.grow</code> constructor, and produces a value of the target type <code>(C (flower.grow name))</code> for that specific <code>flower</code>.
<li>The return type is <code>(Π (n : flower), C n)</code>, that is, a function that takes a flower <code>n</code> and returns a value of the target type <code>(C n)</code> for that flower.
</ul>
<p>
For example, we can try (and succeed) to use <code>flower.rec</code> to write the obvious function <code>flower → pretty</code>:
<pre><code>def can_forget : flower → pretty :=
  @flower.rec
    (fun _, pretty)  -- The target type is `pretty` for all flowers
    (fun name, pretty.grow name)  -- The callback for `flower.grow`
</code></pre>
]]></content>
  </entry>
  <entry>
    <title>Exploring syscalls with GCC and GDB</title>
    <link href="https://www.pointedset.ca/blog/2018/06/09/gcc-gdb.html"/>
    <id>tag:pointedset.ca,2018-06-09:exploring-syscalls-with-gcc-and-gdb</id>
    <updated>2018-06-09T13:12:30+00:00</updated>
    <content><![CDATA[<p>
In the previous post, my first instinct figuring out how to make a Linux syscall in assembly was to look at specs and documentation, but a former colleague wrote in to ask why I didn't just look at the output of GCC. Also, why not just <code>return 110</code> from <code>main</code> rather than calling <code>exit(110)</code>? These are good questions, so let's discuss them!
<p>
Philosophically speaking, knowledge I get from reading specs and knowledge I get from reverse engineering feel different; from specs I expect to get a full picture of how things are <em>supposed</em> to work, and from reverse engineering I expect to see a particular example of how things <em>do</em> work in at least some cases. It really depends on the situation which of these two approaches is more useful or true! And even though I prefer knowledge from specs, reverse engineering can be a good way to figure out what specs to even look at and how to interpret them.
<p>
Practically speaking, when I tried looking at the output of GCC for this particular problem, it wasn't very useful:
<pre><code>$ cat &gt;with_exit.c &lt;&lt;'EOF'

#include &lt;stdlib.h&gt;
int main(void) { exit(110); }

EOF
$ gcc -S with_exit.c -o /dev/stdout
  .file "with_exit.c"
  .text
  .globl main
  .type main, @function
main:
.LFB2:
  .cfi_startproc
  pushq %rbp
  .cfi_def_cfa_offset 16
  .cfi_offset 6, -16
  movq %rsp, %rbp
  .cfi_def_cfa_register 6
  movl $110, %edi
  call exit
  .cfi_endproc
.LFE2:
  .size main, .-main
  .ident "GCC: (Ubuntu 5.4.0-6ubuntu1~16.04.9) 5.4.0 20160609"
  .section .note.GNU-stack,"",@progbits
</code></pre>
<p>
The two relevant lines are <code>movl $110, %edi</code> and <code>call exit</code>, but that's like saying "to exit, call the exit function", which isn't very enlightening. The problem is that in C, every syscall is wrapped by a standard library function, and what we're looking for is inside there rather than in our program. I thought it would maybe get inlined if I turned on optimizations, but no:
<pre><code>$ gcc -O2 -S with_exit.c -o /dev/stdout
  .file "with_exit.c"
  .section .text.unlikely,"ax",@progbits
.LCOLDB0:
  .section .text.startup,"ax",@progbits
.LHOTB0:
  .p2align 4,,15
  .globl main
  .type main, @function
main:
.LFB15:
  .cfi_startproc
  subq $8, %rsp
  .cfi_def_cfa_offset 16
  movl $110, %edi
  call exit
  .cfi_endproc
.LFE15:
  .size main, .-main
  .section .text.unlikely
.LCOLDE0:
  .section .text.startup
.LHOTE0:
  .ident "GCC: (Ubuntu 5.4.0-6ubuntu1~16.04.9) 5.4.0 20160609"
  .section .note.GNU-stack,"",@progbits
</code></pre>
<p>
What about using <code>return</code> instead? I got similarly unenlightening results:
<pre><code>$ cat &gt;with_return.c &lt;&lt;'EOF'

int main(void) { return 110; }

EOF
$ gcc -S with_return.c -o /dev/stdout
  .file "with_return.c"
  .text
  .globl main
  .type main, @function
main:
.LFB0:
  .cfi_startproc
  pushq %rbp
  .cfi_def_cfa_offset 16
  .cfi_offset 6, -16
  movq %rsp, %rbp
  .cfi_def_cfa_register 6
  movl $110, %eax
  popq %rbp
  .cfi_def_cfa 7, 8
  ret
  .cfi_endproc
.LFE0:
  .size main, .-main
  .ident "GCC: (Ubuntu 5.4.0-6ubuntu1~16.04.9) 5.4.0 20160609"
  .section .note.GNU-stack,"",@progbits
$ gcc -O2 -S with_return.c -o /dev/stdout
  .file "with_return.c"
  .section .text.unlikely,"ax",@progbits
.LCOLDB0:
  .section .text.startup,"ax",@progbits
.LHOTB0:
  .p2align 4,,15
  .globl main
  .type main, @function
main:
.LFB0:
  .cfi_startproc
  movl $110, %eax
  ret
  .cfi_endproc
.LFE0:
  .size main, .-main
  .section .text.unlikely
.LCOLDE0:
  .section .text.startup
.LHOTE0:
  .ident "GCC: (Ubuntu 5.4.0-6ubuntu1~16.04.9) 5.4.0 20160609"
  .section .note.GNU-stack,"",@progbits
</code></pre>
<p>
In both the unoptimized and the optimized version, <code>return 110</code> simply translates to the two instructions <code>movl $110, %eax</code> and <code>ret</code> (return).
<p>
This is as far as GCC gets us, but in the spirit of reverse engineering, let's turn to GDB for help!
<h2 id="what-happens-inside-the-stdlib">
What happens inside the stdlib
</h2>
<p>
I thought it would be reasonably simple to use GDB to step through all the assembly instructions that happen when <code>exit(110)</code> gets called, and it <em>is</em> simple (the only GDB knowledge I used was the commands <code>break main</code> to set up a breakpoint, <code>run</code> to get to it, <code>display/i $pc</code> to show the next assembly instruction at each step, and <code>stepi</code> to start stepping through the code), but it's also extremely long. You can see <a href="gdb-with-exit.txt">the output of my GDB session</a> if you're curious, but be warned that it's stepping through about 1375 instructions. As far as I can tell, some of the reasons are:
<ul>
<li>The standard library functions are dynamically linked, so the program actually has to look up whether a function called <code>exit</code> even exists, where it lives in memory, etc., and this seems to involve hash tables and string comparisons.
<li>Once it's found, <code>exit</code> does a bunch of cleanup:
  <ul>
  <li>Look for global destructors and call them.
  <li>Look for functions registered with <code>atexit</code> or <code>on_exit</code> and call them.
  <li>Various bits of IO cleanup.
  </ul>
</ul>
<p>
In the end though, it does use the <code>syscall</code> instruction to invoke the Linux kernel. As for the arguments passed to the kernel, I'm not sure I would want to extract them from this GDB session.
<h2 id="what-happens-after-main">
What happens after main
</h2>
<p>
So that's what happens when calling <code>exit</code>, but what if we just <code>return</code> from <code>main</code>? We saw above that this turns into the <code>ret</code> instruction, but that can't be used to jump back into the kernel directly, so there has to be something implicitly running <em>after</em> <code>main</code>. This can be confirmed by looking at some of the ELF information:
<pre><code>$ gcc with_return.c -o with_return
$ readelf --file-header --symbols with_return
ELF Header:
...
  Entry point address:               0x4003e0
...
Symbol table '.symtab' contains 66 entries:
   Num:    Value          Size Type    Bind   Vis      Ndx Name
...
    59: 00000000004003e0    42 FUNC    GLOBAL DEFAULT   14 _start
...
    61: 00000000004004d6    11 FUNC    GLOBAL DEFAULT   14 main
...
</code></pre>
<p>
The first instructions from this program that get executed are at the entry point <code>0x4003e0</code>, which is the address of the symbol <code>_start</code>, not <code>main</code>. So <code>_start</code> is responsible for doing a bunch of setup before actually calling <code>main</code>. And once <code>main</code> is done, it returns to whatever <code>_start</code> was doing for the cleanup. We can see this in GDB:
<pre><code>$ gdb with_return
GNU gdb (Ubuntu 7.11.1-0ubuntu1~16.5) 7.11.1
Copyright (C) 2016 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later &lt;http://gnu.org/licenses/gpl.html&gt;
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
&lt;http://www.gnu.org/software/gdb/bugs/&gt;.
Find the GDB manual and other documentation resources online at:
&lt;http://www.gnu.org/software/gdb/documentation/&gt;.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from with_return...(no debugging symbols found)...done.
(gdb) break main
Breakpoint 1 at 0x4004da
(gdb) run
Starting program: with_return

Breakpoint 1, 0x00000000004004da in main ()
(gdb) display/i $pc
1: x/i $pc
=&gt; 0x4004da &lt;main+4&gt;: mov    $0x6e,%eax
(gdb) stepi
0x00000000004004df in main ()
1: x/i $pc
=&gt; 0x4004df &lt;main+9&gt;: pop    %rbp
(gdb)
0x00000000004004e0 in main ()
1: x/i $pc
=&gt; 0x4004e0 &lt;main+10&gt;: retq
(gdb)
__libc_start_main (main=0x4004d6 &lt;main&gt;, argc=1, argv=0x7fffffffde38,
    init=&lt;optimized out&gt;, fini=&lt;optimized out&gt;, rtld_fini=&lt;optimized out&gt;,
    stack_end=0x7fffffffde28) at ../csu/libc-start.c:325
325 ../csu/libc-start.c: No such file or directory.
1: x/i $pc
=&gt; 0x7ffff7a2d830 &lt;__libc_start_main+240&gt;: mov    %eax,%edi
(gdb)
0x00007ffff7a2d832 325 in ../csu/libc-start.c
1: x/i $pc
=&gt; 0x7ffff7a2d832 &lt;__libc_start_main+242&gt;:
    callq  0x7ffff7a47030 &lt;__GI_exit&gt;
(gdb)
__GI_exit (status=110) at exit.c:104
</code></pre>
<p>
Once it's done, <code>main</code> returns to <code>__libc_start_main</code>, which then calls <code>__GI_exit</code>. In <a href="gdb-with-exit.txt">the previous GDB session</a>, this appears on line 2971 out of 5784, more than halfway through! So it looks like the extra work for looking up <code>exit</code> rather than using <code>return</code> is a pretty big chunk.
]]></content>
  </entry>
  <entry>
    <title>The simplest system call: exiting</title>
    <link href="https://www.pointedset.ca/blog/2018/05/18/eleventy.html"/>
    <id>tag:pointedset.ca,2018-05-18:the-simplest-system-call-exiting</id>
    <updated>2018-05-18T22:44:12+00:00</updated>
    <content><![CDATA[<p>
From the previous post, we know how to make an x86-64 ELF program start, but how do we make it stop? Let's make a program that will exit, say with exit code 110.
<p>
In C, this would be easy to do:
<pre><code>$ cat &gt;eleventy.c &lt;&lt;'EOF'

#include &lt;stdlib.h&gt;
int main(void) { exit(110); }

EOF
$ gcc eleventy.c -o eleventy
$ ./eleventy; echo $?
110
</code></pre>
<p>
Our goal is to write some assembly code to do this, so we'll have to unwrap a few layers of niceness that the C standard library provides for us. The first stop is <code>man 3 exit</code>, which tells us that <code>exit</code> does a bunch of stuff and eventually calls <code>_exit</code>, with an underscore in front. The next stop is <code>man 2 _exit</code>, which tells us:
<pre><code>   C library/kernel differences
       In glibc up to version  2.3,  the  _exit()  wrapper
       function invoked the kernel system call of the same
       name.   Since  glibc  2.3,  the  wrapper   function
       invokes exit_group(2), in order to terminate all of
       the threads in a process.
</code></pre>
<p>
I love these "C library/kernel differences" sections that are in the man pages for most system calls. Concise and useful!
<p>
Alright, so now we know we should call either the <code>exit</code> (number 60) or the <code>exit_group</code> (number 231) system call. (Those numbers come from either searching online or looking at the file <code>arch/x86/entry/syscalls/syscall_64.tbl</code> in the Linux kernel source tree.)
<p>
Next up, the specification in <a href="https://github.com/hjl-tools/x86-psABI/wiki/x86-64-psABI-1.0.pdf">x86-64-psABI-1.0.pdf</a> tells us on page 148 how to actually make a system call in assembly:
<img alt="calling conventions for x86-64 system calls" src="syscall-convention.png">
<p>
(Incidentally, this tells us that the whole <code>errno</code> thing in C is an abstraction added by C itself; in assembly, the error information is right there in the return value of the system call.)
<p>
Summarizing, the only code we need to exit with 110 is:
<pre><code>mov rax, 0xe7    # system call number 231 (decimal) = e7 (hex)
mov rdi, 0x6e    # exit code 110 (decimal) = 6e (hex)
syscall
</code></pre>
<h2 id="how-to-move-constants-into-registers">
How to move constants into registers
</h2>
<p>
According to the <a href="24594.pdf">AMD64 Architecture Programmer's Manual, Volume 3</a>, page 421, it's pretty easy to encode the <code>syscall</code> instruction:
<img alt="0f 05" src="syscall-encoding.png">
<p>
Just the two bytes <code>0f 05</code> and we're done.
<p>
It's a bit harder to encode the <code>mov</code> instructions, because there are so many ways of doing it. There are eight variants just for moving a constant into a register (page 225):
<img alt="eight forms of the mov instruction" src="mov-encoding.png">
<p>
If you look closely at the encoding, though, you'll notice that the encodings for the 16-bit, 32-bit and 64-bit variants look the same on the binary level. That can't be right! How does the processor know the size of the operands? The answer is hidden in a few other tables and figures and sections in the manual, which specify how various instruction prefixes modify the meaning of instructions. Table 1-2 on page 8 tells us how:
<img alt="how operand sizes are determined in each operating mode" src="operand-size-overrides.png">
<p>
Our program will be operating in the 64-bit submode of long mode, so to use a 64-bit operand size, we need a REX prefix with <code>REX.W = 1</code>. After reading the flow-chart on page 2, section 1.2.7 "REX Prefix", and section 2.5.2 "Opcode Syntax", we can finally figure out that <code>mov rax, 0xe7</code> can be encoded as:
<pre><code>48 b8 e7 00 00 00 00 00 00 00
</code></pre>
<ul>
<li>The first byte, <code>48</code>, is the REX prefix which specifies that we want wide operands, because it has the bit <code>REX.W = 1</code>.
<li>The second byte, <code>b8</code>, together with the bit <code>REX.B = 0</code>, encodes <code>mov rax, {some 64-bit constant}</code>.
<li>The next 8 bytes give the 64-bit constant <code>0xe7</code>, in little-endian format.
</ul>
<p>
But there's more than one way to do it! Let's take this as an invitation to do some premature optimization for code size. We can do the same thing in fewer bytes if we look at figure 2-3 and/or read the surrounding text:
<img alt="a table of the general-purpose registers and how they overlap" src="general-registers.png">
<p>
This specifies that the 32-bit register <code>eax</code> lives in the lower half of the 64-bit register <code>rax</code>, and when we save a result to <code>eax</code> the processor sets the upper half of <code>rax</code> to zero automatically. Our constant has its upper half equal to zero, so we could use the encoding:
<pre><code>b8 e7 00 00 00
</code></pre>
<ul>
<li>There's no more REX prefix, so we're using 32-bit operands by default.
<li>The byte <code>b8</code> and the absence of <code>REX.B</code> encodes <code>mov eax, {some 32-bit constant}</code>.
<li>The next 4 bytes give the 32-bit constant <code>0xe7</code>, again in little-endian.
</ul>
<p>Can we do even better? Our constant fits into a single byte, so we could hope to just stuff it into the 8-bit register <code>al</code>, which lives in the lowest byte of <code>rax</code>. This would be encoded as:
<pre><code>b0 e7
</code></pre>
<p>
However, as figure 2-3 above tells us, the rest of <code>rax</code> doesn't get set to zero when we put something in <code>al</code>. We could do that ourselves, and the section on the <code>mov</code> instruction on page 224 tells us:
<img alt="to initialize a register to 0, use the xor instruction with identical destination and source operands" src="mov-zero.png">
<p>
After reading about the <code>xor</code> instruction and the encoding of ModRM bytes, we can encode <code>xor eax, eax</code> (which sets <code>rax</code> to zero) as:
<pre><code>31 c0
</code></pre>
<p>
So this brings us down to four bytes for something roughly equivalent to <code>mov rax, 0xe7</code>:
<pre><code>31 c0 b0 e7
</code></pre>
<p>
Can we do the same thing for the other value we have to set, <code>mov rdi, 0x6e</code>? Not really! Figure 2-3 tells us that the register <code>dil</code>, which lives in the lowest byte of register <code>rdi</code>, can only be used if we have a REX prefix (so that we can have <code>REX.B = 1</code>), which would bring us back up to five bytes:
<pre><code>31 ff 41 b7 6e
</code></pre>
<ul>
<li>Two bytes <code>31 ff</code> to encode <code>xor edi, edi</code>, which sets <code>rdi</code> to zero.
<li>The byte <code>41</code> is a REX prefix with <code>REX.B = 1</code> so that we can use the register <code>dil</code>.
<li>The byte <code>b7</code> together with <code>REX.B = 1</code> encodes <code>mov dil, {some 8-bit constant}</code>.
<li>The last byte is the constant <code>0x6e</code>.
</ul>
<p>
However, it turns out that we can encode the instruction <code>mov edi, eax</code> with only two bytes:
<pre><code>89 c7
</code></pre>
<p>This means that we can do the following sequence in just 10 bytes:
<pre><code>31 c0    # xor eax, eax
b0 6e    # mov al, 6e
89 c7    # mov edi, eax
b0 e7    # mov al, e7
0f 05    # syscall
</code></pre>
<p>
and the effect is essentially the same as the sequence we started with, which uses 22 bytes:
<pre><code>48 b8 e7 00 00 00 00 00 00 00    # mov rax, 0xe7
48 bf 6e 00 00 00 00 00 00 00    # mov rdi, 0x6e
0f 05                            # syscall
</code></pre>
<p>
(It's only <em>essentially</em> the same because the 10-byte version affects the status flags before doing the syscall, which we don't care about here. Infinite minutiae!)
<p>
Wrap it all up in an ELF file and we have a program that can make a system call:
<pre><code>$ cat &gt;make-eleventy &lt;&lt;'EOF'
#!/usr/bin/python3

s = """
7f 45 4c 46 02 01 01 00
00 00 00 00 00 00 00 00
02 00 3e 00 01 00 00 00
78 00 01 00 00 00 00 00
40 00 00 00 00 00 00 00
00 00 00 00 00 00 00 00
00 00 00 00 40 00 38 00
01 00 40 00 00 00 00 00

01 00 00 00 05 00 00 00
78 00 00 00 00 00 00 00
78 00 01 00 00 00 00 00
00 00 00 00 00 00 00 00
0a 00 00 00 00 00 00 00
0a 00 00 00 00 00 00 00
00 10 00 00 00 00 00 00

31 c0 b0 6e 89 c7 b0 e7
0f 05
"""

open('eleventy', 'wb').write(bytes([int(b, 16) for b in s.split()]))

EOF
$ chmod +x make-eleventy
$ ./make-eleventy
$ chmod +x eleventy
$ ./eleventy
$ echo $?
110
</code></pre>
<h2 id="what-s-the-point">
What's the point?
</h2>
<ul>
<li>Making system calls in x86-64 assembly on Linux is actually not that hard!
<li>There are lots of opportunities for code golf if you find this fun.
<li>Other than as a learning exercise, this process is a tremendous waste of time.
</ul>
]]></content>
  </entry>
  <entry>
    <title>Hand-crafting an ELF file</title>
    <link href="https://www.pointedset.ca/blog/2018/04/10/loopy.html"/>
    <id>tag:pointedset.ca,2018-04-10:hand-crafting-an-elf-file</id>
    <updated>2018-04-10T21:28:04+00:00</updated>
    <content><![CDATA[<p>
This week I suddenly got a burning desire to write some x86 assembly code and have it run on my computer, in as minimal an environment as possible. I decided that it wouldn't be too much to ask my computer to run the code <code>eb fe</code> (that's two bytes of code, written in hexadecimal), which is just an infinite loop, the equivalent of <code>label: goto label</code>. No input, no output, not even an exit, just a single observable side-effect: running forever, or at least until ctrl-C.
<p>
As a first step to doing this, I installed <a href="https://www.qemu.org/">the excellent QEMU project</a> and spent a day trying to understand which command options to give it to disable most of the bells and whistles it gives you but still load a file with some binary code. Unsuccessfully.
<p>
The next day, I vaguely remembered having read <a href="https://www.muppetlabs.com/~breadbox/software/tiny/teensy.html">an excellent blog post about creating tiny ELF files</a>, so I decided that sticking my two bytes of assembly code in a valid ELF file might be easier. The smart thing would have been to immediately Google for this blog post and read it, tweaking as necessary, but instead I tried to look for specs and found:
<ul>
<li><a href="https://en.wikipedia.org/wiki/Executable_and_Linkable_Format">the Wikipedia page about the ELF file format</a> for a list of the fields;
<li><a href="http://man7.org/linux/man-pages/man5/elf.5.html"><code>man 5 elf</code></a> for more details;
<li><code>less /usr/include/elf.h</code> for some of the numeric constants;
<li><code>readelf -a /usr/bin/lefty</code> and <code>hd /usr/bin/lefty</code> for a worked example.
</ul>
<p>
Eventually I ended up with a file called <code>make-loopy</code>:
<pre><code>#!/usr/bin/python3

s = """
7f 45 4c 46 02 01 01 00
00 00 00 00 00 00 00 00
02 00 3e 00 01 00 00 00
78 00 01 00 00 00 00 00
40 00 00 00 00 00 00 00
00 00 00 00 00 00 00 00
00 00 00 00 40 00 38 00
01 00 40 00 00 00 00 00

01 00 00 00 05 00 00 00
78 00 00 00 00 00 00 00
78 00 01 00 00 00 00 00
00 00 00 00 00 00 00 00
02 00 00 00 00 00 00 00
02 00 00 00 00 00 00 00
00 10 00 00 00 00 00 00

eb fe
"""

open('loopy', 'wb').write(bytes([int(b, 16) for b in s.split()]))
</code></pre>
<p>
It's just a Python script that writes a bunch of hex bytes to a file called <code>loopy</code>, and the glorious result is:
<pre><code>$ ./make-loopy && ./loopy
...wait for however long you like...
^C
$ readelf --file-header --program-headers loopy
ELF Header:
  Magic:   7f 45 4c 46 02 01 01 00 00 00 00 00 00 00 00 00
  Class:                             ELF64
  Data:                              2's complement, little endian
  Version:                           1 (current)
  OS/ABI:                            UNIX - System V
  ABI Version:                       0
  Type:                              EXEC (Executable file)
  Machine:                           Advanced Micro Devices X86-64
  Version:                           0x1
  Entry point address:               0x10078
  Start of program headers:          64 (bytes into file)
  Start of section headers:          0 (bytes into file)
  Flags:                             0x0
  Size of this header:               64 (bytes)
  Size of program headers:           56 (bytes)
  Number of program headers:         1
  Size of section headers:           64 (bytes)
  Number of section headers:         0
  Section header string table index: 0

Program Headers:
  Type           Offset             VirtAddr           PhysAddr
                 FileSiz            MemSiz              Flags  Align
  LOAD           0x0000000000000078 0x0000000000010078 0x0000000000000000
                 0x0000000000000002 0x0000000000000002  R E    1000
</code></pre>
<p>
It works! It's a valid ELF file! Success! And it only took 120 well-crafted bytes of overhead.
<h2 id="gdb-isn-t-always-helpful">
<code>gdb</code> isn't always helpful
</h2>
<p>
As a side note, my first several attempts at this resulted in
<pre><code>$ ./make-loopy && ./loopy
Segmentation fault
</code></pre>
<p>and
<pre><code>$ ./make-loopy && ./loopy
Segmentation fault
</code></pre>
<p>and more
<pre><code>$ ./make-loopy && ./loopy
Segmentation fault
</code></pre>
<p>but sometimes
<pre><code>$ ./make-loopy && ./loopy
Segmentation fault (core dumped)
</code></pre>
<p>instead, before returning some more
<pre><code>$ ./make-loopy && ./loopy
Segmentation fault
</code></pre>
<p>
so at some point in the process I decided to try using <code>gdb</code> instead of just poking at the hex values semi-randomly. And while <code>gdb</code> is sometimes very helpful in telling you exactly what's going on and letting you prod at your code and memory, in this case all I got was:
<pre><code>$ ./make-loopy && gdb ./loopy
Reading symbols from ./loopy...(no debugging symbols found)...done.
(gdb) run
Starting program: ./loopy
During startup program terminated with signal SIGSEGV, Segmentation fault.
(gdb) info registers
The program has no registers now.
(gdb) x 0x10000
0x10000: Cannot access memory at address 0x10000
</code></pre>
<p>
Unfortunately, the executable was causing segfaults <em>while being loaded</em>, so there was no program state for <code>gdb</code> to tell me about. <i lang=fr>À l'impossible nul n'est tenu</i>, so I can't really blame <code>gdb</code>.
<h2 id="an-annotated-version">
An annotated version
</h2>
<p>
If you're curious about the details, the following version of <code>make-loopy</code> is probably more informative and tweakable. You too could get your computer to run some lovingly-crafted bytes!
<pre><code>#!/usr/bin/python3.6

magic_bytes  = "7f 45 4c 46" # "This is an ELF file"
elf_class    = "02"          # 64-bit architecture
byte_order   = "01"          # little-endian
file_type    = "02 00"       # an executable
architecture = "3e 00"       # x86-64

# the memory address where execution starts
entry_point = "78 00 01 00 00 00 00 00"

# mark the program's memory as executable and readable
# (for writable as well, use "07 00 00 00")
code_segment_flags = "05 00 00 00"

# the file offset where the code starts
# (right after the headers)
code_offset = "78 00 00 00 00 00 00 00"

# there are two bytes of code
code_size = "02 00 00 00 00 00 00 00"

# this value works on my machine
alignment = "00 10 00 00 00 00 00 00"

s = f"""
{magic_bytes} {elf_class} {byte_order} 01 00
00 00 00 00 00 00 00 00
{file_type} {architecture} 01 00 00 00
{entry_point}
40 00 00 00 00 00 00 00
00 00 00 00 00 00 00 00
00 00 00 00 40 00 38 00
01 00 40 00 00 00 00 00

01 00 00 00 {code_segment_flags}
{code_offset}
{entry_point}
00 00 00 00 00 00 00 00
{code_size}
{code_size}
{alignment}

eb fe
"""

open('loopy', 'wb').write(bytes([int(b, 16) for b in s.split()]))
</code></pre>
<p>
I only marked a few of the ELF fields, but do look up the Wikipedia page if you're curious about the other values in there. Some points:
<ul>
<li>Unlike in Brian Raiter's blog post, this is an x86-64 executable, rather than a 32-bit i386 executable. Other than that, it's very similar to his shortest version which conforms to the ELF standard.
<li>The smallest usable memory address seems to be 0x10000 if you want to avoid segfaults, which partly explains why <code>entry_point</code> is 0x10078.
<li>The other reason for that choice of address is that, experimentally on my machine, the low 12 bits of <code>entry_point</code> and <code>code_offset</code> have to match (as recorded also in the value of <code>alignment</code>).
<li>You might notice that <code>code_size</code> appears twice in the file. That's because one of them is the code size <em>in the file</em> and the other one is the code size <em>in memory once loaded</em>. The code size in memory can be larger, in which case the end will be padded with zeros, but this only works if you change the <code>code_segment_flags</code> to make the program memory writable (as well as readable and executable).
</ul>
]]></content>
  </entry>
</feed>
