tl;dr

• Carefully arranging structs on stack so as to overwrite saved rip , without corrupting the stack canary.
• Leak libc with puts and execute a ret2libc to get shell

tl;dr

• Challenge is a VM implemented over signals and ptrace
• Reverse Instruction types and implementation
• Use gdb scripting to find the executed code and get the pseudo VM code
• Find out the algorithm (Max triangle sum) from VM instructions
• Find an more optimized way to solve the problem (Or lazy solve it!).

tl;dr

• Challenge is a VM implemented over signals and ptrace
• Reverse Instruction types and implementation
• Use gdb scripting to find the executed code and get the pseudo VM code
• Reverse the VM functionality (Hill cipher) for flag and profit