Dear Readers, Hope you all are doing great. In the previous post, we saw how we could create a shellcode for egghunting and ended up creating one of the shortest egghunter shellcode with just under 12 bytes. In this post,  we will only work on encoding the shellcode by combining 3 different encoding schemes. The shellcode we will choose to demo our custom encoder will be a simple /bin/sh shell invoking shellcode. You can download a copy of this simple shellcode from here.  We will write a simple python script which will encode the bytes of the /bin/sh shellcode with our custom encoding scheme. Here is the source of the python script:

Running the python program, we can see that we have encoded shellcode which we can use in decoder stub:

As we can see this is a pretty straightforward Encoder which encodes each byte of the /bin/sh shellcode by XORing it with 0xAA, then XORing it again with 0xCF, then performing a NOT operation on the byte and finally doing a ROT shift of 131 on the byte. Therefore, to create a decoder stub for the shellcode, we will write a simple assembly program as follows:

We can see that it’s a pretty straightforward decoder stub. The first operation it performs is to simply remove the ROT shift of 131 by subtracting the byte with 131. Next, it performs a NOT operation and then finally do a double XOR operation with 0xCF and 0xAA one after the other. After completing these four activities for all the bytes, it directly jumps to the Shellcode. Extracting the bytes of Shellcode from this program, we can just create a C program and test its functionality as follows:

Compiling the C code, we can execute the program and see if works or not:

Works!! Great. We can see that the length of the shellcode is 76 bytes. Let’s run this program in GDB and analyze the step by step decoding process as follows:

We can see that our original /bin/sh encoded shellcode of 25 bytes is visible in the above screenshot. We are currently doing a ROT shift of 131 by subtracting it from the first byte of our encoded shellcode which is 0x2e. This will change the byte 0x2e to 0xab:

Similarly, the next operation is to perform a NOT of the byte which will transform the byte 0xab to 0x54:

Next are two XOR operations, the first with 0xCF which will change the byte to 0x9b and the one with 0xaa will convert the byte to 0x31 which is the original byte of our /bin/sh execve shellcode:

Similarly, all the above operations will happen for rest of the 24 bytes of the shellcode, and before jumping to the decoded shellcode, we will have the following shell code:

This is nothing but the original shellcode of /bin/sh program as we can see in the python encoder Yeepee! We have successfully decoded the entire shellcode with ease. We can name this encoder as RNX2 encoder as XOR-XOR-NOT-ROT operations are followed for encoding and ROT-NOT-XOR-XOR for decoding.

All the files for this tutorial/ exercise are availiable at:

This blog post has been created for completing the requirements of the SecurityTube Linux Assembly Expert certification:

Student-ID: SLAE-1080