CTF Writeup - Flare-On 2016 - 05: smokestack

• Name - smokestack
• Category - Reverse Engineering
• Points - 5
• Description - n/a
• Binary - Download here

Supplying the binary with random arguments doesn't yield us much so let's see what it binary does and if it accepts an argument, what the restrictions are. The 1st validation is pretty obvious:

Our input has to be 10 chars long, at least (later in the program extra chars are disregarded so essentially len(argv) has to be 10). The binary then continues by adding a couple of bytes to the string, perform MD5 0x100000 times on the result and then uses this to RC4 decrypt a blob in memory. But we're really not interested in this part. The magic happens at function sub_401610().

This part of the program is really complicated and looks like some sort of a VM. It starts by initializing a function table (__cfltcvt_init) and a few global variables and then loops 386 times, each time calling one of the functions in the table:

Each of the functions increments word_40DF1E, which acts like a pointer to our next instruction/data, which is held in word_40A140. Analysing a bit further we notice that one of the functions, sub_401000 writes bytes to our array containing the input string whilst sub_401080 reads from it.

Out of all the functions, sub_401300 is the only one that does a comparison of 2 bytes and as we'll see, it's the function that will confirm or deny if a character of our input is right or wrong:

It starts by calling sub_401080 twice, which reads 2 bytes off our input string, compares them, calls sub_401000, which writes to our input array, with a 1 if the are equal and a 0 if they aren't, and increments word_40DF1E. Essentially this is equal to the CMP instruction in a stack-based VM; POP 2 values off the stack, compare them, PUSH result.

So all we have to do is monitor this function and we're done? Well, yes and no. Yes, because this will be the most important function and no because sometimes we don't need to monitor it; the CMP instruction is not only called to compare 2 bytes directly related to our input, it is used when performing loops too. The latter scenario is easily distinguished though. If the result of the first call to sub_401080 is 0, then the VM is in a loop (in fact you can see the 2nd value obtained from sub_401080 decreasing with each iteration) whereas if it isn't, the 2 values are related to our input.

So what we can do is, we can reprogram the entire VM in python and add a few lines of code to function sub_401080 that will inform us when a comparison is successful or not. We loop this over all printable characters to obtain the first, which is the last in this case since it starts from the end, character of our input. Found below is the massive python script to solve the first char of the input:

 import sys
 import string
 
 dword_40DEE0 = ["sub_401030", "sub_4010C0", "sub_4010E0", "sub_401130", "sub_401180", "sub_4011F0", "sub_401260", "sub_4012B0", "sub_401300", "sub_401360", "sub_4013C0", "sub_4013D0", "sub_401480", "sub_401520"]
  
 word_40A140 = [0x00,0x00,0x21,0x00,0x02,0x00,0x00,0x00,0x91,0x00,0x08,0x00,0x00,0x00,0x16,0x00,0x00,0x00,0x0C,0x00,0x09,0x00,0x0A,0x00,0x0B,0x00,0x00,0x00,0x00,0x00,0x0C,0x00,0x02,0x00,0x0C,0x00,0x00,0x00,0x00,0x00,0x1D,0x00,0x0A,0x00,0x0B,0x00,0x00,0x00,0x00,0x00,0x63,0x00,0x02,0x00,0x0C,0x00,0x00,0x00,0x00,0x00,0x18,0x00,0x06,0x00,0x00,0x00,0x54,0x00,0x08,0x00,0x00,0x00,0x33,0x00,0x00,0x00,0x29,0x00,0x09,0x00,0x0A,0x00,0x0B,0x00,0x00,0x00,0x00,0x00,0x2C,0x00,0x02,0x00,0x0C,0x00,0x00,0x00,0x00,0x00,0x3D,0x00,0x0A,0x00,0x00,0x00,0x0E,0x00,0x01,0x00,0x0B,0x00,0x00,0x00,0x00,0x00,0x59,0x00,0x02,0x00,0x0C,0x00,0x00,0x00,0x0B,0x00,0x00,0x00,0x00,0x00,0x00,0x00,0x0C,0x00,0x01,0x00,0x00,0x00,0x09,0x00,0x0C,0x00,0x00,0x00,0x0B,0x00,0x01,0x00,0x00,0x00,0x02,0x00,0x02,0x00,0x0C,0x00,0x01,0x00,0x0B,0x00,0x00,0x00,0x00,0x00,0x01,0x00,0x03,0x00,0x0C,0x00,0x00,0x00,0x0B,0x00,0x00,0x00,0x00,0x00,0x00,0x00,0x08,0x00,0x00,0x00,0x47,0x00,0x00,0x00,0x60,0x00,0x09,0x00,0x0A,0x00,0x0C,0x00,0x00,0x00,0x0B,0x00,0x01,0x00,0x03,0x00,0x00,0x00,0x5D,0x00,0x08,0x00,0x00,0x00,0x7C,0x00,0x00,0x00,0x6E,0x00,0x09,0x00,0x0A,0x00,0x0B,0x00,0x00,0x00,0x00,0x00,0x07,0x00,0x03,0x00,0x0C,0x00,0x00,0x00,0x00,0x00,0x5B,0x00,0x0C,0x00,0x01,0x00,0x00,0x00,0x87,0x00,0x0A,0x00,0x00,0x00,0x36,0x00,0x0C,0x00,0x01,0x00,0x0B,0x00,0x00,0x00,0x0B,0x00,0x01,0x00,0x02,0x00,0x0C,0x00,0x01,0x00,0x0B,0x00,0x01,0x00,0x00,0x00,0x58,0x00,0x02,0x00,0x06,0x00,0x00,0x00,0xF9,0x00,0x08,0x00,0x00,0x00,0xA0,0x00,0x00,0x00,0x96,0x00,0x09,0x00,0x0A,0x00,0x0B,0x00,0x00,0x00,0x00,0x00,0x4D,0x00,0x06,0x00,0x0C,0x00,0x00,0x00,0x00,0x00,0xAE,0x00,0x0A,0x00,0x00,0x00,0x23,0x03,0x00,0x00,0x2B,0x01,0x03,0x00,0x0C,0x00,0x01,0x00,0x0B,0x00,0x00,0x00,0x0B,0x00,0x01,0x00,0x02,0x00,0x0C,0x00,0x01,0x00,0x0C,0x00,0x01,0x00,0x0B,0x00,0x01,0x00,0x0B,0x00,0x01,0x00,0x00,0x00,0x01,0x00,0x03,0x00,0x0C,0x00,0x01,0x00,0x00,0x00,0x03,0x00,0x02,0x00,0x0B,0x00,0x01,0x00,0x00,0x00,0x00,0x00,0x08,0x00,0x00,0x00,0xB2,0x00,0x00,0x00,0xC7,0x00,0x09,0x00,0x0A,0x00,0x07,0x00,0x00,0x00,0x77,0xFE,0x08,0x00,0x00,0x00,0xD8,0x00,0x00,0x00,0xD1,0x00,0x09,0x00,0x0A,0x00,0x0B,0x00,0x00,0x00,0x00,0x00,0x58,0x00,0x02,0x00,0x0C,0x00,0x00,0x00,0x00,0x00,0x03,0x00,0x04,0x00,0x00,0x00,0x8C,0x00,0x02,0x00,0x00,0x00,0x94,0x60,0x08,0x00,0x00,0x00,0xEE,0x00,0x00,0x00,0xE7,0x00,0x09,0x00,0x0A,0x00,0x0B,0x00,0x00,0x00,0x00,0x00,0xE7,0x00,0x02,0x00,0x0C,0x00,0x00,0x00,0x0B,0x00,0x01,0x00,0x02,0x00,0x00,0x00,0x0C,0x00,0x06,0x00,0x00,0x00,0x74,0x00,0x08,0x00,0x00,0x00,0x07,0x01,0x00,0x00,0xFD,0x00,0x09,0x00,0x0A,0x00,0x0B,0x00,0x00,0x00,0x00,0x00,0x09,0x00,0x03,0x00,0x0C,0x00,0x00,0x00,0x00,0x00,0x1D,0x01,0x0A,0x00,0x00,0x00,0x0A,0x00,0x0C,0x00,0x01,0x00,0x0B,0x00,0x01,0x00,0x00,0x00,0x01,0x00,0x03,0x00,0x0C,0x00,0x01,0x00,0x0B,0x00,0x01,0x00,0x00,0x00,0x00,0x00,0x08,0x00,0x00,0x00,0x0B,0x01,0x00,0x00,0x1D,0x01,0x09,0x00,0x0A,0x00,0x00,0x00,0x06,0x00,0x05,0x00,0x00,0x00,0xC0,0x1D,0x08,0x00,0x00,0x00,0x33,0x01,0x00,0x00,0x29,0x01,0x09,0x00,0x0A,0x00,0x0B,0x00,0x00,0x00,0x00,0x00,0x71,0x00,0x02,0x00,0x0C,0x00,0x00,0x00,0x00,0x00,0x3D,0x01,0x0A,0x00,0x0B,0x00,0x00,0x00,0x00,0x00,0x77,0x00,0x02,0x00,0x0C,0x00,0x00,0x00,0x00,0x00,0x3D,0x01,0x0A,0x00,0x00,0x00,0x16,0x00,0x02,0x00,0x00,0x00,0x0E,0x00,0x03,0x00,0x00,0x00,0x61,0x00,0x08,0x00,0x00,0x00,0x53,0x01,0x00,0x00,0x4C,0x01,0x09,0x00,0x0A,0x00,0x0B,0x00,0x00,0x00,0x00,0x00,0x2C,0x00,0x03,0x00,0x0C,0x00,0x00,0x00,0x0C,0x00,0x01,0x00,0x0B,0x00,0x01,0x00,0x00,0x00,0x2C,0x21,0x0B,0x00,0x01,0x00,0x00,0x00,0x01,0x00,0x03,0x00,0x0C,0x00,0x01,0x00,0x00,0x00,0x07,0x00,0x03,0x00,0x0B,0x00,0x01,0x00,0x00,0x00,0x00,0x00,0x08,0x00,0x00,0x00,0x59,0x01,0x00,0x00,0x6E,0x01,0x09,0x00,0x0A,0x00,0x00,0x00,0xCA,0x01,0x06,0x00,0x00,0x00,0xF5,0x1F,0x08,0x00,0x00,0x00,0x81,0x01,0x00,0x00,0x7A,0x01,0x09,0x00,0x0A,0x00,0x0B,0x00,0x00,0x00,0x00,0x00,0x12,0x00,0x02,0x00,0x0C,0x00,0x00,0x00,0x0D,0x00]
  
 word_40DF18 = 0
 word_40DF1A = 0
 word_40DF1C = 9
 word_40DF1E = 0
 word_40DF20 = [0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0]
 
 failure = False
 checking_for_char = 1
 checking_for_char_currently = 0
 
 def sub_401000(input):
  global word_40DF1C
  global word_40DF20
  
  word_40DF1C = word_40DF1C + 1
  input1 = (input & 0xFF00) >> 8
  input2 = input & 0xFF
  word_40DF20[word_40DF1C * 2] = input2
  word_40DF20[(word_40DF1C * 2) + 1] = input1
  
  
 def sub_401030():
  global word_40DF1E
  
  word_40DF1E = word_40DF1E + 1
  result1 = word_40A140[word_40DF1E * 2]
  result2 = word_40A140[(word_40DF1E * 2) + 1]
  result =  (result2 << 8) | result1
  sub_401000(result)
  word_40DF1E = word_40DF1E + 1
  
 def sub_4010C0():
  global word_40DF1E
 
  word_40DF1E = word_40DF1E + 1
  sub_401080()
 
 def sub_401080():
  global word_40DF1C
  
  result1 = word_40DF20[word_40DF1C * 2]
  result2 = word_40DF20[(word_40DF1C * 2) + 1]
  result =  (result2 << 8) | result1
  word_40DF1C = word_40DF1C - 1
  return result
  
 def sub_4010E0():
  global word_40DF1E
  
  result1 = sub_401080()
  result2 = sub_401080()
  result3 = result1 + result2
  sub_401000(result3)
  word_40DF1E = word_40DF1E + 1
  
 def sub_401130():
  global word_40DF1E
 
  result1 = sub_401080()
  result2 = sub_401080()
  result3 = result2 - result1
  sub_401000(result3)
  word_40DF1E = word_40DF1E + 1
  
 def sub_401180():
  global word_40DF1E
 
  result1 = sub_401080()
  result2 = sub_401080()
  result3 =  ((result2 << (16 - result1))  |  (result2 >> result1)) & 0xFFFF
  sub_401000(result3)
  word_40DF1E = word_40DF1E + 1
  
 def sub_4011F0():
  global word_40DF1E
 
  result1 = sub_401080()
  result2 = sub_401080()
  result3 = ((result2 >> (16 - result1))  |  (result2 << result1)) & 0xFFFF
  sub_401000(result3)
  word_40DF1E = word_40DF1E + 1
  
 def sub_401260():
  global word_40DF1E
 
  result1 = sub_401080()
  result2 = sub_401080()
  result3 = result1 ^ result2
  sub_401000(result3)
  word_40DF1E = word_40DF1E + 1
  
 def sub_4012B0():
  global word_40DF1E
  
  result1 = sub_401080()
  result2 = ~result1 & 0xFFFF
  sub_401000(result2)
  word_40DF1E =  word_40DF1E + 1
 
 def sub_401300():
  global word_40DF1E
  global failure
  global checking_for_char_currently
 
  result1 = sub_401080()
  result2 = sub_401080()
  if result1 == result2:
   sub_401000(1)
  else:
   sub_401000(0)
   
  if (result1 != 0): 
   if result1 == result2:
    checking_for_char_currently = checking_for_char_currently + 1
   else:
    failure = True
   
  word_40DF1E = word_40DF1E + 1
  
 def sub_401360():
  global word_40DF1E
  
  result1 = sub_401080()
  result2 = sub_401080()
  result3 = sub_401080()
  if result3 == 1:
   sub_401000(result1)
  else:
   sub_401000(result2)
  word_40DF1E = word_40DF1E + 1
  
 def sub_4013C0():
  global word_40DF1E 
 
  result = sub_401080()
  word_40DF1E = result
  
 def sub_4013D0():
  global word_40DF1E
  global word_40DF18
  global word_40DF1A
  global word_40DF1C
  
  temp = 0
  word_40DF1E =  word_40DF1E + 1
  result = word_40A140[word_40DF1E * 2]
  if result == 0:
   temp = word_40DF18
  elif result == 1:
   temp = word_40DF1A
  elif result == 2:
   temp = word_40DF1C
  elif result == 3:
   temp = word_40DF1E
  sub_401000(temp)
  word_40DF1E = word_40DF1E + 1
   
 def sub_401480():
  global word_40DF1E
  global word_40DF18
  global word_40DF1A
  global word_40DF1C
  
  word_40DF1E =  word_40DF1E + 1
  result = word_40A140[word_40DF1E * 2]
  result1 = sub_401080()
  if result == 0:
   word_40DF18 = result1
  elif result == 1:
   word_40DF1A = result1
  elif result == 2:
   word_40DF1C = result1
  elif result == 3:
   word_40DF1E = result1
  word_40DF1E = word_40DF1E + 1
 
 def sub_401520():
  global word_40DF1E
  word_40DF1E = word_40DF1E + 1
  
 def call_function(index): 
  result = getattr(sys.modules[__name__], dword_40DEE0[word_40A140[index]])()
  return result
  
 if __name__ == "__main__":
  for x in string.printable:
   #initialize global variables
   failure = False
   word_40DF20 = [0,0, 0,0, 0,0, 0,0, 0,0, 0,0, 0,0, 0,0, 0,0, ord(x),0, 0,0, 0,0]
   word_40DF18 = 0
   word_40DF1A = 0
   word_40DF1C = 9
   word_40DF1E = 0
   
   #perform algorithm
   while word_40DF1E < 386:
    call_function(word_40DF1E * 2)
    if failure == True:
     break
 
   if checking_for_char_currently < checking_for_char:
    checking_for_char_currently = 0
   else:
    print "Character is", x
    break

The functions are a direct python implementation of the ones found in the VM. The main function loops over all printable characters to find the 10th character of our input string. As mentioned earlier, our input is copied into a global variable of Words, and hence for each character there are 0's (lines 12 and 196). An extra 2 Words were intialised at the end as they're required by the program as work space. The program then executes the VM and if the failure variable is set to True, we break.

Lines 119 - 123 have been added and are not part of the original VM. The purpose of these is to inform us if the 2 values popped from the stack are equal. If they are then we have successfully bruteforced a single character of our input, and we don't need to look further.

Lines 208 - 212 ensure that if we have the same number of successes as the number of char we're bruteforcing, then we have found the right input char. This is necessary as the sub_401300 records a success pass for each of the right inputs. For example, if we're bruteforcing the 3rd char and we get 2 success passes, it does not mean that we found the right char; it only confirms that the other 2 chars we've given it are right, which we would've already known from the previous 2 bruteforce attempts. If we're bruteforcing the 3rd char, we require 3 successes.

Running the program we get:

Command Prompt

C:\>python vm_implementation.py Character is p C:\>

We then modify the program to get the second character. These are the modifications required:

 [ .. snip .. ]

 checking_for_char = 2

 [ .. snip .. ]

 word_40DF20 = [0,0, 0,0, 0,0, 0,0, 0,0, 0,0, 0,0, 0,0, ord(x),0, ord('p'),0, 0,0, 0,0]

 [ .. snip .. ]

We discover that the 2nd to last char is 'L'. We keep on going until we get to the last character:

 [ .. snip .. ]

 checking_for_char = 10

 [ .. snip .. ]

 word_40DF20 = [ord(x),0, ord('Y'),0, ord('w'),0, ord('x'),0, ord('C'),0, ord('b'),0, ord('J'),0, ord('o'),0, ord('L'),0, ord('p'),0, 0,0, 0,0]

 [ .. snip .. ]

At the end we get:

Command Prompt

C:\>smokestack.exe kYwxCbJoLp A_p0p_pu$H_&_a_Jmp@flare-on.com C:\>