SSE and AVX Mutation Idea (xlate)

Table of contents

SSE and AVX Mutation Idea (xlate)

All Streaming SIMD Extensions (SSE) instructions using legacy encoding can be translated to Advanced Vector Extensions (AVX) encoding. This is already something that most compilers offer these days when using the correct compilation flag (i.e., -QxAVX with MSVC) to compile all legacy SSE instructions into their AVX counterpart.

Compilers will mostly do that for optimisation reason; however, in our case, this is not necessarily what we are looking for. What is more interesting is to be able to modify the encoding of instructions whilst not modifying their operands and the result of their execution. For this very reason a module for a mutation engine could be developed to translate all legacy SSE instructions to AVX format or from AVX to Legacy SSE format to modify the signature of a piece of code.

Therefore, the objective of this paper is to shortly discuss what SSE and AVX are and how it is possible to switch from one format to another without too many difficulties.

Streaming SIMD Extensions (SSE)

Before SSE there was the MMX facilities. The MMX facilities is the first to implement the concept of Single-Instruction, Multiple Data (SIMD), an instruction set that can perform arithmetic and logical operations on multiple data (i.e., bytes, words, or dwords) – hence the name SIMD. The goal being to reduce memory access operations which take a lot of time to process. This is quite useful for modern media, communication and graphic applications.

Example of SIMD operation: addss xmm1,xmm2

However, MMX data registers are aliases for the low 64-bit part of the x87 FP data registers (i.e., ST(0) – ST(7)) on top of limitations or edge cases when a routine uses both instruction sets (e.g., data lose and performance issues). Additionally, MMX facilities does not support operations on floating point values.

This is what the first version of the SSE facilities tries to address via new SIMD and non-SIMD instructions. Also, with it comes completely new 128-bit data registers (i.e., XMM0 – XMM7) which can be used to operate on scalar (single integer/FP for non-SIMD) or packed (for SIMD) operands, such as:

• 16 bytes packed integers
• 8 words packed integers
• 4 dwords packed integers
• 2 qword packed integers
• 4 single-precision (SP) floating point
• 2 double-precision (DP) floating point

Other features have been implemented such as:

• Enhancement of specific type of memory write operation on cacheable memory via Non-Temporal stores (streaming stores) instructions, for example: MOVNTPS, MOVNTQ, or MASKMOVQ;
• Video and media-specific instructions, for example: PAVD, LDDQU;
• Thread synchronisation instructions, for example: PAUSE, MONITOR, MWAIT; and
• Legacy prefix branch hints to help with misprediction when dealing with conditional branches.

Requesting the feature flags via the CPUID instruction can be used to identify whether a specific SSE instruction set is available.

Simple MASM code that can be used in conjunction with the above table, example for SSE:

SSESupport PROC 
    xor eax, eax
    inc eax
    cpuid

    and edx, 002000000h
    shr edx, 19h
    xchg eax, edx 
    ret
SSESupport ENDP 

Advanced Vector Extensions (AVX)

Introduced with Sandy Bridge (Q1 2011) Intel processors, the AVX facilities continue to enhance SIMD functionalities and offers new features such as:

• New 256-bit data register (i.e., YMM0 – YMM7) with the lowest 128-bit being aliases for XMM data registers.
• New encoding with the new 2- or 3-byte VEX prefix for legacy SSE instructions and new AVX instructions; and
• Non-destructive operand operations to reduce the number of copies and load operations; and
• Up to three source operands (with 4 operands instructions) using the upper 4-bit of a 8-bit immediate data value (i.e., VEX[vvvv] + ModRM[rm] + ModRM[reg] + imm8[7:4]).

Note that for this very paper there I will not go too much into AVX2 and I will not cover AVX-256 and Fused-Multiply-Add (FMA) extensions, which brings even more functionalities. Also, with AVX-256 another encoding is possible with the EVEX prefix (i.e., replacement of VEX).

Before being able to use AVX it is important to check that both the processor and the operating system does support the AVX instruction set and 256- and 128-bit data registers. The following MASM code can be used to check the aforementioned from either User-Mode (UM) or Kernel-Mode (KM):

AVXSupport PROC
    ; Get features flags 
    xor eax, eax 
    inc eax 
    cpuid 

    mov eax, ecx 
    mov ebx, ecx 

    ; Check for OSXSAVE and AVX support 
    and eax, 008000000h
    shr eax, 1Bh
    and ebx, 010000000H
    shr ebx, 1Ch

 
    ; Get return value 
    and al, bl
    jz return

get_xcr0: 
    xor ecx, ecx 
    XGETBV 
    mov ebx, eax 

    ; Check for XMM and YMM registers 
    and eax, 04h
    shr eax, 02h
    and ebx, 02h
    shr ebx, 01h

    ; Get return value
    and al, bl 
    return:
    ret 
AVXSupport ENDP

AVX and the new VEX

As mentioned in the previous section, AVX offers a new way to encode instructions (including legacy SSE) with a compact 2- or 3-byte Vector Extension (VEX) prefix, respectively starting with C5h and C4h bytes. The composition of the 2- and 3-byte VEX prefix can be found in the Intel documentation:

Fields are as follows:

• R: like REX[R] in 1’s complement (inverted) form:
  • 1b: Same as REX[R] = 0b (must be set to this in 32-bit mode otherwise LES/LDS)
  • 0b: Same as REX[R] = 1b (64-bit mode only)
• X: Like REX[X] in 1’s compliment (inverted) form:
  • 1b: Same as REX[X] = 0b (must be set to this in 32-bit mode otherwise LES/LDS)
  • 0b: Same as REX[X] = 1b (64-bit mode only)
• B: Like REX[B] in 1’s compliment (inverted) form:
  • 1b: Same as REX[B] = 1b (ignored in 32-bit mode)
  • 0b: Same as REX[B] = 0b (64-bit mode only)
• W: This can either be used like REX[W] or as an additional escape extension. This will be opcode specific.
• m-mmmm: Used to specify opcode escape sequence (will always be 0Fh when using 2-byte VEX prefix):
  • 00000b: Reserved and will #UD
  • 00001b: Implied 0Fh escape opcode (Table 2)
  • 00010b: Implied 0F38h escape opcodes (Table 3)
  • 00011b: Implied 0F3Ah escape opcodes (Table 4)
  • 00100b-11111b: Reserved will #UD
• vvvv: Used in conjunction with a ModRM byte to specify an additional register as source or destination. This is encoded in 1’s complement form (inverted) or 1111b if unused:
  • 1111b: XMM0/YMM0
  • 1110b: XMM1/YMM1
  • 1101b: XMM2/YMM2
  • 1100b: XMM3/YMM3
  • 1011b: XMM4/YMM4
  • 1010b: XMM5/YMM5
  • 1001b: XMM6/YMM6
  • 1000b: XMM7/YMM7
  • 0111b: XMM8/YMM8
  • 0110b: XMM9/YMM9
  • 0101b: XMM10/YMM10
  • 0100b: XMM11/YMM11
  • 0011b: XMM12/YMM12
  • 0010b: XMM13/YMM13
  • 0001b: XMM14/YMM14
  • 0000b: XMM15/YMM15
• L: Vector length bit used to promote operands to 256-bit:
  • 0b: scalar or 128-bit vector operand
  • 1b: 256-bit vector operand
• pp: Specify a SIMD prefix used as an additional escape opcode:
  • 00b: None
  • 01b: 66h
  • 10b: F3h
  • 11b: F2h

Additionally, for a very small subset of AVX2 instructions, a new Vector SIB (VSIB) byte can be used for memory addressing. This is a special case that will not be discussed there. You can find the list of AVX instructions encoded with a VEX prefix that do uses a VSIB byte later in this paper.

Finally, like all the other prefixes (e.g., Operand Size Override or REX), VEX must be positioned before the primary opcode. Additionally, as shown in the above schema, VEX has bit fields equivalent to the REX prefix, to encode escape opcodes (i.e.,0Fh, 0F38h and 0F3Ah) and SIMD prefixes (i.e., 66h, F2h and F3h). Therefore, if any of them are used with a VEX prefix, an Undefined Instruction (i.e., #UD) exception will be raised.

Translation between Legacy SSE to AVX

As aforementioned, all legacy SSE instructions can be converted into AVX format. However, it does not mean that all AVX instructions can be encoded in legacy SSE format. Over the years SSE has been deprecated and newest instructions can only be encoded using VEX (or EVEX if we consider AVX-512).

Additionally, there is three things to be careful about when translating instructions:

• First, some instructions when encoded via VEX uses an additional non-destructive (ND) operand to limit the number of read/write access to/from registers and memory addresses. It means that the ND operand which is encoded with VEX[vvvv] needs to be interpreted. Note that this ND operand can be either a source or a destination; however, only a subset on AVX instructions (13) uses VEX[vvvv] as a destination operand (1st operand) and only AVX2 instructions using VSIB byte and few others uses (13) VEX[vvvv] as a second source operand (3rd operand).
• Second, again, some AVX instructions does not exist in a legacy encoding format (lot of AVX2 and all AVX-512). Therefore, attending to encode them in a legacy format will produce nonsense at best or #UD
• Thirdly, any AVX instructions operating on 256-bit operands (i.e., VEX[L] = 1b) cannot be encoded in legacy SSE format because of the data limitation of the XMM data registers (i.e., 128-bits).

Later in this paper the lists of instructions using an additional non-destructive operand and AVX only instructions are provided. There is also the list of AVX2 instructions using VSIB for vector memory access addressing.

Example 1: Basic 2-byte VEX Encoded Instruction

2-byte VEX prefix will be used because no non-destructive additional register is required, the general-purpose data register do not need to be promoted to 64-bit, and, the instruction operates on 128-bit XMM register. Elements to consider:

• SIMD prefix 66h: VEX[pp] = 01b
• Escape opcode 0Fh: implied with a 2-byte VEX prefix
• 64-bit register promotion W0: VEX[W] = 0b (not used in 2-byte VEX version)
• 128-bit only operands: VEX[L] = 0b

Example 2: Basic 3-byte VEX Encoded Instruction with 64-bit

3-byte VEX prefix will be used because the general-purpose data register DO need to be promoted to 64-bit. Even if non-destructive additional register is required and the instruction operates only on 128-bit XMM register. Elements to consider:

• SIMD prefix 66h: VEX[pp] = 01b
• Escape opcode 0Fh: VEX[m-mmmm] = 00001b
• 64-bit register promotion W1: VEX[W] = 0b
• 128-bit only operands: VEX[L] = 0b

Example 3: 3-byte VEX Encoded Instruction with SIB

Similar instruction as example two but with a SIB byte and an immediate data value. Also, special case when the AVX instruction does not care about VEX[W] bit field (i.e., WIG). Elements to consider:

• SIMD prefix 66h: VEX[pp] = 01b
• Escape opcode 0Fh: VEX[m-mmmm] = 00001b
• 64-bit base register promotion: VEX[B] = 0b
• 128-bit only operands: VEX[L] = 0b

Example 4: With Non-Destructive Operand

In this example there is an instruction using an additional non-destructive operand which will be encoded in the VEX[vvvv] bit field. The complete list of such instruction can be found in one of the tables below. It is important to note that the destination operand need to be encoded twice to prevent unwanted read/write to/from a register or memory address. Elements to consider:

• SIMD prefix 66h: VEX[pp] = 01b
• Source operand XMM4: VEX[vvvv] = 1011b
• Escape opcode 0Fh: implied with a 2-byte VEX prefix
• 128-bit only operands: VEX[L] = 0b

WIB and Synonymous Mutation

Some AVX instructions does not care about the state of the VEX[W] bit – it will be ignored. It means that when encoding an AVX instructions with such specification, it is possible to set the bit to either 0 or 1 and therefore generate again another encoding. The modification is small but enough to modify a byte and thus break signature in some cases.

Let’s take for example the following 2 instructions:

C4 C1 79 D6 4C 4A 01		vmovq mmword ptr [r10+rcx*2+1],xmm1
C4 C1 F9 D6 4C 4A 01 		vmovq mmword ptr [r10+rcx*2+1],xmm1

In the first version VEX[W] = 0b while on the second version VEX[W] = 1b. Both are valid ways to encode this instruction, will execute and not #UD. Obviously, it will work only when instruction is encoded using 3-byte VEX prefix.

Final Notes

First, when looking at all the examples, it is possible to see that when translating from SSE to AVX or from AVX to SSE encoding the ModRM (and potentially SIB) byte, memory displacement and immediate data value are not affected at all. Only the legacy prefixes used as SIMD prefixes, REX prefix, escape opcode and primary opcodes will be modified, making the whole process easier.

It should be noted that mixing both legacy SSE code and AVX is impacting badly the performances of the CPU. AVX modify the upper bits of the YMM data registers while Legacy SSE instructions cannot modify them. As a result, the upper bits can be in a clean, modified and unsaved (also known as dirty), or preserved/Non_INIT state. As a result, when executing an SSE instruction after an AVX instruction, and vis versa, the processor need to save the state of the register (equivalent to a XSAVE instruction).

Therefore, if different code block can be identified, the VZEROUPPER instruction should be executed before and after executing AVX instructions to clean the upper bits of the YMM registers and set them in a clean mode.

To assite in the understanding of this paper, the following resources can be consulted:

• Volume 1: Basic Architecture - Chapter 10: Programming with Intel Streaming SIMD Extensions (Intel SSE)
• Volume 1: Basic Architecture - Chapter 14: Programming with AVX, FMA and AVX2
• Volume 2: Instruction Set Reference, A-Z - 2.1 Instruction Format for Protected Mode, Real-Address Mode, and Virtual-8086 Mode
• Volume 2: Instruction Set Reference, A-Z - 2.2 IA-32e Mode
• Volume 2: Instruction Set Reference, A-Z - 2.3 Intel Advanced Vector Extensions (Intel AVX)
• Chapter 11: Optimization for Intel AVX, FMA and AVX2 - 11.3 Mixing AVX Code with SSE Code

Appendix A : Non-Destructive Operands Instructions

The table below list all the instructions that can be directly translated from AVX VEX encoding to legacy SSE and from legacy SSE encoding to AVX VEX.

Appendix B : AVX Only Instructions

The table below lists all the instructions (mostly AVX2) that can only be encoded with a VEX prefix. Table to Markdown

Appendix C : VSIB Instructions

The table below list all the AVX instructions encoded with a VEX prefix that do use a VSIB.

Appendix D : VEX.vvvv Has Destination Operand Instructions

The table below list all the AVX instructions that are using the VEX[vvvv] bit field to define a destination operand.

Appendix E : VEX.vvvv Has Third Operand

The table below list all the AVX instructions that are using the VEX[vvvv] bit field to define a second source operand (3rd operand).

Appendix F: WIB Instructions

The table below list all the AVX instructions that silently ignore the VEX[W] bit when encoded with a 3-byte VEX prefix.