The three relevant bits from the Counting Mode register are below.  You probably need to adjust the Index_Phase
 
G0_Index_Phase  bit: 6..5  
This bitfield determines the state of the A and B quadrature signals, and is where the index (or Z signal) is acted upon. The Z index can span multiple phases of the quadrature, but you should reload the counter in only one of the phases. 
0: A low, B low
1: A low, B high
2: A high, B low
3: A high, B high
 
G0_Index_Mode  bit: 4
The effect of setting this bit depends on the counting mode. 
Quadrature Mode:  The index (Z) reloads the counter in the phase determined by the value of G1_Index_Phase when the Z input is high.
Two-Pulse Mode:  The counter reloads while Z is active.
Fast Gate Mode:  The gate is synchronized to the counter clock.
Synchronous Source Mode:  Disables edge detection on the source pin, allowing it to be asynchronous.
 
G0_Counting_Mode  bit: 2..0
This field determines the counting mode to use to interface to different encoders and applications. 
0: Normal counting mode
1: Quadrature Mode X1
2: Quadrature Mode X2
3: Quadrature Mode X4
4: Two-Pulse Mode
5: Fast Gate Mode