The three relevant bits from the Counting Mode register are below. You probably need to adjust the Index_Phase
G0_Index_Phase bit: 6..5
This bitfield determines the state of the A and B quadrature signals, and is where the index (or Z signal) is acted upon. The Z index can span multiple phases of the quadrature, but you should reload the counter in only one of the phases.
0: A low, B low
1: A low, B high
2: A high, B low
3: A high, B high
G0_Index_Mode bit: 4
The effect of setting this bit depends on the counting mode.
Quadrature Mode: The index (Z) reloads the counter in the phase determined by the value of G1_Index_Phase when the Z input is high.
Two-Pulse Mode: The counter reloads while Z is active.
Fast Gate Mode: The gate is synchronized to the counter clock.
Synchronous Source Mode: Disables edge detection on the source pin, allowing it to be asynchronous.
G0_Counting_Mode bit: 2..0
This field determines the counting mode to use to interface to different encoders and applications.
0: Normal counting mode
1: Quadrature Mode X1
2: Quadrature Mode X2
3: Quadrature Mode X4
4: Two-Pulse Mode
5: Fast Gate Mode