Several little tidbits:
1. You refer to "externally controlled", which I assume means that you're configuring the counter for edge counting, with externally controlled count direction. You really should use one of the "position measurement" counter modes which will decode your encoder's quadrature input more reliably.
2. If you're using PFI 35 (Source 1) for channel A, you *should* be using PFI 33 (Aux 1) for channel B. PFI 8 corresponds to a default output pin from a different counter. Somewhere you've gotten bad info on pin assignments.
3. Since your "B" signal is in the wrong place, then when you specify externally controlled count direction, there's no signal truly controlling it. That empty terminal is just floating at the board default. If memory serves, it floats high due to an onboard weak pull-up resistor which in turn causes perpetual decrements.
4. In MAX when you specify "edge counting", you cannot specify externally controlled count direction. So it defaults to standard "increment on every edge" behavior.
5. You can easily convert unsigned 32-bit values like 4.2 G into more manageable values like -1,-2,-3,... by using the "To I32" numeric conversion function. (In a subpalette under the numeric functions).
For more general info, examples, etc., check the shipping examples, docs, and this site using keywords like "position measurement", quadrature, encoder, etc.
-Kevin P.
ALERT! LabVIEW's subscription-only policy came to an end (finally!). Unfortunately, pricing favors the captured and committed over new adopters -- so tread carefully.