I am still struggling with the interrupt stopping for no apparent reason. It seems to have something to do with system load -- the higher the load is, the less the chance becomes that the interrupt stops.
In such an event, it can be that my ISR is still called (because it is in the chain with other ISRs), and the AO Overrun bit is set in the control register, while the Interrupt B bit is clear. So the card works, but stopped sending FIFO interrupts.