PXIe board generated TLPs not arriving at host

[GraemeP]

We're running a 1062Q rack with an 8370+8371 and MXI-e x4 cable. No other cards are present in the rack (with the exception of the ones we are trying to test; see below).

The host PCs (3 tested) are based on either Gigabyte or ASUS motherboards (all latest BIOS and drivers installed).

We have developed two PCIe card; one a standard PCIe card (PC based, selectable as either 1-lane or 4-lane) and the other a PXIe card for the 1062Q rack (only 1-lane).

The PCIe based card works flawlessly in the PC. The PXIe card also works in the PC (via an in-house adaptor). Either card in the PC or 1062Q works in as much as host PCIe enumeration is sucessful and their internal operational control registers and memory can be written and read.

The problem arrises when the PXIe plug-in card attempts to perform MEMW TLPs (that is, DMA writes to the host memory) when it is in the 1062Q rack. The DMA writes never arrive at the host and eventually the host will lock-up.

When either card is directly connected to the host, then the DMA works fine.

We're currently at a loss in explaining this, and have started to look for a PXIe bus exerciser.

Any pointers for what we can do next? 

[Robert__R]

That's an interesting problem.  Everything works separately, but they don't work together.

It sounds to me like the DMA is writing to the wrong location.  The fact that the host locks up is the biggest clue for that.  I'm guessing that the DMA starts going, then eventually hits somewhere that matters.  The bigger question is "Why?".

There are several possibilities, depending on how you count. 

1.  Something in the driver is hard-coded for a direct-connect slot (less likely).

2.  Additional latency across the MXI cards exposes a race condition in the driver.

3.  Maybe the resource assignments in the chassis are different in a critical way.  It could introduce an interrupt sharing that causes the driver to get an interrupt before the HW sends it.  Or there could be a bug in the BAR register.

4.  The max packet size could be mis-programmed, causing a loss of packets (an easy check, I'd start here). 

Have you looked at the error reporting registers in your device and the 8370 switch?  You'll want to look at the specific port just above your device.  You'll probably need to clear any previous errors left over from startup.

Can you have your device only initiate a few TLPs?  Then check to see if it thinks it's writing to the same place you think it should.

- Robert

[ericg]

Hi,

Is it possible that the BIOS on the PC is mis-configuring the Max_Payload_Size field for the endpoints in this system?  We've observed that on some systems, the BIOS will incorrectly program the Max_Payload_Size field; it should be using a "least common denominator" approach (i.e., program the MPS on each device to the smallest MPS value supported in the system), but sometimes the BIOS will just write a larger value to an endpoint or root port, ignoring the capabilities of the switches in between.  If a TLP with an unsupported payload size arrives at a switch, the switch will report an "unsupported request" error and the system could hang.

I'd check the BIOS for a way to tweak the Max_Payload_Size field, and if it exists, try setting it to 128B.  If no such BIOS feature exists, you can tweak the MPS directly using a PCI configuration program (we can suggest some if you don't have one already).

Thanks,

Eric G.

[GraemeP]

Thanks for the reply. Yes, it is an interesting problem.

We're using windriver on the host at present. This can lock and reserve memory and provides a physical address which is passed to the add-in card. It also produces a user address for the host app; both of these addresses, although looking different, point to the same physical place.

Then we can generate just a single TLP with any payload. We've attempted payloads from 8 bytes to 128 bytes.

As this PCIe core is based on an FPGA, we can monitor the internal signals to see how it's behaving. It all looks good, the packet address is the one we supplied via the windriver/host application. But when the packet has left our board, it doesn't end up in the host's memory.

We can run these packets without any interrupts.

The host motherboard has a max packet size of 128 bytes, and this seems to be the same vallue programmed into our interface.

No reported errors in our device. I'll see what I can find out about the 8370.

[ericg]

Graeme,

Are you familiar with the Windows Debugger (WinDbg)?  It's a system level debugger that can be useful for debugging problems like this.  We use it extensively here at NI to debug similar problems with our own PCIe systems.  Basically, you attach a host system to the target-to-debug using a serial cable, and once the system is "dead", you can break in and interrogate it.  For PCIe problems, we typically look for PCI/PCIe error reporting information like SERR asserted, Advanced Error Reporting information, etc.  If WinDbg is a tool that's available to you, I'd recommend you set it up and try to break in to the target system after it stalls and look for PCI error reporting information.  This is not always possible, but when it is, it's usually a quick path to debugging these types of problems.

Also, if you can dump the PCI configuration for the portion of the fabric in between and including your endpoint and the PCIe root port, we can look for possible configuration errors.  WinDbg can do this, or there are other tools that sometimes easier to use (PCIScope is an example).

Thanks,

Eric

[Robert__R]

An analyzer would certainly make this easier.  You don't happen to be in Austin?

Can you also generate reads from the board?  It would be interesting to see if you could read back what you wrote (assuming it went to the wrong place). 

Or read/write another board (peer-to-peer).  You could try targetting your PCIe card from your PXIe card.  The nice thing about that is there's no confusion about physical/virtual address space since you can look it up in the BAR.  The problem is that P2P isn't necessarily supported by the chipset (though it probably is).

Let me know if you find anything else.

- Robert

[GraemeP]

We have WinDbg in house, but I'm not familiar with it. I'll see if I can get our software man on the case now.

I'm not sure what you mean by "dump the PCI configuration for the portion of the fabric", how to do it, or what it will tell me.

[GraemeP]

Austin, no. What's the weather like? We're in the UK.

You have given me another idea to try. Although I can't see why a DMA destination address could be corrupted as it goes through the various bridges before it reaches the host, we could target one of our boards from the other. They both have memory.

[GraemeP]

We've just made quite a critical discovery.

All of our PCIe cores are using 64-bit long addressing (as it makes interfacing easier for us as our FPGA's internal datapath is also 64 bits). But, according to the PCI express spec, addresses below 4GB MUST use the 32-bit format.

Changing our core to use 32-bit addressing solves the problem. Now our DMAs go where we expect 

[Robert__R]

Ah.  I'm glad you figured it out.  Thanks for posting the resolution.

- Robert