You probably do not want to set the NI High Performance GigE Vision driver to use jumbo frames for "All Applications." You would only configure it like this if you wanted to use jumbo frames with software besides IMAQdx. If you turn it on for all applications, you risk potentially having issues sending unrelated network traffic to other systems over a local network that doesn't fully support jumbo frames. This is why in general jumbo frames are turned off by default in your network driver settings. With the NI High Performance GigE Vision driver though, you get the best of both worlds because the driver uses jumbo frames for GigE Vision but not for everything else. There is no specific maximum size setting either because the driver will automatically enable the largest size the particular hardware you are using supports.
My hunch would be that you might just be low on bandwidth in the system. Is the network card you are using PCI or PCI express? If it is PCI you are likely going to run into issues because the PCI bus does not have quite enough bandwidth to acquire full-rate from a gigabit connection once overhead is factored in. Adjusting the packet size in one direction or another might help slightly by changing the characteristics of how the data is transferred across the bus, but if you do not have enough bandwidth it won't matter.
One thing you can change is under the "Advanced Ethernet Settings" category in the camera attributes tree there is an attribute called "Maximum Bandwidth Desired". You can change that to something lower than 1000Mbit/sec (try 600Mbit/sec for PCI) and see if it helps. It should tell the camera to send the data at a slower speed.
In all cases, if you are changing attributes in MAX, be sure you click the Save button to save the settings to the configuration file if you want those default settings applied when you open the camera within your own application.
Hope this helps,
Eric
