Most of the files you want to play with are \AT91SAM9DS256\Source\*_bt.*. You can edit them using any C editor, including the Kickstart version of the IAR Embedded Workbench for ARM (which won't allow you to build the firmware, but will allow you to compile it and at least check for compile errors).
What you want to do is probably going to require a pretty thorough understanding of the ARM architecture and how it interfaces with the BlueCore chip (which is described in the LEGO Bluetooth Developer Kit). If you expand the size of the firmware, you may run into other issues when you load it.
Once you think you have the source code fixed, you need to download the 30 day evaluation (not Kickstart) version if IAR Embedded Workbench for ARM v 4.42. At this point, I believe that is the only solution that allows you to build the standard firmware. The NXT doesn't have any easy way to debug your firmware, so unless you are willing to open it up, debugging is pretty much trial and error.
Good luck.