Do not play memory games and deploy front end/application servers on 64-bit

I was recently visiting a large customer who had multiple Web Front End/Application servers (combined PWA and PSI) and each server had 12GB of memory. They were using Windows Server 2003 32-bit version.

In a 32-bit environment only a fraction of the RAM was being utilized and it's important to remember that the /3GB is not supported either: The Windows Server 2003 /3GB switch is not supported in Windows SharePoint Services 2.0 or in later versions or in SharePoint Portal Server 2003 SP2 or in later versions http://support.microsoft.com/kb/933560

As stated in the SharePoint documentation: if you are running 32-bit front-end Web servers, we recommend that you do not use the /3gb switch in Windows Server® 2003 to change the 2 GB of virtual address space to 3 GB for all  user mode processes. We recommend against using the /3gb switch because most SharePoint site traffic involves sending large amounts of data through the operating system, and leaving only 1 GB of address space for the operating system can destabilize the computer.

So in the end the only option is to re-install the servers on a 64-bit operating system like Windows Server 2008 for instance :)

I encourage you to check the following article on MSDN: Memory Limits for Windows Releases