You're mostly there.
If you are using Spektrum satellite receivers as your 3GX receivers, you are correct in that channel 5 is the one that is to be used for controlling the governor.
Back to your 9503. In the system setup menu, open up the Device Select menu and enable the GOVERNOR as shown here:
You will now have a new selection in your menu called "GOVERNOR", and the Flight Mode Switch will control the RPM settings in this menu.
When you go to the GOVERNOR menu, you will see place holders for RPM setting values for the Governor in Normal mode, ST-1 mode, ST-2 mode, and HOLD mode.
These values are what controls the governed RPM for the selected mode (again, controlled through the Flight Mode Switch).
Now that you have the Governor menu operating out of channel 5 of your TX, you'll want to go to the 3GX menu, to this page:
Enter the main gear/pinion gear ratio in the top box. (If you don't know the gear ratio, either look it up in your heli's manual, or count the number of teeth on the big main gear, and count the number of teeth on the pinion gear. Divide the number of teeth on the main gear by the number of teeth on the pinion...that is the gear ratio.
(Example: Main gear has 100 teeth, pinion has 10 teeth. Ratio is 100 / 10 = 10).
With the gear ratio entered in the 3GX menu box, select the flight mode for which you wish to set an RPM, then use the 9503 roller wheel to increase the number while watching the resulting RPM on the 3GX display page.
Set the desired RPM for each of the flight modes. You'll see in my pictures above that I have the Governor disabled in Normal and Hold mode. ST-1 and ST-2 have the values I ended up with, when setting desired RPMs for my Gaui NX4.
Make sure that you have gone through the calibration sequence of your X9503/3GX combo as described in the 3GX menu stuff -- Use a linear 0 to 100% throttle curve when you do this, and your pitch curve for - full pitch to + full pitch (probably your ST-2 setting) when you do this.
Don't forget to put those back where they should be, before flying.
* Making the World Better -- One Helicopter at a time! *