Colors in R can be defined by name (blue, black, darkgoldenrod, cornsilk, etc.), according to RGB values, according to HSL and HCL values (variously, permutations of hue, saturation, chroma and luminescence, see: Wikipedia:HSL and HSV and Wikipedia:Munsell color system for more information). Most of these values are freely definable either as decimal values between 0 to 1 or hexadecimal values between 0 and F.
See the colors command for a list of built in colros.
See rgb, hsv and hcl commands for more on defining custom colors.
See heat.colors, topo.colors, cm.colors, and rainbow for automatic generation of appealing color spectra.
For more sophisticated techniques, you can create your own color palette with the colorRamp and colorRampPalette functions. For instance the 'jet color' scheme used in Matlab can be created by:
jet.colors <- colorRampPalette(c("#00007F","blue","#007FFF","cyan","#7FFF7F","yellow","#FF7F00","red","#7F0000"))
Here are a few links to handy R color charts: