We present light curves of the afterglow of GRB 050502A, including very early data at t-t$_GRB$<60 s. The light curve is composed of unfiltered ROTSE-IIIb optical observations from 44 s to 6 hr postburst, R-band MDM observations from 1.6 to 8.4 hr postburst, and PAIRITEL JHK$_s$ observations from 0.6 to 2.6 hr postburst. The optical light curve is fit by a broken power law, where t$^α$ steepens from α=-1.13+/-0.02 to -1.44+/-0.02 at åisebox-0.5ex 5700 s. This steepening is consistent with the evolution expected for the passage of the cooling frequency ν$_c$ through the optical band. Even in our earliest observation at 44 s postburst, there is no evidence that the optical flux is brighter than a backward extrapolation of the later power law would suggest. The observed decay indices and spectral index are consistent with either an ISM or a wind fireball model, but slightly favor the ISM interpretation. The expected spectral index in the ISM interpretation is consistent within 1 σ with the observed spectral index β=-0.8+/-0.1 the wind interpretation would imply a spectral index slightly (i̊sebox-0.5ex 2 σ) shallower than observed. A small amount of dust extinction at the source redshift could steepen an intrinsic spectrum sufficiently to account for the observed value of β. In this picture, the early optical decay, with the peak at or below 4.7×10$^14$ Hz at 44 s, requires very small electron and magnetic energy partitions from the fireball.