We report on optical and near-infrared observations obtained during and after the 2004 December discovery outburst of the X-ray transient and accretion-powered millisecond pulsar IGR J00291+5934. Our observations monitored the evolution of the brightness and the spectral properties of IGR J00291+5934 during the outburst decay toward quiescence. We also present optical, near-infrared, and Chandra observations obtained during true quiescence. Photometry of the field during outburst reveals an optical and near-infrared counterpart that brightened from Rsimeq 23 to Rsimeq 17 and from K = 19 to Ksimeq 16. Spectral analysis of the RIJHK broadband photometry shows excess in the near-infrared bands that may be due to synchrotron emission. The Hα emission line profile suggests the orbital inclination is simeq22°-32°. The preferred range for the reddening toward the source is 0.7 <= E(B - V) <= 0.9, which is equivalent to 4.06 × 10$^21$ cm $^-2$ <= N$_H$ <= 5.22 × 10$^21$ cm$^-2$. The Chandra observations of the pulsar in its quiescent state gave an unabsorbed 0.5-10 keV flux for the best-fitting power-law model to the source spectrum of (7.0 +/- 0.9) × 10$^-14$ ergs cm$^-2$ s$^-1$ (adopting a hydrogen column of 4.6 × 10$^21$ cm$^-2$). The fit resulted in a power-law photon index of 2.4$^+ 0.5$$_-0.4$. The (R - K)$_0$ color observed during quiescence supports an irradiated donor star and accretion disk. We estimate a distance of 2-4 kpc toward IGR J00291+5934 by using the outburst X-ray light curve and the estimated critical X-ray luminosity necessary to keep the outer parts of the accretion disk ionized. Using the quiescent X-ray luminosity and the spin period, we constrain the magnetic field of the neutron star to be <3 × 10$^8$ G.