This study examined the association between estimated age of first exposure (eAFE) to contact sport participation and neurocognitive performance and symptom ratings in U.S. service academy National Collegiate Athletic Association (NCAA) athletes. Male cadets (N = 891), who participate in lacrosse (n = 211), wrestling (n = 170), ice hockey (n = 81), soccer (n = 119), rugby (n = 10), or non-contact sports (n = 298), completed the Immediate Post-Concussion Assessment and Cognitive Testing (ImPACT) test before the season. Generalized linear modeling was used to predict each neurocognitive domain score and total symptom severity score. Predictor variables were entered in the following order: group (contact vs. non-contact); eAFE (eAFE <12 years vs. eAFE ≥12 years); group-by-eAFE; and covariates for learning accommodation status, concussion history, and age. The group-by-eAFE interaction was not significant for any of the ImPACT composite scores (Verbal Memory, Wald χ2 = 0.073, p = 0.788; Visual Memory, Wald χ2 = 2.71, p = 0.100; Visual Motor Speed, Wald χ2 = 0.078, p = 0.780; Reaction Time, Wald χ2 = 0.003, p = 0.955; Symptom Severity, Wald χ2 = 2.87, p = 0.090). Learning accommodation history was associated with lower scores on Visual Motor Speed (χ2 = 6.19, p = 0.013, B = −2.97). Older age was associated with faster reaction time (χ2 = 4.40, p = 0.036, B = −0.006) and lesser symptom severity (χ2 = 5.55, p = 0.019, B = −0.068). No other parameters were significant. We observed no association between eAFE, contact sport participation, neurocognitive functioning, or subjectively experienced symptoms in this cohort. Earlier eAFE to contact sport participation is not related to worse neurocognitive performance or greater subjectively experienced symptoms in male U.S. service academy NCAA athletes.