The refractive index of the material of a prism is different for different colors of light. As different colors have different speeds in the material of a prism. The prism also has non-parallel surfaces, the change of speed of light rays creates angle of deviation for each wavelength.