One thing to keep in mind when you're talking about accessibility is that it does not mean a specific disability. There are very VERY common disabilities such as color blindness that effect anywhere between 12% and 20% of the population. Color blindness is probably the single most overlooked accessibility checkpoint.
Add to that a few percent of the population with physical disabilities, legal blindness, and other disabilities and I'm sure you'll realize that 20% of the population does, in fact, have a disability that can potentially stop them from accessing content on a site.
Making a site accessible means that everyone, regardless of device they're using, will be able to access the content. Making a site with a red-green or yellow-blue (the most common color blindness types) color scheme makes them dependent on a device that can read the screen content, or other device because they may not be able to read the content themselves.