Ok so.
A lot of misconceptions about the real world are being used as parallels for this setting.
Did you know colonial era people had no idea the diseases they had would spread like they did?
Did you know that any agrarian society domesticates animals if they can?
Domesticating animals en masse creates the varying plagues that we are all used to.
Tuberculosis, small pox, typhoid, etc...
Having people live in close proximity to animals causes us to eventually contract their diseases.
STATED HOW YOU HAVE IT COMMON ELVES WOULD HAVE JUST AS DEADLY DISEASES AS THE HUMANS.
The only reason that colonial era "new world" relations ended the way they did is that many of the nations and tribes of the Americas was due to lack of domesticatable large sized work/ food animals which would allow for large scale development.
There were a few exceptions to this such as the Mayans which used mass slave labor from capturing nearby tribes and the Incas who had some help in the form of Alpacas and Llamas.
But both of those animals are surly and their ability to be used as beasts of burden is laughable when you compare them to the capabilities of horses and donkeys.
But seeing how you have the elves as a well developed multi-city agrarian society it would be safe to assume they have some form of domesticated beast of burden capable of supporting that society.
WHICH WOULD CAUSE PLAGUES TO THE HUMANS WITH NO IMMUNITY TO SAID DISEASES.
The way you set up the world would require the elves to have literal magic. But you suggest that magic is simply folklore in this world (except when it comes to faeries for some reason)