Thanks for the file.
It is not the Hair that goes initial crazy; it is the small polygon object.
Create a larger cloth. This stabilizes the motion of the cloth. Use only a portion of it, or use a Surface Deformer to place the object you would like to have the Hair connected to it.
Cache everything, and use the Simulation> Cloth Cache Tool to refine your results.
In Hair, perhaps use the Dynamics> Riged option, and similar parameters to create a more straw effect.
All the best