Hi psussi,
Thanks for the file.
It is not the Hair that goes initial crazy; it is the small polygon object.
Create a larger cloth. This stabilizes the motion of the cloth. Use only a portion of it, or use a Surface Deformer to place the object you would like to have the Hair connected to it.
Cache everything, and use the Simulation> Cloth Cache Tool to refine your results.
In Hair, perhaps use the Dynamics> Riged option, and similar parameters to create a more straw effect.
Example
https://www.amazon.com/clouddrive/share/R2NnSV5tT69P5L2ASkHY5K2muL3qp5XsBtqeRXT6yBp
All the best