I have a theory about that.
I think that what people want, when they talk about a pastoralist fantasy is actually an anti-capitalistic fantasy: i noticed, even from my experience, that most people don't mind phisical labour if it gives them results: actual, tangible, results.
Once my boss asked me to copy every article from a website and paste them in the new one. It took me roughly four hours for three days to do and my soul was slowly leaving my body. It was easy work, i mean who wouldnt want to earn money to just click here and click there, rinse and repeat? But it was boring, ripetitive and basically useless.
But when I take some time and clean my house, i sweat, i am tired but... satisfied. I see in front of me the result of my hard labour and I am happy, or at least i don't think i wasted my time.
So the fantasy of working hard but at least getting something out of it is appealing: why do people work in kitchens? Or bakeries and wake up at dawn to make bread? Or any hard job like that? I knew a guy that had the possibility of having every job he wanted, but he opened a bar and couldnt be happier.
This is my idea, i'm not a student in sociology or anything but I hope i made a point.