What is hard about being an INFJ?

Depends on your outlook. It's actually easier being an INFJ

So far as I can see, it is far easier being an INFJ than hard, which is quite a luxury considering how other non-INFJ personalities fare.

Look at these factors for example:

1. INFJs love being alone. Their love of solitude brings them more positivities than 'love of people' to others. Their alone time connects them to their inner self, to nature, to creative thinking pursuits which are exactly the areas where 'people-loving' people miss out on and have to work really really hard to experience. For them being alone is depressing, distressing and leading to serious psychological problems.
   (Meaningful 'me-time, or alone-time is rejuvenating and healing even for people who hate being alone. Imagine: others have to force themselves hard to do what we do naturally and love it. What is more powerful and preferable? Being unable to exist alone or loving being alone? )
2. For this reason others are almost morbidly, painfully dependent on social relationships such as INFJs never are which makes INFJs extraordinarily free, independent, self reliant and creating their own happiness within themselves. They also get to bypass so much of drama, stress, pains of entanglement, and unnecessary complexities, approval for external validation, seeking happiness outside who cannot live without people. You just could not wish for more.
3. Their selective introversion makes them wonderfully fit for one-on-one or group interactions. They hate crowds and parties and social gatherings. Nowadays more and more people are waking up to the detrimental effects of 'party-mind', 'continuous plugged-in-personality and cluttering life with meaningless connections.
4. INFJs are naturally intelligent which is again a big plus point. With intelligence comes naturally also the challenges of dealing with not-so-intelligent people in day to day life but can one really complain about long flights if one is an eagle? If you are an eagle, would you really want to fly like butterflies and if you can't, complain about being an eagle?
5. Some might say overthinking is the hard part. Not quite. Turning over thoughts and trying to sort them minutely from a huge jumble of other thoughts, thinking endlessly, concentrating only on thoughts etc etc are part and parcel of having an intelligence that must find answers for itself. At any day, it is better than having an intelligence that doesn't think and doesn't know the huge benefits of thinking for an intelligent approach to any given situation in life. Thinking doesn't kill; non-thinking does.
6. INFJs see through people. They are like walking lie-detectors. I don't think anybody in their right mind can complain about this. There are millions and millions of people who get easily fooled in their life, relationships, business, ventures or thoughts just because they do not have this ability. I know of many people in my life who bumble blindly through relationships just because they can't see the truth that is so obvious to others/me.
7. INFJs are deeply authentic, truthful and honest. They may not have hundred friends but they genuinely care about people as humans and would cry easily to see someone in pain. They go out of their way to help people without wanting so much as a 'Thank you' in return. I would prefer being friendless but weeping over a stranger's pain and offering them my heartfelt pity and help, than have thousand friends and not caring about how they feel and what's happening in their personal life.
8. INFJs do not chase power, money, ego and avoid superficiality, spotlight, flattery, rewards, returns, name, fame, and self-promotion. Their integrity, dignity, and modesty is at human heart level, the healthiest one can think of. All other people have to strive and sacrifice and work really really hard to climb up to this level and still not be able to understand what it all is about.
9. If there is a hard part in being an INFJ it is just that people don't understand us and can't understand even if we explain to them. They have just no idea how we function and mistake us to be something we aren't. But, hey, who can blame them for that? And does it even matter to us what they think of us? We are fine being who we are.
10. INFJs are like an eagle. Eagles are solitary birds. Equipped with eyes that can see for miles ahead, fly higher than most and meant for long flights. They have capacity that sometimes puts four legged strong powerful animals to shame. They shouldn't bury themselves in self doubt and fear of 'hard'. If there's anything hard, its being forced not to fly, or fly like a butterfly.

But that's just my opinion. Thank you for reading!