If he hasn't, the next closest fast travel point is in Svorlag, a village larger than Hov. There's a local famed arena on Hov, and many warriors want to compete in it to prove their worth. Geralt can travel to a fast point near the arena if he's already unlocked Hov's location previously in the game. The quest is available in the village of Hov. The reason for this is that Geralt doesn't need his fighting skills in the quest, so he can undertake at almost any time once the main story of the game took him to Skellige. RELATED: 10 Of The Most Wholesome Things You Can Do In The Witcher 3
However, it's possible to finish on a lower level as well. It's a side quest and it has the recommended level of 14. If he wants to complete the quest 'Fists of Fury: Champion of Champions' and become the champion of fist-fighting, he first needs to finish the quest 'Master of the Arena'.
He meets one of them when he's in Skellige, on the island Spikeroog. Geralt encounters many strange beings during his journeys.