Running a hidden service with safety can be really complicated.
Hidden service administrator has to make sure that no application running on the server can be exploited remotely. Α hidden website may be vulnerable to the same threats every other website is. If the software a server uses to show a website is vulnerable, for example apache server or a wordpress installation or the database, then an adversary may use these vulnerabilities to gain access to the server, perhaps get privilege escalation(in many cases that won't be necessary) and finally be able to find your real IP address.
A hidden service may leak various other information that characterize your server and might lead an adversary to the real location of the service. This can be software's versions, encoding, time zone, or even the content itself.
Depending on the resources of the adversary, some adversaries might validate their suspicions about the location of the hidden service, by attacking part of the network the server resides in. Correlating when a certain part of a network is unreachable with the downtime of your hidden service, might indicate the vague location of your hidden service. This may also happen accidentally, when for example your provider has some kind of outage.
Don't forget that a hidden service just as an ordinary user of Tor Browser, makes use of Tor as a client. Using Tor can be easily distinguishable if your connection is under surveillance. Depending on the depth of surveillance, an adversary apart from telling that you're using Tor as a client (adversary can easily tell that you're not a relay by reading the public consensus of Tor), he may use time and data quota correlation for your outgoing traffic.
As a last point, the answer really depends of the resources of the attacker and whether he already has some kind of clue about the Tor user.