I'd like to know if there is a way I can block / allow specific ip's address to access my Exit-relay.
You cannot stop arbitrary client IP addresses from building circuits that involve your exit node. How would your exit even know what clients are using it? It doesn't know.
You can configure your exit node to not connect out to specific IP addresses and ports by using the
ExitPolicy torrc options.