You can navigate to chrome://settings or click on the hamburger icon and click settings.
Or if you still want Chrome to ask you, you can choose "Ask when a site wants to show desktop notifications".
If you're allowing Chrome to show notifications for approved sites, you can remove sites that you've already approved by clicking "Manage exceptions..."
Click on the site you want to remove, and click on the x that shows up to the right of the line. This will remove the exception.