You really need an invite from the Stable.
Sumo stables are certainly an experience. One stable, which has now closed down, used to have chanko events around the time of the tournaments, where the wrestlers in the evening after fighting would serve chanko to the guests. In addition, it is permissible for men to view the training area. Women are prohibited unfortunately.