Last time I passed through Taipei they had a water dispenser by the restrooms that had options for searing hot and “cold” water (it was lukewarm at best). I’d imagine Japan would have something similar.

Some of the premium ones are self heating so you just can add bottled water