“I guess I don’t know. Sorry, I didn’t mean anything by it.” With a look on his face that clearly shows confusion at why you spent two whole responses about something as insignificant (in his mind) as potatoes. Everyone else probably has similar looks.

For small talk like that you get one response on the topic. If someone said I should order potatoes because I’m Irish I’d lean so far into it, adapt an obvious accent, and say “Oh I do loove me potatoes.” If I wanted to backhand him a little I’d tack on “Except during the famine when there were no potatoes. Those were daark days” to the first statement. There’s enough humor in the accent to cover the callout mass starvation he probably unwittingly referenced.