For millions of years, nature has basically been getting by with just a few elements from the periodic table. Carbon, calcium, oxygen, hydrogen, nitrogen, phosphorus, silicon, sulfur, magnesium and potassium are the building blocks of almost all life on our planet (tree trunks, leaves, hairs, teeth, etc). However, to build the world of humans—including cities, health care products, railways, airplanes and their engines, computers, smartphones, and more—many more chemical elements are needed.
There’s already been issues like this in using copper sulfate (which is inorganic in terms of molecular structure and persistent as an element like you mention, but labeled as organic in some jurisdictions in terms of agricultural treatments and food marketing) as a fungicide. It’s a very short lived fungicide for leaf borne fungal diseases, meaning it must be present on the leaf to prevent infection (it is easily washed away by rain or errant irrigation) and applied repeatedly, but is long lived in the soil meaning it can build up and kill off mycorrhizae and other beneficial soil fungus causing longer term drops in yield.
So, yeah, your worry is valid. And, that’s just one example.