>every viable plot of land is used for farming.

It is? Not in the US!

Try using Google Maps in Satellite View. You'd be hard pressed to find land that isn't used agriculturally.

IIUC, the land that is unutilized or has forest, is either protected, too hilly, too dry, etc.

Consider New England, where in its early days most inhabitants consisted of farming families. Where there were farms and cow pastures, there is now mostly forest.

There are a few farms remaining in New England, yes, but in general farmers consider the soil there not worth farming compared to places like Iowa and Southern Illinois (where all the land really is utilized for farming) even though in the past most the (sizeable) population of New England made a living farming.

We know that most of the forest in New England is suitable for farming because there are still stone walls running through it: these wall were made of stones encountered by farmers while plowing (when the forest was farmland).

Ehh US farmland usage has been dropping for many decades. That said, it is only because we are more than ever reliant upon fossil fuel derived fertilizer and over utilize a lot of arid/desert farming