This is the one book everyone reads too late into the hobby, it will teach you more about how to build a layout than almost any other source. Many, many mistakes will be avoided.
Idk how relevant it might be to the game, but for American railroads, much of my knowledge on the subject comes from Track Planning for Realistic Operation by John Armstrong. It's a very interesting read, even if you're not into modeling. It's way more detailed than you can use in OpenTTD, but knowing the principles can help you make networks that look and feel like something you'd see irl.
That is really very thoughtful of you! Yes O scale is generally a lot more expensive than HO, not only because the models are twice as big, but because it's a smaller market and doesn't enjoy the economy of scale that HO does. Also O scalers tend to be more older and more affluent and can afford more expensive models, so the models are more expensive.
Another pitfall is that model railroaders can be absurdly particular about what models they like. Not only prototype and era, but even manufacturer and features. It would suck to give him a gift and it's the wrong thing.
So you might want to take an alternate approach and maybe suggest going to a hobby shop to have a look around, and you can either buy something for him right there or take mental notes of what he likes and buy something behind his back.
Even being proactively supportive of the hobby without buying him anything is very rare and will mean a lot. Something as simple as offering up a room or more space in a shared room to build in, etc.
Another idea is books. If he has a favorite prototype railroad, then literally any book about it will be a big hit. If he doesn't have it already then Track Planning For Realistic Operation is a must have. America's Colorful Railroads is one of the best coffee table picture books ever done. And they're both inexpensive used on Amazon.
Good luck!
I strongly recommend starting with a simple operations plan that covers what trains will do on your layout. You don't need to use prototype rules but if your trains have nothing to do besides go round and round for no reason, you will get bored after a couple weeks. Knowing why your railroad exists will inform your track plan much better than what will fit in your space.
For example, I'm building a railroad which carries supplies to a dam construction site.
Track Planning for Realistic Operation by John Armstrong is the bible for model railroad track planning.
I also like Byron Henderson's website for learning basics while waiting for the book to arrive from Amazon, and for tips and tricks which the book doesn't cover.