Denis Noble's book and papers on biological relativity (2012) changed a lot of the way I see intelligence. As you start getting in depth, things start to get incredibly counterintuitive.
The problem with a lot of AI research, because there is a lot of disconnect between CS/Math and biology and physics, is that we often focus on the 'intuitive' biology and physics. Intuitive physics/biology is understanding that S=D/T or that neurons fire probabilistically. The counterintuitive physics and biology would be understanding that objects, sound, and touch are hallucinations of reality and don't actually exist (read Donald Hoffman's literature + Andy Clark on predictive processing). Intelligence is fundamentally shaped by the counterintuitive aspects. It shapes the way we communicate, reason, and generalise off of very little information.
To get AGI right, I think we need to take into account (I'm not saying fully replicating, nor am I trying to anthropomorphise AI too much) both the intuitive and the counterintuitive biology and physics.
We'll know we've reached AGI or something close to it (or more) when we start reading research papers and they don't make intuitive sense anymore.