Data is the lifeblood of artificial intelligence. As LLM models grow exponentially in size, with newer models like GPT-4 having been trained on approximately 13T tokens, the need for training data is continuously growing. We understand that the “world model” we all possess so clearly as humans may still be out of touch for current AI models, whether that be due to a lack of high quality data, architectures, training strategies, or some combination of these. Even so, researchers remain fixated on the end form of AI: artificial general intelligence.

Said pursuit of data has catalyzed advancements across AI, notably in natural language processing and image recognition. LLMs stand out in this field due to their ability to decode and replicate seemingly natural human communication. These models leverage transformer architectures which enable them to encode the context and meaning behind words into multi-dimensional vector spaces, transform these vectors in vector space, and decode them back into human readable language. Models like these perform as a function of the quantity and quality of training data provided. It’s no wonder that GPT-4's staggering consumption of over 10 trillion tokens has seemingly exhausted the internet's textual resources. These data appetites underscore the models' dependency on expansive corpora to refine their linguistic mimicry.

A similar technology, image diffusion models, work by introducing noise into image data, mimicking the natural process of diffusion. These models then learn to reverse the diffusion process to generate coherent images. DALL-E, OpenAI’s image model, is no less voracious than its textual counterpart, using over 400 million image-caption pairs for its training.

Now, it seems that the insatiable appetite for data within AI research has stifled the development of new models. Synthetic data generation has offered a controlled yet expansive playground for AI systems to learn from, but synthetic data can only go so far. It won’t be long before LLM generated content is essentially all that’s left online. The architecture of current models, while impressive, has not achieved the efficiency or integration seen in the human brain. A consensus is emerging that a significant breakthrough in model design is necessary to fulfill the vision of AGI - from merely amassing training data to rethinking the fundamental structures that underpin the transformer architecture.

Our vision of AGI stretches far beyond current AI capabilities, aiming for a breadth of competence that rivals human intellect. Broad competence in AGI is about adaptability, where learning and adjusting to new environments and tasks happens without explicit reprogramming. AGI must grasp and reason through complex ideas, much like humans do, making sense of abstract concepts and applying logic in diverse scenarios. Transfer learning is key, allowing an AGI to apply knowledge from one domain to an entirely new one with ease. AGI would need to autonomously learn from raw data and experiences, processing sensory inputs like sight and sound without human guidance. With self-directed goals, an AGI would set and pursue its own objectives. And, at the pinnacle of this vision lies the contentious possibility of AGI experiencing consciousness, a trait that remains speculative but captures the essence of what it means to be truly intelligent.

Navigating the path to AGI, we realize data alone won't be enough. Yann LeCun of Meta believes that real-world interaction, not language, is the bedrock of learning. Today's AI can ace tests but lacks the common sense to navigate a car like a teenager. This gap signals a need for a paradigm shift. Could novel hardware and software solutions bridge this chasm? Understanding visual input doesn't equate to comprehending the surrounding world. Language, while being a dense and complex way of communicating intent, emotion, and data, is just the tip of the iceberg. AGI isn't just about more data or processing power; it's about an AI's embodied experience in the physical world.

Robotics might hold the key to the next leap in AI's evolution. By providing sensory inputs and a physical form, robots could offer AI a means to understand the world more like a human does. Consider this: all the language data we've fed into AI amounts to 1013 bytes. A young child, on the other hand, absorbs 1015 bytes through their optical nerve. That’s two orders of magnitude more data, absorbed far more efficiently. Innovative methods, such as equipping robots with cameras or even strapping cameras to infants, could unlock a kind of data that is vital for developing the sophisticated understanding that underpins human intelligence.

With every stride towards AGI, we’re walking on thin ice between groundbreaking innovation and an uncharted ethical landscape. Robots and AI systems that meet our goals of AGI will inevitably surpass human intelligence, and this raises profound questions about the role of AI in a meshed society.