philip lelyveld The world of entertainment technology

Learning to Play Minecraft with Video PreTraining (VPT) (Virtual Beings and AI)

OpenAI have trained an AI to play Minecraft by watching Youtube videos. They took 2,000 hours of footage which also contained the keyboard and mouse input, which is used to determine the actions from 70,000 hours of Youtube videos to enable the AI to learn how to chop down trees and craft various items. It's all open-sourced and people can compete for $100,000+ in prizes for getting it to perform the more complex tasks.https://openai.com/blog/vpt/

Facebook Comment by Edward Saatchi, Fable Studios and the Admin of the Facebook Virtual Beings group
I think this really validates a lot of our focus in the simulation - that we could in the virtual beings space consider moving away from ‘brain-in-jar’ chatbots (even one’s with bodies) to AIs that can do things in worlds - robots are not the place for this for decades most likely but videogames and the metaverse provide us with the ability to have AIs in simulated worlds having lives, doing things alongside you and - aside from being way more fun - that serves as a more coherent, persistent prompt for language models which lose coherence so fast. It also provides a hint at memory - another huge failing for language models - simply by using the game database of actions taken. Finally this opens up a new area beyond the text, image data from the 2D web toward data of users playing videogames and mimicking their physical behavior - which were very excited about in the simulation. Overall I’m much more excited about this than DALL•E - that demos well and will definitely help ideation but this gets us closer to a metaverse where the majority of avatars are rich, characterful AIs and for the first time we’re interacting with AI virtual beings on a level playing field. Amazing !