New AI: Photos Go In, Reality Comes Out! 🌁

  • Published on Nov 15, 2021
  • ❤️ Check out Lambda here and sign up for their GPU Cloud:
    📝 The paper "ADOP: Approximate Differentiable One-Pixel Point Rendering" is available here:
    ❤️ Watch these videos in early access on our Patreon page or join us here on TheXvid:
    🙏 We would like to thank our generous Patreon supporters who make Two Minute Papers possible:
    Aleksandr Mashrabov, Alex Haro, Andrew Melnychuk, Angelos Evripiotis, Benji Rabhan, Bryan Learn, Christian Ahlin, Eric Martel, Gordon Child, Ivo Galic, Jace O'Brien, Javier Bustamante, John Le, Jonas, Kenneth Davis, Klaus Busse, Lorin Atzberger, Lukas Biewald, Matthew Allen Fisher, Mark Oates, Michael Albrecht, Michael Tedder, Nikhil Velpanur, Owen Campbell-Moore, Owen Skarpness, Rajarshi Nigam, Ramsey Elbasheer, Steef, Taras Bobrovytsky, Thomas Krcmar, Timothy Sum Hon Mun, Torsten Reil, Tybie Fitzhugh, Ueli Gallizzi.
    If you wish to appear here or pick up other perks, click here:
    Thumbnail background design: Felícia Zsolnai-Fehér -
    Meet and discuss your ideas with other Fellow Scholars on the Two Minute Papers Discord:
    Károly Zsolnai-Fehér's links:
    Instagram: twominutepa...
    Twitter: twominutepapers
  • Science & TechnologyScience & Technology

Comments • 597

  • Luke Costello
    Luke Costello 6 months ago +966

    This is literally the magic that I thought computers could do when I was a kid. Now its a reality.

    • Tuhin Mitra
      Tuhin Mitra 4 months ago

      Thanks to you for presenting this to the world

    • Luiz Henrique Lima
      Luiz Henrique Lima 4 months ago

      I was writing exactly the same thing you wrote. So I make of your words my words. Amazing!

    • Nugget Bugget
      Nugget Bugget 6 months ago

      Technically the computers from when you were a kid could do this, it would just take years and you would have to manually input everything it requires to function.

    • ziplock9000
      ziplock9000 6 months ago

      Me too. There's a Star Trek: TNG episode where Geordie is on the holodeck and he asks the computer to extrapolate features of a very blurred figure based on other metrics. I knew one day we get that, but this is even more impressive than that!

    • MadsterV
      MadsterV 6 months ago

      @Lynn Harrod good, he is right

  • EMOJO_2001
    EMOJO_2001 6 months ago +640

    "Don't expect too much"
    Ai: *makes a perfect video*

  • SentireAeris
    SentireAeris 6 months ago +324

    I can really foresee this being refined and packaged for mobile use, when computing power gets a bit better. Samsung or Apple would pay a LOT of money for this feature if a phone could run it.
    You take a few photos on holiday, and this app recreates the scene. like a movie panorama, wow.

    • Bordeaux 17
      Bordeaux 17 5 months ago +1

      And hook up to VR headset and it would be perfect

    • Xavier Magnus
      Xavier Magnus 6 months ago +1

      Games where you can feed pictures at runtime at it can make the environment as needed...

    • my name
      my name 6 months ago +1

      @Hemant K thats possible now

    • my name
      my name 6 months ago +1

      it will be even more useful for vr and ar

    • Hemant K
      Hemant K 6 months ago +3

      If this ever comes to a mobile, I guess it'll be an application that takes in the photos, runs the process on the backend servers, and produces the rendered video to the user.

  • MadCreativity
    MadCreativity 6 months ago +112

    I've been very proud of being able to hold onto my papers until now, but you have successfully made me drop them. I was expecting it to be good, because otherwise it wouldn't be here, but I was not expecting it to be *that* good. Stunning work.

  • Marian Szewczyk
    Marian Szewczyk 6 months ago +258

    Looking forward to visiting all places from my old photo album in 360° 3D VR soon...

    • Kristjan Kask
      Kristjan Kask 5 months ago

      I'd prefer some nice classic movie scenes

    • Opec Light
      Opec Light 5 months ago

      @Kliman Khmeron he means in the future

    • Kliman Khmeron
      Kliman Khmeron 5 months ago +1

      Unless you took 35 pictures of the same location

    • Dislike Button 👎
      Dislike Button 👎 6 months ago +1

      @Leonard Choo nightmare!

    • JC MSchott
      JC MSchott 6 months ago +39

      Now it's time to apologize to those friends or relatives who used to take a lot of pictures from a specific place, person or building, when we thought only one was enough. They were collecting training data for today use.

  • KCR World Birthday Songs
    KCR World Birthday Songs 6 months ago +154

    There are just so many applications of AI in the CG World. Looking forward to creative AI that are capable of learning and building 3d worlds including cities, economies, etc.

    • KCR World Birthday Songs
      KCR World Birthday Songs 6 months ago

      @zdenek burian cool

    • HappyBuggy
      HappyBuggy 6 months ago

      It will be much cheaper to make movies and we will actually have Hollywood level effects for low budget sitcoms

    • zdenek burian
      zdenek burian 6 months ago +2

      imagine if these AI agents could search inside online storages of every kind of data, photos, books, etc to reach the knowledge to rebuild the parts lacking since you provided only a limited set of informations; they could ie rebuild an entire walkthrough of a city as it was centuries ago learning from historical researches

    • WebDev
      WebDev 6 months ago +1

      The problem is as more automatic as harder to control the output and use for professional work

    • Alex Walters
      Alex Walters 6 months ago +7

      Instead of rendering a whole CGI scene, render certain frames from story boards, run this AI to get a preview of the whole scene before you render it fully. Added bonus, you can probably use your final renders as training data to refine the network for next time

  • Raki Jr
    Raki Jr 6 months ago +7

    It's gonna be pretty cool when Google Earth implements this for full VR

  • Rohan Asokan
    Rohan Asokan 6 months ago +50

    I do research in this field and my research team discussed this paper a week ago.....its nice to see Karoly pick these great topics and share them with the general public. Love the work you are doing, Karoly. One day I'll work hard enough to get my paper shown on this channel - idc if it makes to SIGGRAPH Asia, if it makes to Two Minute Papers then its game won :)

  • Daniel Moore
    Daniel Moore 6 months ago +22

    This will dramatically change and simplify certain workflows in the film industry. Imagine taking just a few high quality photos and getting this as an output. Unreal.

  • Jack Bauer
    Jack Bauer 6 months ago +8

    Also: Does this mean that we will be soon able to recreate full 3D environments from just a few pictures?

    • [esa]A
      [esa]A 6 months ago +4

      I’d guess that using the output from this one as input for a 3D scanning application would be possible with some tweaking.
      Source: none, i have no idea what i’m talking about

  • Tom Pfeiffer
    Tom Pfeiffer 6 months ago +27

    Okay I'm definitely going to use this some day. This is awesome, I was really close to dropping that paper

  • Jorge C. M.
    Jorge C. M. 6 months ago +20

    2:10 It couldn't make an interior but it managed to make the background, this AI is pure wizardry

    • tomahzo
      tomahzo 6 months ago +1

      IKR :D! How the hell did it figure out how to put the fence right there? That's so amazingly clever!

  • Chris Z
    Chris Z 6 months ago +10

    I've been waiting for years for Ai to enter the photogrammetry field. Both this filling out areas part - but I hope also AI to remove shadows and detect reflections will appear. It will be very interesting to see this implemented in the future in metashape or realityCapture type software.

  • Joshua Consiglio
    Joshua Consiglio 6 months ago +4

    I'd love to see this used in tandem with photoscanning. Usually to do it well you need a bunch of images, but if this can fairly accurately interpolate most of them, that could save a lot of time!

  • Zac D
    Zac D 6 months ago +13

    It's amazing that it really picks up on the surface/material qualities. Metal looks and reflects like metal, plenty of specular reflections and highlights on most surfaces where it should be. Would much rather have a bit of noise than those details be missing.

  • Bernon Aarey
    Bernon Aarey 6 months ago +30

    OK, so this has been assigned a fairly mechanical task, but the fact that it far exceeds my expectations does give me pause for thought. Watching this makes me realise that I really have no idea what AI will be capable of in the coming years.

    • James
      James 6 months ago

      Yea this one broke me lol

    • zdenek burian
      zdenek burian 6 months ago

      Nobody should invest time in a steep learning career in 2d graphics, 3d or video until things clear

  • Sébastien Pautot
    Sébastien Pautot 6 months ago +20

    It could be used to improve current photogrammetry programs ! Since it outputs a video, which is basically a group of pictures... it could be used to create even more photo references for the programs.

    • Powerbyte7
      Powerbyte7 5 months ago

      This is an AI that's able to understand and render a scene based on observations. It doesn't really help with the photogrammetry process, it in fact needs it to function in the first place.
      NeRF (Neural Radience Fields) are a far better candidate if you're concerned with actual 3D reconstruction as it generates a viewable volume of your subject, which contains depth information. However, a NeRF mesh reconstruction loses the data of view-dependant changes (Transparency, Reflections, Lighting), which makes it a lot less useful. We're missing a way to convert that data into usable PBR values for meshes. That's where the real potential lies if you ask me.

    • Thomas Donalek
      Thomas Donalek 6 months ago +1

      I am preparing to go shoot a somewhat complex site for photogrammetry next week, so I was wondering something similar. The idea is to give the photogrammetry system as much information as possible in order to produce as accurate a point cloud/mesh as possible. But the original set of photos only has a limited amount of information in them to build the spatial data from. One possibility is that using the output of this system to feed into a photogrammetry system would result in a more dense point cloud, but not necessarily more accurate. The other is that the existing standard photogrammetry systems are not extracting as much information as is possible from the initial data set. I hope someone with more knowledge of this puts this type of system together with photogrammetry and figures out wether we can get better end results for the point cloud from initial photo sets.

  • Luciano Rivera
    Luciano Rivera 6 months ago +4

    It is absolutely incredible how it manages to even recreate the tree density from different perspectives without washing out the details.
    And sure, some particular scenes where there is nothing to draw detail from, it won't even attempt to do so, but I wouldn't even complain if this recreation was done by hand.
    Truly astounding. Can't wait to see how this could later be also applied to upscaling to refine image quality.

  • M JAY saratchandra
    M JAY saratchandra 6 months ago +14

    I remember watching a doraemon episode where a gadget creates a full recreation of a scenery from a part of it in an image, which they use to find out who was walking a particular dog or something. to think that something similar exists in reality is mind boggling!

  • Netux
    Netux 6 months ago +3

    For those wondering, the "consumer GPU" seems to be an RTX 2080 Ti rendering 1080p frames

  • Culmacatrane
    Culmacatrane 6 months ago +15

    This is amazing ...we can now stockpile images of places/buildings that might not be here in another 10/20/30 years and then make a 3d representation of them in VR for future generations to see.

    • The Potato
      The Potato 6 months ago

      @J M so if it was built from point cloud, then it's just photogrammetry then >.>
      well, but at least they need much less input.
      And I also quickly forgotten that it emphasize on the in between. maybe it's just pseudo 3d.

    • J M
      J M 6 months ago

      at least that's what it seems like from the paper. It needs point cloud as input

    • J M
      J M 6 months ago

      @The Potato It is real 3d because it builds those images from the point cloud

    • The Potato
      The Potato 6 months ago

      @J M No, you're talking about photogrammetry. this isnt that. this one is just pixels put together by AI to produce a 3D looking video. not real 3d.

    • J M
      J M 6 months ago

      @Michael Leue It's a point cloud rendered as a video. It is 3d

  • i p
    i p 6 months ago +6

    This is so good. Can we directly export as 3D models or use photogrammetry on the new footage, maybe this can be used to enhance photogrammetry software? How are flight paths determined?

    • Powerbyte7
      Powerbyte7 5 months ago

      This is an AI that's able to understand and render a scene based on observations. It doesn't really help with the photogrammetry process, it in fact needs it to function in the first place.
      NeRF (Neural Radience Fields) are a far better candidate if you're concerned with actual 3D reconstruction as it generates a viewable volume of your subject, which contains depth information and can be converted into a mesh. However, a NeRF mesh reconstruction loses the data of view-dependant changes (Transparency, Reflections, Lighting), which makes it a lot less useful. We're missing a way to convert that data into usable PBR values for meshes.

  • FreshBakedClips
    FreshBakedClips 6 months ago +3

    Glad that I take photos from time to time, whether it is mundane or important events.
    The photos that I took are the only legacy (historical records) that I can hand down to my future descendants.
    It's up to them how they will use it, as long as I will be remembered in a form of a digital reality.

  • EpicZEVEN
    EpicZEVEN 6 months ago +3

    This is incredible!!
    The digitalization of cities is very close!

  • LumberApple
    LumberApple 6 months ago +17

    This could be used to improve google maps street view. You no longer have to teleport around and can instead just slowly drive across the road.

    • J M
      J M 6 months ago

      Google already uses photogrammetry although not in street view but when you zoom into a city, and like all photogrammetry which this is, you need to see things from multiple angles which google street view currently is only seeing from the road. If they sent drones rather than cars then they could generate point clouds like this

    • Ryan Hellyer
      Ryan Hellyer 6 months ago +2

      If they can use this sort of tech to create a photorealistic 3D model, then we could move about within Google Street View in first person. This would be super useful for game development Imagine race car games, where you could drive across the entire planet :)

    • Q
      Q 6 months ago +3

      @Michael Leue Google can precompute all video footage needed and serve the videos in the player one after another seamlessly.

    • Blue Mamba
      Blue Mamba 6 months ago

      Would be better if google started taking 360 videos instead. Although, it would be pretty trippt seeing people when reversing the video etc.

    • Michael Leue
      Michael Leue 6 months ago

      I mean, sure, if "slowly" means an hour per meter.

  • Soos
    Soos 6 months ago +3

    This feels like dreams, close to reality but with some bits missing.

  • Rohit Agrawal
    Rohit Agrawal 6 months ago

    Wow, this is amazing! The amount of potential this has for creating 3D assets and environments is UNREAL!!

  • tomahzo
    tomahzo 6 months ago +1

    This truly is incredible! I can't wait for the kind of applications that might come from this. You could integrate it into 3D modelling tools, video editing suites, image editors, so many things. And think about the applications in video compression :D.
    I remember some paper from Google a few years back when they had a bunch of photos of the exterior of a building and people were losing their shit over how it was possible to move the camera around virtually to generate new images from new angles. In that paper they could only move like a few feet to the side before the model started to fall apart but that was impressive back then. This is a whole new world in comparison!

  • Jaface
    Jaface 6 months ago +5

    I wonder if the data it's using in the learning algorithm could be translated to a 3d model for use in normal 3d rendering... I'll just have to wait for 1-2 more papers down the line.

  • MokouAwesome
    MokouAwesome 6 months ago

    Really impressed about how such a smooth and high quality video can be produced from so little pictures! Love learning this kind of stuff.

  • James Hughes
    James Hughes 6 months ago +1

    I wonder if this works well enough to feed the results into photogrammetry software? the progress is stunning

  • psiga
    psiga 6 months ago +1

    Amazing. Now I'd like to see something like that generate volumetric point clouds that can be viewed in VR.

  • The Best Stooge
    The Best Stooge 6 months ago +4

    The time is fast approaching where we will never know if what our senses tells us is based in reality or some augmented AI simulation.

  • ozzi9816
    ozzi9816 6 months ago

    I can see it being used as-is as an assistant for creating animations! You can take a few pictures on your phone, feed it into this, and have an entire video to rotoscope for fancy dynamic shots.

  • Gal Grünfeld
    Gal Grünfeld 6 months ago +2

    Holy moly, this is amazing!
    An immediate application I can think of is for are:
    • Google's Street View and similar services
    • Spying and intelligence
    • Virtual tours
    • Reality-inspired/based maps (e.g for games)
    • Architecture (modeling and exterior and interior design)
    And I think that it's amazing that it's so simple to train. Especially for Street View I can imagine a google car driving around, taking pictures and processing them in/close to real time into these videos as it drives around and takes them.

  • Boris Maganic
    Boris Maganic 6 months ago

    This is mind blowing... I would really love to connect with you or one of the AI experts who developed and/or engineered the algorythms because this type of intelligent "fill in the blanks" rendering could be precicely what I have been envisioning as a solution to a future iteration of a visualization product/production service that my company offers to our clients.

  • Adam Zinzan
    Adam Zinzan 6 months ago

    Would love to see how this could be applied to turning a series of drawings or architectural sketches into a 3D model for presentation to a client. potential applications for design, photography and 3D modelling work seem pretty incredible!

  • Melon Collie
    Melon Collie 6 months ago +2

    They should totally use this plus a more thorough google street view sweep to a 3d roadmap of the entire lower 48 in the US or even Europe.
    MS flight sim has it, but what about truck simulator?

  • Ryan Favelle
    Ryan Favelle 6 months ago

    Would be curious to also know the input resolution limits, I have a dataset of half tera-pixel scenes of the same city from multiple lookouts, would be interesting for a way to explore the scene at intermediate angles

  • Grim Of Doom
    Grim Of Doom 6 months ago

    This would be amazingly cool to see used in larger image datasets such as... google maps street view

  • ‏‏‎
    ‏‏‎ 6 months ago

    I wonder how well this would work as frame interpolation to upscale 24/30 fps to 60

  • Nick Bosshard
    Nick Bosshard 6 months ago

    If the speed of this increases well enough a few papers down the line, this would be great to render missing frames or even reduce rendered frames of a 3D scene and synthesise the rest.

  • Edward Whitton
    Edward Whitton 6 months ago +2

    I love your videos. But it would be great to have some sort of overview of how this is done? Otherwise it really is just magic. AI research and programming is zooming off into the future so fast its hard to keep up. Keep up the good work!!

  • Ahmadx1x1 the best nickname

    Imagine this being used for seamless transitions in Film/TV/Etc., that would extremely impress me. This is so ADVANCED when it comes to 3D Scene Creation.

  • Grayve Rose
    Grayve Rose 6 months ago

    This would work wonders for making smooth 60 frame animations that are low frame rate, like anime or any film

  • Caleb Willden
    Caleb Willden 6 months ago

    Absolutely incredible! Literally jaw dropping for me.

  • Felixxenon
    Felixxenon 6 months ago +3

    What about letting the a.i create a 3d space from the information from the pictures to get rid of artifacts? It probably needs to be combined with some other program, but it would be a great way to create games (and other experiences) in the future. What a time to be alive♡

    • Rohan Asokan
      Rohan Asokan 6 months ago

      @Michael Leue It is possible...we can do it. I work in the field and one direction that is being explored is how we can build tools that handle the way these neural networks represent things u know. But as u said there is a huge way we have to go. Another thing is its one AI thingy for one scene so things blow up when u wanna design game scenes and all...

    • Michael Leue
      Michael Leue 6 months ago +1

      The previous paper on this channel, about generating a 3D mesh based on pictures of an object, could be used to do this, but comparing the processing requirements between the two shows that we're pretty far away from realistically doing THIS with THAT. Also, this requires a completely static environment, which wouldn't be a huge improvement over existing creation techniques in games.

  • Joe
    Joe 6 months ago

    this combination of AI and AR is going to make the metaverse very interesting

  • Jerry Flowers
    Jerry Flowers 6 months ago

    Wow. It starts looking like complete 3D scenes.
    Glad I held on to my papers.

  • lumenwrites
    lumenwrites 6 months ago

    Incredible. Really looking forward to some future version of Google Earth using this.

    LEUKICK 6 months ago +1

    This is actually scary. I'm incredibly amazed, and really enjoy looking at this. But where are we headed in the world at this rate??

  • Hall 00117
    Hall 00117 6 months ago

    Awesome! I wonder if this could help with getting cleaner photogrammetry data somehow?

  • DaFox
    DaFox 6 months ago

    So glad that this channel even exists. Always mind-blowing things to learn about, thanks a lot

  • Vincent Boudewijn
    Vincent Boudewijn 6 months ago

    This is insane for VFX. Im a matte painter and some of my work is reconstructing the environment and reproject photos on it. This AI just does all of that for you. This is crazy

  • William Alten
    William Alten 6 months ago

    It seems to reproduce lighting behavior quite accurately as well. Very impressive.

  • Solarpowered
    Solarpowered 6 months ago +2

    Can't wait to see whats two papers down the line!

  • MrGTAmodsgerman
    MrGTAmodsgerman 6 months ago

    WOW. This would be insane to use for Photogrammetry when you have not enough data.

  • Crispy K
    Crispy K 6 months ago

    Need this to make 3d models for blender etc...
    Interesting vid as always, going to keep an eye on this ;)

  • Nicolas Silva A.
    Nicolas Silva A. 6 months ago

    unbelievable, imagine what the future of mapping could lead with this

  • Martin Ask
    Martin Ask 6 months ago

    That is insane! Is it possible to export this not as a video but as a perfectly projected 3D model with applied textures?

  • Prince Westerburg
    Prince Westerburg 6 months ago +4

    So, presumably you could output a tumbling camera version to photogrammetry software that could build the scene in 3D and texture it?

  • Tom Pfeiffer
    Tom Pfeiffer 6 months ago +3

    That title is really promising, thanks for all the great content man!

  • Samuel Sägesser
    Samuel Sägesser 5 months ago

    This could be manually replicated with photogrammetry and classical 3d camera animation rendering, so nothing that was impossible but much simpler for the end user. Fantastic how it also improves detail.

  • Marc Lawrenson
    Marc Lawrenson 6 months ago

    cant wait for someone to train an AI on 3D images, so you give it a photograph and it generates a 3d model. awesome.

  • Tristan Wegner
    Tristan Wegner 6 months ago

    Very impressive. The only performance metric I wish you would have mentioned is the rendering speed. Like how many seconds per frame on the mentioned consumer graphics card.

  • vastcast
    vastcast 6 months ago +6

    technology is evolving too fast, not that it's a bad thing

    • Fred Nethel
      Fred Nethel 6 months ago

      i just wanna say thanks to/for our robot ai overlords.....ha ha just kidding ha ha ha no seriously they are here they win :)

  • HeCedTooMuch
    HeCedTooMuch 6 months ago

    I love you for bringing us the bleeding edge in AI!

  • Lasse Hansen
    Lasse Hansen 6 months ago

    I'm imagining combining this with photogrammetry to create 3d models from a few photos

  • John Sherfey
    John Sherfey 6 months ago

    Can these be turned into SDFs, is my question.
    If so then maybe some extra layered effects could be added, or maybe virtual objects added directly to the scene and have them properly lit.

  • timtheli0n
    timtheli0n 5 months ago

    Could you use this for video compression? Feed the algorithm keyframes, check the output, and then overwrite the places that it fails to do a good job? That way you'd only have to send the keyframes as well as smaller bits of video that would fill in the messed up places...

  • yaelm631
    yaelm631 6 months ago +1

    I think this is better than photogrammetry for the same few amount of photos
    It achieves background and thin objects really well.
    It uses a point cloud, I hope we can export a mesh from it because it would look awesome for VR (They support VR view, but I don't know if it is 6dof)

  • bdot02
    bdot02 6 months ago

    Absolutely cannot wait for what the next paper on this has in store

  • James
    James 6 months ago

    WHOA this is certainly one of the really spectacular ones, mind blown. I thought I had been desensitized to dropping my papers and I know photogrammetry has been around for awhile but yowza that’s clean!! And from so few pics! I’m heartbroken I can’t easily jump in and play with it this second

  • Kashur Intellectual
    Kashur Intellectual 6 months ago

    Perfect, now we need an AI that recreates from a bunch of photos a complete 3D model of the environment!

    • Rohan Asokan
      Rohan Asokan 6 months ago

      u can do it withh this one too.....

  • Matte Hues
    Matte Hues 6 months ago +1

    This could be integrated into Google Maps.. would be amazing

  • Shoot Red Barrels
    Shoot Red Barrels 6 months ago

    Okay, so this could create AI assisted photogrammetry with a very limited set of source photos... crazy.

  • LifeAsANoun
    LifeAsANoun 6 months ago

    Amazing vid. What, specifically, do I have to learn to be able to do this to my pictures?

  • Artificial Intelligence and Blockchain

    This is so great! When are we going to watch fully AI created movies?

  • TheHighwayMan
    TheHighwayMan 6 months ago

    Imagine this technology weaving together Google street view images seamlessly. You could drive down any road anywhere with photoreal results.

  • Elliot Marks
    Elliot Marks 6 months ago +3

    I was sitting on 2 legs of my chair when I saw this, and I literally lost balance and fell off. This is INSANE! Holy f***. WOW

  • Dev Pareek
    Dev Pareek 6 months ago

    This can also be used in part with metaverse or future gaming applications with ar and vr. Imagine an artist making few scenes and the gaps will be filled by this.. Then another software will convert it into 3d models and boom.. We have the space for the game ready.. Amazing

  • Alex Tuduran
    Alex Tuduran 6 months ago +34

    This will be gold for game development. Forget ray-tracing. Forget map design, forget mesh design forget materials. Just go outside, take photos => Ready to play level. Hyper-realistic graphics on the way. Jokes aside, this is groundbreaking and will make a huge impact in a lot of domains.

    • Alex Tuduran
      Alex Tuduran 6 months ago

      @Jeroen Goyvaerts This is well different from photogrametry. The only thing they have in common is the process of images acquisition and that's about it. In photogrametry, you end up with actual geometry and textures. Here it's a different thing. You end up with a system that can synthesize a photo without actually knowing what it synthesizes. It's like a neural network having a dream in which you control the camera. There's nothing tangible that you can extract besides the renders themselves. For sure the system has an understanding of what it represents, but that understanding is deeply embedded into the networks' model data (ex. weights of the neural networks used in the system). That's why this paper is so groundbreaking. Perspective interpolation is a thing many seek for tens of years and now it's finally here. Even if not mature yet, the results are extremely promising.

    • Jeroen Goyvaerts
      Jeroen Goyvaerts 6 months ago

      Similar technology to this already existed for quite some time its called photogrammetry and it does make 3D models. Look at quixel megascans for example a lot of their 3D models are made using that technique.

    • Alex Tuduran
      Alex Tuduran 6 months ago +1

      @Polygonist Perhaps a game could be done without even using polygons. We are so deeply embedded into the classic rasterization approach in 3D computer graphics, that we forget 3D data can be represented in other ways (ex. SDFs) or perhaps not represented at all. If a game gives you the illusion that you can move freely in a 3D space, do you really care if the rendering of that 3D space is actually based on meshes? For all I know, everything could be a hallucination of a ANN. And if I can interact and play, for me that's a game by all means. With technologies like the one presented in the paper, we're really stepping in a zone where the gap between videos (highly realistic, less interactive) and games (highly interactive, less realistic) could be shorten and perhaps get the best of both worlds some day. Also, cool name.

    • Polygonist
      Polygonist 6 months ago

      This doesn't generate 3D meshes, and therefore cannot be used to make a conventional game in any capacity. Combine this with newer, improved photogrammetry techniques and maybe you're on to something

    • Larion233
      Larion233 6 months ago

      @Michael Leue It looks to me like it forms a 3D environment with depth data. One can take that data and implement it into a game... no? Maybe not right now, but it's certainly possible.

  • Darian Stephens
    Darian Stephens 6 months ago +1

    Something I'd really like to see at some point, is to be able to provide a single picture or small group of pictures, and then just be able to explore an infinite AI-generated world.
    Not just filling in spaces or smoothing video, but being able to delve deeper and deeper in to a photo based with AI, including exploring entirely-unseen areas that the AI generates.
    Also, not just a regular generation algorithm with AI visuals slapped on.

  • Dristan Ross VII
    Dristan Ross VII 6 months ago

    I wonder if we could train this on cartoon objects/information as well - it would mean a lot for hand-drawn animations if one could create essentially 3d objects and scenes from a few select key frames.

  • Thanh Nguyen
    Thanh Nguyen 6 months ago +1

    This is god sent. Imagine the cost of movies and animation studios drop dramatically as we can create amazing and dynamic content using such little amount of key frames.

  • sour_patch_armadillo
    sour_patch_armadillo 6 months ago

    Imagine reconstructing scenes from a family vacation or photos of an old family home. Amazing research.

  • Maciej Jabłoński
    Maciej Jabłoński 6 months ago

    This is a stunning result. Far better than I would have expected!

  • hu an
    hu an 6 months ago

    is that possible to change it to a 3D model?

  • Eoku
    Eoku 6 months ago

    This could really speed up the photogrammetry process!!

  • Josh Kariakin
    Josh Kariakin 6 months ago

    waiting for the day this can be used easily to pull the real world into VR worlds in near real-time

  • Ilja
    Ilja 6 months ago +1

    I've never experienced how not realistic photorealism feels/looks before.

  • Andrew900460
    Andrew900460 6 months ago +1

    So basically, this is "Start with 1 frame per second, and get back 60 frames per second."
    very impressive interpolation I'd say

  • Hildebert
    Hildebert 6 months ago

    I hope stuff like this will be implemented in the gaming industry to make engines look absolutly real.

  • PizzaCutHD
    PizzaCutHD 5 months ago +1

    This would be useful for street view, you could view places where cars can't even go

  • BeansGalaxy
    BeansGalaxy 6 months ago

    I think its called photogrammetry, or where you take a TON of photos then use software to 3D model something. What this did with the playground slide is amazing compared to that

  • openroomxyz openroomxyz

    This looks to good to be true. I see many creative usage of this in future.

  • Dmitry Fatiev
    Dmitry Fatiev 6 months ago

    This is a step towards 3d scene from a picture (or several). This can be used in CG (imagine an artist sketches a scene and it's in 3d moments later) and if performance allows, even in real/near-real-time applications like computer vision and autonomous robotics.

    • Dmitry Fatiev
      Dmitry Fatiev 6 months ago

      Also a great thing for the nascent metaverses of all kinds. A few photos of the area and you have your simulation environment to train more AIs.

  • Daniel Ramskill
    Daniel Ramskill 6 months ago +1

    Taking virtual house tours in real estate to a whole new level!

  • kStuen
    kStuen 6 months ago +6

    I wonder if such technology can be used to make a driving game from Google Earth images.

    • Moritz Kaufmann
      Moritz Kaufmann 6 months ago

      This would be a great idea!

    • matthew 2
      matthew 2 6 months ago

      Could be used to create highly accurate 3d model of streets anywhere

  • Юрий Бурак
    Юрий Бурак 6 months ago

    Damn, starting to want to test it on some fake photos, or photos of smth with really complex form...
    Or rotate from inside, not outside
    Or change the environment while making photos(then, could it be used as scary transformation videoeffect?)
    Or shoot the drawings/3D images on screen