Early AI „everything turns into dog heads“ vibes. Beautiful.
drcongo 17 minutes ago [-]
I miss those. Anyone know if it's still possible to get the models etc. needed to generate them?
schneehertz 59 minutes ago [-]
san check, 1d10
ghurtado 4 hours ago [-]
Seth Brundle has entered the chat.
Dumbledumb 13 minutes ago [-]
In Chapter D.7 they describe: "The complex reflection in water is interpreted by the network as a distant mountain, therefore the water surface is broken."
This is really interesting to me because the model would have to encode the reflection as both the depth of the reflecting surface (for texture, scattering etc) as well as the "real depth" of the reflected object. The examples in Figure 11 and 12 already look amazing.
Long tail problems indeed.
Leptonmaniac 3 hours ago [-]
Can someone ELI5 what this does? I read the abstract and tried to find differences in the provided examples, but I don't understand (and don't see) what the "photorealistic" part is.
emsign 3 hours ago [-]
Imagine history documentaries where they take an old photo and free objects from the background and move them round giving the illusion of parallax movement. This software does that in less than a second, creating a 3D model that can be accurately moved (or the camera for that matter) in your video editor. It's not new, but this one is fast and "sharp".
Gaussian splashing is pretty awesome.
kurtis_reed 2 hours ago [-]
What are free objects?
ferriswil 2 hours ago [-]
The "free" in this case is a verb. The objects are freed from the background.
Retr0id 2 hours ago [-]
Until your comment I didn't realise I'd also read it wrong (despite getting the gist of it). Attempted rephrase of the original sentence:
Imagine history documentaries where they take an old photo, free objects from the background, and then move them round to give the illusion of parallax.
necovek 1 hours ago [-]
I'd suggest a different verb like "detach" or "unlink".
tzot 1 hours ago [-]
> Imagine history documentaries where they take an old photo, free objects from the background
Even using commas, if you leave the ambiguous “free” I suggest you prefix “objects” with “the” or “any”.
2 hours ago [-]
ares623 3 hours ago [-]
Takes a 2D image and allows you to simulate moving the angle of the camera with correct-ish parallax effect and proper subject isolation (seems to be able to handle multiple subjects in the same scene as well)
I guess this is what they use for the portrait mode effects.
derleyici 3 hours ago [-]
It turns a single photo into a rough 3D scene so you can slightly move the camera and see new, realistic views. "Photorealistic" means it preserves real textures and lighting instead of a flat depth effect. Similar behavior can be seen with Apple's Spatial Scene feature in the Photos app: https://files.catbox.moe/93w7rw.mov
eloisius 3 hours ago [-]
From a single picture it infers a hidden 3D representation, from which you can produce photorealistic images from slightly different vantage points (novel views).
avaer 3 hours ago [-]
There's nothing "hidden" about the 3d represenation. It's a point cloud (in meters) with colors, and a guess at the the "camera" that produced it.
(I am oversimplifying).
uh_uh 2 hours ago [-]
"Hidden" or "latent" in a context like this just means variables that the algo is trying to infer because it doesn't have direct access to them.
eloisius 3 hours ago [-]
Hidden in the sense of neural net layers. I mean intermediary representation.
avaer 3 hours ago [-]
Right.
I just want to emphasize that this is not a NERF where the model magically produces an image from an angle and then you ask "ok but how did you get this?" and it throws up its hands and says "I dunno, I ran some math and I got this image" :D.
avaer 3 hours ago [-]
It makes your picture 3D. The "photorealistic" part is "it's better than these other ways".
carabiner 2 hours ago [-]
Black Mirror episode portraying what this could do: https://youtu.be/XJIq_Dy--VA?t=14. If Apple ran SHARP on this photo and compared it to the show, that would be incredible.
Agreed, this is a terrible presentation. The paper abstract is bordering on word salad, the demo images are meaningless and don’t show any clear difference to the previous SotA, the introduction talks about “nearby” views while the images appear to show zooming in, etc.
derleyici 3 hours ago [-]
Apple's Spatial Scene in the Photos app shows similar behavior, turning a single photo into a small 3D scene that you can view by tilting the phone. Demo here: https://files.catbox.moe/93w7rw.mov
Traubenfuchs 2 hours ago [-]
It‘s awful and often creates a blurry mess in the imaginated space behind the object.
Photoshop content aware fill could do equally or better many years ago.
CUDA is needed to render side scrolling video, but there is many ways to do other things with result.
matthewmacleod 2 hours ago [-]
This is specifically only for video rendering. The model itself works across GPU, CPU, and MPS.
yodon 4 hours ago [-]
> photorealistic 3D representation from a single photograph in less than a second
tartoran 4 hours ago [-]
Impressive but something doesn't feel right to me.. Possibly too much sharpness, possibly a mix of cliches, all amplified at once.
arjie 4 hours ago [-]
This is incredibly cool. It's interesting how it fails in the section where you need to in-paint. SVC seems to do that better than all the rest, though not anywhere close to the photorealism of this model.
Is there a similar flow but to transform either a video/photo/NeRF of a scene into a tighter, minimal polygon approximation of it. The reason I ask is that it would make some things really cool. To make my baby monitor mount I had to knock out the calipers and measure the pins and this and that, but if I could take a couple of photos and iterate in software that would be sick.
avaer 3 hours ago [-]
Is there a link with some sample gaussian splat files coming from this model? I couldn't find it.
Without that that it's hard to tell how cherry-picked the NVS video samples are.
I thought this was going to be the Super Troopers version
harhargange 4 hours ago [-]
TMPI looks just as good if not better.
wfme 3 hours ago [-]
Have a look through the rest of the images. TMPI has some pretty obvious shortcomings in a lot of them.
1. Sky looks jank
2. Blurry/warped behind the horse
3. The head seems to move a lot more than the body. You could argue that this one is desirable
4. Bit of warping and ghosting around the edges of the flowers. Particularly noticeable towards the top of the image.
5. Very minor but the flowers move as if they aren't attached to the wall
jjcm 4 hours ago [-]
Disagree - look at the sky in the seaweed shot. It doesn't quite get the depth right in anything, and the edges of things look off.
shwaj 3 hours ago [-]
Agreed. The head of the fly also seems to have weird depth.
diimdeep 3 hours ago [-]
Works great, model file is 2.8 GB, on M2 rendering took a few seconds, result is guassian .ply file but repo implementation requires CUDA card to render video, I have used one of webgl live renderers from here https://github.com/scier/MetalSplatter?tab=readme-ov-file#re...
yodon 4 hours ago [-]
See also Spaitial[0] which announced today full 3D environment generation from a single image
The best I've seen so far is Marble from World Labs, though that gives you a full 360 environment and takes several minutes to do so.
dag11 4 hours ago [-]
I'm confused, does it actually generate environments from photographs? I can't view the galleries since I didn't sign up for emails but all of the gallery thumbnails are AI, not photos.
jrflowers 2 hours ago [-]
> I'm confused, does it actually generate environments from photographs?
It’s a website that collects people’s email addresses
andsoitis 4 hours ago [-]
Why are all their examples of rooms?
Why no landscape or underwater scenes or something in space, etc.?
jaccola 3 hours ago [-]
Constrained environments are much simpler.
I believe this company is doing image (or text) -> off the shelf image model to generate more views -> some variant of gaussian splatting.
So they aren't really "generating" the world as one might imagine.
benatkin 4 hours ago [-]
That is really impressive. However, it was a bit confusing at first because in the koala example at the top, the zoomed in area is only slightly bigger than the source area. I wonder why they didn't make it 2-3x as big in both axes like they did with the others.
ballpug 4 hours ago [-]
[dead]
IlikeKitties 4 hours ago [-]
[flagged]
calvinmorrison 4 hours ago [-]
I understand AI for reasoning, knowledge, etc. I haven't figured out how anyone wants to spend money for this visual and video stuff. It just seems like a bad idea.
accurrent 4 hours ago [-]
Simulation. It takes a lot of effort today to bring up simulations in various fields. 3 D programming is very nontrivial and asset development is extremely expensive. If I have a workspace I can take a photo of and just use it to generate a 3d scene I can then use it in simulations to test ideas out. This is particularly useful in robotics and industrial automation already.
rv3392 3 hours ago [-]
This specific paper is pretty different to the kind of photo/video generation that has been hyped up in recent years. In this case, I think this might be what they're using for the iOS spatial wallpaper feature, which is arguably useless but is definitely an aesthetic differentiator to Android devices. So, it's indirectly making money.
re-thc 4 hours ago [-]
Do people not spend on entertainment? Commercials? It's probably less of a bad idea than knowledge. AI giving a bad visual has less negatives than giving the wrong knowledge leading to the wrong decision.
Rendered at 09:33:57 GMT+0000 (Coordinated Universal Time) with Vercel.
This is really interesting to me because the model would have to encode the reflection as both the depth of the reflecting surface (for texture, scattering etc) as well as the "real depth" of the reflected object. The examples in Figure 11 and 12 already look amazing.
Long tail problems indeed.
Gaussian splashing is pretty awesome.
Imagine history documentaries where they take an old photo, free objects from the background, and then move them round to give the illusion of parallax.
Even using commas, if you leave the ambiguous “free” I suggest you prefix “objects” with “the” or “any”.
I guess this is what they use for the portrait mode effects.
(I am oversimplifying).
I just want to emphasize that this is not a NERF where the model magically produces an image from an angle and then you ask "ok but how did you get this?" and it throws up its hands and says "I dunno, I ran some math and I got this image" :D.
Or if you prefer Blade Runner: https://youtu.be/qHepKd38pr0?t=107
Photoshop content aware fill could do equally or better many years ago.
https://github.com/apple/ml-sharp#rendering-trajectories-cud...
CUDA is needed to render side scrolling video, but there is many ways to do other things with result.
Is there a similar flow but to transform either a video/photo/NeRF of a scene into a tighter, minimal polygon approximation of it. The reason I ask is that it would make some things really cool. To make my baby monitor mount I had to knock out the calipers and measure the pins and this and that, but if I could take a couple of photos and iterate in software that would be sick.
Without that that it's hard to tell how cherry-picked the NVS video samples are.
EDIT: I did it myself, if anyone wants to check out the result (caveat, n=1): https://github.com/avaer/ml-sharp-example
1. Sky looks jank 2. Blurry/warped behind the horse 3. The head seems to move a lot more than the body. You could argue that this one is desirable 4. Bit of warping and ghosting around the edges of the flowers. Particularly noticeable towards the top of the image. 5. Very minor but the flowers move as if they aren't attached to the wall
[0]https://www.spaitial.ai/
It’s a website that collects people’s email addresses
Why no landscape or underwater scenes or something in space, etc.?
I believe this company is doing image (or text) -> off the shelf image model to generate more views -> some variant of gaussian splatting.
So they aren't really "generating" the world as one might imagine.