Depth Anything 3 Tests - SUPER IMPRESSIVE RESULTS!

[QuickscopingFTW]

ive tested da3 and it is a generational upgrade over 2, i can run full 1080p videos at 1920x1080 depth resolution that runs faster and looks better than VDA_L and other things like depthcrafter, the only issues i have run into so far is slight flicker and outside scenes with the sky seem kind of bugged, as well as things with a solid colour background, i think it uses a new feature for those which was not working properly in the gui i used for these tests, i used this gui here:
https://github.com/enoky/Depth-Anything-3-GUI
here are some video examples of the depth maps:

001.Depth.mp4

Img.0997.Depth.1.mp4

Img.1183.Depth.1.mp4

[QuickscopingFTW]

after some more testing it does seem to have some problems with crushing and clipping into pure white values and the falloff seems to be very narrow by default where a lot of the foreground is clipped white and a lot of the background will be clipped black making the resulting 3D look like 2D billboards instead of true 3D, idk if this is an issue with the gui i am using currently or a model issue, i would guess probably a gui issue, are there any plans to see if da3 will work with iw3 in the near future?

[nagadomi]

I haven't looked at it much yet,
but regarding the sky, it seems to be performing sky segmentation and maximum distance clipping.
The issue with the foreground being clipped might be the same problem as #503.
I still don't know whether the output is distance or disparity, but I suspect there might be some confusion around that area (diparity convertion and 0-1 normalization).

[QuickscopingFTW]

I ran another test on a 30 second movie clip, any scene that includes the sky seems to degrade the image quite a bit overall, at 6 seconds the snow machine fog thing is picked up very well, at 16 seconds as you can see why the sky isnt bugging the whole shot its very crisp and detailed, even picks up individual hair strands, its still overblown though into white so you cant see any detail on the actual face, at 20 seconds the dust flying past is picked up really well and the particles of sand that pop up look like they all get picked up too even some of the water at 27 seconds is picked up, although not the main rain which i think is a bit unrealistic anyway.
The bitrate is kind of bad cause i had to compress the 4k clips to under 10MB to put here:
Depth Map:

Zootopia_Depth.mp4

Reference:

Zootopia_Ref.mp4

With all the bugs sorted out this will be a hell of a lot more powerful and quicker than any of the current models used in iw3, cant wait will its fully working

[diverswan9]

Ran @QuickscopingFTW 's vid + depthmap through iw3.
https://drive.google.com/file/d/1jydeNiFek0TYJUwYRXWg3Q-UfMUqfOMt/view

[nagadomi]

It requires the use of a sky mask. The following is pseudocode. Maybe the author of Depth-Anything-3-GUI would understand.

def normalize_depth_with_sky_mask(depth, sky_mask):
    # batch = 1
    median_rel_dist = torch.median(depth)
    depth = depth / (median_rel_dist + 1e-6)
    max_rel_dist = torch.quantile(depth, 0.98)
    depth = torch.where(torch.logical_or(sky_mask, depth > max_rel_dist), max_rel_dist, depth)
    depth = 1.0 / (depth + 0.1)  # convert to disparity. try 0.1, 0.01, 0.5                                          
    depth = (depth - depth.min()) / (depth.max() - depth.min())

    return depth

out = da3mono_model(x)
depth = out["depth"].float()
sky_mask = out["sky"] > 0.3
depth = normalize_depth_with_sky_mask(depth, sky_mask)

[QuickscopingFTW]

the author of Depth-Anything-3-GUI implemented the patch and it has drastically improved the quality, here is a before and after, both runs were at 1080p res:

Before:

Snowy.Forest.30Fps.Depth.2.mp4

After:

Snowy.Forest.30Fps.Depth.1.mp4

[QuickscopingFTW]

really incredible the detail it has picking out the small branches on the tops of trees, im sure if it was a 4k video it would be even better too, i think the sky is a tad brighter because of the gamma settings i had

[QuickscopingFTW]

ok this is pretty insane, check this out, this was run at 3402x1428 with the new fix, i dont think ive ever seen a depth map so clear in my life, straight up looks like a z-depth buffer of a 3D scene:

Zootopia.2016.4K.Hdr.Dv.2160P.Bdremux.Ita.Eng.X265-Nahom.Depth.mp4

just a short snippet for now, full 30 seconds is processing

[QuickscopingFTW]

make sure to full screen it as its a near 4k clip, even the individual moving leaves behind the sign are tracked through the holes somehow

[QuickscopingFTW]

Here is the full clip, the fix did work overall a lot better but there is still some obvious artifacting around some edges and on smoke and particles, the smoke and particles in particular look worse than before imo
overall a big improvement though:

Zootopia.2016.4K.Hdr.Dv.2160P.Bdremux.Ita.Eng.X265-Nahom.Depth.1.mp4

[QuickscopingFTW]

x264 version if that doesnt play:

Zootopia.2016.4K.Hdr.Dv.2160P.Bdremux.Ita.Eng.X265-Nahom.Depth.1.mp4

[nagadomi]

I also added Any_V3_Mono to iw3.
The formulas for the sky mask and disparity conversion may be changed later.
This does not use the official API (wrapper), so if you notice anything unusual compared to Depth-Anything-3-GUI, please let me know.

[QuickscopingFTW]

and here is the full 1920x1080 res one from the gui, which only used around 8-10Gb of vram

gui_1920x1080.mp4

[QuickscopingFTW]

i am running the inpainting branch if that helps with anything

[QuickscopingFTW]

ive noticed on iw3 that most of the depth maps look a bit blocky around small details, like everything is made out of small squares, i noticed this back when using VDA_L and thought it was just how that model is but ive noticed it with this new model now too, it doesnt happen when using the model outside of iw3, i have no clue why:

in iw3:

out of iw3:

is this intentional? if so what does it do?

[QuickscopingFTW]

disregard this last message, i found out after asking that it was the edge fix setting that does this, whoops

[QuickscopingFTW]

seems i made a little mistake, iw3 depth input is the vertical whereas the gui was the horizontal, the difference in vram usage isnt nearly what i though it was, it does still seem that it uses slightly higher than the gui though, sorry again for the mixup

[QuickscopingFTW]

used 1080p depthmap to get this video, used the inpainting too, its cross eye view mode, i think its very good at a glance, would have to watch in vr to make sure though:

Snowy.Forest.30Fps.Rlf.Cross.mp4

[QuickscopingFTW]

can only notice the tiniest of flicker on some of the thinnest of branches at the tops

[QuickscopingFTW]

it seems to be working great all things considered, looks amazing with the inpainting, only things really are the weird artefacts from the sky sometimes and (maybe) the increased vram usage but idk if its just overhead from other stuff the program is doing or if its not running as optimally as it could be, anyway great job implementing it so fast and even adding it to the inpaint branch

[nagadomi]

Regarding the increase in VRAM usage, I think the main reason is that the head (the part other than DINOv2) is running in FP32.
However, this comes from the official code.
I plan to check if it can use FP16 as much as possible.

[nagadomi]

Also, da3mono-large is not the main model of DA3. There are still many aspects that can be improved using DA3, but I probably will not work on them right away.
Since da3mono-large gives not good results with inputs that include black bars, I plan to handle that kind of minor work first.

[TeeJay-NLD]

@nagadomi ... as always your dedication and speed in order to implement and thus support the latest ai-models just makes me very grateful. So much appreciated!

Currently I am running my first full feature conversion setting depth resolution at 518 VRAM usage with 10,1 GByte is just fine and comparable with VDA_L with the same settings. In my humble opinion nothing special to report here.

Unfortunately when it comes especially to those nasty distortions at very narrow structures I had hoped that DAV3 would have been much better than any previous versions. But... no... right now it isn't. But see for yourself...

Depth Resolution at 518

As soon as the depth resolution has been increased, in this example to 1080, the distortion largely disappears and the narrow structure is much better aligned, showing here...

Depth Resolution at 1080

... which of course drops the average fps conversion speed dramatically.

Both example images are taken from an interlaced screenshot, meaning, the left and right eye picture are merged together from a Full SBS conversion into one frame for a better comparison.

But as @nagadomi already mentioned this is just a first implementation of DAV3 with plenty of space for improvements. Anyways, looking forward how my first entire feature film is going to perform on a big screen tonight using this brand new ai-model.

Thanks for each and everything and write to you soon...

TeeJay-NLD

[diverswan9]

@TeeJay-NLD - the 518 looks 1\2 as deep as the 1080

[TeeJay-NLD]

@diverswan9 ... hi there!

Thanks for your comment. For a better reference I also snapshot-ed the same scene in anaglyph which does not show two times more depth with 1080.

I also included my current favorite setting using VDA_B at a depth resolution of 675 resulting in very minor distortion artifacts from my perspective of view. But of course feel free to judge for yourself :-).

DAV3, depth resolution 518

DAV3, depth resolution 1080

VDA_B, depth resolution 675 (my current favorite)

Looking forward what comes next... tonight it will be "The Goonies" (1985) as my first DAV3 feature film. Curious, what it will look like overall in comparison to my last version converted with VDA_B. All conversions will be presented of course in FullHD using a MK3D-MVC container :-).

Best wishes... TeeJa-NLD

[diverswan9]

I wrote this for another place but it's about DA3,
"A quick take on some comparisons between DAV2 (or Distill), DA3, and VDA. At first glance they look very similar but there's some differences. Da3 handles bodies/faces/people very similar to VDA. They are more upright and forward than DAV2 (Distill). With VDA they come forward just a little more. I've noticed with Dav2 (Distill) people are always leaning back just a little bit. For close-up of faces DA3 gives a bit more sculptedness than VDA (flatter). One area where DA3 fails 3D-wise is objects in the background and windows (I haven't really explored landscape stuff yet with DA3). So far (I was testing with 4:3 aspect 1080p TV shows - so lots of people) if I had to use a model for these types of film I would prefer using VDA. We'll see how things develop with DA3 down the road."

[andy500]

just convert a 5min clip with the new depth model but still not as good as vda the new model as a issues with the depth it can't keep the depth it drop the depth in some frames it is not as good as vda which l find no issue with the depth

[TeeJay-NLD]

... this is what I can confirm, too, and why I stopped watching my first entire feature film tonight :-).

[andy500]

l do my 3d movies with vdas and l do not use the imprint as there are issues with that l do my clips at 5mins 3d depth 2 .0 foreground scale 1 edge fix 3 flicker reduction 24 full tb and it works ok for me

[QuickscopingFTW]

Ok, I've figured out the vram problem kind of, when using the export format it seems to behave exactly like the gui does, using around the same for the same res, but as soon as anything else is involved it jumps significantly, even when its not doing any stereo stuff like if i select RGB-D it still uses around 5Gb more vram than just export, is this a bug or intended? and if it is, is there no way that you could run the stereo step separately from the depth step? i know there is the export and .yml method but is there no all in 1 solution without manually going back for each video done? this would be especially helpful for things like the inpainting fill as it uses a lot of vram by default limiting depth res if you were to do it all in 1.

Here is a test render of the Zootopia scene done in iw3 this time, the things that don't seem to be working right is of course the sky mask having artifacts and it seems some depth values are off as you can see at around 6 seconds with the stadium lights, the ground has no depth at all and is entirely flat, not sure if this is correct or not but looks kind of odd, apart from that and the flickering it seems to be working as intended:

iw3_Zootopia_1428x3402.mp4

[nagadomi]

Regarding VRAM usage, could it be caused by the batch size, worker threads, or NVENC?
Is it the same even when Low VRAM is checked and libx264 is used instead?

[QuickscopingFTW]

I've done some testing, hevc_nvenc seems to use around 1-2 extra GB of VRAM, libx264 and libx265 seems to not add any, low_vram seems to save around 3-4 Gb of VRAM, a weird thing is RGB-D seems to use 1 Gb more of VRAM than a full conversion using forward_fill

[QuickscopingFTW]

idk if this is possible but could there be a resolution setting for the inpainting? because it is pretty much unusable over 1440p especially at 4K which would require at least a 5090 with 32Gb of vram, is there no way it could internally render a lower res version and apply it to the higher res video? Also i had a question, when you set the depth resolution does it always run at that resolution or does it scale to the nearest valid resolution for the video that is being processed? This only seems to be an issue now as the new da3 model can run depth up to 4k unlike in the past with resolutions near 500p, like if you had the res set to say 1080 and the video its processing is 480 or 720 instead of running that at 1080 it just sets it to the highest res it can for 720?

[QuickscopingFTW]

depth batch size didnt seem to make a difference, and i couldnt test torch compile because there isnt a new patch for the new python version yet

[nagadomi]

When Low VRAM is enabled, Batch Size is ignored (always set to 1).

[diverswan9]

Here's something I came across. A drop in the 3D. Maybe something to do with Scene Boundary Detection or Flicker Reduction?

Below is VDA on the left and DA3 on the right. The upper pair are just about a dozen frames apart from the lower pair.

These dramatic drops in 3D were happening on and off throughout this 20minute film with DA3.
DA3 - https://drive.google.com/file/d/1gFOFLDSVhjDXgjGLr0swGAckaaqKL0Xb/view?usp=drive_link
VDA - https://drive.google.com/file/d/1zqINwxjMUU9LenTYPl7TBITI4L5tM5Jl/view?usp=drive_link
(same depth + convergence)

[diverswan9]

To put it simply... it's like DA3 overreacts to an object coming into the foreground and sends the background way back and compresses the 3d layers. In one scene in this film it literally goes flat-2D into the back. By the way, I tested this against both VDA and DAV2 and neither of them do this.

[nagadomi]

Yes, I was also concerned about that issue.
I compared the results of DA2 and DA3 using various convertion parameters, and the current settings seemed to be the best on average.
I think the fundamental problem is that it maps scenes at all distances to 0-1.

[QuickscopingFTW]

i tested the same scene on the new patch and some areas have less of the glitches than before but some scenes seem to have worse glitches than before for some reason:

iw3_Zootopia_1428x3402_new_patch.mp4

[QuickscopingFTW]

vram usage seemed the same as before but i only did the depth map, haven't tested with the actual stereo conversion stuff yet

[nagadomi]

da3mono-large outputs relative distance, so it cannot determine the metric distance in the scene (e.g., indoor or outdoor), forcing it to apply the same transformation to all scenes.
DA2 likely performs the same transformation during training, but differences in loss functions or other factors mean that converting da3mono-large's relative distance output to disparity does not yield the same results as DA2's disparity output.

Depth representation. Unlike DA2, which predicts scale–shift-invariant disparity, our teacher outputs scale–shift-invariant depth. Depth is preferable for downstream tasks, such as metric depth estimation and multiview geometry, that directly operate in depth space rather than disparity. To address depth’s reduced sensitivity for near-camera regions comparing to disparity, we predict exponential depth instead of linear depth, enhancing discrimination at small distances.

I suspect this exponential depth is the cause, but I don't fully understand it yet.

Using da3metric-large provides the maximum distance per scene, so it should be possible to do more.

[TeeJay-NLD]

Hi @nagadomi,

I am not quite sure if the following question relates to this topic but please allow me to try.

I noticed a new setting "Inpaint Max With" within the IW3 GUI. Unfortunately, I could not find any explanation on this in the given documentation. Therefore, I am curious on how this setting affect the overall inpainting (quality)? Does this setting only relates to certain "Depth Models" or other settings?

Sure, I can take a look at the underlying changes to the code but some describing words would be much more helpful at least for me. If I have overlooked any, I sincerely apologize.

Regarding your extensive to do-list I am super curious what comes next for IW3 :-).

Thanks for reading and as always happy coding...

TeeJay-NLD

[diverswan9]

"iw3: Add --inpaint-max-width option
This option simply downscales the output size.
For videos, using --vf scale option is more efficient."

See - dd89d6a

@TeeJay-NLD
I tried this using a 2160p source. It downscaled to 1080p. It's quicker to implement than entering code into -vf (src) but when running a conversion the processing speed is slower by a little. (for me 4fps using inpaint shortcut - 5fps using -vf (src))

[diverswan9]

Some comparisons... I used Any V3 Mono and VDA. I tried Any V2 but the results were very similar to VDA so...

Windows and Glass - DA3 misses most of the 3D separation of the glass and windows in these scenes but did better with catching the edge of the sliding glass (girl scene). There's 2 vid clips to compare.
https://drive.google.com/file/d/1LesolLlsM9CECZ5XhF3c3OGDLWD0ZNnC/view?usp=drive_link
https://drive.google.com/file/d/18KpqYvP25Ui2e09dstyXLLoLDNt69ehC/view?usp=drive_link
DA3

VDA

DA3

VDA

DA3

VDA

Knive Blade
Here's a clip where DA3 mono keeps the knive blade in the correct 3d plane when the camera de-focuses but VDA loses it...
DA3
https://drive.google.com/file/d/1AcgpRbHyo_OVugWUJly0Zrb65dV8ud_8/view?usp=sharing
VDA
https://drive.google.com/file/d/1yzCDXK_STkjP_FVsyv9eEzQXA4g6OPJ2/view?usp=sharing

[QuickscopingFTW]

I've done some more testing with da3 and it seems the quality/accuracy of the depth degrades the higher the res goes, around 1080p seems like the sweet spot where its only a tiny bit less stable then the 504 res training resolution, the issues with the background/sky seem to be a lot worse the higher res but still show up on the 504 res version, TTA setting seems to work but i dont notice much or any difference in quality or stability

[nagadomi]

I tested it with various images, and when there's no depth in front of the subject, it looks very good.
The 3D geometry of objects appears more accurate than with other models.
However, if there's depth closer than the subject, it becomes flat.
Additionally, borders (including black bars) and vignette effects seem to be detected as the frontmost object.

[nagadomi]

I feel like something different from my understanding is happening with convergence.
Trying automatic convergence might help me understand something.

[francdn]

I tested it with various images, and when there's no depth in front of the subject, it looks very good. The 3D geometry of objects appears more accurate than with other models. However, if there's depth closer than the subject, it becomes flat. Additionally, borders (including black bars) and vignette effects seem to be detected as the frontmost object.

Yes, I noticed in some scenes when the subject is very close to the camera, the subject becomes completely flat in 3d while the background has 3D depth. This does not happen with V2 where both the subject and background appear in 3D.

[nagadomi]

If DA2's scaling looks good, I plan to try a machine learning model that converts da3mono's output to DA2's output (da3 depth to da2 disparity convertion).
Currently, a simple conversion formula is used, but it doesn't look very similar.

[melam98]

Hello, @nagadomi or anyone, can you please help me? Can you tell me how to install Depth Anything V3, since it’s in safetensors format and not pth? When I update IW3, I only see Any_V3_Mono. I’d love to try the others. I don’t know if there’s anywhere that explains updates and their details when they’re released, but if so, it would be great to know, especially for those of us who aren’t computer-savvy and use Windows. Thanks.

[diverswan9]

At this point for iw3 there is only Any_V3_Mono. It's still early stages and they are still working the bugs out. Eventually there probably will be some other variants available.

For info on 'updates' I usually go here...
https://github.com/nagadomi/nunif/commits/dev/
or here...
https://github.com/nagadomi/nunif/activity
and/or following the 'Discussions' and 'Issues' sections.

... as for actually understanding what these technical updates do takes a bit if work for me anyways.

[Tilkanator]

I downloaded the mono model and placed it into the directory for iw3 checkpoints. Do I have to rename it to something specific for it to work?

[QuickscopingFTW]

the mono model should auto download when u try to use it

[QuickscopingFTW]

@nagadomi I've done a couple more tests, I used 2 clips from the game Arc Raiders that I recorded, one of the clips is fully outside and the other fully inside.

Outside clip:

Arc_Outside_1080p.mp4

Inside clip:

Arc_Inside_1080p.mp4

As you can see the clip outside has serious degradation and errors compared to the inside clip, idk if it is fixable currently but i have looked at the official Depth Anything 3 github and they seem to have added a lot more stuff which might help fix the problem.

[QuickscopingFTW]

here is a link to the uncompressed for github depth videos and source videos i used:
https://pixeldrain.com/l/sxdya63J

[QuickscopingFTW]

if its at all useful to you

[nagadomi]

The only thing I can handle on my side is depth scaling (mapping to disparity).
Issues such as sky segmentation or HUD artifacts will probably not be fixed in DA3.

[QuickscopingFTW]

ah, so its a problem with the base model? wonder if they will ever fix it

[QuickscopingFTW]

Just tested the new patch and the depth scale seems more stable overall, used the same settings as last time:

Zootopia_3402x1428_Depth.mp4

at the 3 second mark when the text shows up at the bottom the old version used to shift the entire scenes depth a lot darker, seems to stay the same now, overall there might be a tiny bit less flicker too but its hard to tell, unfortunate that the model has a problem with the sky thing, i hope they patch it sometime or at least hope it wont be and issue if they release a video version like VDA

[QuickscopingFTW]

after some more testing the depth scaling is defo a lot more stable, good job with that update

[QuickscopingFTW]

i rendered out the scene at 1920x1080p depth res and 1920x1080p inpaint res with the custom inpaint model that was made, it looks really solid now and from what i can see those visual glitches present in the depth map dont really show up visually in the actual render, is this just a bug on the depth map export then maybe?

video is a little compressed to fit on github

Zootopia.2016.4K.Hdr.Dv.2160P.Bdremux.Ita.Eng.X265-Nahom.Rlf.Cross.1.mp4

[nagadomi]

Depth glitches in low-texture regions have almost no effect on the SBS view.
For example, in an extreme case, the depth of a solid black region may become very unstable, but since a solid black area moving left or right cannot be perceived in the RGB image, it does not affect the visual result.
RGB/Depth

Cross eyed

[QuickscopingFTW]

yea thats what i thought might be happening, thanks for confirming, im very happy with how da3 is running in iw3 now, dont have any issues left with it at all tbh, only things with it that could be improved are basically what the video version will fix when it releases, mainly the flicker, great job with the speedy implementation and fixes!

[diverswan9]

I ran a test after 'Update' with those clips I shared from above. There's less of a depth jump between some scenes... but also there's a little less depth in general for the same depth settings.

It still is not showing 3D behind glass windows.

[nagadomi]

This is not a change from VDA, so there is no point in comparing them.
If VDA is better, simply use VDA.

[nagadomi]

This is an indoor scene and there is not much distance between the woman and the wall behind her.
Previously, the wall disparity was 0, but now it is not. This is because the wall is not 100 meters away.
There should be no problem when viewing on a VR device. However, if you care about anaglyph or 3DTV, it might be better to create a separate model that scales disparity to 0-1.

[diverswan9]

I just realized I made a mistake when running these tests.... (hits forehead). The source clip has black bars that I forgot to crop.... so sorry. I deleted my above posts. Cropping can make a huge difference.... that is for certain.

[nagadomi]

OK. However, considering anaglyph and 3D TV, I thought it would be better to have a 0-1 scaled version, so I added Any_V3_Mono_01. This version always stretches disparity to the 0-1 range.

Note that for SBS for VR devices, the Any_V3_Mono version is recommended. Any_V3_Mono_01 can be used when you need a 0-1 range disparity image and it prevents depth from disappearing on anaglyph or 3D TV. However, on VR devices, it can cause either indoor scenes to appear at an abnormal scale or outdoor scenes to become flat. (Which problem occurs depends on the 3D strength.)

[diverswan9]

ok... Thanks!

[francdn]

Even with strength at 5, Any_V3_Mono and Any_V3_Mono_01 produce a flat subject (or almost flat) when the subject is very close to the camera.

[nagadomi]

It has the parameter, and I think it would be nice if it could be adjusted with foreground-scale, but it's not possible yet.
Try setting shift = 0.

nunif/iw3/depth_anything_v3_model.py

Lines 45 to 47 in 9ca17ac

	# TODO: This value should ideally be adjustable via the foreground scale option,
	# but currently it is not possible.
	shift = 0.2

If that has no effect, it is an issue on the model or training data side (minimum distance clipping).

[francdn]

I tried with shift = 0 and it seems to help but it does not get rid of the problem completely.