About inpainting- or the art of persuading you, Nagadomi :-)

[TeeJay-NLD]

Dear Nagadomi,

With so much joy and pleasure I am using your awesome application for quite some time now. Frankly, I am deeply impressed and very grateful about your continuing efforts to improve nunif constantly!

With lots of experiences converting into Full SBS using various settings with iw3 in my humble opinion I would very much like to see some more improvements in regard to...

... inpainting methods because even using from my understanding the most advanced Method "forward fill" still does produce some annoying "smearing" artifacts like you can see here...

I would agree, that this left and right image might show a more or less very difficult scene but within an entire movie scenes like these are common.

... some more improvements with "Flicker Reduction". While my favorite Depth Model is "Distill_Any_L" even with a lookahead buffer size of "150" I still recognize some good amount of flickering especially with some more delicate narrow objects panning through some scenes.

Dear Nagadomi, maybe you can give your community and me some outlook on your future plans regarding these two areas. I certainly would appreciate it so much!

Once again thank you very much for sharing your fantastic work with us all.

My most warm regards... TeeJay-NLD

P.S. What is your preferred way to "buy you a coffee" :-)
P.P.S. I am using your dev branch

[nagadomi]

hi,
iw3 does not currently use proper inpainting. row_flow_v3 and mlbw use image warping and forward_fill simply copies the neighbor pixels.
The only technical problem remaining in the stereo generation method is inpainting so I am planning to work on it sometime.
Depth flickering is a problem of the depth estimation model. If it is a concern try using VDA.

P.S. What is your preferred way to "buy you a coffee" :-)

There is a patreon link on the right side of the repository page.
https://www.patreon.com/c/nagadomi

[TeeJay-NLD]

Dear Nagadomi,

Your prompt reply is very much appreciated. Thank you for that!
I asked myself if you would or could like to be a bit more "time-specific" about your "sometime", couldn't you ;-)?
Unfortunately, I am not very happy with the "VDA" results in terms of wrapping artifacts compared to "Distill_Any_L". That's why I prefer the latter much more. Anyhow, thanks for your appreciated advice.
Looking so much forward for any upcoming improvement.

Most warm regards... TeeJay-NLD

P.S. You now have one more paid patreon member :-).

[nagadomi]

No time can be promised.
However, I have previously tried training inpainting models and most of the related code already exists.
It is not published because it is worse than row_flow_v3.
The reason the results are not good is known, but fixing it is troublesome so it has been left alone.
I am currently working on a task separate from iw3, but since it is stuck I might work on it next week as a break. However, if it cannot be done in about a week it might be left alone again ;_;

Unrelated to inpainting, VDA_B, VDA_Metric_B, VDA_Metric_S was just released. So I'll probably add it tomorrow.
VDA seems to be under active development, so I expect various improvements will come along eventually.
Also, if the problem is not distortion in 3D space but image warping artifacts, then it is an issue of the stereo generation method (inpainting or antialiasing/edge dilation) rather than the depth estimation model.

[TeeJay-NLD]

Nagadomi, With...

Also, if the problem is not distortion in 3D space but image warping artifacts, then it is an issue of the stereo generation method (inpainting or antialiasing/edge dilation) rather than the depth estimation model.

... you nailed the difference in visual quality between "Distill_Any_L" and "VDA_L". It seems that the generation methods used for "VDA_L" do indeed produce much more anti-aliasing artifacts around most edges of the emphasized stereoscopic objects. Therefore I am so much looking for a state-of-the-art new inpainting method. This indeed would be one of the most important and necessary improvements using AI related stereoscopic generation.

Anyways, looking forward to run some tests with the soon to be implemented VDA_B and metric variants.

Not to forget, it seems that next week is going to be an interesting "inpainting week" somehow ;-).

Thanks again, Nagadomi!

Best wishes... TeeJay-NLD

[TeeJay-NLD]

Dear @nagadomi ,

Interesting. I just recognized a new directory named "inpainting" coming with the dev branch. And I wonder the deeper meaning of it ;-). Does this indicate some pause on your other project and thus you are about to resolve the inpainting issues within iw3 once and for all? Whatever, I'm keeping my fingers crossed for you and holding my breath about any possible progression in the coming days.

Good luck, and a huge thanks, @nagadomi !

TeeJay-NLD

[nagadomi]

Please do not expect too much.
I need to develop something that can achieve 20 FPS at 1080p resolution.
The usual expected result is that it is impossible or it becomes very bad.
I only hope this will be a trigger for me to challenge this problem.

[TeeJay-NLD]

... of course not, dear @nagadomi ! Call it my positive attitude that I believe you can achieve some satisfactory results at least :-). May I ask you why trying to achieve 20 fps at 1080p? On what kind of GPU? As for me it is quality that counts thus a compromise between quality and speed would be appropriate. In this regard maybe you want to implement two settings in the future. One prioritizing speed, another quality?
If it's all right for you please allow me to close just by wishing you some surprisingly good new inpainting results :-).
Thanks for your work!
TeeJay-NLD

[nagadomi]

See #389 (comment)
If you do not care about VRAM or processing time, use StereoCrafter or StereoMaster that uses it.
That does not match my approach.

[TeeJay-NLD]

... many thanks for pointing at StereoCrafter and/ or StereoMaster. Your clarification is totally understandable. Thanks again. But... in my humble opinion there are many advantages using your fabulous IW3 despite of StereoCrafter and/ or StereoMaster. Even with those two I could not achieve visually much better inpainting results compared to your IW3 "forwad_fill" method together with "Distill_Any_X". In this regard please take it as a real big compliment, @nagadomi .
Thanks for your reply.
Best wishes... TeeJay-NLD

[andy500]

Hi nagadomi what model is your inpaint use for l am using forward fill and VDA L and thank you for all the updates .

[nagadomi]

I plan to support forward_fill and mlbw_l2. Since inpainting is implemented only on the stereo generation method side, it can support any depth model.

[andy500]

Thank you for your message nagadomi

[TeeJay-NLD]

Hi there @nagadomi,

Would you mind sharing some actual information about the current state of your new inpainting solution approaches? And what maybe your outlook and what you think the prospects are for that?
Looking forward to some more details if there is any mentionable progress.

Many thanks and all the best...

TeeJay-NLD

[nagadomi]

The basic parts of inpainting for images and for videos are already done.
The inpainting model itself is not worse than I expected but often it is already distorted by depth mismatch or warp before inpainting so the impression is not good.
I am still adjusting now but unfortunately I will be away for about one month from 9/22.

What I found after trying is that it will probably not give good results unless the whole image is redrawn.
Diffusion based models also in practice do not inpaint only the holes but redraw the whole image.
However if I do that it will cause a new problem where the texture changes between left and right views and stereo/3d artifacts appear over the whole image so I do not plan to go that far. (As a very basic point regarding this, since the area requiring impainting is only visible from one eye, the original issue is not stereo/3D artifacts.)
In the end, it will probably just add one more model that is not much different from the existing row_flow_v3 or mlbw.

[TeeJay-NLD]

Dear @nagadomi,
Your detailed reply is very much appreciated. Thank you for that. And a huge thank you for trying to tackle that persistent inpainting problem, kind of the holy grail for this matter it seems, with your mentioned approach. I am still a bit unsure on how to assess your achieved results. Thus it seems we all have to be patient until you completed and released that new method to see it with our own eyes. As soon as you find some time for this project again there may be more headroom for tweaking and improving the quality. In any case, it sounds to me like you're on the right track.
Looking much forward to the new method.
Thanks again, @nagadomi. And have a good trip!
TeeJay-NLD
P.S. Time for me to buy you another coffee :-).

[nagadomi]

I plan to make it possible to test it in its current state within this week. I am currently working on iw3_inpaint branch, so I plan to make it usable by switching to it just like dev branch.
However, as mentioned above, it is not impressive.
It is possible to provide examples that look good, but if one actually tries it, it is possible to find countless bad examples.

[andy500]

just see you have added a new code Add --method forward_inpaint but can not find this models
VIDEO_MODEL_URL = "models/video_inpant_v4/inpaint.light_video_inpaint_v1.pth"
IMAGE_MODEL_URL = "models/inpant_v7_gan4_ffc6/inpaint.light_inpaint_v1.pth"

[nagadomi]

I haven't uploaded them yet.
I'll let you know when it's available.

[andy500]

thank you

[nagadomi]

@TeeJay-NLD @andy500
See #484

[TeeJay-NLD]

... therefore just a brief note on how thankful I am for what you already achieved providing us with your "very private tool" of IW3, especially your latest inpainting-improvements. I really keep my fingers crossed that you will maintain and improve this great tool as soon as you get back in a couple of weeks.

@nagadomi All the best and looking so much forward for everything that is coming up regarding IW3.

Best wishes... TeeJay-NLD

[diverswan9]

iw3 inpaint test...Aspect Ratio.=7:8 Top is 'row-flow-v3', bottom is 'mlbw_l2_inpaint' mask '1' in+out (observe around Batman's head > left side or right side) > https://drive.google.com/file/d/1ZuUdlXiZ-NJ0PKnek_-1hEZSa3YBMKIQ/view?usp=drive_link

[nagadomi]

Most of the distortion comes from depth mismatches caused by Edge Fix(--edge-dilation).
If you specify --method mask_mlbw_l2 in the CLI, inpainting mask will be output without inpainting.
If the image is already distorted at that stage, it will not be fixed.

[francdn]

What would be the recommended settings for mask dilation? Also, is mask dilation related to edge fix?

[andy500]

hi nagadomi am impressed with your inpaint does a good job thank you

[TeeJay-NLD]

@nagadomi
... after watching a couple of feature movies a big shout out to you how impressed I am already with your current inpainting solution. I would be very delighted to improve this even further as soon as you are back in "iw3-town" giving us a brief overview on how we could train the inpainting method by ourself with our own training data :-).
Have a good time wherever you might be!
Best wishes... TeeJay-NLD

[nagadomi]

The error seems to be due to insufficient image files ( less than 12 pair) in one of the folders under data\video_inpaint_v1\train, not in data\video_inpaint_v1\eval.

[TeeJay-NLD]

... you nailed it, @nagadomi . I only get 8 pairs per sequence even if I "force" 16 using the switch --seq 16.

Do you have an explanation for this behavior?

Best wishes... TeeJay-NLD

I found the cause of this issue... it's simply model based. Using your example with "Any_V2_B" creates a pair of 16. But I am training with "VDA_L" which only creates 8 pairs of training images. Maybe you can verify this and post a workaround in order to train also "VDA_L"? Much appreciated @nagadomi !

[nagadomi]

Ah, VDA is not supported.

I also changed to show which folder is causing the error. 8e09536

[TeeJay-NLD]

... are you planing to support "VDA" then? I guess, I must update the inpainting-branch then!?!

[nagadomi]

I don't have any plans to support it at the moment, but there's no strong reason not to, so it might be added in the future.
From my perspective, it can be replaced with Any_V2, and the lack of temporal stability might actually help train a more robust model.

[KolaKater]

When the inpaint method is applied, the images regularly exhibit this kind of shadowing, very similar to the Baseline-Poly method of SpatialMe:

[nagadomi]

As noted above
Inpainting is not applied to areas that are not included in the inpainting mask.

• Mask is misaligned with edge fix
• Mask is not in the area
• Foreground or background pixel is leaking out of the mask (mask is too small)

These can be checked by outputting masks.

[nagadomi]

In the case of video, motion blur or interlacing can sometimes make it appear like that.

[John9520]

Hi,
It is possible to add the inpaint method with Iw3 Desktop ?
Thank you for reading me.

[nagadomi]

No, the current inpaint model does not support stream input.
Also, the current inpaint model alone runs at around 35 FPS on 1080p (single side view) RTX 3070 Ti, so combined with other processing it will likely drop to around 5-15 FPS.

[TeeJay-NLD]

Hi there 3D-folks and @nagadomi of course,

Just a brief note that I successfully trained my first inpainting inpaint.light_video_inpaint_v1.pth running all the documented epochs plus fine tuning. As expected I am very much impressed with the final result using my own *.pth which frankly seems to outperform the default inpainting *.pth from what I see. As soon as I have some more time I even will try to further improve my data with some longer and more diverse video input footage.

Anyhow, I am extremely grateful that @nagadomi made this all possible for us just because the visual benefit from this new AI-inpainting is such a treat and was long awaited. At least for me.
Looking forward for the next leap coming. Maybe by implementing the upcoming DINOv3 and VDA_v3 based on DAv3!?? I am quite sure if this is possible IW3 will not disappoint in this regard.

My perspective about some rather huge visual quality improvements for 2026 couldn't be brighter.

With my best wishes to all of you... TeeJay-NLD

[diverswan9]

A friend made this...

''Here is a newly trained model for iw3 video inpainting. I have runs tests and it surpasses the current model's color-matching prediction.
It has been trained on a 130GB dataset for 400 epochs + 30 epochs with GAN discriminator. (I am going to train it more with another 200 epochs + a ton of GAN epochs)

Just drop it in your checkpoints folder: nunif/iw3/pretrained_models/hub/checkpoints

I named it the same as the old model so backup your current model if you wish to switch back to it later. (you can also re-download the original model from github)''

"Update: trained with an additional 200 epochs on a larger dataset. Training evaluation dropped from 0.00364 to 0.00294 (lower is better)"

"Update: here is the latest version of the video inpaint model for iw3. See instructions above. Trained with 800 epochs on over 500K image/mask pairs. Enjoy!"

https://drive.google.com/file/d/1lgwmIQNdtpFpI68ch57ShcFtWX_pkChB/view?usp=sharing
<<<(updated 11.26.25 3rd version)>>>

@enoky - a Large model > see post below...
#470 (comment)
<<<(12.5.25)>>>

His Git Overview page...
https://github.com/enoky

[nagadomi]

The inpaint model configuration file shown above has been supported in #552 .
If you encounter any strange behavior or have any requests, please submit an issue.

[ViRT1ST]

Just drop it in your checkpoints folder: nunif/iw3/pretrained_models/hub/checkpoints

After that just use mlbw_12_inpaint?
I don't understand how iw3 will use Enoki's inpaint model if I still select mlbw_12_inpaint in the UI.

[diverswan9]

@ViRT1ST - rename enoky's inpaint file to

iw3_light_video_inpaint_v1_20250919-Aether.pth

Put it in the checkpoints folder.
Run update.bat
Download this and put in nunif/iw3 folder...
inpaint_models.yml

When choosing 'Method' when you choose either of the 'inpaint' methods you should see aether's (enoky's) there...

[KolaKater]

@ViRT1ST - rename enoky's inpaint file to

iw3_light_video_inpaint_v1_20250919-Aether.pth

Put it in the checkpoints folder. Run update.bat Download this and put in nunif/iw3 folder... inpaint_models.yml

When choosing 'Method' when you choose either of the 'inpaint' methods you should see aether's (enoky's) there...

do you still make updates for the inpaint model?

[diverswan9]

@KolaKater - he's working on something training-wise,

"For anyone interested, I will be releasing a new iw3 inpaint model trained on a custom generated version of the Kubric dataset."
-aether (enoky)

It sounds like it's a lot more involved so it's taking some time (a few days maybe?). We'll see...

[andy500]

how do l use the link above when there is no link Enoky's inpaint model updated (11.26.25) ?

[enoky]

this is the link above: #470 (comment)

Here is an even newer version trained with 800 epochs:
iw3_light_video_inpaint_v1_20250919.zip

[andy500]

thank you for letting your work available to use

[diverswan9]

updated my link above for enoky's 3rd version
#470 (comment)

[diverswan9]

Enoky's 1st inpaint model and third version...

[KolaKater]

I just tested v3 from Enoky. One of the main issues is still with the hair: the brighter the hair, the more ghosting shadows appear.
here is a screenshot with setting:Distill_Any_B_mlbw_l2_inpaint_d25_c05_di[0, 0]_fs0.0_ipd0_ema99_LR:

[diverswan9]

Aether (@enoky ) has finished another inpaint model for us to play with (test).

"Here is the iw3 video inpaint Large model: https://mega.nz/file/VsVRjDJQ#KpI5ppL93OLPra97oT5zk9DLlEU4ZhN-XF6IzxiGe58
instructions are in the readme.txt file
The large model uses a fair bit more VRAM so I'm training a medium model as well" - enoky

[andy500]

does not show v3 model only the iw3 video inpaint Large model: how do l show both models

[diverswan9]

@andy500
Seems to be a little finicky perhaps... mine's working. Also just for kicks run update.bat.
try this...
inpaint_models.yml

[andy500]

Thank you working now

[Cazforshort]

Seems to be working very well.

I dont see a reason to inpaint both either. Picking right alone seems to have the same effect with this depth in my limited AB testing.

[diverswan9]

Another aether (enoky) model update... Medium. Supposedly works even more efficiently than his Large inpaint model.

"aether
—
<<uploaded-12.6.25>>
Here is the iw3 Medium inpaint model: https://mega.nz/file/490BQShI#X43L5IwFwse6tso0oGEMtyQzl-y8V9rbB22Iw72420A
install instructions are in the readme.txt"

"Update: here is the best video inpaint model yet for iw3 according to the benchmarks..." -enoky

[nagadomi]

Thank you for sharing the benchmark results.
In my local benchmark, Aether small performs reasonably well, but the large and medium models perform worse in comparison.
I suspected that the issue was caused by changes in the training code, but if the benchmark results for the large and medium models are fine on your side, it might be due to data bias (synthetic data?).
I recommend also benchmarking with evaluation data that contains only normal(real) movie data, which represents real-world use cases.
If you are using a modified training code only for the medium and large models, I recommend fine-tuning with the original code to confirm that the modifications have not caused any degradation.

Also, PSNR does not measure sharpness. Smoother and blurred results tend to achieve higher PSNR. It is more of a measure of fidelity to the original. Perceptual losses (LPIPS) and GAN are used to avoid high-PSNR models that produce blurry results.

[KolaKater]

Another aether (enoky) model update... Medium. Supposedly works even more efficiently than his Large inpaint model.

"aether — <<uploaded-12.6.25>> Here is the iw3 Medium inpaint model: https://mega.nz/file/490BQShI#X43L5IwFwse6tso0oGEMtyQzl-y8V9rbB22Iw72420A install instructions are in the readme.txt"

"Update: here is the best video inpaint model yet for iw3 according to the benchmarks..." -enoky

Could you please tell us the Depth Resolutions you used for the 3 differen models? for example Light model 512?

[diverswan9]

It seems to give better results as the depth-res increases from what I've seen.