[Severian] Tenki no Ko (Weathering with You)(Best Extensive HC Sub Cleanup 23.976fps)[1080p]

hardcoded subs have been blended as best as possible with the surrounding pixels

I mean, based on those images you provided, it's not blending well at all. It appears to look worse than the other release that attempted this, just covering less of the video this time. Thanks for the attempt though!

@Astral, you input is appreciated. But please download it and take a look. The blended area is much smaller compared to the almost smear/smudge on the other versions and since its taking the average RGB values of the surrounding pixels the HC subs are almost non-noticeable in many scenes. This being said, the blending works much better for darker scenes (which Tenki no Ko is mostly comprised of), however the brighter scenes are as manageable as the other versions are. Perhaps a neural network to identify the HC letters and some machine learning to estimate the background images behind those letters would really be the proper solution. If anyone is willing to attempt this, Severian is more than willing to assist.

If anyone is willing to attempt this, Severian is more than willing to assist.

I'd be interested to work on this, although probably with a conventional video inpainting algorithm instead of something deep learning based. (But we can try deep learning too maybe)

I think I can handle the inpainting if you can supply a B&W mask of where text is. (Which I think your current approach does really well) Did you say you were planning on uploading your code to GitHub? Let me know once you have and I'd love to see if we can improve it.

Can you use better image hosting?

Wont the bluray be available in another 4 weeks or so?

yea, should be out may 27th

@PromareWeekOne this video inpainting algorithm may work really nicely with the python script if the two could be integrated somehow. Hence why machine learning was mentioned as it is accessible through python as well. However that should not deter the endeavour. Here is a link to the python script with a somewhat compiled guide on how to use it https://github.com/Severian-desu-ga/Hard-Coded-Subs-Film-Blender. Please feel free to edit it to your preferences.

@Godim apologies, the hosting site used seemed not to work. Here are links to two screenshots (https://ibb.co/CPXNg0m, https://ibb.co/HCFnLw9). These are the same as the previous video's uploads as very little has changed in those two scenes specifically.

@KaoriScarlet1997 @Ctkirito is correct, however there are some of us that would rather watch it now... considering how many are in quarantine and a month stuck at home waiting is too much. However for the others who do stick it out, the wait will totally be worth it when it does finally arrive!

Thanks for the upload Severian! If you end up making another version though, can you crop off the black bars? The aspect ratio of the video is incorrect (squished a bit) and if the black bars are hardcoded in it makes it more difficult to stretch the video to 16:9 when playing it. I know it's not your fault (source video is wrong ratio), but it makes it really hard to watch with everyone so unnaturally skinny lol.

Total idiot here, is this supposed to have English subs because I don't see any, tried a couple of video players and they don't exist??

@alepolo101 it's an absolute pleasure. Yes you are right, the black bars do seem to deform the characters' figures. Have you tried to play around with any other aspect ratios on your media player? That's not really the right solution, but cropping the movie and resizing it on this end can't be guaranteed. However there are no definites. Perhaps this is a movie to watch twice when the BD release comes out...

@iwolf this is strange. VLC, Totem, and MPV have had no issues here. Not to state the obvious, but sometimes the player does not automatically detect the subtitles. That shouldn't really be an issue though since almost all players should have a manual setting to enable it. If that is not the case then this is concerning!

@Severian Never mind after doing the same thing on VLC for the 10th time, enabling the sub track wasn't working either, it just randomly started working lol thanks though

@Severian do you think AI/machine learning/etc from nvidia will help with this kind of cleanup? from the technical streams i watched they seem to be working a lot on this stuff and the results are pretty amazing

@opol, it absolutely can! Here is a good example of an AI that can perform multiple cleaning up tasks https://www.youtube.com/watch?v=qkHK1QdQ2Fk As you will see it can more than just repair images/videos with watermarks eg: HC subs, but it can also repair areas that have been corrupted by means of inpainting, denoising, and super-resolution techniques. The only issue is the development and implementation time. Even using an existing A.I, say the one above, becomes extremely computationally expensive as to train the neural network first (eg: feed it a bunch of data to learn from) and then implement it is time consuming. Most researchers have access to HPCs (high performance clusters... pretty much a parallel network of many heavy duty CPUs) to achieve these outcomes. Even so, that should not deter anyone! The code for the above A.I is available on GitHub https://github.com/yossigandelsman/DoubleDIP. Perhaps it may be worthwhile!

Can you attach some screenshots?

@Severian i link you the stuff from nvidia i remember
https://news.developer.nvidia.com/new-ai-imaging-technique-reconstructs-photos-with-realistic-results/
https://github.com/NVlabs/noise2noise
https://github.com/yu4u/noise2noise

so i guess you need an rtx card to work well with this stuff?

Thank you but there is too much artifacting which is making it hard to watch because it artifacts over scenes without subs. And the subs are a mess due to lyrics being mashed in with general dialogue.

@Severian It's totally fixable by stretching, it's just a bit harder to do with the black bars hardcoded in. I just meant if you crop out the black bars and leave it at the resolution it is now (I believe this would make it 1846x1080) then you can just change aspect ratios in VLC or MPV which would stretch it automatically with just the press of one button. That way whoever watches can choose easier to watch it at its native res or at it's intended ratio. The original webrip was already widescreen as well, which leads me to believe the video was meant to be stretched automatically by the original websites video player.

@psycrax sorry, image hosting issues. Screenshot links have been added to the torrent description

@opol Those are some interesting software, especially the third link! It can't ever hurt to have a better GPU, definitely adds a significant boost to any form of image processing. Though the nvadia software seems tailor made for their hardware, so yeah an rtx card is probably a good fit.

@ AoiOdin2113 it's a pleasure. You are absolutely correct there are artefacts that appear due to blending, this was unavoidable. Even still, would like to say this is still a better version available. The subs were unedited from Hina (@LnlyHikikomori), thinking now, perhaps some editing may have been better...

@alepolo101 yes you are right. That would be a decent solution to fixing it. However, the current download rate between this torrent and the previous upload which is not as good as this one have been stuck at 50% each. Meaning, people are still downloading the torrents equally. To upload a new fixed/even better version seems quite futile considering there will probably still be more people downloading the first two uploads.

Even still, everyone's input has been much appreciated and a lot more will be taken into consideration when/if the next project arises. Until then, happy watching and thank you for your patronage, Severian.

@alepolo101 you can crop and stretch the video with mkvmerge without reencoding.

I've been experimenting with different ways to remove the subtitles. I've come up with a better algorithm for masking the text, which I'm pretty happy with now. It results in less artifacts outside of the text region. Now I'm trying to find a good way to inpaint the text area. I tried using this approach which seemed promising, but after spending a few hours trying to get it to work I have discovered that the results on anime are somewhat inconsistent. Also it would take over 100 hours to process the entire movie, even on my GTX 1080Ti. Based on the fact that Deep-Flow-Guided-Video-Inpainting is designed specifically for video inpainting, and that it appears to be a best-in-class approach, I don't think it's worth pursuing any other ML approaches, at least not ones that have been pre-trained on live-action images.

I have a couple more ideas to try. Maybe DeepCreamPy can help, but this isn't really what it was designed to do, and the results probably won't be stable enough for video. Next thing I'm going to try is a conventional inpainting algorithm, since deep-learning solutions are incredibly dependant on what kind of media they were trained on, and I don't have the time or skill to train a model for anime.

@PromareWeekOne 100 hours does seem a tad bit long even with the decent GPU you have. DeepCreamPy looks interesting! Your masking and inpainting of the HC subs looks good. Perhaps the parameters need to be fiddled around with some more to get a better fix. However, here is another option that worked albeit better than expected for it's versatility to time regards. OpenCV2 also has an inpainting method, 2 actually... but both outcomes are relatively the same in animated scenes. Here is an example this torrents fix https://ibb.co/7zzYMzf, and here is one with the inpainting https://ibb.co/dDppnVJ. Check it out.

Also this has prompted a new upload of a Tenki no Ko fix that will include this inpainting, removal of all artefacts non-HC-sub related using a double pass filter. And removal of all the music lyrics to ease the viewer with jumbled text scenes. Cropping the black edges still was not done, sorry it the one fix that was neglected.