StableSR - [IJCV2024] Exploiting Diffusion Prior for Real-Wor

项目说明

<p align="center">
<img src=" height=40>
</p>

Exploiting Diffusion Prior for Real-World Image Super-Resolution

Paper | Project Page | Video | WebUI | ModelScope | ComfyUI

<a href=" src=" alt="google colab logo"></a> ![Hugging Face]() ![Replicate]() ![OpenXLab]() !

Jianyi Wang, Zongsheng Yue, Shangchen Zhou, Kelvin C.K. Chan, Chen Change Loy

S-Lab, Nanyang Technological Uni

<img src="assets/网络.png" width="800px"/>

:star: If StableSR is helpful to your images or projects, please help star this repo. Thanks! :hugs:

Update

2024.06.28: Accepted by IJCV. See the latest Full paper.

2024.02.29: Support StableSR with SD-Turbo. Thank Andray for the finding!

Now the ComfyUI ![GitHub Stars]() of StableSR is also a. Thank gameltb and WSJUSA for the implementation!

2023.11.30: Code Update.

- Support DDIM and negati
- Add CFW training scripts
- Add FaceSR training and test scripts

2023.10.08: Our test sets associated with the results in our paper are now a at [HuggingFace] and [OpenXLab]. You may have an easy comparison with StableSR now.

2023.08.19: Integrated to :hugs: Hugging Face. Try out online demo! ![Hugging Face]().

2023.08.19: Integrated to :panda_face: OpenXLab. Try out online demo! ![OpenXLab]().

2023.07.31: Integrated to :rocket: Replicate. Try out online demo! ![Replicate]() Thank Chenxi for the implementation!

2023.07.16: You may reproduce the LDM baseline used in our paper using LDM-SRtuning ![GitHub Stars]().

2023.07.14: :whale: ModelScope for StableSR is released!

2023.06.30: :whale: New model trained on SD-2.1-768v is released! Better performance with fewer artifacts!

2023.06.28: Support training on SD-2.1-768v.

2023.05.22: :whale: Improve the 代码 to save more GPU memory, now 128 --> 512 needs 8.9G. Enable start from intermediate steps.

2023.05.20: :whale: The WebUI ![GitHub Stars]() of StableSR is a. Thank Li Yi for the implementation!

2023.05.13: Add Colab demo of StableSR. <a href=" src=" alt="google colab logo"></a>

2023.05.11: Repo is released.

TODO

[x] ~~Code release~~

[x] ~~Update link to paper and project page~~

[x] ~~Pretrained models~~

[x] ~~Colab demo~~

[x] ~~StableSR-768~~

[x] ~~Replicate demo~~

[x] ~~HuggingFace demo~~

[x] ~~StableSR-face released~~

[x] ~~ComfyUI support~~

Demo on real-world SR

<img src="assets/imgsli_1.jpg" height="223px"/> <img src="assets/imgsli_2.jpg" height="223px"/> <img src="assets/imgsli_3.jpg" height="223px"/>
<img src="assets/imgsli_8.jpg" height="223px"/> <img src="assets/imgsli_4.jpg" height="223px"/> <img src="assets/imgsli_5.jpg" height="223px"/>
<img src="assets/imgsli_9.jpg" height="214px"/> <img src="assets/imgsli_6.jpg" height="214px"/> <img src="assets/imgsli_7.jpg" height="214px"/> <img src="assets/imgsli_10.jpg" height="214px"/>

For more e, please refer to our paper for details.

Demo on 4K Results

StableSR is capable of achieving arbitrary upscaling in theory, below is an 4x example with a result beyond 4K (4096x6144).

<img src="assets/main-fig.png" width="800px"/>

More examples.

- 4K Demo1, which is a 4x SR on the image from here.
- 4K Demo2, which is a 8x SR on the image from here.
- More comparisons can be found here and here.

Dependencies and Installation

Pytorch == 1.12.1

CUDA == 11.7

pytorch-lightning==1.4.2

xformers == 0.0.16 (Optional)

Other required packages in en.yaml

Running Examples

#### Train
Download the pretrained Stable Diffusion models from [HuggingFace]

Train Time-aware encoder with SFT: set the ckpt_path in config files (Line 22 and Line 55)

Train CFW: set the ckpt_path in config files (Line 6).

You need to first generate training data using the finetuned diffusion model in the first stage.

The data folder should be like this:

Then you can train CFW:

#### Resume

#### Test directly

Download the Diffusion and autoencoder pretrained models from [HuggingFace | OpenXLab].
We use the same color correction scheme introduced in paper by default.
You may change for better color correction.
You may also disable color correction by

StableSR-Turbo: Get the ckpt first from [HuggingFace or OpenXLab]. Then you just need to modify and set to 4. See examples below:

DDIM is supported now. See here

Test on 128 --> 512: You need at least 10G GPU memory to run this script (batchsize 2 by default)

Test on arbitrary size w/o chop for autoencoder (for results beyond 512): The memory cost depends on your image size, but is usually above 10G.

Test on arbitrary size w/ chop for autoencoder: Current default setting needs at least 18G to run, you may reduce the autoencoder tile size by setting and .

Note the min tile size is 512 and the stride should be smaller than the tile size. A smaller size may introduce more border artifacts.

For test on 768 model, you need to set , and . You can also adjust , and accordingly. We did not finetune CFW.

#### Test FaceSR
You need to first generate reference images using [CodeFormer] or other blind face models.
Pretrained Models: [HuggingFace | OpenXLab].

#### Test using Replicate API

You may see here for more information.

Citation

If our work is useful for your research, please consider citing:

{wang2024exploiting,
author = {Wang, Jianyi and Yue, Zongsheng and Zhou, Shangchen and Chan, Kelvin C.K. and Loy, Chen Change},
title = {Exploiting Diffusion Prior for Real-World Image Super-Resolution},
article = {International Journal of Computer },
year = {2024}
}

License

This project is licensed under <a rel="license" href=" S-Lab License 1.0</a>. Redistribution and use should follow this license.

Acknowledgement

This project is based on stablediffusion, latent-diffusion, SPADE, mixture-of-diffusers and BasicSR. Thanks for their awesome work.

Contact

If you have any questions, please feel free to reach me out at .

项目信息

开发语言：Python

Star 数：2671

技术要点

该项目属于黑客技术方向的开源资源，代码结构清晰，适合安全研究人员学习参考。项目围绕[IJCV2024] Exploiting Diffusion Prior for Real-World Image Super-Resolution展开，提供了可运行的实现，对理解相关安全机制、工具原理与攻防技术具有较高的研究价值。

从技术实现角度看，此类项目的核心在于对底层原理的深入理解与工程化落地。开发者在研究过程中可以结合官方文档与社区讨论，梳理项目的初始化流程、核心模块职责与数据流转，逐步掌握其中的技术细节与设计思路。

使用与学习建议

1. 下载源码压缩包，解压到本地环境。

2. 根据项目类型准备对应的运行环境与依赖。

3. 按照 README 中的步骤完成配置与部署。

4. 在隔离的测试环境中运行验证，避免影响生产系统。

免责声明

本资源来源于开源社区，仅供安全研究、技术学习与合法授权范围内的攻防演练使用，请勿用于任何非法或侵害他人权益的用途。使用本项目产生的一切后果由使用者自行承担。