control-edgedrawing-cv480edpf-fastdup-fp16-checkpoint-45000

README.md

@@ -8,18 +8,20 @@ tags:
 - controlnet
 ---
 
-Controls image generation by edge maps generated with [Edge Drawing](https://github.com/CihanTopal/ED_Lib). Edge Drawing comes in different flavors: original (_ed_), parameter free (_edpf_), color (_edcolor_).
+Controls image generation by edge maps generated with [Edge Drawing](https://github.com/CihanTopal/ED_Lib). Note that Edge Drawing comes in different flavors: original (_ed_), parameter free (_edpf_), color (_edcolor_).
 
 * Based on my monologs at [github.com - Edge Drawing](https://github.com/lllyasviel/ControlNet/discussions/318)
 * For usage see the model page on [civitai.com - Model](https://civitai.com/models/149740).
-* To generate edpf maps you can use the [space](https://huggingface.co/spaces/GeroldMeisinger/edpf) or script from [gitlab.com - edpf.py](https://gitlab.com/-/snippets/3601881).
-* For evaluation see the corresponding .zip with images in "files".
-* To run your own evaluations you can use the script [gitlab.com - inference.py](https://gitlab.com/-/snippets/3602096).
+* To generate edpf maps you can use [this space](https://huggingface.co/spaces/GeroldMeisinger/edpf) or [this script gitlab.com](https://gitlab.com/-/snippets/3601881).
+* For evaluation see the corresponding .zip with images at the files.
+* To run your own evaluations you can use [this script at gitlab.com](https://gitlab.com/-/snippets/3602096).
 
 **Edge Drawing Parameter Free**
 
 ![image/png](https://cdn-uploads.huggingface.co/production/uploads/64c0ec65a2ec8cb2f589233a/jmdCGeMJx4dKFGo44cuEq.png)
 
+_Clear and pristine! Wooow!_
+
 **Example**
 
 sampler=UniPC steps=20 cfg=7.5 seed=0 batch=9 model: v1-5-pruned-emaonly.safetensors cherry-picked: 1/9
@@ -28,8 +30,6 @@ prompt: _a detailed high-quality professional photo of swedish woman standing in
 
 ![image/png](https://cdn-uploads.huggingface.co/production/uploads/64c0ec65a2ec8cb2f589233a/2PSWsmzLdHeVG-i67S7jF.png)
 
-_Clear and pristine! Wooow!_
-
 **Canndy Edge for comparison (default in Automatic1111)**
 
 ![image/png](https://cdn-uploads.huggingface.co/production/uploads/64c0ec65a2ec8cb2f589233a/JZTpa-HZfw0NUYnxZ52Iu.png)
@@ -66,10 +66,10 @@ accelerate launch train_controlnet.py ^
 To evaluate the model it makes sense to compare it with the original Canny model. Original evaluations and comparisons are available at [ControlNet 1.0 repo](https://github.com/lllyasviel/ControlNet), [ControlNet 1.1 repo](https://github.com/lllyasviel/ControlNet-v1-1-nightly), [ControlNet paper v1](https://arxiv.org/abs/2302.05543v1), [ControlNet paper v2](https://arxiv.org/abs/2302.05543) and [Diffusers implementation](https://huggingface.co/takuma104/controlnet_dev/tree/main). Some points we have to keep in mind when comparing canny with edpf in order not to compare apples with oranges:
 * canny 1.0 model was trained on 3M images with fp32, canny 1.1 model on even more, while edpf model so far is only trained on a 180k-360k with fp16.
 * canny edge-detector requires parameter tuning while edpf is parameter free.
-* Do we manually fine-tune canny to find the perfect input image or do we leave it at default? We could argue that "no fine-tuning required" is the usp of edpf and we want to compare in the default setting, whereas canny fine-tuning is subjective.
-* Would the canny model actually benefit from a edpf pre-processor and we might not even require a edpf model? (2023-09-25: see `eval_canny_edpf.zip` but it seems as if it doesn't work and the edpf model may be justified)
+* Should we manually fine-tune canny to find the perfect input image or do we leave it at default? We could argue that "no fine-tuning required" is the usp of edpf and we want to compare in the default setting, whereas canny fine-tuning is subjective.
+* Would the canny model actually benefit from a edpf pre-processor and we might not even require a specialized edpf model? (2023-09-25: see `eval_canny_edpf.zip` but it seems as if it doesn't work and the edpf model may be justified)
 * When evaluating human images we need to be aware of Stable Diffusion's inherent limits, like disformed faces and hands, and don't attribute them to the control net.
-* When evaluating style we need to be aware of the bias from the image dataset (`laion2b-en-aesthetics65`), which might tend to generate "aesthetic" images, and not actually work "intrisically better".
+* When evaluating style we need to be aware of the bias from the image dataset (`laion2b-en-aesthetics65`), which might tend to generating "aesthetic" images, and not actually work "intrisicly better".
 
 # Versions
 
@@ -129,16 +129,50 @@ see experiment 3.0. restarted from 0 with `--proportion_empty_prompts=0.5` => re
 
 **Experiment 4.1 - 2023-09-26 - control-edgedrawing-cv480edpf-drop50-fp16-checkpoint-90000**
 
-resumed from 45000 steps with left-right flipped images => results are still not good, 50% is probably too much for 45k steps. guessmode still doesn't work and tends to produces humans. abort.
+resumed from 45000 steps with left-right flipped images until 90000 steps => results are still not good, 50% is probably also too much for 90k steps. guessmode still doesn't work and tends to produces humans. aborting.
+
+** Experiment 5.0 - 2023-09-28 - control-edgedrawing-cv480edpf-fastdup-fp16-checkpoint-45000 **
+
+see experiment 3. cleaned original images following the [fastdup introduction](https://nbviewer.org/github/visual-layer/fastdup/blob/main/examples/cleaning-image-dataset.ipynb) resulting in:
+```
+180210 images in total
+ 67854 duplicates
+   644 outliers
+    26 too dark
+   321 too bright
+    57 blurry
+ 68621 unique removed (that's 38%!)
+```
+
+restarted from 0 with left-right flipped images and `--mixed-precision="no"` to create a master release and converted to fp16 afterwards.
+
+** Experiment 6.0 - control-edgedrawing-cv480edpf-rect-fp16-checkpoint-XXXXX **
+
+see experiment 5.0.
+* included images with aspect ratio > 2
+* resized images with shortside to 512 which gives us rectangular images instead of 512x512 squares
+* center-cropped images to 512x(n)*64 (to make them SD compatible) and max longside 1024
+* sorted duplicates by `similarity` value from `laion2b-en-aesthetics65` to get the best `text` of all duplicates
+
+```
+183410 images in total
+ 75686 duplicates
+   381 outliers
+    50 too dark
+   436 too bright
+    31 blurry
+ 76288 unique removed (that's 42%!)
+```
+
+restarted from 0 and `--mixed-precision="fp16"`.
 
 # Ideas
 
-* fine-tune from canny
-* cleanup image dataset (l65)
-* uncropped mod64 images
+* make conceptual captions for laion
 * integrate edcolor
-* bigger image dataset (gcc)
-* cleanup image dataset (gcc)
+* try to fine-tune from canny
+* image dataset with better captions (cc3m)
+* remove images by semantic (use only photos, paintings etc. for edge detection)
 * re-train with fp32
 
 # Question and answers

control-edgedrawing-cv480edpf-fastdup-fp16-checkpoint-45000.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:ea1d8bf1f7e7b5dbb501aeb3c294cf60120c6b56575c16173f43ffacc68f4a8d
+size 722598616