Cloud2Sketch: Augmenting Clouds with Imaginary Sketches

Abstract

Have you ever looked up at the sky and imagined what the clouds look like? This work studies the task of augmenting clouds in the sky with imagined sketches. Different from generic image-to-sketch translation, cloud augmentation requires recognizing ambiguous cloud shapes, retrieving plausible sketches, and aligning those sketches to irregular cloud contours.

Cloud2Sketch is a self-supervised pipeline that first extracts cloud contours, retrieves geometrically similar sketches, and then uses a sketch translation model with built-in free-form deformation to align sketches with clouds. The project also introduces SketchyZoo, an icon-based sketch collection used for training.

Method Review

Downloads

These files are mirrored here so the dataset remains accessible even if the original Notion page changes. The slides file is also mirrored locally.

Video

The project video is hosted on YouTube: Cloud2Sketch video.

Citation

@inproceedings{10.1145/3503161.3547810,
  author = {Wan, Zhaoyi and Xu, Dejia and Wang, Zhangyang and Wang, Jian and Luo, Jiebo},
  title = {Cloud2Sketch: Augmenting Clouds with Imaginary Sketches},
  year = {2022},
  isbn = {9781450392037},
  publisher = {Association for Computing Machinery},
  address = {New York, NY, USA},
  url = {https://doi.org/10.1145/3503161.3547810},
  doi = {10.1145/3503161.3547810},
  booktitle = {Proceedings of the 30th ACM International Conference on Multimedia},
  pages = {2441--2451},
  numpages = {11},
  keywords = {cloud augmentation, sketch synthesis, shape alignment, image-to-sketch generation},
  location = {Lisboa, Portugal},
  series = {MM '22}
}