# How to Segment and Monitor Skin Anomalies Over Time Using Computer Vision

> Techniques for segmenting, mapping, and tracking changes in skin anomalies by computer vision CNNs, U-Nets, and transformers using OpenCV and PyTorch

- Canonical URL: https://www.funvisiontutorials.com/blog/segment-monitor-skin-anomalies-time-computer-vision/
- Author: Vladimir Kucera
- Published: 2025-04-01
- Topics: Deep learning, Face detection & landmarks, Computer vision, Object detection

I'll explore techniques for segmenting, mapping, and tracking changes in skin anomalies (e.g., moles, spots, vessels) using dermatological images captured with everyday devices (e.g., mobile phones). I'll focus on classical computer vision methods (e.g., morphology, filtering) as well as state-of-the-art deep learning techniques – including CNNs, U-Nets, and transformers. I’ll include supervised, unsupervised, and non-learning methods, with practical Python implementations using OpenCV and PyTorch, as well as research papers and open-source tools suitable for long-term tracking of skin changes, including under low contrast conditions.

[Image: Title graphic reading Skin Lesion Segmentation Methods in Computer Vision, beside a phone photographing a mole with the segmented region highlighted in magenta]

## Classical Non-Learning Methods (Thresholding, Edges, and Morphology)

These algorithmic approaches process image data without the need for training. A basic example is **thresholding** pixel brightness or color. **Otsu’s method** provides unsupervised segmentation based on image histograms ([PubMed](https://pubmed.ncbi.nlm.nih.gov/35204435)). Thresholding typically isolates darker lesions from lighter surrounding skin. Under poor contrast, adaptive thresholds and histogram equalization improve performance.

**Morphological operations** (dilation, erosion, opening/closing) help eliminate noise and artifacts, smoothen edges, and refine segmentation masks. These techniques improve lesion masks post-thresholding ([PubMed](https://pubmed.ncbi.nlm.nih.gov/35204435)).

To remove **hairs**, which appear as thin lines, edge detection filters or clustering-based techniques are used. For example, hair inpainting based on max filters and **DBSCAN** clustering was demonstrated in [this Stack Overflow example](https://stackoverflow.com/questions/71679345/image-segmentation-of-skin-lesion).

Other classical methods include **edge detection** (Canny, Sobel) and **active contours** (snakes, level-sets). Active contours iteratively adjust curves to match lesion boundaries and are especially effective when initialized well ([Elsevier](https://www.sciencedirect.com/science/article/pii/S1389041718309859)).

More advanced techniques combine methods, such as **watershed transforms** with morphological filtering and fuzzy clustering. Rout et al. (2021) used a **multiscale morphological filter**, watershed, and **Fuzzy C-Means** (FCM) to separate lesions from healthy skin ([MDPI](https://www.mdpi.com/2073-8994/13/11/2085)).

### 🔧 Open-Source Tools:

-   **OpenCV**: thresholding, edge detection, morphology, watershed, inpainting.

-   **scikit-image**: `threshold_otsu`, active contours, watershed.

-   **scikit-learn**: clustering methods like DBSCAN.

## Unsupervised Segmentation Methods (Clustering, Generative Models)

Unsupervised methods detect patterns **without labeled data**. One example is **color clustering** using **k-means** or **Fuzzy C-Means** to classify pixels into lesion vs. background groups. This is effective where color contrast is present ([MDPI](https://www.mdpi.com/2073-8994/13/11/2085)).

More advanced: **Generative Adversarial Networks (GANs)**. In 2023, Innani et al. introduced **Efficient-GAN (EGAN)**, which segments lesions without labeled masks ([PubMed](https://pubmed.ncbi.nlm.nih.gov/37596306)). EGAN uses adversarial and morphological losses, achieving Dice ~90%, Jaccard ~83.6% on the ISIC dataset. **Mobile-GAN**, a lightweight version, is suitable for smartphones.

### 🛠 Open-Source Tools:

-   **scikit-learn / scikit-image**: clustering and fuzzy segmentation.

-   **PyTorch**: build custom GANs.

-   **SAAL (Self-supervised Active Learning)**: [GitHub](https://github.com/jacobzhaoziyuan/SAAL) offers self-supervised training and active learning.

## Deep Convolutional Networks (CNNs)

**Supervised CNNs** can learn detailed segmentation from labeled datasets. Early CNNs were used for lesion classification; modern ones support **semantic segmentation**.

### 🔍 Key Architectures:

-   **Mask R-CNN**: combines detection and segmentation; used in melanoma diagnosis ([PMC](https://pmc.ncbi.nlm.nih.gov/articles/PMC10825805)).

-   **U-Net**: a fully convolutional encoder-decoder network, originally from biomedical imaging ([MDPI](https://www.mdpi.com/1424-8220/20/6/1601)).

Monica et al. (2023) showed Mask R-CNN effectively highlights lesion contours ([PMC](https://pmc.ncbi.nlm.nih.gov/articles/PMC10825805)).

### 📚 Open-Source Resources:

-   **Detectron2**, **MMDetection**: Mask R-CNN frameworks.

-   **segmentation\_models.pytorch**: U-Net, FPN, DeepLab, PSPNet.

-   [skin-lesion-segmentation projects](https://github.com/topics/skin-lesion-segmentation): GitHub repos with ISIC challenge models.

-   [Unet-Pytorch](https://github.com/JoshuaSXA/Unet-Pytorch): Example training code.

-   [LB-UNet](https://github.com/xuxuxuxuxuxjh/LB-UNet): Lightweight model for low-resource settings.

## U-Net and Variants

U-Net remains a top choice for lesion segmentation, especially with enhancements:

-   **ResUNet**: adds ResNet encoders ([MDPI](https://www.mdpi.com/1424-8220/20/6/1601)).

-   **Attention U-Net**, Dual U-Net: improve precision under low contrast.

Trained U-Nets adapt well even to smartphone macro images. Color normalization, hair removal, and filtering are helpful pre-processing steps.

### 🛠 Tools:

-   [nnU-Net](https://github.com/MIC-DKFZ/nnUNet): auto-configuration pipeline.

-   [MONAI](https://monai.io/): medical PyTorch toolkit.

-   [Segmentation Models (Keras)](https://github.com/qubvel/segmentation_models): for TensorFlow users.

## Transformer-Based Approaches

Transformers (e.g., **ViT**) model global context but lack fine-grained edge localization. Hybrid models (CNN + Transformer) perform best.

### Notable Models:

-   **TC-Net**: Combines ViT and CNN features. Improved Dice by ~2.5% over Swin-UNet ([PLOS One](https://journals.plos.org/plosone/article?id=10.1371/journal.pone.0277578)).

-   **TAFM-Net (2024)**: Combines EfficientNetV2 + Transformer ([arXiv](https://arxiv.org/html/2411.17556v1)).

Other architectures: **TransUNet**, **DuaSkinSeg**, **SPA-Former** – often available on GitHub.

### 🛠 Tools:

-   **timm** PyTorch library: Vision Transformers.

-   **HuggingFace Transformers**: ViT support.

-   Custom U-Net + ViT hybrids using PyTorch.

## Monitoring Lesion Changes Over Time

To track lesion changes across images:

1.  **Image Registration**: Use OpenCV’s `findHomography` or `estimateAffinePartial2D` to align images.

2.  **Change Detection**: XOR masks, compare Dice coefficients, measure area or longest diameter changes.

**3D mapping** (e.g., [Revisiting Lesion Tracking in 3D](https://arxiv.org/html/2412.07132v2)) offers advanced body tracking but is less practical for home use.

### Mobile Tools:

-   **[MoleMapper](https://www.ohsu.edu/war-on-melanoma/molemappertm-mole-tracking-app)**: maps moles by body zone, tracks over time.

-   **[Mel GitHub](https://github.com/aevri/mel)**: catalog, rotate, and compare mole photos.

For change detection, you can also train a CNN that compares image pairs (similar to satellite change detection).

## Summary

Combining classical, deep, and unsupervised segmentation with tracking enables comprehensive skin analysis from mobile photos. Open-source tools like OpenCV, PyTorch, and U-Net implementations allow for robust lesion mapping and long-term monitoring. Whether for research or personal use, these methods empower early detection and dermatological insights.

**References**:

-   [Otsu thresholding and preprocessing](https://pubmed.ncbi.nlm.nih.gov/35204435)

-   [Hair removal example – Stack Overflow](https://stackoverflow.com/questions/71679345/image-segmentation-of-skin-lesion)

-   [Watershed + Fuzzy C-Means – MDPI](https://www.mdpi.com/2073-8994/13/11/2085)

-   [EGAN – PubMed](https://pubmed.ncbi.nlm.nih.gov/37596306)

-   [Mask R-CNN in dermatology – PMC](https://pmc.ncbi.nlm.nih.gov/articles/PMC10825805)

-   [U-Net review – MDPI](https://www.mdpi.com/1424-8220/20/6/1601)

-   [U-Net vs Mask R-CNN – ResearchGate](https://www.researchgate.net/publication/342115218_A_Brief_Analysis_of_U-Net_and_Mask_R-CNN_for_Skin_Lesion_Segmentation)

-   [Transformer-CNN hybrid – PLOS One](https://journals.plos.org/plosone/article?id=10.1371/journal.pone.0277578)