Unsupervised Anomaly Detection (UAD) methods aim to identify anomalies in test samples comparing them with a normative distribution learned from a dataset known to be anomaly-free. Approaches based on generative models offer interpretability by generating anomaly-free versions of test images, but are typically unable to identify subtle anomalies. Alternatively, approaches using feature modelling or self-supervised methods, such as the ones relying on synthetically generated anomalies, do not provide out-of-the-box interpretability. In this work, we present a novel method that combines the strengths of both strategies: a generative cold-diffusion pipeline (i.e., a diffusion-like pipeline which uses corruptions not based on noise) that is trained with the objective of turning synthetically-corrupted images back to their normal, original appearance. To support our pipeline we introduce a novel synthetic anomaly generation procedure, called DAG, and a novel anomaly score which ensembles restorations conditioned with different degrees of abnormality. Our method surpasses the prior state-of-the art for unsupervised anomaly detection in three different Brain MRI datasets.
@article{marimont2024disyre,title={Ensembled Cold-Diffusion Restorations for Unsupervised Anomaly Detection},author={Naval Marimont, Sergio and Siomos, Vasilis and Baugh, Matthew and Tzelepis, Christos and Kainz, Bernhard and Tarroni, Giacomo},journal={arXiv preprint arXiv:2407.06635 (Accepted in MICCAI 2024)},volume={},number={},pages={},year={2024},url={},doi={}}
Are CLIP features all you need for Universal Synthetic Image Origin Attribution?
Dario Cioni, Christos Tzelepis, Lorenzo Seidenari, and
1 more author
The steady improvement of Diffusion Models for visual synthesis has given rise to many new and interesting use cases of synthetic images but also has raised concerns about their potential abuse, which poses significant societal threats. To address this, fake images need to be detected and attributed to their source model, and given the frequent release of new generators, realistic applications need to consider an Open-Set scenario where some models are unseen at training time. Existing forensic techniques are either limited to Closed-Set settings or to GAN-generated images, relying on fragile frequency-based “fingerprint” features. By contrast, we propose a simple yet effective framework that incorporates features from large pre-trained foundation models to perform Open-Set origin attribution of synthetic images produced by various generative models, including Diffusion Models. We show that our method leads to remarkable attribution performance, even in the low-data regime, exceeding the performance of existing methods and generalizes better on images obtained from a diverse set of architectures. We make the code publicly available at: https://github.com/ciodar/UniversalAttribution
@article{cioni2024ecccvw,title={Are CLIP features all you need for Universal Synthetic Image Origin Attribution?},author={Cioni, Dario and Tzelepis, Christos and Seidenari, Lorenzo and Patras, Ioannis},journal={ECCVW / TWYN},volume={},number={},pages={},year={2024},url={},doi={}}
Bilinear Models of Parts and Appearances in Generative Adversarial Networks
James Oldfield, Christos Tzelepis, Yannis Panagakis, and
2 more authors
IEEE Transactions on Pattern Analysis and Machine Intelligence (T-PAMI), 2024
Recent advances in the understanding of Generative Adversarial Networks (GANs) have led to remarkable progress in visual editing and synthesis tasks, capitalizing on the rich semantics that are embedded in the latent spaces of pre-trained GANs. However, existing methods are often tailored to specific GAN architectures and are limited to either discovering global semantic directions that do not facilitate localized control, or require some form of supervision through manually provided regions or segmentation masks. In this light, we present an architecture-agnostic approach that jointly discovers factors representing spatial parts and their appearances in an entirely unsupervised fashion. These factors are obtained by applying a semi-nonnegative tensor factorization on the feature maps, which in turn enables context-aware local image editing with pixel-level control. In addition, we show that the discovered appearance factors correspond to saliency maps that localize concepts of interest, without using any labels. Experiments on a wide range of GAN architectures and datasets show that, in comparison to the state of the art, our method is far more efficient in terms of training time and, most importantly, provides much more accurate localized control. Code to reproduce our results and explore our model is available at: http://github.com/james-oldfield/PandA
@article{oldfield2023tpami,title={Bilinear Models of Parts and Appearances in Generative Adversarial Networks},author={Oldfield, James and Tzelepis, Christos and Panagakis, Yannis and Nicolaou, Mihalis A. and Patras, Ioannis},journal={{IEEE} Transactions on Pattern Analysis and Machine Intelligence (T-PAMI)},volume={},number={},pages={},year={2024},url={},doi={}}
DISYRE: Diffusion-Inspired SYnthetic REstoration for Unsupervised Anomaly Detection
Sergio Naval Marimont, Matthew Baugh, Vasilis Siomos, and
3 more authors
arXiv preprint arXiv:2311.15453 (Accepted in ISBI 2024), 2024
Unsupervised Anomaly Detection (UAD) techniques aim to identify and localize anomalies without relying on annotations, only leveraging a model trained on a dataset known to be free of anomalies. Diffusion models learn to modify inputs x to increase the probability of it belonging to a desired distribution, i.e., they model the score function ∇xlogp(x). Such a score function is potentially relevant for UAD, since ∇xlogp(x) is itself a pixel-wise anomaly score. However, diffusion models are trained to invert a corruption process based on Gaussian noise and the learned score function is unlikely to generalize to medical anomalies. This work addresses the problem of how to learn a score function relevant for UAD and proposes DISYRE: Diffusion-Inspired SYnthetic REstoration. We retain the diffusion-like pipeline but replace the Gaussian noise corruption with a gradual, synthetic anomaly corruption so the learned score function generalizes to medical, naturally occurring anomalies. We evaluate DISYRE on three common Brain MRI UAD benchmarks and substantially outperform other methods in two out of the three tasks.
@article{marimont2023disyre,title={{DISYRE}: Diffusion-Inspired SYnthetic REstoration for Unsupervised Anomaly Detection},author={Marimont, Sergio Naval and Baugh, Matthew and Siomos, Vasilis and Tzelepis, Christos and Kainz, Bernhard and Tarroni, Giacomo},journal={arXiv preprint arXiv:2311.15453 (Accepted in ISBI 2024)},volume={},number={},pages={},year={2024},url={},doi={}}
One-shot Neural Face Reenactment via Finding Directions in GAN’s Latent Space
Stella Bounareli, Christos Tzelepis, Vasileios Argyriou, and
2 more authors
International Journal of Computer Vision (IJCV), 2024
In this paper, we present our framework for neural face/head reenactment whose goal is to transfer the 3D head orientation and expression of a target face to a source face. Previous methods focus on learning embedding networks for identity and head pose/expression disentanglement which proves to be a rather hard task, degrading the quality of the generated images. We take a different approach, bypassing the training of such networks, by using (fine-tuned) pre-trained GANs which have been shown capable of producing high-quality facial images. Because GANs are characterized by weak controllability, the core of our approach is a method to discover which directions in latent GAN space are responsible for controlling head pose and expression variations. We present a simple pipeline to learn such directions with the aid of a 3D shape model which, by construction, inherently captures disentangled directions for head pose, identity, and expression. Moreover, we show that by embedding real images in the GAN latent space, our method can be successfully used for the reenactment of real-world faces. Our method features several favorable properties including using a single source image (one-shot) and enabling cross-person reenactment. Extensive qualitative and quantitative results show that our approach typically produces reenacted faces of notably higher quality than those produced by state-of-the-art methods for the standard benchmarks of VoxCeleb1 and 2.
@article{bounareli2023ijcv,title={One-shot Neural Face Reenactment via Finding Directions in {GAN}'s Latent Space},author={Bounareli, Stella and Tzelepis, Christos and Argyriou, Vasileios and Patras, Ioannis and Tzimiropoulos, Georgios},journal={International Journal of Computer Vision (IJCV)},volume={},number={},pages={},year={2024},url={},doi={}}
Improving Fairness using Vision-Language Driven Image Augmentation
Moreno D’Incà, Christos Tzelepis, Ioannis Patras, and
1 more author
In IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), 2024
Fairness is crucial when training a deep-learning discriminative model, especially in the facial domain. Models tend to correlate specific characteristics (such as age and skin color) with unrelated attributes (downstream tasks), resulting in biases which do not correspond to reality. It is common knowledge that these correlations are present in the data and are then transferred to the models during training. This paper proposes a method to mitigate these correlations to improve fairness. To do so, we learn interpretable and meaningful paths lying in the semantic space of a pre-trained diffusion model (DiffAE) – such paths being supervised by contrastive text dipoles. That is, we learn to edit protected characteristics (age and skin color). These paths are then applied to augment images to improve the fairness of a given dataset. We test the proposed method on CelebA-HQ and UTKFace on several downstream tasks with age and skin color as protected characteristics. As a proxy for fairness, we compute the difference in accuracy with respect to the protected characteristics. Quantitative results show how the augmented images help the model improve the overall accuracy, the aforementioned metric, and the disparity of equal opportunity.
@inproceedings{dinca2024vlbc,title={Improving Fairness using Vision-Language Driven Image Augmentation},author={D'Inc\`a, Moreno and Tzelepis, Christos and Patras, Ioannis and Sebe, Nicu},booktitle={IEEE/CVF Winter Conference on Applications of Computer Vision (WACV)},year={2024}}
2023
Parts of Speech-Grounded Subspaces in Vision-Language Models
James Oldfield, Christos Tzelepis, Yannis Panagakis, and
2 more authors
In Advances in Neural Information Processing Systems (NeurIPS), 2023
Latent image representations arising from vision-language models have proved immensely useful for a variety of downstream tasks. However, their utility is limited by their entanglement with respect to different visual attributes. For instance, recent work has shown that CLIP image representations are often biased toward specific visual properties (such as objects or actions) in an unpredictable manner. In this paper, we propose to separate representations of the different visual modalities in CLIP’s joint vision-language space by leveraging the association between parts of speech and specific visual modes of variation (e.g. nouns relate to objects, adjectives describe appearance). This is achieved by formulating an appropriate component analysis model that learns subspaces capturing variability corresponding to a specific part of speech, while jointly minimising variability to the rest. Such a subspace yields disentangled representations of the different visual properties of an image or text in closed form while respecting the underlying geometry of the manifold on which the representations lie. What’s more, we show the proposed model additionally facilitates learning subspaces corresponding to specific visual appearances (e.g. artists’ painting styles), which enables the selective removal of entire visual themes from CLIP-based text-to-image synthesis. We validate the model both qualitatively, by visualising the subspace projections with a text-to-image model and by preventing the imitation of artists’ styles, and quantitatively, through class invariance metrics and improvements to baseline zero-shot classification.
@inproceedings{oldfield2023part,title={Parts of Speech-Grounded Subspaces in Vision-Language Models},author={Oldfield, James and Tzelepis, Christos and Panagakis, Yannis and Nicolaou, Mihalis A. and Patras, Ioannis},booktitle={Advances in Neural Information Processing Systems (NeurIPS)},year={2023}}
HyperReenact: One-Shot Reenactment via Jointly Learning to Refine and Retarget Faces
Stella Bounareli, Christos Tzelepis, Vasileios Argyriou, and
2 more authors
2023 IEEE/CVF International Conference on Computer Vision (ICCV), 2023
In this paper, we present our method for neural face reenactment, called HyperReenact, that aims to generate realistic talking head images of a source identity, driven by a target facial pose. Existing state-of-the-art face reenactment methods train controllable generative models that learn to synthesize realistic facial images, yet producing reenacted faces that are prone to significant visual artifacts, especially under the challenging condition of extreme head pose changes, or requiring expensive few-shot fine-tuning to better preserve the source identity characteristics. We propose to address these limitations by leveraging the photorealistic generation ability and the disentangled properties of a pretrained StyleGAN2 generator, by first inverting the real images into its latent space and then using a hypernetwork to perform: (i) refinement of the source identity characteristics and (ii) facial pose re-targeting, eliminating this way the dependence on external editing methods that typically produce artifacts. Our method operates under the one-shot setting (i.e., using a single source frame) and allows for cross-subject reenactment, without requiring any subject-specific fine-tuning. We compare our method both quantitatively and qualitatively against several state-of-the-art techniques on the standard benchmarks of VoxCeleb1 and VoxCeleb2, demonstrating the superiority of our approach in producing artifact-free images, exhibiting remarkable robustness even under extreme head pose changes.
@article{bounareli2023iccv,title={{HyperReenact}: One-Shot Reenactment via Jointly Learning to Refine and Retarget Faces},author={Bounareli, Stella and Tzelepis, Christos and Argyriou, Vasileios and Patras, Ioannis and Tzimiropoulos, Georgios},journal={2023 IEEE/CVF International Conference on Computer Vision (ICCV)},volume={},number={},pages={},year={2023},url={},doi={}}
Attribute-preserving Face Dataset Anonymization via Latent Code Optimization
Simone Barattin*, Christos Tzelepis*, Ioannis Patras, and
1 more author
In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) [* denotes co-first authorship], 2023
This work addresses the problem of anonymizing the identity of faces in a dataset of images, such that the privacy of those depicted is not violated, while at the same time the dataset is useful for downstream task such as for training machine learning models. To the best of our knowledge, we are the first to explicitly address this issue and deal with two major drawbacks of the existing state-of-the-art approaches, namely that they (i) require the costly training of additional, purpose-trained neural networks, and/or (ii) fail to retain the facial attributes of the original images in the anonymized counterparts, the preservation of which is of paramount importance for their use in downstream tasks. We accordingly present a task-agnostic anonymization procedure that directly optimizes the images’ latent representation in the latent space of a pre-trained GAN. By optimizing the latent codes directly, we ensure both that the identity is of a desired distance away from the original (with an identity obfuscation loss), whilst preserving the facial attributes (using a novel feature-matching loss in FaRL’s deep feature space). We demonstrate through a series of both qualitative and quantitative experiments that our method is capable of anonymizing the identity of the images whilst – crucially – better-preserving the facial attributes. We make the code and the pre-trained models publicly available.
@inproceedings{tzelepis2023falco,title={Attribute-preserving Face Dataset Anonymization via Latent Code Optimization},author={Barattin*, Simone and Tzelepis*, Christos and Patras, Ioannis and Sebe, Nicu},booktitle={IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) [* denotes co-first authorship]},year={2023}}
Self-Supervised Video Similarity Learning
Giorgos Kordopatis-Zilos, Giorgos Tolias, Christos Tzelepis, and
3 more authors
In IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshop (CVPRW), 2023
We introduce S^2VS, a video similarity learning approach with self-supervision. Self-Supervised Learning (SSL) is typically used to train deep models on a proxy task so as to have strong transferability on target tasks after fine-tuning. Here, in contrast to prior work, SSL is used to perform video similarity learning and address multiple retrieval and detection tasks at once with no use of labeled data. This is achieved by learning via instance-discrimination with task-tailored augmentations and the widely used InfoNCE loss together with an additional loss operating jointly on self-similarity and hard-negative similarity. We benchmark our method on tasks where video relevance is defined with varying granularity, ranging from video copies to videos depicting the same incident or event. We learn a single universal model that achieves state-of-the-art performance on all tasks, surpassing previously proposed methods that use labeled data.
@inproceedings{Kordopatis2023s2vs,title={Self-Supervised Video Similarity Learning},author={Kordopatis{-}Zilos, Giorgos and Tolias, Giorgos and Tzelepis, Christos and Kompatsiaris, Ioannis and Patras, Ioannis and Papadopoulos, Symeon},booktitle={IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshop (CVPRW)},volume={},year={2023}}
PandA: Unsupervised Learning of Parts and Appearances in the Feature Maps of GANs
James Oldfield, Christos Tzelepis, Yannis Panagakis, and
2 more authors
In The Eleventh International Conference on Learning Representations , 2023
Recent advances in the understanding of Generative Adversarial Networks (GANs) have led to remarkable progress in visual editing and synthesis tasks, capitalizing on the rich semantics that are embedded in the latent spaces of pre-trained GANs. However, existing methods are often tailored to specific GAN architectures and are limited to either discovering global semantic directions that do not facilitate localized control, or require some form of supervision through manually provided regions or segmentation masks. In this light, we present an architecture-agnostic approach that jointly discovers factors representing spatial parts and their appearances in an entirely unsupervised fashion. These factors are obtained by applying a semi-nonnegative tensor factorization on the feature maps, which in turn enables context-aware local image editing with pixel-level control. In addition, we show that the discovered appearance factors correspond to saliency maps that localize concepts of interest, without using any labels. Experiments on a wide range of GAN architectures and datasets show that, in comparison to the state of the art, our method is far more efficient in terms of training time and, most importantly, provides much more accurate localized control. Our code is available at: https://github.com/james-oldfield/PandA.
@inproceedings{oldfield2023panda,title={{PandA}: Unsupervised Learning of Parts and Appearances in the Feature Maps of {GAN}s},author={Oldfield, James and Tzelepis, Christos and Panagakis, Yannis and Nicolaou, Mihalis and Patras, Ioannis},booktitle={The Eleventh International Conference on Learning Representations },year={2023},url={https://openreview.net/forum?id=iUdSB2kK9GY}}
StyleMask: Disentangling the Style Space of StyleGAN2 for Neural Face Reenactment
Stella Bounareli, Christos Tzelepis, Vasileios Argyriou, and
2 more authors
In 17th IEEE International Conference on Automatic Face and Gesture Recognition, FG 2023, Waikoloa Beach, HI, USA, January 5-8, 2023, 2023
In this paper we address the problem of neural face reenactment, where, given a pair of a source and a target facial image, we need to transfer the target’s pose (defined as the head pose and its facial expressions) to the source image, by preserving at the same time the source’s identity characteristics (e.g., facial shape, hair style, etc), even in the challenging case where the source and the target faces belong to different identities. In doing so, we address some of the limitations of the state-of-the-art works, namely, a) that they depend on paired training data (i.e., source and target faces have the same identity), b) that they rely on labeled data during inference, and c) that they do not preserve identity in large head pose changes. More specifically, we propose a framework that, using unpaired randomly generated facial images, learns to disentangle the identity characteristics of the face from its pose by incorporating the recently introduced style space S of StyleGAN2, a latent representation space that exhibits remarkable disentanglement properties. By capitalizing on this, we learn to successfully mix a pair of source and target style codes using supervision from a 3D model. The resulting latent code, that is subsequently used for reenactment, consists of latent units corresponding to the facial pose of the target only and of units corresponding to the identity of the source only, leading to notable improvement in the reenactment performance compared to recent state-of-the-art methods. In comparison to state of the art, we quantitatively and qualitatively show that the proposed method produces higher quality results even on extreme pose variations. Finally, we report results on real images by first embedding them on the latent space of the pretrained generator. We make the code and pretrained models publicly available
@inproceedings{bounareli2023stylemask,title={StyleMask: Disentangling the Style Space of StyleGAN2 for Neural Face Reenactment},author={Bounareli, Stella and Tzelepis, Christos and Argyriou, Vasileios and Patras, Ioannis and Tzimiropoulos, Georgios},booktitle={17th {IEEE} International Conference on Automatic Face and Gesture Recognition, {FG} 2023, Waikoloa Beach, HI, USA, January 5-8, 2023},pages={1--8},publisher={{IEEE}},year={2023},doi={10.1109/FG57933.2023.10042744}}
"Just To See You Smile": SMILEY, a Voice-Guided GUY GAN
Qi Yang, Christos Tzelepis, Sergey Nikolenko, and
2 more authors
In Proceedings of the Sixteenth ACM International Conference on Web Search and Data Mining, WSDM 2023, Singapore, 27 February 2023 - 3 March 2023, 2023
In this technical demonstration, we present SMILEY, a voice-guided virtual assistant. The system utilizes a deep neural architecture ContraCLIP to manipulate facial attributes using voice instructions, allowing for deeper speaker engagement and smoother customer experience when being used in the "virtual concierge" scenario. We validate the effectiveness of SMILEY and ContraCLIP via a successful real-world case study in Singapore and a large-scale quantitative evaluation.
@inproceedings{yang2023smiley,title={"Just To See You Smile": {SMILEY}, a Voice-Guided {GUY} {GAN}},author={Yang, Qi and Tzelepis, Christos and Nikolenko, Sergey and Patras, Ioannis and Farseev, Aleksandr},booktitle={Proceedings of the Sixteenth {ACM} International Conference on Web Search and Data Mining, {WSDM} 2023, Singapore, 27 February 2023 - 3 March 2023},editor={Chua, Tat{-}Seng and Lauw, Hady W. and Si, Luo and Terzi, Evimaria and Tsaparas, Panayiotis},pages={1196--1199},publisher={{ACM}},year={2023},doi={10.1145/3539597.3573031}}
2022
DnS: Distill-and-Select for Efficient and Accurate Video Indexing and Retrieval
Giorgos Kordopatis-Zilos, Christos Tzelepis, Symeon Papadopoulos, and
2 more authors
International Journal of Computer Vision (IJCV), 2022
In this paper, we address the problem of high performance and computationally efficient content-based video retrieval in large-scale datasets. Current methods typically propose either: (i) fine-grained approaches employing spatio-temporal representations and similarity calculations, achieving high performance at a high computational cost or (ii) coarse-grained approaches representing/indexing videos as global vectors, where the spatio-temporal structure is lost, providing low performance but also having low computational cost. In this work, we propose a Knowledge Distillation framework, called Distill-and-Select (DnS), that starting from a well-performing fine-grained Teacher Network learns: a) Student Networks at different retrieval performance and computational efficiency trade-offs and b) a Selector Network that at test time rapidly directs samples to the appropriate student to maintain both high retrieval performance and high computational efficiency. We train several students with different architectures and arrive at different trade-offs of performance and efficiency, i.e., speed and storage requirements, including fine-grained students that store/index videos using binary representations. Importantly, the proposed scheme allows Knowledge Distillation in large, unlabelled datasets – this leads to good students. We evaluate DnS on five public datasets on three different video retrieval tasks and demonstrate a) that our students achieve state-of-the-art performance in several cases and b) that the DnS framework provides an excellent trade-off between retrieval performance, computational speed, and storage space. In specific configurations, the proposed method achieves similar mAP with the teacher but is 20 times faster and requires 240 times less storage space. The collected dataset and implementation are publicly available at https://github.com/mever-team/distill-and-select.
@article{kordopatis2022dns,title={{DnS}: Distill-and-Select for Efficient and Accurate Video Indexing and Retrieval},author={Kordopatis{-}Zilos, Giorgos and Tzelepis, Christos and Papadopoulos, Symeon and Kompatsiaris, Ioannis and Patras, Ioannis},journal={International Journal of Computer Vision (IJCV)},volume={130},number={10},pages={2385--2407},year={2022},url={https://doi.org/10.1007/s11263-022-01651-3},doi={10.1007/s11263-022-01651-3}}
ContraCLIP: Interpretable GAN generation driven by pairs of contrasting sentences
Christos Tzelepis, James Oldfield, Georgios Tzimiropoulos, and
1 more author
This work addresses the problem of discovering non-linear interpretable paths in the latent space of pre-trained GANs in a model-agnostic manner. In the proposed method, the discovery is driven by a set of pairs of natural language sentences with contrasting semantics, named \textitsemantic dipoles, that serve as the “limits” of the interpretation that we require by the trainable latent paths to encode. By using the pre-trained CLIP encoder, the sentences are projected into the vision-language space, where they serve as dipoles, and where RBF-based warping functions define a set of non-linear directional paths, one for each semantic dipole, allowing in this way traversals from one semantic pole to the other. By defining an objective that discovers paths in the latent space of GANs that generate changes along the desired paths in the vision-language embedding space, we provide an intuitive way of controlling the underlying generative factors and address some of the limitations of the state-of-the-art works, namely, that a) they are typically tailored to specific GAN architectures (i.e., StyleGAN), b) they disregard the relative position of the manipulated and the original image in the image embedding and the relative position of the image and the text embeddings, and c) they lead to abrupt image manipulations and quickly arrive at regions of low density and, thus, low image quality, providing limited control of the generative factors. We provide extensive qualitative and quantitative results that demonstrate our claims with two pre-trained GANs, and make the code and the pre-trained models publicly available at: https://github.com/chi0tzp/ContraCLIP.
@article{tzelepis2022contraclip,title={{ContraCLIP}: Interpretable {GAN} generation driven by pairs of contrasting sentences},author={Tzelepis, Christos and Oldfield, James and Tzimiropoulos, Georgios and Patras, Ioannis},journal={CoRR},volume={abs/2206.02104},year={2022},url={https://doi.org/10.48550/arXiv.2206.02104},doi={10.48550/arXiv.2206.02104},eprinttype={arXiv},eprint={2206.02104}}
2021
Estimating continuous affect with label uncertainty
Niki Maria Foteinopoulou, Christos Tzelepis, and Ioannis Patras
In 9th International Conference on Affective Computing and Intelligent Interaction, ACII 2021, Nara, Japan, September 28 - Oct. 1, 2021, 2021
Continuous affect estimation is a problem where there is an inherent uncertainty and subjectivity in the labels that accompany data samples – typically, datasets use the average of multiple annotations or self-reporting to obtain ground truth labels. In this work, we propose a method for uncertainty-aware continuous affect estimation, that models explicitly the uncertainty of the ground truth label as a uni-variate Gaussian with mean equal to the ground truth label, and unknown variance. For each sample, the proposed neural network, estimates not only the value of the target label (valence and arousal in our case), but also the variance. The network is trained with a loss that is defined as the KL-divergence between the estimation (valence/arousal) and the Gaussian around the ground truth. We show, that in two affect recognition problems, with real data, the estimated variances are correlated with measures of uncertainty/error in the labels that are extracted either by considering multiple annotations of the data, or by manually cleaning the dataset.
@inproceedings{foteinopoulou2022acii,title={Estimating continuous affect with label uncertainty},author={Foteinopoulou, Niki Maria and Tzelepis, Christos and Patras, Ioannis},booktitle={9th International Conference on Affective Computing and Intelligent Interaction, {ACII} 2021, Nara, Japan, September 28 - Oct. 1, 2021},pages={1--8},publisher={{IEEE}},year={2021},url={https://doi.org/10.1109/ACII52823.2021.9597425},doi={10.1109/ACII52823.2021.9597425}}
Uncertainty Propagation in Convolutional Neural Networks: Technical Report
In this technical report we study the problem of propagation of uncertainty (in terms of variances of given uni-variate normal random variables) through typical building blocks of a Convolutional Neural Network (CNN). These include layers that perform linear operations, such as 2D convolutions, fully-connected, and average pooling layers, as well as layers that act non-linearly on their input, such as the Rectified Linear Unit (ReLU). Finally, we discuss the sigmoid function, for which we give approximations of its first- and second-order moments, as well as the binary cross-entropy loss function, for which we approximate its expected value under normal random inputs. A PyTorch implementation of the presented “uncertainty-aware” layers is available under the MIT license here: https://github.com/chi0tzp/UncPropCNN.
@article{tzelepis2021uncprop,title={Uncertainty Propagation in Convolutional Neural Networks: Technical Report},author={Tzelepis, Christos and Patras, Ioannis},journal={CoRR},volume={abs/2102.06064},year={2021},eprinttype={arXiv},eprint={2102.06064}}
WarpedGANSpace: Finding non-linear RBF paths in GAN latent space
Christos Tzelepis, Georgios Tzimiropoulos, and I. Patras
2021 IEEE/CVF International Conference on Computer Vision (ICCV), 2021
This work addresses the problem of discovering, in an unsupervised manner, interpretable paths in the latent space of pretrained GANs, so as to provide an intuitive and easy way of controlling the underlying generative factors. In doing so, it addresses some of the limitations of the state-of-the-art works, namely, a) that they discover directions that are independent of the latent code, i.e., paths that are linear, and b) that their evaluation relies either on visual inspection or on laborious human labeling. More specifically, we propose to learn non-linear warpings on the latent space, each one parametrized by a set of RBF-based latent space warping functions, and where each warping gives rise to a family of non-linear paths via the gradient of the function. Building on the work of Voynov and Babenko, that discovers linear paths, we optimize the trainable parameters of the set of RBFs, so as that images that are generated by codes along different paths, are easily distinguishable by a discriminator network. This leads to easily distinguishable image transformations, such as pose and facial expressions in facial images. We show that linear paths can be derived as a special case of our method, and show experimentally that non-linear paths in the latent space lead to steeper, more disentangled and interpretable changes in the image space than in state-of-the art methods, both qualitatively and quantitatively. We make the code and the pretrained models publicly available at https://github.com/chi0tzp/WarpedGANSpace.
@article{tzelepis2021warpedganspace,title={{WarpedGANSpace}: Finding non-linear {RBF} paths in {GAN} latent space},author={Tzelepis, Christos and Tzimiropoulos, Georgios and Patras, I.},journal={2021 IEEE/CVF International Conference on Computer Vision (ICCV)},year={2021},pages={6373-6382}}
2019
A deep generic to specific recognition model for group membership analysis using non-verbal cues
Wenxuan Mou, Christos Tzelepis, Vasileios Mezaris, and
2 more authors
@article{mou2019group,title={A deep generic to specific recognition model for group membership analysis using non-verbal cues},author={Mou, Wenxuan and Tzelepis, Christos and Mezaris, Vasileios and Gunes, Hatice and Patras, Ioannis},journal={Image and Vision Computing},volume={81},pages={42--50},year={2019},url={https://doi.org/10.1016/j.imavis.2018.09.005},doi={10.1016/j.imavis.2018.09.005}}
2018
Linear Maximum Margin Classifier for Learning from Uncertain Data
Christos Tzelepis, Vasileios Mezaris, and Ioannis Patras
IEEE Transactions on Pattern Analysis and Machine Intelligence (T-PAMI), 2018
In this paper, we propose a maximum margin classifier that deals with uncertainty in data input. More specifically, we reformulate the SVM framework such that each training example can be modeled by a multi-dimensional Gaussian distribution described by its mean vector and its covariance matrix-the latter modeling the uncertainty. We address the classification problem and define a cost function that is the expected value of the classical SVM cost when data samples are drawn from the multi-dimensional Gaussian distributions that form the set of the training examples. Our formulation approximates the classical SVM formulation when the training examples are isotropic Gaussians with variance tending to zero. We arrive at a convex optimization problem, which we solve efficiently in the primal form using a stochastic gradient descent approach. The resulting classifier, which we name SVM with Gaussian Sample Uncertainty (SVM-GSU), is tested on synthetic data and five publicly available and popular datasets; namely, the MNIST, WDBC, DEAP, TV News Channel Commercial Detection, and TRECVID MED datasets. Experimental results verify the effectiveness of the proposed method.
@article{tzelepis2018svmgsu,title={Linear Maximum Margin Classifier for Learning from Uncertain Data},author={Tzelepis, Christos and Mezaris, Vasileios and Patras, Ioannis},journal={{IEEE} Transactions on Pattern Analysis and Machine Intelligence (T-PAMI)},volume={40},number={12},pages={2948--2962},year={2018},url={https://doi.org/10.1109/TPAMI.2017.2772235},doi={10.1109/TPAMI.2017.2772235}}
2017
Generic to Specific Recognition Models for Membership Analysis in Group Videos
Wenxuan Mou, Christos Tzelepis, Vasileios Mezaris, and
2 more authors
In 12th IEEE International Conference on Automatic Face & Gesture Recognition, FG 2017, Washington, DC, USA, May 30 - June 3, 2017, 2017
Automatic understanding and analysis of groups has attracted increasing attention in the vision and multimedia communities in recent years. However, little attention has been paid to the automatic analysis of group membership, i.e., recognizing which group the individual in question is part of. This paper presents a novel two-phase Support Vector Machine (SVM) based specific recognition model that is learned using an optimized generic recognition model. We conduct a set of experiments using a database collected to study group analysis from multimodal cues while each group (i.e., four participants together) were watching a number of long movie segments. Our experimental results show that the proposed specific recognition model (52%) outperforms the generic recognition model trained across all different videos (35%) and the independent recognition model trained directly on each specific video (33%) using linear SVM.
@inproceedings{mou2017group,title={Generic to Specific Recognition Models for Membership Analysis in Group Videos},author={Mou, Wenxuan and Tzelepis, Christos and Mezaris, Vasileios and Gunes, Hatice and Patras, Ioannis},booktitle={12th {IEEE} International Conference on Automatic Face {\&} Gesture Recognition, {FG} 2017, Washington, DC, USA, May 30 - June 3, 2017},pages={512--517},publisher={{IEEE} Computer Society},year={2017},url={https://doi.org/10.1109/FG.2017.69},doi={10.1109/FG.2017.69}}
