Deep learning-based Point Cloud Representation

Funder

Organizational Unit

Publications

Report on PCC Core Experiment 1.1: Performance of the Verification Model under Consideration

Publication . Guarda, André F. R.; Rodrigues, Nuno M. M.; Pereira, Fernando

This document reports the performance results of the first version of the JPEG Pleno Point Cloud Coding Verification Model under Consideration, following the Call for Proposals on JPEG Pleno Point Cloud Coding issued in January 2022 [1].

2022-10Report

Restricted access

IT/IST/IPLeiria Response to the Call for Proposals on JPEG Pleno Point Cloud Coding

Publication . Guarda, André F. R.; Rodrigues, Nuno M. M.; Ruivo, Manuel; Coelho, Luís; Seleem, Abdelrahman; Pereira, Fernando

This document describes a deep learning (DL)-based point cloud (PC) geometry codec and a DL-based PC joint geometry and colour codec, submitted to the Call for Proposals on JPEG Pleno Point Cloud Coding issued in January 2022 [1]. These proposals have been originated by research developed at Instituto de Telecomunicações (IT), in the context of the project Deep-PCR entitled “Deep learning-based Point Cloud Representation” (PTDC/EEI-COM/1125/2021), financed by Fundação para a Ciência e Tecnologia (FCT).

2022-10Report

Restricted access

Deep Learning-Based Point Cloud Coding and Super-Resolution: a Joint Geometry and Color Approach

Publication . Guarda, André F. R.; Ruivo, Manuel; Coelho, Luís; Seleem, Abdelrahman; M. M. Rodrigues, Nuno; Pereira, Fernando

In this golden age of multimedia, realistic content is in high demand with users seeking more immersive and interactive experiences. As a result, new image modalities for 3D representations have emerged in recent years, among which point clouds have deserved especial attention. Naturally, with this increase in demand, efficient storage and transmission became a must, with standardization groups such as MPEG and JPEG entering the scene, as it happened before with other types of visual media. In a surprising development, JPEG issued a Call for Proposals on point cloud coding targeting exclusively learningbased solutions, in parallel to a similar call for image coding. This is a natural consequence of the growing popularity of deep learning, which due to its excellent performances is currently dominant in the multimedia processing field, including coding. This paper presents the coding solution selected by JPEG as the best-performing response to the Call for Proposals and adopted as the first version of the JPEG Pleno Point Cloud Coding Verification Model, in practice the first step for developing a standard. The proposed solution offers a novel joint geometry and color approach for point cloud coding, in which a single deep learning model processes both geometry and color simultaneously. To maximize the RD performance for a large range of point clouds, the proposed solution uses down-sampling and learningbased super-resolution as pre- and post-processing steps. Compared to the MPEG point cloud coding standards, the proposed coding solution comfortably outperforms G-PCC, for both geometry, color, and joint quality metrics.

2023-11Journal article

Restricted access

Adaptive bridge model for compressed domain point cloud classification

Publication . Seleem, Abdelrahman; Guarda, André F. R.; Rodrigues, Nuno M. M.; Pereira, Fernando

The recent adoption of deep learning-based models for the processing and coding of multimedia signals has brought noticeable gains in performance, which have established deep learning-based solutions as the uncontested state-of-the-art both for computer vision tasks, targeting machine consumption, as well as, more recently, coding applications, targeting human visualization. Traditionally, applications requiring both coding and computer vision processing require frst decoding the bitstream and then applying the computer vision methods to the decompressed multimedia signals. However, the adoption of deep learning-based solutions enables the use of compressed domain computer vision processing, with gains in performance and computational complexity over the decompressed domain approach. For point clouds (PCs), these gains have been demonstrated in the single available compressed domain computer vision processing solution, named Compressed Domain PC Classifer, which processes JPEG Pleno PC coding (PCC) compressed streams using a PC classifer largely compatible with the state-of-the-art spatial domain PointGrid classifer. However, the available Compressed Domain PC Classifer presents strong limitations by imposing a single, specifc input size which is associated to specifc JPEG Pleno PCC confgurations; this limits the compression performance as these confgurations are not ideal for all PCs due to their diferent characteristics, notably density. To overcome these limitations, this paper proposes the frst Adaptive Compressed Domain PC Classifer solution which includes a novel adaptive bridge model that allows to process the JPEG Pleno PCC encoded bit streams using diferent coding confgurations, now maximizing the compression efciency. Experimental results show that the novel Adaptive Compressed Domain PC Classifer allows JPEG PCC to achieve better compression performance by not imposing a single, specifc coding confguration for all PCs, regardless of its diferent characteristics. Moreover, the added adaptability power can achieve slightly better PC classifcation performance than the previous Compressed Domain PC Classifer and largely better PC classifcation performance (and lower number of weights) than the PointGrid PC classifer working in the decompressed domain.

2024-06-08Journal article

Open access

IT/IST/IPLeiria Report on JPEG PCC Core Experiment 4.1: Attention Models

Publication . Ghafari, Mohammadreza; Guarda, André; Rodrigues, Nuno; Pereira, Fernando

Context and Objective: In the JPEG Pleno PC dataset, there are some PCs (e.g., sparse PCs) which are more ‘difficult’ to code and may benefit from improvements in the JPEG PCC VM DL coding model.

2023-07Other

Restricted access