To submit to our competition, please use the Submit link. We ask that each team only sign up with one account, as this team name will be used for tracking. We also ask that there be only one team per organization (lab or corporation). If you believe there is a need to have more than two, please reach out to the organizers, either directly or through the forums.
Each task of the competition will take place in two phases: validation and test. For each phase datasets will be provided and people must submit a decoder, a set of encoded images and a selected docker container in the Submit link on the navigation.
The submissions run in a selected docker container by either running your executable or by first unzipping your zip file and running ./decode. At this point, your decoder should be able to decode the images or videos, including unzipping them, into a local subfolder (we recommended images/ or videos/ for easier debugging).
It is important that you submit a good working decoder in this phase or you will not be eligible in the test phase.
Also, you must have a paper ID and a paper submitted to DCC before the test phase begins.
Once the test phase begins, you must upload an exact decoder binary or zip file as you did in the validation phase. If they are not hashed to the same value, your decoder will be rejected and not ran.
We will also release the test data. You must encode this test data locally and be able to submit a previous decoder and file the represents the entirety of the compressed dataset.
To be prize eligible, you must submit to all the bit-rates of a track (all three image rate points and both of the video rate points). You are not limited to using the same decoder for all the rate points, but that decoder has to have been submitted in the validation phase.
The goal is to compress a corpus of images to three bitrates:
- 0.3 bpp*
- 0.15 bpp*
- 0.075 bpp*
The winners will be chosen based on a human rating task. The raters compare the submitted images to each other, using the uncompressed image as a reference, and chose the preferred submission.
For guidance, several objective metrics will be shown on the leaderboard but not taken into account for prizes.
The validation set is derived from the CLIC 2022 test set. We will have a similar amount of images and resolutions for the test set.
- 30 images, collected from Unsplash.
- Provided as
- Images resized such that the longer side is 2048.
- Download here (130 MB)
- Download test set here (102 MB)
The images are released using the Unsplash license. Please contact the organizers if you have objections to any of the released pictures.
- The decoder size should not exceed 4GB.
The goal is to compress a corpus of videos to an average bitrate of
- 0.05 mbps*
- 0.5 mbps*
The winners will be chosen based on a human rating task. The raters compare the compressed videos to the uncompressed videos and chose the preferred codec.
For guidance, several objective metrics will be shown on the leaderboard but not considered for prizes.
We provide a validation set that reflects the (hidden) test data. Validation set statistics:
- 292 videos, split into 3 sets: 4 near lossless, 40 webcam, 248 general.
- Provided as
- Videos are either 720p or 1080p.
- Duration is exactly 10 seconds for all.
- The FPS varies from 15fps to 60fps, with the bulk of the videos in the 25-30 fps range.
- Download the full validation set here (2.2 GB).
- During the validation phase, we evaluate on the 30 video IDs published here.
- Download the 30 video subset needed for the validation phase here (367 MB).
- Use the devkit for converting to PNG if needed.
- Download test set here (300 MB)
- Note: Make sure to use the latest version of the CLIC devkit if you use it for converting MP4 to PNG.
Please contact the organiziers if you have objections to any of the released videos.
- The decoder size should not exceed 4GB.
- IMPORTANT: Submissions only need to compress 30 videos of the validation set, see above.
- We provide the videos as mp4, but we will expect submissions to output three PNGs per frame: one for Y, U, and V channels. Please use the devkit for converting to PNGs, if you want to train on PNGs. Or look at the Example Submission below.
Example submissionThe devkit contains an example submission based on H264. It showcases the format expected by the server.
Image Perceptual Model
In the image perceptual model track, you need to design a model to evaluate the submissions of the image compression task, for instance by proposing a metric. Given a pair of images (one being the original and the other being a distorted image), your metric can generate a score. Methods A and B can be compared by generating two scores d(O, A) and d(O, B), where O is the original image. If d(O, A) < d(O, B), the metric prefers method A over method B.
In our image compression evaluation, human raters are presented with three images (O, A, B) and asked to pick one of A or B. The winner of the perceptual model challenge is chosen based on how often each submission matches these binary decisions of the raters.
The development kit is provided as an example of what is expected from participants. It is in no way intended to contain data representative of the final challenge simply because that is not possible. The final test set will be created from the files uploaded by the participants in the compression challenge, and as a result it is impossible for us to provide data which will match that distribution in the validation set.
To get the devkit, use:
git clone https://github.com/spacl-ua/CLIC-devkit.git
- Note: The validation dataset is the same as last year's.
- Download here (2.1 GB)
- Test set will be provided during test phase.
The full details of the tasks are contained in the README file of the repository.
The 2021 test data may be used to train new models for this year's competition. Please note that the link above contains pointers to the raw data (unfiltered) which we collected from raters. Therefore, it is possible that certain pairs of images might have been rated both ways (i.e., one rater might have disagreed with the other). It is up to the participants to do any data cleanup that might be necessary.
Video Perceptual Model *NEW*
In our new video perceptual model track you will need to design a model to predict MOS scores on the video compression task. Given a video, your method will need to generate a predicted mean opinion score (MOS) in the interval [1,5], as is generated by Microsoft's P.910 implementation on CLIC videos. The evaluation of your model will be the MOS over the entire dataset. Similar to the Image Perceptual Model track, teams will upload a CSV containing with each line containing two fields
<video_name>,<mos score> where the video name will have the full subpath, for example
The metric used to rank the winners is Kendall’s tau-b. The confidence interval of the P.910 results will be used to account for potential ties. Pearson’s Correlation Coefficient (PCC) and Spearman’s Rank Correlation Coefficient (SRCC) will also be computed for reference.
- Training data is from CLIC 2022's compression participants and contain results from the 1.0 mbps track at the time (folders ending with _10) and 0.1 mbps track (folders ending with _01).
- Reconstructions from CLIC 2022 video compression task (7.3 GB)
- Originals for reference (these will not be provided for the test set) (353 MB)
- The test set will be provided during test phase and will be results generated from CLIC 2024. Please note that the video compression task has changed bitrates this year, so developing a robust perceptual metric is important.
* bpp: bits per pixel. One pixel consists of three color components.
* mbps: megabits per second. 0.05 mbps = 50 000 bits per second.