Official Round: Completed

ImageCLEF 2022 Fusion - Media Interestingness

ImageCLEF

2011

Note: ImageCLEF Fusion includes 2 subtasks. This page is about the Media Interestingness subtask. For information about the Result Diversification subtask click here. Both challenges' datasets are shared together, so registering for one of these challenges will automatically give you access to the other one.

Note: Do not forget to read the Rules section on this page. Pressing the red Participate button leads you to a page where you have to agree with those rules. You will not be able to submit any results before agreeing with the rules.

Note: Before trying to submit results, read the Submission instructions section on this page.

Challenge description

While deep neural network methods have proven their predictive power in many tasks, there are still several domains where a single deep learning network is not enough for attaining high precision, e.g., prediction of subjective concepts such as violence, memorability, etc. Late fusion, also called ensembling or decision-level fusion, represents one of the approaches that researchers in machine learning employ to increase the performance of single-system approaches. It consists of using a series of weaker learner methods called inducers, whose prediction outputs are combined in the final step, via a fusion mechanism to create a new and improved super predictor. These systems have a long history and are shown to be particularly useful in scenarios where the performance of single-system approaches is not considered satisfactory [Ştefan2020, Constantin2021a, Constantin2022].

The ImageCLEFfusion 2022 task challenge participants to develop and benchmark late fusion schemes. The participants will receive a dataset of real inducers and are expected to provide a fusion mechanism that would allow to combine them into a super system yielding superior performance compared to the highest performing individual inducer system. The provided inducers were developed to solve two real tasks: (i) prediction of visual interestingness (for more information see [Constantin2021b]), and (ii) diversification of image search results (for more information see [Ionescu2020]).

This task would allow to explore various aspects of late fusion mechanisms, such as the performance of different fusion methods, the methods for selecting inducers from a larger set, the exploitation of positive and negative correlations between inducers, and so on.

Data

As soon as the data are released they will be available under the "Resources" tab.

The devset can be found in the "2022-ImageCLEFfusion-MediaInterestingness-devset".

ImageCLEFfusion-int. The data for this task is extracted and corresponds to the Interestingness10k dataset [Constantin2021b]. We will provide output data from 29 inducers, representing visual interestingness predictions for 2435 images, split into:

devset - representing the training data, composed of outputs from the 29 inducers for 1877 images
testset - representing the training data, composed of outputs from the 29 inducers for 558 images

This fusion task corresponds to a regression task, where participants are tasked with the problem of image interestingness prediciton. Participants to this task must use the inducer system outputs we provide in order to create better, stronger image interestingness prediction results. Results will be judged according to the MAP@10 metric.

FOLDER STRUCTURE

We will provide the following folder for the data splits:

inducers - folder containing outputs from the 29 inducers for all the images in the devset or testset in .txt format
scripts - folder containing useful scripts for calculating the metrics associated with this task
gt - folder containing ground truth values - only for devset
performance - the performance of the provided inducers given the ground truth data and the performance metrics - only for devset

For the testset only the inducers folder will be made available.

INDUCERS FOLDER

The inducers folder contains outputs from the 29 inducers for all the images in the devset or testset in .txt format. Filenames are created using the data split name (devset or testset) followed by the ID of the inducer (a number from 1 to 29). Each entry in this file contains the following fields, separated by commas:

video_id - represents the id of the video from which the image was extracted
image_id - represents the id of the image
classification - inducer classification output, where 1 represents interesting images and 0 represents non-interesting images
interestingness_score - represents the interestingness score prediction for the images in the dataset

An example is represented below:

video_78,164_131-197.jpg,1,0.500603
video_78,207_198-217.jpg,1,0.713791
...
video_78,2367_2358-2376.jpg,1,0.494209
video_79,148_99-198.jpg,1,0.332301
video_79,242_199-285.jpg,1,0.294029
...

SCRIPTS FOLDER

Contains the results_to_trec.py python script that transforms the result files from the format listed above, in the "inducers" folder into a format that is compatible with the metric calculation tool. The script can be used by calling the following command from the terminal, that generates and writes a new file in the same folder as the prediction file you use, giving it a .trec format:

python results_to_trec.py <prediction_file>

Also contains the trec_eval tool, used for calculating the performance of interestingness predicting methods. The script can be used by calling the following command from the terminal:

treceval -M10 <gt_file.qrels> <prediction_file>

The -M10 command instructs the script to compute the metric values with a 10 cutoff value. With this script you can see the performance of the inducers or of your own fusion method. While the script in itself displays many metrics at different cutoff values, we will only take into account the MAP@10 metric.

GT FOLDER

Contains the ground truth values for the images in the devset, all stored in one file, containing the following information:

video_id - represents the id of the video from which the image was extracted
iter - not used
image_id - represents the id of the image
classification - represents the classification of the image, where 1 represents an interesting image and 0 represents a non-interesting one

An example of such a file is presented below:

video_82 0 962_955-970.jpg 0
video_82 0 984_971-997.jpg 1
...
video_82 0 3076_3071-3081.jpg 1
video_83 0 31_11-52.jpg 0
video_83 0 68_53-84.jpg 0
...

PERFORMANCE FOLDER

The performance folder contains the performance metrics for all the 33 inducers, in files in txt format. While trec_eval tool outputs a large number of different metrics, we are only interested in the official metric for this competition, namely MAP@10, which you can find in the txt file in a line that displays map score like:

...
map            	all	0.0311
...

Submission instructions

As soon as the submission is open, you will find a “Create Submission” button on this page (next to the tabs).

Before being allowed to submit your results, you have to first press the red participate button, which leads you to a page where you have to accept the challenge's rules.

These are the instructions for your submissions; please pay close attention to all the details and let us know if you have any questions.

The submission file has the following columns, all separated by a space character (the file extension does not matter):

video_id - string value - represents the id of the video from which the image was extracted (example: video_100, video_101)
iter - integer value - not used value is 0
image_id - string value - represents the id of the image (example: 1016_1003-1030.jpg)
classification - integer value but only 0 or 1 - represents the classification of the image (0 - not interesting, 1 - interesting)
interestingness_score - floating value between 0 and 1 - represents the interestingness score for the image (example: 0.3411)
run_name - string value - represents a general name the participant given to the submission (example: submission_run1.txt)

A short example of this would be:

...
video_100 0 1016_1003-1030.jpg 1 0.317 submission.txt
video_100 0 101_67-135.jpg 1 0.3446 submission.txt
video_100 0 1038_1031-1045.jpg 0 0.2445 submission.txt
...

When submitting your files, the following rules must be respected:

all the 6 tokens previously mentioned (video_id, iter, image_id, ...) must be present for every entry in the submission file
please respect the limits and requested values previously indicated (iter must always be 0, classification must be 0 or 1, interestingness_score must be between 0 and 1)
video_id must only have the values associated with the testset (video_100, video_101, ... video 107), also image_id must also be the ones associated with the testset
do not leave out any video_id associated with the testset from your submission
keep the same string for run_name throughout a submission file
pairs of (video_id, image_id) combinations must be unique (e.g., no two entries for (video_100 7690_7498-7883.jpg))

Finally, when submitting your runs, please note that the scores (metrics) are not automatically calculated - therefore the system will display a score of 0.00. We will download the submissions manually after you submit them, calculate your scores, and update the website and communicate the results to you.

Rules

Note: In order to participate in this challenge you have to sign an End User Agreement (EUA). You will find more information on the 'Resources' tab.

ImageCLEF lab is part of the Conference and Labs of the Evaluation Forum: CLEF 2022. CLEF 2022 consists of independent peer-reviewed workshops on a broad range of challenges in the fields of multilingual and multimodal information access evaluation, and a set of benchmarking activities carried in various labs designed to test different aspects of mono and cross-language Information retrieval systems. More details about the conference can be found here.

Submitting a working note with the full description of the methods used in each run is mandatory. Any run that could not be reproduced thanks to its description in the working notes might be removed from the official publication of the results. Working notes are published within CEUR-WS proceedings, resulting in an assignment of an individual DOI (URN) and an indexing by many bibliography systems including DBLP. According to the CEUR-WS policies, a light review of the working notes will be conducted by ImageCLEF organizing committee to ensure quality. As an illustration, ImageCLEF 2021 working notes (task overviews and participant working notes) can be found within CLEF 2021 CEUR-WS proceedings.

Important

Participants of this challenge will automatically be registered at CLEF 2022. In order to be compliant with the CLEF registration requirements, please edit your profile by providing the following additional information:

First name
Last name
Affiliation
Address
City
Country
Regarding the username, please choose a name that represents your team.

This information will not be publicly visible and will be exclusively used to contact you and to send the registration data to CLEF, which is the main organizer of all CLEF labs

Participating as an individual (non affiliated) researcher

We welcome individual researchers, i.e. not affiliated to any institution, to participate. We kindly ask you to provide us with a motivation letter containing the following information:

the presentation of your most relevant research activities related to the task/tasks
your motivation for participating in the task/tasks and how you want to exploit the results
a list of the most relevant 5 publications (if applicable)
the link to your personal webpage

The motivation letter should be directly concatenated to the End User Agreement document or sent as a PDF file to bionescu at imag dot pub dot ro. The request will be analyzed by the ImageCLEF organizing committee. We reserve the right to refuse any applicants whose experience in the field is too narrow, and would therefore most likely prevent them from being able to finish the task/tasks.

Citations

[Ionescu2020] Ionescu, B., Rohm, M., Boteanu, B., Gînscă, A. L., Lupu, M., & Müller, H. (2020). Benchmarking Image Retrieval Diversification Techniques for Social Media. IEEE Transactions on Multimedia, 23, 677-691.

[Ştefan2020] Ştefan, L. D., Constantin, M. G., & Ionescu, B. (2020, June). System Fusion with Deep Ensembles. In Proceedings of the 2020 International Conference on Multimedia Retrieval (pp. 256-260).

[Constantin2021a] Constantin, M. G., Ştefan, L. D., & Ionescu, B. (2021, June). DeepFusion: Deep Ensembles for Domain Independent System Fusion. In the International Conference on Multimedia Modeling (pp. 240-252). Springer, Cham.

[Constantin2021b] Constantin, M. G., Ştefan, L. D., Ionescu, B., Duong, N. Q., Demarty, C. H., & Sjöberg, M. (2021). Visual Interestingness Prediction: A Benchmark Framework and Literature Review. International Journal of Computer Vision, 1-25.

[Constantin2022] Constantin, M. G., Ştefan, L. D., & Ionescu, B. (2022). Exploring Deep Fusion Ensembling for Automatic Visual Interestingness Prediction. In Human Perception of Visual Information (pp. 33-58). Springer, Cham.

Prizes

Publication

ImageCLEF 2022 is an evaluation campaign that is being organized as part of the CLEF initiative labs. The campaign offers several research tasks that welcome participation from teams around the world. The results of the campaign appear in the working notes proceedings, published by CEUR Workshop Proceedings (CEUR-WS.org). Selected contributions among the participants will be invited for publication in the following year in the Springer Lecture Notes in Computer Science (LNCS) together with the annual lab overviews.

Resources

Contact us

Discussion Forum

You can ask questions related to this challenge on the Discussion Forum. Before asking a new question please make sure that question has not been asked before.
Click on Discussion tab above or direct link: https://www.aicrowd.com/challenges/imageclef-2022-fusion-media-interestingness/discussion

Alternative channels

We strongly encourage you to use the public channels mentioned above for communications between the participants and the organizers. In extreme cases, if there are any queries or comments that you would like to make using a private communication channel, then you can send us an email at :

liviu_daniel [dot] stefan [at] upb [dot] ro
mihai [dot] constantin84 [at] upb [dot] ro
dogariu [dot] mihai8 [at] gmail [dot] com

More information

You can find additional information on the challenge here: https://www.imageclef.org/2022/fusion