Commons:Wiki Science Competition/Evaluation

The worflow of an international event with non-standardized upload is quite intense.

This is a list all the general guidelines for WSC evaluation process originally drafted by User:alexmar983 for the 2017 edition and than enlarged.

See also Commons:Wiki Science Competition/File management fur further information.

Key steps

• refine categories and remove unsuitable files
• assign jurors to a task, link to the a preliminary look at the category.
• Prepare the "lists of set of images".
• select method of evaluation.
• get prefinalists (es: 2nd step on Montage)
• arrange the prefinalists by categories.
• double check the "non photographic media" category.
• double check the "set of images" category.
• brainstorming for the finalists (and the national winners)
• publish the national finalists (and winners) with the right layout.
• write down all the statistics.
• last double check of the finalists per countries, few removals and few additions.
• inform the first-level international jury.
• prepare the final additional trans-national categories for evaluation.
• international winners and runner-ups.

Merging local categories

The first step for organizers and academic committee is to decide which country has a separate sub-jury, and which categories are simply merged to the global "rest of the world" one.

Since the evaluation of the quality is up to the independent jurors, we could only agree on a quantitative approach, that is the number of files. Of course this is not simply the brutal final number of files. in facts, there are two types of files that have no chance to win:

• "obviously unsuitable", that is files that are blurry, confused or basically "out-of-scope". In other words, crap that noone will ever reuse. We will put those files in specific subcategories for statistical purpose and to be eliminated probably, but for categories with less than 20-30 upload you cans imply count them manually
• "disqualifed", that is files with watermark or small size.

Now, in theory, all national categories should be merged if they have not enough files once both "obviously unsuitable" and "disqualifed" files are removed. This is clearly practical but the spirit of WSC is to fill a gap between the academic communities and the wikimedia platforms. We therefore decided consistently to remove only countries that had at least 10-12 non "obviously unsuitable" files but to include disqualified files in such sum.

According to this rule, that is not considering the disqualified files, some country selection can exist... of course the file below 2 megapixels or with watermarks will be discarded as possible finalists, but their role is recognized as part of the effort by the uploaders to put their country on the map, which is a good thing per se, especially when it will time to prepare the categories in the next editions.

In addition to that, it is worth spending few minutes to evaluate a solid scientific file if someone makes the effort to share it with free license, no matter if in a big or small categories, because it can be useful. In the end, we want our jurors to also improve categorization and reuse, and that includes files of low resolutions or requiring cropping.

So, in the interest of wikimedia platforms, even if some of them a will probably fail to get a finalist for the next level, the jurors will evaluate the images from small countries in the same way they evaluate also the images that will be disqualified in other countries with bigger upload.

Countries removed after the original local deadline

Some countries can be removed before the deadline because of zero or minimal upload, while other after the deadline for practical purpose. These removal should if possible be listed by year as a form of transparency and for statistical purpose.

• Commons:Wiki Science Competition 2017/Evaluation
• Commons:Wiki Science Competition 2019/Evaluation

Jurors

Second-level jurors should be associated to the countries based mostly on their language and geographical background, and on the type of uploaded files. If the majority of pictures for examples are related to flora and fauna, a biologist will be preferred to a physicist, for example.

Methods

Countries with fewer uploads will go through a simple manual screening. In these cases, there will be basically two or three jurors, with also a key coordinato that can act, based on his/her "global" expertise and overview, as one of the juror or a devil's advocate and a reviewer of the selection, assuring that the possible quality gap among national finalists is not too big.

The other countries can use the Montage tool with at least three jurors, especially when the uploaded files are very diverse, of similar quality and/or more than 50-100. In this case, unless necessary, the coordinators of the Montage campaigns are not jurors.

Criteria

The image guidelines of Commons can be shown to the jurors to get an idea of the type of selection required for standard images (not the ones produced with advanced instrumentation).

For more specific images, it can be also possible to look into the categories for similar images. This is a possible way to understand the originality of an image in fields where jurors are less expert.

Summary table

In the subpage by year a work-in-progress table with all second-level jurors associated with the countries can be prepared.

In the table one column is the country (flag and codice ISO), one is the country subcategory with the number of total files and unsuitable files in italics, one is the methods used for the selection, than you can read the surnames of the jurors involved, and a summary of the final output.

The global coordinator of the jury duty table is also indicated.

National winners and finalists

They are listed in ... category to be created.

A scaffolding for every country with jurors, dates, prefinalists, finalists for the international jury, winners should be prepared.

All countries with a nice amount of good candidates using montage or a national jury could provide also a set of pre-finalists, that is all the files whose quality is considered good, listed just before their final selection. These lists of prefinalists will be presented also as a possible reservoir for good files, i.e. valued, featured and quality images.

A scaffolding for every country with jurors, dates, prefinalists, finalists for the international jury, winners can be found at Commons:Wiki Science Competition/FinalistsLayout.

In the end the national finalists are the files selected for the main level jury. In that set the country-level jury or the international second-level jury can select further winners, with specific subcategories. Usually countries have less winners than finalists. If necessary, the role of the international second-level jury is also to give advice to national committee on the possible selections of additional finalists.

It is important to point out that with small sets of files and limited jurors, fluctuations are always stronger than with "bigger countries". Also for this reason as many country with no national jury as possible should be merged in an international "rest of the world" category, especially when their number of files or quality is limited.

The number of files selected as finalists is based on their overall number and quality in the category.

When selecting the finalists of country without local committee however, especially when no prize was involved, it seems that sometimes a bigger tolerance in border-line cases for their selection can occur. This is also a way to thank the users who uploaded them and made possible a separate country selection, which can act as catalyst for a more complete competition next time. In this case it is possible to state that an image is a national finalist but cannot be a real finalist.

For countries with limited upload and no prize, it is possible that sometimes the same file is proposed as a finalist in the "set of images" and in another category. it's not a usual occurrence but as stated in the file management page, the key question is if the presence of the file drastically changes the quality of the set. if it does not, a duplicate finalist is possible.

In any case they will all be equal to the evaluation of the final international jury.

Statistics

Statics about total uploaded files, eligible files, winners can be listed by year.

• ...