User talk:Rotatebot/Archive/2011

From Wikimedia Commons, the free media repository
Jump to: navigation, search
Archive This is an archive of past discussions. Do not edit the contents of this page. If you wish to start a new discussion or revive an old one, please do so on the current talk page.

Multiple rotations

The bot has not removed {{rotate}}, so has repeated rotation of a couple of images after 6 hours. Finavon (talk) 18:19, 18 February 2011 (UTC)

File:Petrophile pulchella 1.jpg and earlier edits Finavon (talk) 08:55, 19 February 2011 (UTC)
File:Acacia obtusifolia 2.jpg Finavon (talk) 20:04, 19 February 2011 (UTC)
Not sure why (all looked usual) - maybe a temp. glitch. Seems not to have re-occured. --Saibo (Δ) 19:07, 26 October 2011 (UTC)
This section was archived on a request by: Saibo (Δ) 19:07, 26 October 2011 (UTC)

No rotation

Now has removed {{rotate}} without a new upload at File:West Virginia crag overlooking Germany Valley (5074103993).jpg. It does not look like just a caching problem. Finavon (talk) 06:32, 17 May 2011 (UTC)

Managed at second attempt. Finavon (talk) 19:32, 17 May 2011 (UTC)
Same reason/solution as above --Saibo (Δ) 19:07, 26 October 2011 (UTC)
This section was archived on a request by: Saibo (Δ) 19:07, 26 October 2011 (UTC)

"wrong degree-parameter (°)"

Something seems to be borked, see [1]. --  Docu  at 18:30, 16 July 2011 (UTC)

thx, looking for.--Luxo 18:32, 16 July 2011 (UTC)
fixed. Reason was a change in the api.--Luxo 19:39, 16 July 2011 (UTC)
Thanks for fixing this so quickly. --  Docu  at 20:40, 16 July 2011 (UTC)
BTW, looks like bugzilla:6672 finally gets done. --  Docu  at 20:43, 16 July 2011 (UTC)
This section was archived on a request by: Saibo (Δ) 19:07, 26 October 2011 (UTC)

Glitch

Looks like it skipped the remove-{{rotate}}-step here. Probably just a temporary problem. --  Docu  at 06:35, 28 September 2011 (UTC)

There were two {{rotate}} tags on the image. --  Docu  at 06:38, 28 September 2011 (UTC)
This section was archived on a request by: Saibo (Δ) 18:54, 26 October 2011 (UTC)

Rotatebot - new feature

from here

Darth Vaders along NE Corridor in Washington DC.jpg
CSX River Sub Kingston CP 90.jpg

Please visit File:Darth Vaders along NE Corridor in Washington DC.jpg. This file contains an wrong the EXIF information: error rotation. Photography is correctly vertically (portrait), but the content (area) in the vertical position should be rotated. The result is again a vertical photo (portrait). Example (history): File:CSX River Sub Kingston CP 90.jpg.

Designing a new feature: rotate image area while maintaining the portrait / landscape. --W.Rebel (talk) 08:46, 18 October 2011 (UTC)

Correcting wrong EXIF is no new feature - it is already present: just rotate it to be upright as it has been done at Darth Vader. Cheers --Saibo (Δ) 12:49, 18 October 2011 (UTC)
This section was archived on a request by: Saibo (Δ) 18:54, 26 October 2011 (UTC)

FIFO on Images requiring rotation by bot

If we keep getting higher volumes and if it's easy to change it, maybe RotateBot could process the images added to the category first, before the others. "last change date" as here could be another option. --  Docu  at 12:18, 7 November 2011 (UTC)

According to this diagram, there is a field cl_timestamp in the table categorylinks (ORDER BY cl_timestamp ?); and sorting according to the "added-date" is also exposed by API so it should be possible. -- RE rillke questions? 14:12, 7 November 2011 (UTC)
Maybe it is not much to program, yes - but I'd say it is not really necessary. There simply shouldn't be long wait times. ;-)
Well, I had a closer look: cmsort=timestamp does it. Then the oldest are first. The bot uses cmlimit=x to restrict the number of hits - seems to work: http://commons.wikimedia.org/w/api.php?action=query&list=categorymembers&cmtitle=Category:Images requiring rotation by bot&format=php&cmprop=ids|title|sortkey|timestamp&cmnamespace=6&cmlimit=30&cmsort=timestamp (copy link from code) returns the oldest x entries. The API response is equal except the sorting (checked it with one entry - hopefully the same for all. ;-) )
So, Luxo, if you read here: it would only be the addition of &cmsort=timestamp to the query URL. --Saibo (Δ) 21:27, 7 November 2011 (UTC)
Noch besser:http://commons.wikimedia.org/w/api.php?action=query&list=categorymembers&cmtitle=Category:Images requiring rotation by bot&format=php&cmprop=ids|title|sortkey|timestamp&cmlimit=30&cmsort=timestamp&cmtype=file
Dann könnt die Prüfung auf NS=6 auch raus und ihr erhaltet immer genau soviele Bilder wie angefordert, auch wenn ein Vandale Seiten mit einkategorisiert. -- RE rillke questions? 19:54, 8 November 2011 (UTC)
habs so übernommen. Danke--Luxo 20:17, 8 November 2011 (UTC)
So wie es ausseiht, geht es. --Saibo (Δ) 00:02, 7 December 2011 (UTC)
This section was archived on a request by: Saibo (Δ) 00:02, 7 December 2011 (UTC)

Source code?

Where is the source code for this bot? Badon (talk) 07:22, 8 November 2011 (UTC)

It is not published currently but should be in the near future under an open source license. Do you want to run it for your own wiki? Cheers --Saibo (Δ) 15:07, 8 November 2011 (UTC)
[2]--Luxo 18:06, 8 November 2011 (UTC)
Yes, I would like to experiment with running bots for educational purposes, and this one seems like it might be useful on my own wiki too, since it can make things easier for people using the site. Badon (talk) 04:48, 10 November 2011 (UTC)

Linked it on the bot's user page, too. --Saibo (Δ) 00:03, 7 December 2011 (UTC)

This section was archived on a request by: Saibo (Δ) 00:03, 7 December 2011 (UTC)

Any chance of Rotatebot handling more rotates?

moved from User_talk:Luxo#Any_chance_of_Rotatebot_handling_more_rotates.3F to here. --Saibo (Δ) 18:46, 6 December 2011 (UTC)

Hey Luxo, I just looked at Rotatebot following an OTRS complaint about an image on enwiki being the wrong way around. It currently has a huge backlog: is there any way to get it to handle rotations either more often, or perhaps to have another instance of Rotatebot that could be run elsewhere? It just seems slightly strange to have a situation where someone can nominate something for rotation, and then having to potentially wait five days for it to be handled.

That said, it's great that Rotatebot exists and I've used it a lot while new file patrolling and so on. —Tom Morris (talk) 00:33, 4 December 2011 (UTC)

Well, maybe in first place we should ask the very busy mass taggers at Commons:Bots/Work_requests#Maintenance_category_for_files_with_EXIF_rotation_other_than_0_degrees to stop if ~18 hours backlog is exceeded? I have put in a indication of the backlog on user:rotatebot and in the category intro.
Yes, it can be made faster - at least by about 50%. Although this would mean more load on the toolserver. And the danger of maybe some errors. Lets see what Luxo thinks. I to not have the time to make the changed to the code now. Cheers --Saibo (Δ) 03:21, 4 December 2011 (UTC)
Given the time it takes to process the requests, it could run every 30', at least on a temporary basis. As it now works on a FIFO basis, the risk should be rather limited. I'd rather not have people stop tagging images, otherwise they never get fixed. --  Docu  at 07:55, 4 December 2011 (UTC)
I don't see a viable alternative to ramping up Rotatebot. In fact, fixing this problem seems high enough priority that we might see if we can't temporarily shut down some toolserver tools that are heavily used and put a high load on it, to free up capacity for rotatebot. (That may not be possible, but perhaps a toolserver admin could do it.) This task needs to be completed, and as soon as possible, so that normal service can resume! Rd232 (talk) 11:33, 4 December 2011 (UTC)
Well toolserver capacity is the smaller problem, the script is the bigger problem. This script is not suitable for much more than 20 rotatings/hour because of the program flow. It needs bigger adjustments to fit it for such much requests. At the moment I don't have the time to do this; if you find someone who like to do this, I'm readily to give this bot. But I think this is a temporary problem because of mass tagging of pictures from some users; until yesterday 15images/h was enough. I think it will take one, maybe two weeks then this overflow will be worked off.--Luxo 13:03, 4 December 2011 (UTC)
Hm, I'll have to take your word for it; I assumed that there was some kind of throttling of the script to prevent server overload, rather than an inherent capacity limitation. Rd232 (talk) 13:47, 4 December 2011 (UTC)
I think there was a problem if the bot tries to retrieve more than 30 images at once (exact number seems to depend on filesizes).
Thus, the script could run more frequently, but shouldn't retrieve more images at once. For 30 images, the process didn't seem to take more than 30', so every 30 minutes should be possible. --  Docu  at 14:10, 4 December 2011 (UTC)
Yes, IF the previous run is finished in less than 30 minutes it could run every 30 minutes. But the problem is: the time which is needed by a batch/run depends (AFAIK) heavily on the size of the images (download/upload time). So if we have 40 × 35 Megabyte images the chance is good that it will mess up severely. If we have 40 × 3 Megabyte it'll be fine. Cheers --Saibo (Δ) 15:04, 4 December 2011 (UTC)
Silly question: is there no way for the next run to check if the previous run has finished, to prevent collisions? Rd232 (talk) 15:31, 4 December 2011 (UTC)
Not that silly, yes that would be a relatively simple possibility. Could be done - yes. Simply by a "lock" file while it is running.
Start: if (!file_exists($savepath."rotatebotlock")) {system("touch ".$savepath."rotatebotlock")} else {myDie("Fuuu")}. End: system("rm ".$savepath."rotatebotlock"). All the die( exits need to be changed to e.g. myDie( where the "End:" code is located.
A problem could be unexpected exits (severe errors) - the lock file prevents any further start. Well, not a bad idea anyway since a human should have a look if such an error occurred. Cheers --Saibo (Δ) 16:05, 4 December 2011 (UTC)
@Luxo: if you have time to deploy and test (the normal case and the error case): code +lock feature (not tested). diff (ignore the signature of me, of course). Cheers --Saibo (Δ) 00:21, 5 December 2011 (UTC)
How about if (system("ps aux | pgrep rotatebot") outputs something) {die} else {start}? It would be nice if Rotatebot could rotate more images. Alternatively, tell Rotatebot to delete the lock file if it was created at least several hours ago. --Stefan4 (talk) 15:40, 6 December 2011 (UTC)
I think having it not run automatically if some un-handled error occurred is not the worst idea. ;-) Not you think? But thanks for your suggestion! :-) --Saibo (Δ) 18:49, 6 December 2011 (UTC)
I don't know anything, but should it be possible to copy the script of this bot to a second new bot? That would double the rotation speed and if there are less images to rotate you can always stop one of the bots. In that way you don't have to make difficult changes on this bot, and that's maybe a big benefit? I90Christian (talk) 22:59, 6 December 2011 (UTC)
If two bots try to rotate the same image, the image might be rotated twice, which would not be correct. On the other hand, if it could be designed so that they pick images in different order (e.g. ascending/descending file names), it should be fine. On the other hand, this is probably just a temporary problem. --Stefan4 (talk) 23:09, 6 December 2011 (UTC)
Yes, another instance could run in parallel if it would select other images (e.g. by inverting the sorting). But: we would need someone with a Toolserver account and a bot account here. And the Toolserver bandwidth is limited - so I do not know if the TS folks would like this at all. ;) The other TS users could experience a slow connection if Rotatebot-1 or -2 are constantly uploading or downloading. If the above proposed code goes live we will have about 104 images per 2h. Or 3,8 instead of 10 days backlog. Cheers --Saibo (Δ) 23:50, 6 December 2011 (UTC)
Thank you Saibo, it works. Bot is now started every 15 minutes, so he will run more or less every time.--Luxo 16:30, 7 December 2011 (UTC)
Great, thanks for deploying! So there are some "Could not get lock. Lock file already present. Exit." in the server-side log now, correct? Cheers --Saibo (Δ) 20:54, 7 December 2011 (UTC)
Your "cron" job on nightshade
php rotbot/rotbot.php

produced the following output:

Could not get lock. Lock file already present. Exit.

Gruss--Luxo 21:28, 7 December 2011 (UTC)

Nice. That brought us to a full-load speed of about 1471 images per day. --Saibo (Δ) 00:05, 9 December 2011 (UTC)
This section was archived on a request by: Saibo (Δ) 00:05, 9 December 2011 (UTC)
Seems to steam ahead quite well. Thanks for improving this. --  Docu  at 07:50, 11 December 2011 (UTC)
It is since some days ~ 2181 images per day. (artificial slowdown removed) --Saibo (Δ) 00:01, 12 December 2011 (UTC)

Thanks - you did great work here. I thought my images would be unusable for weeks, and now pretty much everything I put into the category is already rotated. --AndreasPraefcke (talk) 20:47, 12 December 2011 (UTC)

Huge Queue

I would urgently suggest, that the robot should run more often than 40 images every two hours as the queue has now grown to more than 3500 images and every new image takes about 180 hours (more than a week!!!) until it is processed. --GDK (talk) 16:38, 5 December 2011 (UTC)

Symbol keep vote.svg Agree I agree with the suggestion of GDK. I would like a robot that could run like 100 images every two hours. When the box (category) is emtpy, you also could rechange it's speed to 40 or 50 images per two hours. Could you change the robot? I90Christian (talk) 17:03, 5 December 2011 (UTC)
It could be increased slightly (to about 60 per hour max.), but this would require the script to be changed. See User_talk:Luxo#Any_chance_of_Rotatebot_handling_more_rotates.3F for details. --  Docu  at 17:07, 5 December 2011 (UTC)
We are well aware that 40/2h is not quite good currently. But with the current code it is dangerous (not really possible) to increase to more. Sorry for this. From October until now it was fine. But for some reason (also due to an initial, not-really coordinated mass-fix attempt) the we got many many more requests than usual. See bot's log at 2011-12-03. Cheers --Saibo (Δ) 18:54, 6 December 2011 (UTC)

I guess "some reason" is some caching problem: for weeks, the stupid MediaWiki bug didn't cause that much trouble, but suddenly a couple of days ago, many thousands of thumbnails (at least of my uploads) were re-computed making the problem much more urgent in all projects. --AndreasPraefcke (talk) 10:14, 7 December 2011 (UTC)

According to the server admin log they run a thumb-cleaner: "thumb cleaner started up again" -- RE rillke questions? 13:01, 7 December 2011 (UTC)
LOL.. would be nice if they had asked or at least told us ... --Saibo (Δ) 18:57, 7 December 2011 (UTC)
Just had a chat in IRC #wikimedia-tech with apergos: They run a thumb cleaner because they run out of HDD space on the thumb servers. → LOL sorry... He tried to only purge thumbs of files which are not in article use. --Saibo (Δ) 14:25, 8 December 2011 (UTC)

Done per #Any_chance_of_Rotatebot_handling_more_rotates.3F. See #Off-topic_at_Rotation_on_Wikipedia for other discussion places. --Saibo (Δ) 00:09, 9 December 2011 (UTC)

This section was archived on a request by: Saibo (Δ) 00:09, 9 December 2011 (UTC)

Off-topic at Rotation on Wikipedia

Split off from #Rotation on Wikipedia. --Saibo (Δ) 23:51, 8 December 2011 (UTC)

Pictogram voting info.svg Info this ↓↓ is off topic to rotatebot (--Saibo (Δ) 20:49, 7 December 2011 (UTC)):

Isn't it more like vice versa? Scaled down web images usually have no EXIF data, but vertical phtotographs with EXIF data are for some reason sometimes rotated horizontally after downloading from camera. Doesn't MW's automatic rotation feature cause more harm than use? Who would upload images without giving them right orientation (but not knowing it's also needed hack EXIF data) before? 193.40.10.181 15:58, 7 December 2011 (UTC)
Ask the paid devs. → bugzilla: I don't know. -- RE rillke questions? 16:24, 7 December 2011 (UTC)
Dear nameless user, the problem is: Users in fact do care and rotate their images before uploading. And the is the problem. That is because they often use Software which doesn't care about EXIF (like the common Window$ photo gallery). If they would use software like gThumb, Xnview or irfanview no rotation would be necessary (before upload and for viewing) at all because usually the cameras store the right EXIF orientation.
The automatic rotation by MW leads to images with correct EXIF metadata in the end (when we have all fixed). Without the fixes our images would have wrong metadata and would show up rotated if someone uses a EXIF-aware program which does autorotation. More info here: Commons:Rotation. Cheers --Saibo (Δ) 19:03, 7 December 2011 (UTC)
Exactly, that was my thought, users do rotate their images before uploading. Then again, there may be no need to rotate and EXIF orientation could be correct but with its value other than 1 (normal) MW still messes up the thumbnails as according to my current experience MW rerotates thumbnails for everything that has EXIF orientation other than 1. E.g File:Leedu Suursaatkond.IMG 2806.JPG (EXIF orientation to be reset) is on side because it has correct orientation data "Rotated 90° CCW" (the angle at which the camera was when photo was captured) and the thumbnail for File:Loigu keerdkadakas.jpg is generated correctly because orientation is "Normal", though I'd expect it to be 90° for a vertical photo. So I fail to understand how the current fixes provide the correct metadata. 193.40.10.180 20:21, 7 December 2011 (UTC)
A EXIF orientation value other than 1/"normal" means that you need to rotate the image if you want to view it as intended since the image is not saved in the correct orientation. That (rotating for viewing) is what some image viewing tools and now also MW do.
Those images like File:Leedu Suursaatkond.IMG 2806.JPG are already saved in the correct orientation - after the bot has corrected the EXIF orientation it will again show up correctly here. It has correct EXIF orientation metadata then. That is what I meant by "leads to images with correct EXIF metadata in the end". ;-) Certainly(!) the way this software change was rolled-out was (and is) ugly (as expected). Cheers --Saibo (Δ) 20:49, 7 December 2011 (UTC)
One problem is that some people edit images in faulty programs which can't display images properly and that those programs save images in a way which instructs correctly working programs to display the images in an unintended way. Another problem is that some people rotate images in a way so that data is lost during the rotation. Hopefully, this update will make people realise that they need to stop using faulty programs (or expect that the files won't display correctly in those faulty programs) and that they need to stop rotating images in a stupid way. --Stefan4 (talk) 20:59, 7 December 2011 (UTC)
Yes, that is a good side of the update: hopefully less people will rotate their images in a lossy way (altough they did not some image editing where it is not to avoid except if we go for png instead of jpg). --Saibo (Δ) 21:06, 7 December 2011 (UTC)

I resent the common notion among MediaWiki developpers and in the discussion here that the uploaders were at fault here. Nothing in the JPG standard mentions these supid EXIF data, and when uploading this stuff years ago, no one ever mentioned that these EXIF information would ever be used to alter the displaying of the image. I feel betrayed by the Commons, and I think it is an unprecedented assault on the stability of the website's behaviour. If someone uses my images on a Wikimedia project, or, even worse, somewhere outside the Wikimedia universe, this person will not be alerted that the thumbnails are now different from a couple of weeks ago. MediaWiki 1.18 makes Commons looking like a joke, just after we had gradually developped into a site that could be taken seriously. --AndreasPraefcke (talk) 20:09, 8 December 2011 (UTC)

Maybe the wrong place, Andreas. Here you are at a place where volunteers offer a solution to the problems. And as previously said: It is not just Commons. And BTW, if the reusers outside Wikimedia are using the files, they are not affected by Thumb-purging AFAIK because they load the original and scale it their own way. See mw:InstantCommons. -- RE rillke questions? 20:18, 8 December 2011 (UTC)
True, but even here it seems like the uploaders are stupid, when in fact it's the developpers... By he way, Wikimedia actively encourages the direct linking of images (Commons as an image server), so the bug actually does affect sites outside. --AndreasPraefcke (talk) 21:43, 8 December 2011 (UTC)
Andreas, both are each to some extent. You wouldn't drive a car with a big text on it "scratch me!", would you? ;-) The developers and (now also the) server ops really did not do good work here, that's true. --Saibo (Δ) 23:51, 8 December 2011 (UTC)

Bei der Professionalität werde ich wohl in Zukunft keine Exif mehr hochladen. --störfix (talk) 22:15, 8 December 2011 (UTC)

Stimmt. Bilder mit Metadaten reichen uns völlig. -- RE rillke questions? 22:21, 8 December 2011 (UTC)
Meine etwa 500 mit Stripper bearbeiteten Bilder (im Gegensatz zu den übrigen) bringen jedenfalls keinen Beitrag zur Beschäftigungstherapie. --störfix (talk) 22:46, 8 December 2011 (UTC)

Note to all: This is Rotatebot's talk page - please discuss non-Rotatebot-related issues elsewhere: e.g. at *Commons:Village_pump#Why_has_an_image_rotated_without_being_told_to_do_so.3F (en)

That helps us Rotatebot developers/helpers (and so helps you). --Saibo (Δ) 23:51, 8 December 2011 (UTC)

This section was archived on a request by: Saibo (Δ) 23:51, 8 December 2011 (UTC)

new feature: Duplicate Orientation tags workaround / handling

For bugzilla:32868 ("incorrect duplicate EXIF IFD0:Orientation tag handling") here is a workaround or "handling": download (diff). I hope it works. :-D A bit hard to test now...  :-( It would need a deploy and some watching for errors ...

Roughly 2% (9) of the last 500 uploads were affected. And they show up right after Rotatebot - as long as the strange MediaWiki behaviour isn't changed. ;-)

I found those files by filtering rotatebot's upload log for NOT "EXIF". About half of them were not really EXIF Orientation=0 or =1. Instead they had duplicate Orientation tags due to a Apple software bug. Sadly MediaWiki doesn't (currently) ignore that conflicting tags but rotates despite them. Cheers --Saibo (Δ) 02:04, 8 December 2011 (UTC)

Current last 500 uploads by Rotatebot: 1 such a case (0.2 %) --Saibo (Δ) 13:42, 8 December 2011 (UTC)

So this is not really important now. Luxo, if you do not have time to test just continue with the old code. It is more important that Rotatebot runs at all. :-) Cheers --Saibo (Δ) 16:35, 8 December 2011 (UTC)

Live since circa 8 December 2011, 19:07 (next bot run). --Saibo (Δ) 23:59, 8 December 2011 (UTC)

This section was archived on a request by: Saibo (Δ) 23:59, 8 December 2011 (UTC)

Tiffany & co.

Would you rotate File:Tiffany & Co. SF exterior.JPG?--Mauro Tozzi (talk) 08:03, 8 December 2011 (UTC)

Rotation has been requested, so it will be rotated eventually. --Stefan4 (talk) 13:02, 8 December 2011 (UTC)
This section was archived on a request by: Saibo (Δ) 13:28, 8 December 2011 (UTC)


Request for rotation

Can you please correct File:AOK-Gebaeude Hamburg-Borgfelde4.jpg on Commons? - Thank you. --93.205.110.179 22:30, 8 December 2011 (UTC)

Done (at by Denniss). You can use {{rotate}} if you do not want to sign in (only then you have RotateLink available). Cheers --Saibo (Δ) 23:57, 8 December 2011 (UTC)
This section was archived on a request by: Saibo (Δ) 23:57, 8 December 2011 (UTC)

new feature: KillerMode and DontDieMode

code (diff). Needs the two new variables killAllRotatebots and dontDieOnLockProblems in the config file (should be set to "false"). Cheers --Saibo (Δ) 21:28, 9 December 2011 (UTC)

Note: the processname is a guess - needs to be figured out via ps. And: this only works if they all run under the same user. --Saibo (Δ) 14:01, 10 December 2011 (UTC)

live. diff of this new code on luxo's fixed size check (the wrong size check was my faule - sorry).

This section was archived on a request by: Saibo (Δ) 17:50, 10 December 2011 (UTC)

File:Itsukushima pagoda 2011 yoko.JPG

Please don't rotate this image. It's a variant of File:Itsukushima pagoda 2011.JPG used for illustrating the problems with EXIF rotation on Japanese Wikipedia, so it is correct that the sky is to the right. See ja:MediaWiki:Sitenotice. A user requested rotation of the image, but I reverted it. Maybe it would have been better if the file name had contained the word "horizontal" instead of "yoko"... --Stefan4 (talk) 19:06, 10 December 2011 (UTC)

An IP address tried rotating it again while I was translating a Japanese explanation of the orientation into English. Now I've placed the explanations above the licence information to make them more visible. We need to check this image carefully. --Stefan4 (talk) 19:36, 10 December 2011 (UTC)
I simply protected it for the next 2 weeks. -- RE rillke questions? 20:00, 10 December 2011 (UTC)
This section was archived on a request by: Saibo (Δ) 23:16, 12 December 2011 (UTC)

Image rotations

A user incorrectly rotated several images that need to be corrected now. How do I go about getting this done? --EncycloPetey (talk) 04:07, 17 December 2011 (UTC)

Links to images ? --Denniss (talk) 04:56, 17 December 2011 (UTC)
The problem seems to have been corrected already. --EncycloPetey (talk) 02:47, 18 December 2011 (UTC)
This section was archived on a request by: Saibo (Δ) 04:16, 18 December 2011 (UTC)

Why has it stopped working?

The next issued rotation request would take approximately 1 day, 17:28 hours to be processed. Last edit was at 06:53, 18 December 2011. Please start it again quickly! Cheers, --Edelseider (talk) 10:12, 18 December 2011 (UTC)

Thanks for the report!
It was indeed not working between 2011-12-18T06:53:38 and 2011-12-18T12:38:33. Now it is again. Why it stopped - only Luxo can tell. It is working again because Luxo poked it. Cheers --Saibo (Δ) 13:01, 18 December 2011 (UTC)
It was a login error. --Luxo 14:30, 18 December 2011 (UTC)
This section was archived on a request by: Saibo (Δ) 13:01, 18 December 2011 (UTC)

Why has it stopped working again??

The next issued rotation request would take approximately 3 days, 2:18 hours to be processed. Last edit was 18:28, 18 December 2011. Ich sagte es doch: Berliner S-Bahn!--Edelseider (talk) 19:14, 18 December 2011 (UTC)

Thanks! You do not need to look at the time - just the last edit is important. Seems to be stuck again - I will have a look. Cheers --Saibo (Δ) 19:23, 18 December 2011 (UTC)
Runs again. --Saibo (Δ) 19:43, 18 December 2011 (UTC)
Was again a login error... It uses still die() there, I'll fix that...--Luxo 19:48, 18 December 2011 (UTC)
fixed, should restart automatically now. --Luxo 19:53, 18 December 2011 (UTC)
Is it in logindata.php? That is just after the line "// after this line only suicide() should be done instead of die()!" -- meeh. :-D Thanks, Luxo! Cheers --Saibo (Δ) 22:58, 18 December 2011 (UTC)
No, logindata.php contains only password and username of mysql; it was in the upload.php and login.php. See rev 267. Greetings,--Luxo 13:07, 19 December 2011 (UTC)
This section was archived on a request by: Saibo (Δ) 16:39, 19 December 2011 (UTC)

Discussion about batchsize 20

This is split off from #Rotatebot and the WMF. --Saibo (Δ) 17:55, 19 December 2011 (UTC)


Didn't that accelerate it? It might just be a series of smaller than average images. --  Docu  at 06:50, 16 December 2011 (UTC)
On User:Rotatebot it says: "Edit period(s): 24/7. Starts every 5 minutes if the previous run has finished gracefully" so reducing batches from 40 images to 20 images means a 20-image-batch is now done in about 50% of the time of a 40-image-batch and maximum 5 minutes later it will start with the next batch of 20 images. As you can see in the history of the log before that change every hour about 2 batches of 40 images were handled and now that is about 4 batches of 20 images per hour. - Robotje (talk) 10:46, 16 December 2011 (UTC)
Yeah, but in the log I found 4.8 batches/per hour at 20 images (over five hours) and 2.2 batches/per hour at 40 images (over 24 hours). This despite that the average idle time doubles. --  Docu  at 12:22, 16 December 2011 (UTC)
The backlog is getting bigger again so speeding up instead of slowing down shall be the right decision.--Edelseider (talk) 13:29, 16 December 2011 (UTC)
@Docu; My guess is, there is to much variation in the time needed to rotate one image to draw any conclusions yet (2.2 * 40 = 88 images per hour against 4.8 * 20 = 96 images per hour). A huge image takes more time to rotate than a small image, the CPU-load on the server depends also on other tasks running, sometimes only the EXIF information is changed (EXIF-Orientation set from 6 to 1, rotated 0°), etc. You would expect that lowering the number of images per batch has a small negative impact on the number of images that can be processed because, as you already mentioned, the idle time between to batches will roughly double if the number of images to be processed in one batch is halved. If in the long run it turns out that the number of images that can be handled per hour is slightly higher with a batch size of 20 images, then the adjustment by Saibo was a little bit counterproductive. - Robotje (talk) 13:47, 16 December 2011 (UTC)
I was aware that it will not slow down much - but it should be a bit as now the double amount of "wait until the next 5 min trigger arrives" happens per same amount of images. And, yes, I guess that were many smaller images. We should tag faster. ;-) I have prepared some tiny code to maintain a appropriate minimum lag - Luxo if you have time this evening.... ;) Note: have set to full speed again. Note2: Reedy's bot would be available if needed. It has a much higher speed. Rotatebot should be faster too since Ops fixed a LVM issue which limited upload speed (via non old secure servers). Cheers --Saibo (Δ) 14:54, 16 December 2011 (UTC)

To get some numbers, could we have it run in batches of 20 for a day? If the total image size is added to the bot's edit summary, we could attempt to make a better comparison. Personally, I had the impression that already 30 was quicker than 40. --  Docu  at 18:54, 16 December 2011 (UTC)

Comparing the number of images is not really useful - the time varies too much by size of the images. And since the bot processes FIFO it depends which images were tagged. But, okay, let's have a look at the numberz (hope without errors by me):
  • 2011-12-16T04:32:37‎ - 2011-12-16T15:07:07‎: 11.65 h. 41×20 files - 2 errors: 818 files. → 70.21 files / h.
  • 2011-12-15T04:24:26‎ - 2011-12-15T15:16:28: 11.67 h. 26×40 files - 1 error: 1039 files. → 89.03 files / h.
So? What is your conclusion now? ;-) Cheers --Saibo (Δ) 20:55, 16 December 2011 (UTC)
Instead of stopping rotatebot to test the other bot, just turn the sorting from asc to desc then the bots will never hit each other, just if there are < 40 pictures in the cat.--Luxo 21:18, 16 December 2011 (UTC)
@Saibo: Quite convincing. Nothing to add. --  Docu  at 13:03, 18 December 2011 (UTC)
This section was archived on a request by: Saibo (Δ) 17:34, 20 December 2011 (UTC)

Rotation on Wikipedia

Would it be possible to provide rotation on Wikipedia too? Free images can be moved here and rotated here after the move, but fair use images have to stay on Wikipedia and need to be rotated there. At least English Wikipedia has some fair use images with wrong EXIF data, e.g. en:File:1984WorldsFairMetalSign.jpg. --Stefan4 (talk) 23:06, 6 December 2011 (UTC)

Find someone to run the bot, grab the code and do it. ;-) Yes, it should be possible (to my knowledge - I am not the bot operator but have programmed some parts of it). Usefulness depends how many images are affected. As FU images mostly are no photographs but instead scaled down web images I guess that will not occur often. I have fixed the image you mentioned (it was too high res anyway). Cheers --Saibo (Δ) 00:00, 7 December 2011 (UTC)
This section was archived on a request by: Saibo (Δ) 14:53, 22 December 2011 (UTC)

File:Totem Park pole 2.jpg - purge issue

  • Rotation seems to have caused problems with File:Totem Park pole 2.jpg. Are you able to fix it? Melburnian (talk) 00:29, 22 December 2011 (UTC)
    • Surely - where is the problem? Which page? The Wikipedia article pages maybe need a purge (or some days waiting). And be sure to refresh your cache if it still persists. Cheers --Saibo (Δ) 01:25, 22 December 2011 (UTC)
      • see en:Totem_pole#Property. Seems to be fixed now by a blank edit and save of the affected section. I assume a local caching issue as the rotated thumb was not there on editing the section - it had to be generated first. --Denniss (talk) 02:43, 22 December 2011 (UTC)
        • I made edits to the 3 English Wikipedia articles which contain the image to refresh the articles --- all instances look fine on my screen now. Thanks for your help. Melburnian (talk) 03:18, 22 December 2011 (UTC)
  • Have added a short section here: Commons:Rotation#Wikipedia_articles --Saibo (Δ) 04:30, 22 December 2011 (UTC)
This section was archived on a request by: Saibo (Δ) 04:24, 22 December 2011 (UTC)

Problem with an image - broken jpeg

File:Helichrysum sanguineum 7.jpg. First image here. Rotatebot's log:

“Error can't rotate File:Helichrysum_sanguineum_7.jpg:
EXIF had severe errors on write attempt. (ec: 1)”

I attempted to rotate it myself:

$ exiftool -Orientation= Helichrysum_sanguineum_7.jpg
Error: JPEG EOI marker not found - Helichrysum_sanguineum_7.jpg
0 image files updated
1 files weren't updated due to errors

Any ideas? The left portion of the image also seems to be broken. --Stefan4 (talk) 21:31, 23 December 2011 (UTC)

That error was produced by the "BruteForceClean" part of rotatebot's code which is already much more error tolerant as your exiftool command used above. And as we can see from the image the upload seems to have been interrupted - a part of the image is missing. Also the third repetition would not have helped. I will fix this manually. Cheers --Saibo (Δ) 22:29, 23 December 2011 (UTC)
Done. And: thanks for your big help at the rotation stuff! :-) Cheers --Saibo (Δ) 22:47, 23 December 2011 (UTC)
This section was archived on a request by: Saibo (Δ) 22:47, 23 December 2011 (UTC)

Stopped/Stucked again

Please investigate why the bot stopped again. No edits since 1.5 hours, last activity was uploading an image but stuck from then on (notice not removed). --Denniss (talk) 05:56, 22 December 2011 (UTC)

Pictogram voting info.svg Info Bot is stucked since 04:21, 22 December 2011. --Edelseider (talk) 08:45, 22 December 2011 (UTC)

Sorry and thanks for the notice. At it... --Saibo (Δ) 13:50, 22 December 2011 (UTC)

Works again. Investigation:

For whatever reason (Luxo can you have a look in the server-side log?) the bot did not remove the rotate tag at File:Muzeum Orkana - church 23.JPG. Notable also the long time for the next upload after this. Then uploaded the file for File:Muzeum Orkana - inside 21.JPG bot did not remove the tag and was stuck then. I guess a network problem or server problem on the WMF servers. Cheers --Saibo (Δ) 14:28, 22 December 2011 (UTC)

In two of the cases, the bot rotated the image twice and didn't remove the template until after the second rotation. It might be the same problem as above. --Stefan4 (talk) 00:06, 23 December 2011 (UTC)
Hmmm - but in the above case it wasn't stuck afterwards. ;-) Maybe the same root cause, though. We will see what Luxo says. Runs fine currently. --Saibo (Δ) 00:47, 23 December 2011 (UTC)
upload File:Uspenski_Cathedral_in_Helsinki.jpg...
php_network_getaddresses: getaddrinfo failed: temporary name resolution failure (0)
... --Luxo 13:20, 3 January 2012 (UTC)
Thanks - so, as guessed, a network error. In long term there could be made robustness improvements at many code places, I guess. Okay for now. Define: network works at 100 % of the time. Problem solved. ;-) --Saibo (Δ) 15:50, 4 January 2012 (UTC)
This section was archived on a request by: Saibo (Δ) 15:50, 4 January 2012 (UTC)

new code bit (not important) proposal for activation controlling via a template

diff --Saibo (Δ) 23:06, 18 December 2011 (UTC)

don't see the sense of this... what's the difference to var active = true ?--Luxo 18:07, 19 December 2011 (UTC)
The sense is that with the current setup and with a empty queue rotatebot would rotate an image at maximum 5 minutes after tagging it (which is even too short for self-corrections after reading the rotate template). The code (+User:Rotatebot/config.js/active) enables a queue dependent activation of the bot. That means (with the current template setup) if the queue length is below the lower time target (that means how often the bot runs) then the bot is not activated each 5 minutes but at a longer interval. More clear? Cheers --Saibo (Δ) 22:17, 19 December 2011 (UTC)

Solved in #Slow_down_Rotatebot.3F below. Code not live. --Saibo (Δ) 18:18, 10 January 2012 (UTC)

This section was archived on a request by: Saibo (Δ) 18:18, 10 January 2012 (UTC)

Rotatebot and the WMF

Hi, With relation to bugzilla:31509, I'm trying to find out the best way to help you fix up the backlog etc.

Do you know what is constraining you on the toolserver? Memory? Cpu? Network I/O? Combination?

There would be easily the possibility of running a clone bot on the Wikimedia cluster, this will reduce the network rountime, and bandwith constraints. Then the process becomes more CPU/memory bound.

Ideally, I'd like to speak to you in real time via IRC or something, but via talk pages/email can work...

And Saibo has just responded on #wikimedia-commons...

Thanks


Reedy (talk) 22:30, 13 December 2011 (UTC)

As per a conversation with Saibo on IRC, it seems network is the limiting factor. The plan is tomorrow essentially get a Rotatebot clone up on wikimedia servers and hopefully running, at a much increased speed. Some minor refactoring etc of code may take place to help normalise the configuration between the 2 versions. If this is sufficiently faster, we should stop Rotatebot, and we can use either another specific account, or my User:Reedy Bot (unused on commons) to do the renames. The dependencies don't seem too bad, so hopefully we will make some progress! Reedy (talk) 23:46, 13 December 2011 (UTC)
The current backlog would take less than 24 hours to process by Rotatebot. Is higher speed really needed? --Stefan4 (talk) 11:00, 14 December 2011 (UTC)
That's not potentially full backlog though... That's what users have so far tagged to need rotating... Reedy (talk) 16:21, 14 December 2011 (UTC)
Well maybe 2 days then the backlog is obtained. The actual log is because there was a huge accumulation because this bot was really slow. Since he's running full speed, the backlog is dropping continuously. But I don't know how many picture are still out there with old thumbnails... It's your choice, if you like to run a Rotatebot clone - do it! Faster is always better ;) --Luxo 20:39, 14 December 2011 (UTC)
At Commons:Bots/Work_requests#Maintenance_category_for_files_with_EXIF_rotation_other_than_0_degrees there was an estimation on 4th December of around 50k files (of which 20k are in use files) with EXIF Orientation (not all are wrong, of course). Rotatebot has done 14792 files since then. So around 35208 are missing and would take at current speed of 2130 per day around 16.5 days. Although not in use images are not really important. If we assume that currently most rotated images were in use images of the 20k there are only around 5k (minimum - 100 % in use assumed) left. --Saibo (Δ) 00:54, 15 December 2011 (UTC)
Info: Reedy did some testing with user:Reedy_RotateBot (contribs) - not really successful - afaik. The upload wasn't faster than with Toolserver. We do not know why - a test upload from another high speed inet connection was much faster. Maybe the upload.php is a bit too old. ;-) We will see what Reedy does tomorrow. Note: I had suspended Rotatebot for about 1:40 hours to avoid conflicts. Cheers --Saibo (Δ) 01:37, 16 December 2011 (UTC)
Well, well.. currently we are at 7 hours.. and I guess that it will be 0 when I get up again. I have tried to slow down a bit ;-) Apparently we are not fast enough with tagging currently. --Saibo (Δ) 03:17, 16 December 2011 (UTC)
Reedy's bot has stopped after only a few hours work and now the backlog/queue for Rotatebot is again huge (some 1,500 newly tagged files, this morning there were only 340 left). --Edelseider (talk) 16:10, 16 December 2011 (UTC)
Yes, Reedy did just short tests (no full speed running) - he also has no Bot flag here (currently) which means that he shouldn't upload much images as they will spam the new files patrolling. I wouldn't call a backlog of currently 19:49 hours "huge". Will probably shrink again in the night. Cheers --Saibo (Δ) 18:39, 16 December 2011 (UTC)
Reedy can you start your bot processing the images in reverse-order? -- RE rillke questions? 11:51, 19 December 2011 (UTC)
Reedy first needs a bot flag - otherwise he spams the new files. And: processing reverse only is a bit bad since then there is no time to correct wrong taggings. If reedy has a botflag we can simply switch off rotatebot - the WMF rotatebot is so much faster that rotatebot in parallel is not needed. The backlog will be 0 in some hours. What will be the real problem then is that we have enough people checking the logs. Cheers --Saibo (Δ) 17:55, 19 December 2011 (UTC)
Reedy did some more tests - he could start to run it now (freakin fast). As soon as he has a bot flag. Running without bot flag clears the backlog - but simply is disturbing other processes at commons (my reasoning). So if there is a agreement we should do a speedy ;-) bot flag request at our usual place. Cheers --Saibo (Δ) 00:08, 20 December 2011 (UTC)
If the bot is simply a clone, I see no need for such a bureaucratic action. There is nothing to discuss or evaluate. He can assign the bot-flag himself. He is staff. If you go to the bot-approval-page you will have to wait until a 'crat is closing the request. And this won't be done within one week. -- RE rillke questions? 09:55, 20 December 2011 (UTC)
I do not like community actions performed by staff rights. Don't you think that a crat could do a speedy bot flag in that situation? --Saibo (Δ) 14:17, 20 December 2011 (UTC)
Undoing damage that is not done by the community does not need community approval. For technical reasons, staff can assign user-groups. NeilK did this even for a "normal user" who is now sysop. This is a technical reason, there is nothing I can think about to discuss, so just do it. Of course you can ask bureaucrat: "Please assign the botflag. It is a clone of an already approved bot. If you don't do it you are responsible for upset users." -- RE rillke questions? 15:39, 20 December 2011 (UTC)
He would not undo the damage - undoing would have been to disable the rotation directly after it was deployed... Bot flag is a community thing which should be handled by the community - I think we should stick to our principles here - we do not want to have a Jimbo deleting files here.
Commons:Bots/Requests/Reedy RotateBot --Saibo (Δ) 17:21, 20 December 2011 (UTC)
Don't know where you see a Jimbo ;-) It's not a porn-delete-bot. -- RE rillke questions? 17:49, 20 December 2011 (UTC)
I saw vandalizing Jimbo (by abusing his founder flag) in May 2010. --Saibo (Δ) 16:42, 21 December 2011 (UTC)
user:Reedy RotateBot is flagged now. --Saibo (Δ) 16:42, 21 December 2011 (UTC)
Because someone told a 'crat that it would be urgent. -- RE rillke questions? 16:49, 21 December 2011 (UTC)
And.. em.. yes - I know. I would have done so too when I got the message from Reedy that he advanced in preparing the bot for run. Not sure about what you want to comment. --Saibo (Δ) 16:58, 21 December 2011 (UTC)
This section was archived on a request by: Revicomplaint? 03:29, 12 March 2014 (UTC)