Commons:Wikimaps/IEG

From Wikimedia Commons, the free media repository
Jump to navigation Jump to search

Also see https://meta.wikimedia.org/wiki/Grants:IEG/Wikimaps_Warper_2.0

2016 Individual Engagement Grant (IEG)

[edit]

In 2016 the Wikimaps project was funded by an IEG grant from the Wikimedia Foundation.

See https://meta.wikimedia.org/wiki/Grants:IEG/Wikimaps_Warper_2.0 for details about the grant, to see the scope of work involved, to view the people and team and volunteers involved and to view reports.

IEG Project Goals

[edit]

Backend

[edit]

The Warper code will be refactored. User interface is separated from computation to allow more versatile use of the software. Operations inside the code are also separated to make possible developing individual parts separately. New functions will be added. Mass import of already rectified maps will come handy to GLAM users and can be made part of maps uploads. Category imports will be enabled. Developing the new functionalities will start before the project. The API will be enhanced to be the primary means of operating the software.

Frontend

[edit]

After this stage the project will be ready for the next phase where user interfaces will be remade. We decided to do this in steps because of time constraints, and to assure that the backend is ready when the front end is made.

Integrations

[edit]

Wikimedia Finland has an ongoing project for processing metadata for Commons uploads. Warper upload will be made an option in it. We are looking for Commons volunteers to participate in Commons integrations, coding gadgets and templates. Linking with Wikidata will be a future direction.

Design

[edit]

Design decisions will be based on earlier work in the Wikimaps project, and updated to reflect development in sister software such as the OSM iD editor.

Development

[edit]

Code and issues are being developed on GitHub. On the wikimaps_new branch and the IEG milestone.

Dev Log

[edit]

This log will hopefully fulfil some of the requirements for reporting for the grant, provide community members with updates on the work being performed and hopefully be a useful learning resource. See https://meta.wikimedia.org/wiki/Grants:IEG/Wikimaps_Warper_2.0/Timeline for wider / official progress reporting on the grant. Logging by User Chippyy (talk) except where specified.

Previous recent things

[edit]
  • Oauth authentication - leading to the development of a Devise / Omniauth OAuth strategy for mediawiki. see blog post.
  • Commons category import - a Category of maps can be imported all at once into the warper.
  • Import of batch control points - a CSV of control points for many maps can be imported.
  • Download of control points - control points can be easily downloaded for a map
  • Bot edit of wiki page to update bounding box / extent and status - when warped, the warper will act as a bot and update the commons wiki page with the results of the bounding box.

January

[edit]
  • Wikimedia Labs system underwent a system wide reboot following a kernel vulnerability. After the reboot the warper application was offline. Passenger wouldn't start the server. It was taking 11 minutes to run "rake -T" in the command line! (It takes 23 seconds on a 3 year old Atom netbook in comparison!) The root cause was that after the reboot the NFS was slower than before - other apps and tools were experiencing issues. This was an issue because the Ruby environment was loaded via .rvm from a users home. And /home was on the NFS and not local to the virtual machine. A copy of the environment was made to the local /srv partition and symlinked from .rvm and led to "rake -T" taking 1 minute. Passenger gem was recompiled and updated. The new Passenger config was able to increase the timeout value. The application needed about 1-2 mins to start. Still too long. But much better than 11 minutes.
  • Strace and ruby-prof needs to be run to see where bottle necks are, and expert eyes to interpret the results. I think that further rvm work may also improve performance.
  • IEG Milestone created on Github Issues. To add issues, bugs, features to the work plan
  • Prioritising of issues. Conversations were had to prioritise initial work.
  • Github Issues were cleaned and closed. Since many issues were created before we decided to have the warper as API and a separate project for the new front end.
  • API & Feature documentation. More docs were created to help plan for the new API
  • Issue set up to create a bare bones front end to test the API, authentication etc.
  • Fix for a YQL best place service breaking when encountering special characters in wiki / maps titles.
  • British Library maps of the War Office in Africa were discussed on the OSM-GB mailing list https://lists.openstreetmap.org/pipermail/talk-gb/2016-February/thread.html and the wikimaps warper was mentioned and introuced. The maps had been added to the wikimaps warper previously however there was an issue with them not being in the actual category.
    • An issue to fix the import of maps was created and made a priority so that when a category is re-imported any existing maps can be added to that mosaic.

February

[edit]

March

[edit]

April

[edit]
  • Investigated bug with thumbnails in Beta Commons https://phabricator.wikimedia.org/T131472
  • Further work towards better handling of thumbnails
  • Fixed bug where sorting a maps mosaics by the percentage complete column caused an error
  • Set to enabled as default for the "add to new category" checkbox option in the Commons import form
  • Improved code to handle editing and updating wikitext automatically (pre-requisite for new Map template)
  • Started work on new Map Template
  • Updated warper.wmflabs.org with previous bug fixes
  • started work on api
  • Trying rails-api gem
  • Created new api_wikimaps branch for API development
  • Installed FactoryGirl to attempt basic unit tests towards good controller API tests

May

[edit]
  • using JSON API for output response format http://jsonapi.org/
  • Patched Imagemagick on server to workaround vulnerability
  • Created issue to update paperclip gem
  • Created issue to update rails
  • Working on Api_template branch
  • using rails-api and activemodel serialisers
  • developing geojson adapter for activemodel serialisers
  • New maps api
  • new layers api
  • new ground control points api
  • Merged in code for geosearch and index so that it will work with just one endpoint now
  • Fixed bug with sorting a layers maps by percent
  • Adding majority of CRUD actions, adding links to JSON output
  • Added basic GeoJSON output format for some Map and Layer outputs

June

[edit]
  • Added new token based authentication for API
  • Updated server for another imagemagick vuln
  • Developed simple POC app for interacting with new authentication and API https://github.com/timwaters/warper_oauth_token_auth_demo
  • Implemented json-api deserialization (changed API input format from old school Rails JSON format to New Improved json-api formats)
  • Custom Validation of type value for each method is correct using json-api
  • Added error responses for model Not Found
  • Plenty more controller api tests
  • Markdown API docs
  • curl examples of calls
  • added CORS configuration for /api/v1 calls
  • added authentication by user and password
  • added sign out method which disables token validity
  • Deployed api instance to staging server
  • Added API Docs to /api/v1 path
  • Merged in the api branch into the main (wikimaps) branch
  • Deployed API changes to warper.wmflabs.org

July

[edit]

Additional bug fixes and features:

  • Ensure API and web frontend works with HTTPS
  • OAuth Authentication forced to go via HTTPS
  • Update Rails version
  • Update Paperclip library
  • Added notice to users that logging in via mediawiki would also edit the map wiki page
  • Added check and notice if, when an import has been created, the pages within that do not have a Map template
  • Ensured that latitude and longitude are also populated, if they are missing from the Map template
  • Add custom basemap using ZXY tile format to the rectify base map
  • Use HTTP Caching for source (un warped) images so that the webserver and browser can speed up things
  • Use Redis for caching of rectified map images and mosaics with decaching upon every change
Retrieved from "https://commons.wikimedia.org/w/index.php?title=Commons:Wikimaps/IEG&oldid=203035041"