2016 Individual Engagement Grant (IEG)
In 2016 the Wikimaps project was funded by an IEG grant from the Wikimedia Foundation.
See https://meta.wikimedia.org/wiki/Grants:IEG/Wikimaps_Warper_2.0 for details about the grant, to see the scope of work involved, to view the people and team and volunteers involved and to view reports.
IEG Project Goals
The Warper code will be refactored. User interface is separated from computation to allow more versatile use of the software. Operations inside the code are also separated to make possible developing individual parts separately. New functions will be added. Mass import of already rectified maps will come handy to GLAM users and can be made part of maps uploads. Category imports will be enabled. Developing the new functionalities will start before the project. The API will be enhanced to be the primary means of operating the software.
After this stage the project will be ready for the next phase where user interfaces will be remade. We decided to do this in steps because of time constraints, and to assure that the backend is ready when the front end is made.
Wikimedia Finland has an ongoing project for processing metadata for Commons uploads. Warper upload will be made an option in it. We are looking for Commons volunteers to participate in Commons integrations, coding gadgets and templates. Linking with Wikidata will be a future direction.
Design decisions will be based on earlier work in the Wikimaps project, and updated to reflect development in sister software such as the OSM iD editor.
This log will hopefully fulfil some of the requirements for reporting for the grant, provide community members with updates on the work being performed and hopefully be a useful learning resource. See https://meta.wikimedia.org/wiki/Grants:IEG/Wikimaps_Warper_2.0/Timeline for wider / official progress reporting on the grant. Logging by User Chippyy (talk) except where specified.
Previous recent things
- Oauth authentication - leading to the development of a Devise / Omniauth OAuth strategy for mediawiki. see blog post.
- Commons category import - a Category of maps can be imported all at once into the warper.
- Import of batch control points - a CSV of control points for many maps can be imported.
- Download of control points - control points can be easily downloaded for a map
- Bot edit of wiki page to update bounding box / extent and status - when warped, the warper will act as a bot and update the commons wiki page with the results of the bounding box.
- Wikimedia Labs system underwent a system wide reboot following a kernel vulnerability. After the reboot the warper application was offline. Passenger wouldn't start the server. It was taking 11 minutes to run "rake -T" in the command line! (It takes 23 seconds on a 3 year old Atom netbook in comparison!) The root cause was that after the reboot the NFS was slower than before - other apps and tools were experiencing issues. This was an issue because the Ruby environment was loaded via .rvm from a users home. And /home was on the NFS and not local to the virtual machine. A copy of the environment was made to the local /srv partition and symlinked from .rvm and led to "rake -T" taking 1 minute. Passenger gem was recompiled and updated. The new Passenger config was able to increase the timeout value. The application needed about 1-2 mins to start. Still too long. But much better than 11 minutes.
- Phabricator issue was created for this https://phabricator.wikimedia.org/T124538
- Strace and ruby-prof needs to be run to see where bottle necks are, and expert eyes to interpret the results. I think that further rvm work may also improve performance.
- IEG Milestone created on Github Issues. To add issues, bugs, features to the work plan
- Prioritising of issues. Conversations were had to prioritise initial work.
- Github Issues were cleaned and closed. Since many issues were created before we decided to have the warper as API and a separate project for the new front end.
- API & Feature documentation. More docs were created to help plan for the new API
- OAuth provider with 3rd party external JS app researched and investigated
- looking at https://github.com/lynndylanhurley/j-toker which talks to a server running: https://github.com/lynndylanhurley/devise_token_auth
- jtoken-rails-app.herokuapp.com and http://jtoker-frontend-app.herokuapp.com/ set up by us to test strategy
- you can run the j-toker app locally and configure (in config/default.yml) to " apiUrl: '//jtoken-rails-app.herokuapp.com' "
- this rails app will behave as the warper and uses github auth
- CORS would have to be enabled, or a proxy set up on the new UI frontend
- Issue set up to create a bare bones front end to test the API, authentication etc.
- Fix for a YQL best place service breaking when encountering special characters in wiki / maps titles.
- British Library maps of the War Office in Africa were discussed on the OSM-GB mailing list https://lists.openstreetmap.org/pipermail/talk-gb/2016-February/thread.html and the wikimaps warper was mentioned and introuced. The maps had been added to the wikimaps warper previously however there was an issue with them not being in the actual category.
- An issue to fix the import of maps was created and made a priority so that when a category is re-imported any existing maps can be added to that mosaic.
- Further investigation as to NFS and slow startup
- Appears there is confirmation as to issue with NFS in labs
- Using Strace and Ruby-prof
- Blog post of lessons learnt on User:Chippyy personal blog: https://thinkwhere.wordpress.com/2016/03/04/strace-and-ruby-prof-to-identify-slow-rails-startup-on-wikimaps/
- Fixed issue with re-imports for the British Library war office maps
- New issue of resolving thumbnails created
- Prioritisation of issues ongoing
- Work commenced on implementing new Map template
- Wiki-markup code refactored.
- Work towards better wiki text Template / Parameter parsing in Ruby https://github.com/timwaters/mediawiki-template-parse
- Created Phabricator project for wikimaps-warper infrastructure, server issues: https://phabricator.wikimedia.org/T130603
- Identified bug with beta cluster API calls via OAuth authentication (blocking work on new template)
- https://phabricator.wikimedia.org/T130611 "500 status responses and fatal errors for OAuth authenticated API requests on beta servers"
- Added code to gracefully fail when theres trouble with the API like this
- Fixed bug where non logged in users would see an error when viewing an unloaded map
- Fixed bug where map titles had apostrophes escaped
- Spotted increased load from http://graphite.wmflabs.org/dashboard/#maps_warper1 due to press from http://www.eldiario.es/hojaderouter/internet/Wikipedia-mapas-cartografia-creative_commons_0_494851710.html
- TODO Need to add some simple analytics to determine source of spikes
- Fixed bug where long page titles were causing errors when adding new maps
- Oauth bug on beta cluster was fixed, so progress on new template can go ahead
- Fixed bug where layer/mosaic wms layers were not being shown on the to map in the rectify main screen
- Investigated bug with thumbnails in Beta Commons https://phabricator.wikimedia.org/T131472
- Further work towards better handling of thumbnails
- Fixed bug where sorting a maps mosaics by the percentage complete column caused an error
- Set to enabled as default for the "add to new category" checkbox option in the Commons import form
- Improved code to handle editing and updating wikitext automatically (pre-requisite for new Map template)
- Started work on new Map Template
- Created new branch https://github.com/wikimaps-dev/mapwarper/tree/69template for this feature
- Deployed to staging
- Updated warper.wmflabs.org with previous bug fixes
- started work on api
- Trying rails-api gem
- Created new api_wikimaps branch for API development
- Installed FactoryGirl to attempt basic unit tests towards good controller API tests
- using JSON API for output response format http://jsonapi.org/
- Patched Imagemagick on server to workaround vulnerability
- Created issue to update paperclip gem
- Created issue to update rails
- Working on Api_template branch
- using rails-api and activemodel serialisers
- developing geojson adapter for activemodel serialisers
- New maps api
- new layers api
- new ground control points api
- Merged in code for geosearch and index so that it will work with just one endpoint now
- Fixed bug with sorting a layers maps by percent
- Adding majority of CRUD actions, adding links to JSON output
- Added basic GeoJSON output format for some Map and Layer outputs
- Added new token based authentication for API
- Updated server for another imagemagick vuln
- Developed simple POC app for interacting with new authentication and API https://github.com/timwaters/warper_oauth_token_auth_demo
- Implemented json-api deserialization (changed API input format from old school Rails JSON format to New Improved json-api formats)
- Custom Validation of type value for each method is correct using json-api
- Added error responses for model Not Found
- Plenty more controller api tests
- Markdown API docs
- curl examples of calls
- added CORS configuration for /api/v1 calls
- added authentication by user and password
- added sign out method which disables token validity
- Deployed api instance to staging server
- Added API Docs to /api/v1 path
- Merged in the api branch into the main (wikimaps) branch
- Deployed API changes to warper.wmflabs.org
Additional bug fixes and features:
- Ensure API and web frontend works with HTTPS
- OAuth Authentication forced to go via HTTPS
- Update Rails version
- Update Paperclip library
- Added notice to users that logging in via mediawiki would also edit the map wiki page
- Added check and notice if, when an import has been created, the pages within that do not have a Map template
- Ensured that latitude and longitude are also populated, if they are missing from the Map template
- Add custom basemap using ZXY tile format to the rectify base map
- Use HTTP Caching for source (un warped) images so that the webserver and browser can speed up things
- Use Redis for caching of rectified map images and mosaics with decaching upon every change