After a year of work for EU-OSHA Agency (European Agency for Safety & Health at Work), Zylk’s BI team Iñigo Sanchez, Sergio Argerey and Irune Prado, together with Bilbomatica’s team, has concluded a huge and ambitious project where dashboards are delivered as dynamic infographics, using Pentaho Community Dashboard Framework. The result is available in the following link
This project was born in order to satisfy Agency’s requirements of having a mixture of quantitative and qualitative data, where a report’s result is made live, for general public, in order to present its conclusions.
Some of the requirements were the following:
User-friendliness, being compatible with most recent browsers and devices.
Ability to manage a mixed-up composition of both quantitative and qualitative data
For qualitative data, features such as internationalization (i18n), localization (l10n), searching facilities, simplicity of maintenance and theming capabilities were highlighted.
In terms of quantitative data, possibility of transformation, diverse source origin management and format customization.
Availability of a wide graphics catalog, being able to configure them at the deepest detail, including event handling.
Exporting and sharing capabilities
Embedding options, being able to manage incoming and outcoming events.
To be built as a framework, in order to re-use it in future dashboards
SEO friendliness and traceability within Piwik tools
Other features; possibility to pre-configure guided “visual stories”, role definitions for different target users, …
Costs, maintenance and product share in the market
And of course, robustness, Maintainability, Scalability and Performance
A total of 24 requirements were defined with different weighing of importance. Pentaho was analysed, together with Tableau, EIGE Agency-like ad-hoc development and a Drupal ad-hoc development.
The conclusion was that Drupal was a good tool to manage qualitative data, while Pentaho for quantitative data, both of them having bad score in the other part. Mostly, due to the existence of Pentaho’s Community Dashboard Framework being promising enough, it was opted as winner, with very few points of difference.
Because the final goal was quite ambitious and the uncertainty/risk of not being able to achieve it was high, it was decided to make an initial pilot, in order to test the features of Pentaho satisfying the requirements defined during the previous phase.
The following modules were defined and tested during the pilot.
Core Module – As the glue of architecture that makes the dashboard possible. This included the definition of the technology to be used (Angular, Protovis, Pentaho, CDF/CDE, CDA, … modules).
Data Load Module – Based in Pentaho Data Integration for the required ETL transformations
Data Access Module – For the retrieval of the data from the defined database, for both quantitative and qualitative data, taking into consideration possible translations
Embedding Module – Embed the resulting dashboard inside a Drupal container emulating Agency’s corporate website.
Graphics and Maps Module
Exporting Module - both to data-type formats (CSV, XSL, … ) and image types (PNG, SVG, ...)
Sharing Module – For any user to be able to share the section of the dashboard it’s seeing, including it’s state, in terms of the different filters that may be available.
Tracking Module– with Piwik, as par of the Agency’s product stack.
No efforts were performed in terms of design, just the required developments for the client to visualize the capabilities, and have a better understatement of the scope and possibilities, within the defined cost of a pilot.
Having both the client and technical team comfortable enough, All Ages dashboard development started, where final data, graphics design, and all the little details were taken care of.
Investing in graphics ease of understatement was made imperative, leading to configurations such as fluidity, color-code, axis minimum/maximum and step definition. Pampering developments were performed, such as maximization options, where graphics functional and technical legend are presented.
Although Pentaho’s CDF’s graphics catalog is quite big, provided by Protovis library (from the author of D3) , the used graphics were reduced to the simplest, only requiring a few atypical ones, such as pyramid, combined type graphs, and radials.
In terms of qualitative data work, translation efforts were discarded for this iteration, in favor of using Google Translator, due to its results being decent enough.
Although new qualitative data uploads were made quick, it was evidenced that some edition facilities could come handy to the agency, giving them some level of autonomy.
We’ve learned A LOT from Pentaho’s Framework in terms of CDF usage, Protovis graphics and Javascript, and a huge list of good-practices and wishes are currently in our backlog for next developments (thx Taiga.IO and Bunt Planet for making our work easier).
Will let you explore it yourself, please feel free to ask anything in comments, could elaborate a more thorough post based on your inquiries
Show me the code! At EU-OSHA Github (currently in private repository, hope they will make it public at some time)
Other Similar projects
Other European Agencies efforts in order to bring statistical information to general public:
EU-OSHA
ESENER1 https://osha.europa.eu/en/surveys-and-statistics-osh/esener/2009
ESENER2 https://osha.europa.eu/en/surveys-and-statistics-osh/esener/2014
EIGE (European Institute for Gender Equality)
Gender Equality Index http://eige.europa.eu/gender-statistics/gender-equality-index
Gender Statistics Database http://eige.europa.eu/gender-statistics/dgs
Eurofound
European Working Condition Surveys http://www.eurofound.europa.eu/surveys/data-visualisation FRA (Fundamental Rights Agency) survey data explorer http://fra.europa.eu/en/publications-and-resources/data-and-maps/survey-data-explorers
FAO (Food and Agriculture Organization of the United Nations)
FAOSTAT (beta) http://faostat.fao.org/beta/en/#country/203
CEDEFOP (European Centre for the Development of Vocational Training)
Skills Panorama http://skillspanorama.cedefop.europa.eu/en
Institute of Health Metrics and Evaluation
http://vizhub.healthdata.org/gbd-compare/
Excelente artículo Irune, gracias por compartir!
