Digital business demands dictate an ever increasing need for the extraction of information and useful deductions from live/ real time data on the web. Be it manufacturing or sales or any other department, they all require to base their decisions on real time data. Building visualizations from static data however, is very simple. But this does not provide a solution, for example, to a sales management executive who wants to visualize real time sales by product category. This is where there arises a need for a method for extraction and visualization of live data. While SAP Lumira helps best with the visualization, we need a method for extracting this live data into SAP Lumira.
Import.io seems to be the best option to fit the requirement. It is free, easy to use and powerful with lot of features that we can use for extracting the data into SAP Lumira. Read on to find out how to scrape a website and bring in live data for creating visualizations.
I had come across this great blog post: http://scn.sap.com/community/lumira/blog/2014/06/08/using-importio-to-extract-any-data-from-the-web-into-lumira-shuffle-up-and-deal-edition by Ronald Konijnenburg about using import.io in Lumira to extract data from the web where you can follow to setup and expose Data as an API inside import.io
We would need to have live data. Every time we need Lumira to fetch the updated data from the website and so the CSV method would not work. The DA extensions of SAP Lumira would help us connect to the Data source. So let’s build a DA extension for SAP Lumira. For our simplicity of understanding, we are taking the same WSOP data from http://www.wsop.com/players/index.asp?pagenum=1
The process of DA extensions is simple enough to fetch the API and bring the data into Lumira. The data is converted into CSV and is then consumed by Lumira for Visualization. The extension is then installed into Lumira.
Figure 1: Process flow
The extension is built with Python. Requests module is used to send the request to the import.io API URL and the response is then used as data for SAP Lumira for the Visualization. Import.io parses pagination in website and returns data across as pages, so we can mention how many pages of data that we need. EasyGUI is used to capture that input from the user.
Figure 2: Parameters
Finally the Data is printed into Lumira that appears in dimensions and measures that can be used to build the Visualization.
Figure 3: Print data
You can then build the extension into binary file and use it in Lumira.
You could check out the following Lumira - Open Source Data Access Extensions to learn about DA extensions and developing DA extensions for SAP Lumira.
Installing and Executing the Extension
To find out how you can download and install this extension Please take a read @ Website to Visuals – Scraping Website int... | SCN