Friday 28 October 2016

Tapping The Mining Services Goldmine

Tapping The Mining Services Goldmine

In Australia, resources booms tend to come and go. In a recent speech, Reserve Bank Deputy Governor Ric Battellino identified five major booms over the last two hundred years - from the gold rush of the 1850s, to our current minerals and energy boom.

Many have argued that the current boom is different from anything we've experienced before, with the modernisation of the Chinese and Indian economies likely to keep demand high for decades. That's led some analysts to talk of a resources supercycle. And yet a supercycle is still a cycle.

By definition, cycles are uneven, with commodity prices ebbing and flowing in response to demand, economic conditions and market sentiment. And the share prices of resources companies tend to move with them.

Which raises the question: what's the best way for investors to tap into the potential of the mining boom, without the heart-stopping volatility that mining stocks sometimes deliver?
Invest in the store that sells the spade

Legend has it that the people who really profited from Australia's gold rush weren't the miners who flocked to the fields, but the store-owners who sold them their spades and pans. You can put the same principle to work today by investing in mining services and engineering companies.

Here are five reasons to consider giving mining services companies a place in your portfolio:

1. Growing demand

In November, the Australian Bureau of Agricultural and Resource Economics reported that mining and energy companies plan to invest a record $132.9bn in new projects, a 58% increase from the previous year. That includes 72 projects at an advanced stage of development, such as the $43bn Gorgon LNG project and the $20bn Olympic dam expansion. The mining services sector is poised to benefit from all of them.

The sector also stands to benefit from Australia's worsening skills shortage, with more companies looking to contractors to provide essential services in remote locations.

2. Less volatility

Resource stocks tend to fluctuate with commodity prices, which are subject to international economic forces and market sentiment beyond the control of any individual company. As a result, they are among the most volatile companies on the Australian sharemarket. But mining services stocks, while still exposed to the commodities cycle, tend to be more stable.

3. More predictable cash flow

One reason for the comparative volatility of commodity companies is that their cash flow can be very variable. In the development phase, they need to make significant capital expenditure, often leading to negative cash flows. And while they enjoy healthy revenues in the production phase, that revenue may diminish as a resource is exhausted, unless they make further investments in exploration and development.
In contrast, mining services companies require comparatively little capital investment, with more predictable cash flows over the long-term.

4. Higher dividends

Predictable cash flows and lower capital expenditures often allow services companies to pay out more of their earnings as dividends, making them more appealing for income-oriented investors.

5. No need to pick winners

Many miners are highly leveraged to demand for a single commodity, whether it's gold, coal, copper or iron ore. Some are reliant on a single mine or field. Whereas services companies generally have a more diversified customer base.

Source: http://ezinearticles.com/?Tapping-The-Mining-Services-Goldmine&id=5924837

Monday 17 October 2016

How Web Scraping Affects your Revenue Growth

How Web Scraping Affects your Revenue Growth

Web scraping is an indispensable resource when it comes to gaining an edge in the competition with the help of business intelligence. As more and more data gets created on the world wide web, the complexity of extracting it intensifies. Web scraping is a technology that demands an extensive tech stack, high end resources and technically skilled labour. Given this resource hungry nature, many businesses prefer outsourcing it to doing the scraping in-house. Here is a brief walk-through of web scraping so that you can get a grip on the whole process and understand how it could affect your revenue growth as a business.

Business intelligence

The competition among online businesses is at its peak. This has more to do with the ready availability of insightful data. When data acquisition at this scale wasn’t possible in the past, businesses made hit-or-miss decisions upon instincts. Now that every activity can be recorded, extracted as data and analysed to arrive at the best business decisions, companies are making the most of it to boost their revenue. This includes monitoring the activity of competitors on social media, price intelligence, sentiment analysis, gathering data for market research and much more. The use cases of web scraping in business is almost infinite. Business intelligence is extremely helpful for the survival of companies in a market that fluctuates often. Implementing a business intelligence strategy powered by web scraping can definitely give a boost to your revenue growth.
Cost centres involved in in-house Web Scraping

Web scraping, despite being a robust solution for extracting data from the web, is not going to be an easy path if your company is not technically rich already. It involves setting up resources like a tech stack and servers that can run the web crawler by a technically skilled team. Following are the primary cost centres involved in the web scraping process.

1. High end servers

Web scraping is a resource intensive process. Considering the importance of uptime here, the crawlers cannot be run on average performance machines. To have the optimum uptime and avoid crashes, the crawler has to be run on high performing servers located in different parts of the world. The quality of servers is crucial to the consistency of the process. Not to mention, these high end servers makeup for a significant amount of the cost involved in web scraping.

2. Technically skilled labour

Scanning through the source code to identify appropriate tags that hold the required data points and creating a program that can automatically fetch these data points from similar pages’ at large scale requires deep programming skills. It goes without saying that employing skilled people would incur cost that could take a hit on your revenue. Ideally, you will need a team of at least 10 to run a web scraping setup in-house.       

3. An extensive tech stack

Although most of the software being used for web scraping are open source, you will find yourself investing in paid software to make certain things easier or faster. Dealing with open source software might not be as user friendly as the paid ones. In any case, having a tech stack with a lot of options is a necessary aspect of web scraping that would incur additional cost.   

4. Maintenance

Building and running the web scraping setup is only half of the story. Since websites undergo changes often, there is a possibility of the crawler setup breaking from time to time. To avoid or solve this at the earliest, a monitoring system that involves both machines and humans is necessary. Monitoring and maintenance contribute to a considerable cost in the web scraping process.
Data as a service

If data for business is your requirement, a better way to acquire it would be to depend on a company that can deliver it via the data as a service route. Web scraping companies have already set up high-end resources required to run the web crawlers that you can utilize to avail web scraping at a much lower cost than what you would incur by doing it on your own. With this, you can also save yourself from the complications and maintenance headache associated with web scraping. Moreover, with a web scraping service, you can enjoy a much higher return on investment owing to the lowered cost of data acquisition. You can use our ROI calculator to compare between the cost of going with an in-house web scraping setup and a hosted solution.

Source: https://www.promptcloud.com/blog/web-scraping-affects-revenue-growth

Sunday 16 October 2016

Web Scraping with Python: A Beginner’s Guide

Web Scraping with Python: A Beginner’s Guide

In the Big Data world, Web Scraping or Data extraction services are the primary requisites for Big Data Analytics. Pulling up data from the web has become almost inevitable for companies to stay in business. Next question that comes up is how to go about web scraping as a beginner.

Data can be extracted or scraped from a web source using a number of methods. Popular websites like Google, Facebook, or Twitter offer APIs to view and extract the available data in a structured manner.  This prevents the use of other methods that may not be preferred by the API provider. However, the demand to scrape a website arises when the information is not readily offered by the website. Python, an open source programming language is often used for Web Scraping due to its simple and rich ecosystem. It contains a library called “BeautifulSoup” which carries on this task. Let’s take a deeper look into web scraping using python.

Setting up a Python Environment:

To carry out web scraping using Python, you will first have to install the Python Environment, which enables to run code written in the python language. The libraries perform data scraping;

Beautiful Soup is a convenient-to-use python library. It is one of the finest tools for extracting information from a webpage. Professionals can scrape information from web pages in the form of tables, lists, or paragraphs. Urllib2 is another library that can be used in combination with the BeautifulSoup library for fetching the web pages. Filters can be added to extract specific information from web pages. Urllib2 is a Python module that can fetch URLs.

For MAC OSX :

To install Python libraries on MAC OSX, users need to open a terminal win and type in the following commands, single command at a time:

sudoeasy_install pip

pip install BeautifulSoup4

pip install lxml

For Windows 7 & 8 users:

Windows 7 & 8 users need to ensure that the python environment gets installed first. Once, the environment is installed, open the command prompt and find the way to root C:/ directory and type in the following commands:

easy_install BeautifulSoup4

easy_installlxml

Once the libraries are installed, it is time to write data scraping code.

Running Python:

Data scraping must be done for a distinct objective such as to scrape current stock of a retail store. First, a web browser is required to navigate the website that contains this data. After identifying the table, right click anywhere on it and then select inspect element from the dropdown menu list. This will cause a window to pop-up on the bottom or side of your screen displaying the website’s html code. The rankings appear in a table. You might need to scan through the HTML data until you find the line of code that highlights the table on the webpage.

Python offers some other alternatives for HTML scraping apart from BeautifulSoup. They include:

    Scrapy
    Scrapemark
    Mechanize

 Web scraping converts unstructured data from HTML code into structured form such as tabular data in an Excel worksheet. Web scraping can be done in many ways ranging from the use of Google Docs to programming languages. For people who do not have any programming knowledge or technical competencies, it is possible to acquire web data by using web scraping services that provide ready to use data from websites of your preference.

HTML Tags:

To perform web scraping, users must have a sound knowledge of HTML tags. It might help a lot to know that HTML links are defined using anchor tag i.e. <a> tag, “<a href=“http://…”>The link needs to be here </a>”. An HTML list comprises <ul> (unordered) and <ol> (ordered) list. The item of list starts with <li>.

HTML tables are defined with<Table>, row as <tr> and columns are divided into data as <td>;

    <!DOCTYPE html> : A HTML document starts with a document type declaration
    The main part of the HTML document in unformatted, plain text is defined by <body> and </body> tags
    The headings in HTML are defined using the heading tags from <h1> to <h5>
    Paragraphs are defined with the <p> tag in HTML
    An entire HTML document is contained between <html> and </html>

Using BeautifulSoup in Scraping:

While scraping a webpage using BeautifulSoup, the main concern is to identify the final objective. For instance, if you would like to extract a list from webpage, a step wise approach is required:

    First and foremost step is to import the required libraries:

 #import the library used to query a website

import urllib2

#specify the url wiki = “https://”

#Query the website and return the html to the variable ‘page’

page = urllib2.urlopen(wiki)

#import the Beautiful soup functions to parse the data returned from the website

from bs4 import BeautifulSoup

#Parse the html in the ‘page’ variable, and store it in Beautiful Soup format

soup = BeautifulSoup(page)

    Use function “prettify” to visualize nested structure of HTML page
    Working with Soup tags:

Soup<tag> is used for returning content between opening and closing tag including tag.

    In[30]:soup.title

 Out[30]:<title>List of Presidents in India till 2010 – Wikipedia, the free encyclopedia</title>

    soup.<tag>.string: Return string within given tag
    In [38]:soup.title.string
    Out[38]:u ‘List of Presidents in India and Brazil till 2010 in India – Wikipedia, the free encyclopedia’
    Find all the links within page’s <a> tags: Tag a link using tag “<a>”. So, go with option soup.a and it should return the links available in the web page. Let’s do it.
    In [40]:soup.a

Out[40]:<a id=”top”></a>

    Find the right table:

As a table to pull up information about Presidents in India and Brazil till 2010 is being searched for, identifying the right table first is important. Here’s a command to scrape information enclosed in all table tags.

all_tables= soup.find_all(‘table’)

Identify the right table by using attribute “class” of table needs to filter the right table. Thereafter, inspect the class name by right clicking on the required table of web page as follows:

    Inspect element
    Copy the class name or find the class name of right table from the last command’s output.

 right_table=soup.find(‘table’, class_=’wikitable sortable plainrowheaders’)

right_table

That’s how we can identify the right table.

    Extract the information to DataFrame: There is a need to iterate through each row (tr) and then assign each element of tr (td) to a variable and add it to a list. Let’s analyse the Table’s HTML structure of the table. (extract information for table heading <th>)

To access value of each element, there is a need to use “find(text=True)” option with each element.  Finally, there is data in dataframe.

There are various other ways to scrape data using “BeautifulSoup” that reduce manual efforts to collect data from web pages. Code written in BeautifulSoup is considered to be more robust than the regular expressions. The web scraping method we discussed use “BeautifulSoup” and “urllib2” libraries in Python. That was a brief beginner’s guide to start using Python for web scraping.

Source: https://www.promptcloud.com/blog/web-scraping-python-guide

Monday 3 October 2016

Assuring Scraping Success with Proxy Data Scraping

Have you ever heard of "Data Scraping?" Data Scraping is the process of collecting useful data that has been placed in the public domain of the internet (private areas too if conditions are met) and storing it in databases or spreadsheets for later use in various applications. Data Scraping technology is not new and many a successful businessman has made his fortune by taking advantage of data scraping technology.

Sometimes website owners may not derive much pleasure from automated harvesting of their data. Webmasters have learned to disallow web scrapers access to their websites by using tools or methods that block certain ip addresses from retrieving website content. Data scrapers are left with the choice to either target a different website, or to move the harvesting script from computer to computer using a different IP address each time and extract as much data as possible until all of the scraper's computers are eventually blocked.

Thankfully there is a modern solution to this problem. Proxy Data Scraping technology solves the problem by using proxy IP addresses. Every time your data scraping program executes an extraction from a website, the website thinks it is coming from a different IP address. To the website owner, proxy data scraping simply looks like a short period of increased traffic from all around the world. They have very limited and tedious ways of blocking such a script but more importantly -- most of the time, they simply won't know they are being scraped.

You may now be asking yourself, "Where can I get Proxy Data Scraping Technology for my project?" The "do-it-yourself" solution is, rather unfortunately, not simple at all. Setting up a proxy data scraping network takes a lot of time and requires that you either own a bunch of IP addresses and suitable servers to be used as proxies, not to mention the IT guru you need to get everything configured properly. You could consider renting proxy servers from select hosting providers, but that option tends to be quite pricey but arguably better than the alternative: dangerous and unreliable (but free) public proxy servers.

There are literally thousands of free proxy servers located around the globe that are simple enough to use. The trick however is finding them. Many sites list hundreds of servers, but locating one that is working, open, and supports the type of protocols you need can be a lesson in persistence, trial, and error. However if you do succeed in discovering a pool of working public proxies, there are still inherent dangers of using them. First off, you don't know who the server belongs to or what activities are going on elsewhere on the server. Sending sensitive requests or data through a public proxy is a bad idea. It is fairly easy for a proxy server to capture any information you send through it or that it sends back to you. If you choose the public proxy method, make sure you never send any transaction through that might compromise you or anyone else in case disreputable people are made aware of the data.

A less risky scenario for proxy data scraping is to rent a rotating proxy connection that cycles through a large number of private IP addresses. There are several of these companies available that claim to delete all web traffic logs which allows you to anonymously harvest the web with minimal threat of reprisal. Companies such as offer large scale anonymous proxy solutions, but often carry a fairly hefty setup fee to get you going.

Source:http://ezinearticles.com/?Assuring-Scraping-Success-with-Proxy-Data-Scraping&id=248993