We are going to scrape the FT.com website for current stock prices for selected stocks. You’ll learn:
1# import 'urllib2' to open the url and 'sys' so we could take input from console
2import urllib2, sys
3
4# take input from the console and save it as 'symbol'
5symbol = sys.argv[1]
6
7# we are using FT.com for the stock pricing
8url = 'http://markets.ft.com/research/markets/Tearsheets/Summary?s='
9
10# open the URL, and save the output in a variable called 'content'
11content = urllib2.urlopen(url+symbol)
12# read the 'content' and save it in a variable called 'data'
13data = content.read()
14
15print(data)
We are going to import urllib2 and sys so we could a) open a url and b) take input from console. Next, we’ll take input (i.e the stock symbol we want to look up) from console and save it in a variable called symbol. sys.argv[1] basically means the first argument that was given to the system. The URL we are going to scrape is from the Financial Times website. We’ll save that url in a variable aptly names url. Then, we are going to open that URL and save it’s content in a variable called content. We’ll then read that content and save whatever it read in a variable called data.
Lastly, we’ll print that data to make sure everything is working. At this point it’ll output a whole lot of jumbled up code in your terminal (which is basically what you’ll get if you ‘View Page Source’ in the browser)
So you have ALL the data on the page now.. What we really want is just the latest stock price. Say hello to Regex! Regex (aka Regular Expressions) let’s you define an expression and look up that expression in a whole lot of data to get what you want. For a layman, it is the equivalent of doing Ctrl+F in the browser but way more powerful and complex.
But first, we need to know what we are looking for. To do that, i’m going to save the data in a text file, do a search for the price and check for something unique that represents that price so i can look for it every time with the regex i write.
Use the >> redirection operator in the shell to redirect the results (output) of whatever we get after running stock_portfolio.py and save that in a text file called output.txt You need to run the following commadn in the Terminal (console).
python stock_portfolio.py >> output.txt
Now open that output in your favorite code editor. The reason for opening it in a code editor is so that i can view it with syntax highlighting, makes it easier to make sense of the file and look for what we want. Start with searching (ctrl+F) for the exact stock price. Then look for code before and after that stock price that makes it unique.
For example, i looked up the stock APL, i know the stock price was 593.76 at the time i extracted the data so i searched for ‘593.76’ in the output. That gave me the position in the file where the latest stock price is mentioned. Then i looked for unique identifiers before and after the price and found that _last_lastPrice" data-stream-feed="rmds_streamer"> was always mentioned exactly before the latest price and it was only used for the price and nothing else. So, i used that to base my regular expression on.
To be able to use regular expressions in our script, we’ll import the re library.
1import urllib2, sys, re
2
3symbol = sys.argv[1]
4
5url = 'http://markets.ft.com/research/markets/Tearsheets/Summary?s='
6content = urllib2.urlopen(url+symbol)
7data = content.read()
8
9# Regex
10# 'm' is for match, it is frequently used to represent return of a match
11m = re.search('_last_lastPrice" data-stream-feed="rmds_streamer">(\d*[.]\d*)', data)
12# we are searching for the latest stock price (\d*[.]\d*) placed after a specific span tag, in whatever is stored in 'data'.
13
14# if we have a match, then quote is equal to the first group we looked up. the () represent a group in regex.
15quote = m.group(1)
16
17print(quote)
We are running the search method on the re module to search for _last_lastPrice" data-stream-feed="rmds_streamer">(\d*[.]\d*) where (\d*[.]\d*) represents the format of the price 593.76 ( \d is for digits and ***** means all ). Look for this price, it comes immidiately after _last_lastPrice" data-stream-feed="rmds_streamer"> and look for it in the data variable. Save the result of the search in a variable called m.
Use RegExr to find exact match in text and learn about writing regex queries. Getting it right may take you a few tries if you are new to regex. It took me three to get the expression right. This might be the most difficult part of this whole tutorial if you have never used regex before.
At this point the result of print(m) would be <_sre.SRE_Match object at 0x1086c1990> which basically means that it searched and a match exists. Now we save the result of the match in a variable called quote.
And finally, print that quote. print(quote)
593.76
What if the symbol given has a typo, or is incorrect or doesn’t exist? Python by defualt will stop running the script after the error and if you had any other symbols after the error they won’t show since the script stopped running.
To counter that, we are going to add an if/else statement so that if the value exists, it is shown and if not, it shows an error message. In case of an error, the script will show an error message and keep running.
1if m:
2 # if we have a match, then quote is equal to the first group we looked up.
3 quote = m.group(1)
4else:
5 quote = symbol + ' is not a correct symbol or it does not exist'
6
7print(quote)
instead of taking input from the system, i’m going to save the multiple symbols i want to look up in a list. that way i won’t have to type the symbols every time i want to look up the prices.
1# SYMBOLS TO LOOK FOR:
2symbol_list = ['APL', 'PPL', 'HUBC', 'FFC']
3
4# for every Symbol in the list, look it up and print the result
5for symbol in symbol_list:
6 url = 'http://markets.ft.com/research/markets/Tearsheets/Summary?s='+symbol+':KAR'
7 # (FT.com adds :KAR at the end of symbols in KSE)
8 # i'm permanently adding it to the URL because i'm only going to be looking up KSE stocks
9 # and i am too lazy to add :KAR with every symbol manually
10 content = urllib2.urlopen(url).read()
11
12 m = re.search('_last_lastPrice" data-stream-feed="rmds_streamer">(\d*[.]\d*)', content)
13 if m:
14 quote = m.group(1)
15 print(symbol + ' ' + quote)
16 else:
17 quote = symbol + " is not a correct symbol or it doesn't exist"
18 print(quote)
19 ```
20
21
22Final script
23---
24
25```python
26import urllib2, re
27# urllib2 is required to open the url
28# re is required for regular expression (regex)
29
30# SYMBOLS TO LOOK FOR:
31symbol_list = ['APL', 'PPL', 'HUBCO', 'EFERT']
32
33# for every Symbol in the list, look it up and print the result
34for symbol in symbol_list:
35 url = 'http://markets.ft.com/research/markets/Tearsheets/Summary?s='+symbol+':KAR'
36 # (FT.com adds :KAR at the end of symbols in KSE)
37 content = urllib2.urlopen(url).read()
38
39 m = re.search('_last_lastPrice" data-stream-feed="rmds_streamer">(\d*[.]\d*)', content)
40 if m:
41 quote = m.group(1)
42 print(symbol + ' ' + quote)
43 else:
44 quote = symbol + ' is not a correct symbol or it doesn't exist'
45 print(quote)