Fix Python – Can scrapy be used to scrape dynamic content from websites that are using AJAX?

I have recently been learning Python and am dipping my hand into building a web-scraper. It’s nothing fancy at all; its only purpose is to get the data off of a betting website and have this data put into Excel.
Most of the issues are solvable and I’m having a good little mess around. However I’m hitting a massive hurdle over one issue. If a site….

Fix Python – “OSError: [Errno 1] Operation not permitted” when installing Scrapy in OSX 10.11 (El Capitan) (System Integrity Protection)

I’m trying to install Scrapy Python framework in OSX 10.11 (El Capitan) via pip. The installation script downloads the required modules and at some point returns the following error:
OSError: [Errno 1] Operation not permitted: ‘/tmp/pip-nIfswi-uninstall/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/six-1.4.1-py2.7.egg-i….

Fix Python – Scraping: SSL: CERTIFICATE_VERIFY_FAILED error for http://en.wikipedia.org

I’m practicing the code from ‘Web Scraping with Python’, and I keep having this certificate problem:
from urllib.request import urlopen
from bs4 import BeautifulSoup
import re

pages = set()
def getLinks(pageUrl):
global pages
html = urlopen(“http://en.wikipedia.org”+pageUrl)
bsObj = BeautifulSoup(html)
for link in bsObj.findAll(….

Fix Python – Cannot install Lxml on Mac OS X 10.9

I want to install Lxml so I can then install Scrapy.
When I updated my Mac today it wouldn’t let me reinstall lxml, I get the following error:
In file included from src/lxml/lxml.etree.c:314:
/private/tmp/pip_build_root/lxml/src/lxml/includes/etree_defs.h:9:10: fatal error: ‘libxml/xmlversion.h’ file not found
#include “libxml/xmlversion.h”
….