Scrapyd部署scrapy爬虫
Scrapyd部署scrapy爬虫, Scrapyd部署, Scrapyd经验。
1、sudo pip install scrapyd
2、sudo pip install scrapyd-client
3、切换目录到项目所在路径：
修改scrapy.cfg设置
|
1 2 3 |
[deploy:xxxx] url = http://192.168.2.239:6800 project = xxxspider |
发布工程到scrapyd
|
1 2 3 4 5 6 |
scrapyd-deploy xxxx -p xxxspider --version 54(version为版本号,可以方便控制爬虫版本) Packing version 1470208884 Deploying to project "xxxspider" in http://192.168.2.239:6800/addversion.json Server response (200): {"status": "ok", "project": "xxxspider", "version": "1470208884", "spiders": 18, "node_name": "ubuntu-16"} |
4、验证是否发布成功
|
1 |
scrapyd-deploy -L xxxx |
会输出类型“xxxspider”的回显
|
1 |
scrapyd-deploy -l |
会输出类似“target http://192.168.2.239:6800”的回显
5、启动爬虫
|
1 |
curl http://192.168.2.239:6800/schedule.json -d project=Crawler -d spider=CommonSpider |
4、实时监控爬虫
通过页面实时监控：http://192.168.2.239:6800
6、终止爬虫
|
1 |
curl http://192.168.2.239:6800/cancel.json -d project=Crawler -d job= 8270364f9d9811e5adbf000c29a5d5be |