CentOS7分布式部署pyspider

Original howie6879 老胡的储物柜 2022-06-01

1.搭建环境：

系统版本：`Linux centos-linux
python版本： Python3.5.1

1.1.搭建python3环境：

本人在尝试过后选择集成环境Anaconda

1.1.1.编译

# 下载依赖
yum install -y ncurses-devel openssl openssl-devel zlib-devel gcc make glibc-devel libffi-devel glibc-static glibc-utils sqlite-devel readline-devel tk-devel gdbm-devel db4-devel libpcap-devel xz-deve
# 下载python版本
wget https://www.python.org/ftp/python/3.5.1/Python-3.5.1.tgz
# 或者使用国内源
wget http://mirrors.sohu.com/python/3.5.1/Python-3.5.1.tgz
mv Python-3.5.1.tgz /usr/local/src;cd /usr/local/src
# 解压
tar -zxf Python-3.5.1.tgz;cd Python-3.5.1
# 编译安装
./configure --prefix=/usr/local/python3.5 --enable-shared
make && make install
# 建立软链接
ln -s /usr/local/python3.5/bin/python3 /usr/bin/python3
echo "/usr/local/python3.5/lib" > /etc/ld.so.conf.d/python3.5.conf
ldconfig
# 验证python3
python3
# Python 3.5.1 (default, Oct 9 2016, 11:44:24)
# [GCC 4.8.5 20150623 (Red Hat 4.8.5-4)] on linux
# Type "help", "copyright", "credits" or "license" for more information.
# >>>
# pip
/usr/local/python3.5/bin/pip3 install --upgrade pip
ln -s /usr/local/python3.5/bin/pip /usr/bin/pip
# 本人在安装时出现问题将pip重装
wget https://bootstrap.pypa.io/get-pip.py --no-check-certificate
python get-pip.py

1.1.2.集成环境anaconda

# 集成环境anaconda(推荐)
wget https://repo.continuum.io/archive/Anaconda3-4.2.0-Linux-x86_64.sh
# 直接安装即可
./Anaconda3-4.2.0-Linux-x86_64.sh
# 若出错，可能是解压失败
yum install bzip2

1.2.安装mariaDB

# 安装
yum -y install mariadb mariadb-server
# 启动
systemctl start mariadb
# 设置为开机启动
systemctl enable mariadb
# 配置密码默认为空
mysql_secure_installation
# 登录
mysql -u root -p
# 创建一个用户自己设定账户密码
CREATE USER 'user_name'@'localhost' IDENTIFIED BY 'user_pass';
GRANT ALL PRIVILEGES ON *.* TO 'user_name'@'localhost' WITH GRANT OPTION;
CREATE USER 'user_name'@'%' IDENTIFIED BY 'user_pass';
GRANT ALL PRIVILEGES ON *.* TO 'user_name'@'%' WITH GRANT OPTION;

1.3.安装pyspider

本人使用 Anaconda

# 搭建虚拟环境sbird python版本3.*
conda create -n sbird python=3*
# 进入环境
source activate sbird
# 安装pyspider
pip install pyspider
# 报错
# it does not exist. The exported locale is "en_US.UTF-8" but it is not supported
# 执行可写入.bashrc
export LC_ALL=en_US.utf-8
export LANG=en_US.utf-8
#ImportError: pycurl: libcurl link-time version (7.29.0) is older than compile-time version (7.49.0)
conda install pycurl
# 退出
source deactivate sbird
# 若在虚拟机内出现无法访问localhost:5000 可关闭防火墙
systemctl stop firewalld.service
#########直接运行源码==============
mkdir git;cd git
# 下载
git clone https://github.com/binux/pyspider.git
# 安装
/root/anaconda3/envs/sbird/bin/python /root/git/pyspider/run.py

其他方法

# 搭建虚拟环境
pip install virtualenv
mkdir python;cd python
# 创建虚拟环境pyenv3
virtualenv -p /usr/bin/python3 pyenv3
# 进入虚拟环境激活环境
cd pyenv3/
source ./bin/activate
pip install pyspider
# 若pycurl报错
yum install libcurl-devel
# 继续
pip install pyspider
# 关闭
deactivate

本人推荐用anaconda方式安装

若pyspider运行过程中出现错误，参考 anaconda安装部分，至此，访问 localhost:5000可看到页面。

1.4.安装Supervisor

# 安装
yum install supervisor -y
# 若无法检索则添加阿里的epel源
vim /etc/yum.repos.d/epel.repo
# 添加以下内容
[epel]
name=Extra Packages for Enterprise Linux 7 - $basearch
baseurl=http://mirrors.aliyun.com/epel/7/$basearch
http://mirrors.aliyuncs.com/epel/7/$basearch
failovermethod=priority
enabled=1
gpgcheck=0
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-EPEL-7
[epel-debuginfo]
name=Extra Packages for Enterprise Linux 7 - $basearch - Debug
baseurl=http://mirrors.aliyun.com/epel/7/$basearch/debug
http://mirrors.aliyuncs.com/epel/7/$basearch/debug
failovermethod=priority
enabled=0
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-EPEL-7
gpgcheck=0
[epel-source]
name=Extra Packages for Enterprise Linux 7 - $basearch - Source
baseurl=http://mirrors.aliyun.com/epel/7/SRPMS
http://mirrors.aliyuncs.com/epel/7/SRPMS
failovermethod=priority
enabled=0
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-EPEL-7
gpgcheck=0
# 安装
yum install supervisor -y
# 测试是否安装成功
echo_supervisord_conf

1.4.1.Supervisor用法

supervisord #supervisor的服务器端部分启动
supervisorctl #启动supervisor的命令行窗口
# 假设创建进程pyspider01
vim /etc/supervisord.d/pyspider01.ini
# 写入以下内容
[program:pyspider01]
command = /root/anaconda3/envs/sbird/bin/python /root/git/pyspider/run.py
directory = /root/git/pyspider
user = root
process_name = %(program_name)s
autostart = true
autorestart = true
startsecs = 3
redirect_stderr = true
stdout_logfile_maxbytes = 500MB
stdout_logfile_backups = 10
stdout_logfile = /pyspider/supervisor/pyspider01.log
# 重载
supervisorctl reload
# 启动
supervisorctl start pyspider01
# 也可这样启动
supervisord -c /etc/supervisord.conf
# 查看状态
supervisorctl status
# output
pyspider01 RUNNING pid 4026, uptime 0:02:40
# 关闭
supervisorctl shutdown

1.5.安装redis

# 消息队列采用redis
mkdir download;cd download
wget http://download.redis.io/releases/redis-3.2.4.tar.gz
tar xzf redis-3.2.4.tar.gz
cd redis-3.2.4
make
# 或者直接yum安装
yum -y install redis
# 启动
systemctl start redis.service
# 重启
systemctl restart redis.service
# 停止
systemctl stop redis.service
# 查看状态
systemctl status redis.service
# 更改文件/etc/redis.conf
vim /etc/redis.conf
# 更改内容
daemonize no 改为 daemonize yes
bind 127.0.0.1 改为 bind 10.211.55.22(当前服务器ip)
# 重启redis
systemctl restart redis.service

1.6.关于自启动

# Supervisor添加到自启动服务
systemctl enable supervisord.service
# redis添加到自启动服务
systemctl enable redis.service
# 关闭防火墙自启动
systemctl disable firewalld.service

至此，pyspider单个服务器运行环境搭建且部署完毕，启动 localhost:5000进入web界面。

也可编写脚本运行，在 /pyspider/supervisor/pyspider01.log查看运行状态。

2.分布式部署

刚才配置的服务器，将其命名为 centos01，按照这样的配置，再分别部署两台 centos02、centos03。

如下：

服务器名称	ip	说明
centos01	10.211.55.22	redis,mariaDB, scheduler
centos02	10.211.55.23	fetcher, processor, result_worker,phantomjs
centos03	10.211.55.24	fetcher, processor,,result_worker,webui

2.1.centos01

进入服务器 centos01，经过第一步，基本环境已经搭好，首先编辑配置文件 /pyspider/config.json

{
"taskdb": "mysql+taskdb://user_name:user_pass@10.211.55.22:3306/taskdb",
"projectdb": "mysql+projectdb://user_name:user_pass@10.211.55.22:3306/projectdb",
"resultdb": "mysql+resultdb://user_name:user_pass@10.211.55.22:3306/resultdb",
"message_queue": "redis://10.211.55.22:6379/db",
"logging-config": "/pyspider/logging.conf",
"phantomjs-proxy":"10.211.55.23:25555",
"webui": {
"username": "",
"password": "",
"need-auth": false,
"host":"10.211.55.24",
"port":"5000",
"scheduler-rpc":"http:// 10.211.55.22:5002",
"fetcher-rpc":"http://10.211.55.23:5001"
},
"fetcher": {
"xmlrpc":true,
"xmlrpc-host": "0.0.0.0",
"xmlrpc-port": "5001"
},
"scheduler": {
"xmlrpc":true,
"xmlrpc-host": "0.0.0.0",
"xmlrpc-port": "5002"
}
}

尝试运行下：

/root/anaconda3/envs/sbird/bin/python /root/git/pyspider/run.py -c /pyspider/config.json scheduler
# 报错
ImportError: No module named 'mysql'
# 下载 mysql-connector-python
cd ~/git/
git clone https://github.com/mysql/mysql-connector-python.git
# 安装
source activate sbird
cd mysql-connector-python
python setup.py install
# 安装redis
pip install redis
source deactivate
# 运行
/root/anaconda3/envs/sbird/bin/python /root/git/pyspider/run.py -c /pyspider/config.json scheduler
# 输出 ok
[I 161010 15:57:25 scheduler:644] scheduler starting...
[I 161010 15:57:25 scheduler:779] scheduler.xmlrpc listening on 0.0.0.0:5002
[I 161010 15:57:25 scheduler:583] in 5m: new:0,success:0,retry:0,failed:0

运行成功后，可直接更改 /etc/supervisord.d/pyspider01.ini如下：

[program:pyspider01]
command = /root/anaconda3/envs/sbird/bin/python /root/git/pyspider/run.py -c /pyspider/config.json scheduler
directory = /root/git/pyspider
user = root
process_name = %(program_name)s
autostart = true
autorestart = true
startsecs = 3
redirect_stderr = true
stdout_logfile_maxbytes = 500MB
stdout_logfile_backups = 10
stdout_logfile = /pyspider/supervisor/pyspider01.log

# 重载
supervisorctl reload
# 查看状态
supervisorctl status

centos01部署完毕。

2.2.centos02

在 centos02中，需要运行 result_worker、processor、phantomjs、fetcher

分别建立文件：

/etc/supervisord.d/result_worker.ini

[program:result_worker]
command = /root/anaconda3/envs/sbird/bin/python /root/git/pyspider/run.py -c /pyspider/config.json result_worker
directory = /root/git/pyspider
user = root
process_name = %(program_name)s
autostart = true
autorestart = true
startsecs = 3
redirect_stderr = true
stdout_logfile_maxbytes = 500MB
stdout_logfile_backups = 10
stdout_logfile = /pyspider/supervisor/result_worker.log

/etc/supervisord.d/processor.ini

[program:processor]
command = /root/anaconda3/envs/sbird/bin/python /root/git/pyspider/run.py -c /pyspider/config.json processor
directory = /root/git/pyspider
user = root
process_name = %(program_name)s
autostart = true
autorestart = true
startsecs = 3
redirect_stderr = true
stdout_logfile_maxbytes = 500MB
stdout_logfile_backups = 10
stdout_logfile = /pyspider/supervisor/processor.log

/etc/supervisord.d/phantomjs.ini

[program:phantomjs]
command = /pyspider/phantomjs --config=/pyspider/pjsconfig.json /pyspider/phantomjs_fetcher.js 25555
directory = /root/git/pyspider
user = root
process_name = %(program_name)s
autostart = true
autorestart = true
startsecs = 3
redirect_stderr = true
stdout_logfile_maxbytes = 500MB
stdout_logfile_backups = 10
stdout_logfile = /pyspider/supervisor/phantomjs.log

/etc/supervisord.d/fetcher.ini

[program:fetcher]
command = /root/anaconda3/envs/sbird/bin/python /root/git/pyspider/run.py -c /pyspider/config.json fetcher
directory = /root/git/pyspider
user = root
process_name = %(program_name)s
autostart = true
autorestart = true
startsecs = 3
redirect_stderr = true
stdout_logfile_maxbytes = 500MB
stdout_logfile_backups = 10
stdout_logfile = /pyspider/supervisor/fetcher.log

在 pyspider目录中建立pjsconfig.json

{
/*--ignore-ssl-errors=true */
"ignoreSslErrors": true,
/*--ssl-protocol=true */
"sslprotocol": "any",
/* Same as: --output-encoding=utf8 */
"outputEncoding": "utf8",
/* persistent Cookies. */
/*cookiesfile="e:/phontjscookies.txt",*/
cookiesfile="pyspider/phontjscookies.txt",
/* load image */
autoLoadImages = false
}

下载phantomjs至 /pyspider/文件夹，将 git/pyspider/pyspider/fetcher/phantomjs_fetcher.js复制到 phantomjs_fetcher.js

# 重载
supervisorctl reload
# 查看状态
supervisorctl status
# output
fetcher RUNNING pid 3446, uptime 0:00:07
phantomjs RUNNING pid 3448, uptime 0:00:07
processor RUNNING pid 3447, uptime 0:00:07
result_worker RUNNING pid 3445, uptime 0:00:07

centos02部署完毕。

2.3.centos03

部署这三个进程 fetcher,processor,result_worker和 centos02 一样，本服务器主要是在前面的基础上加上 webui

建立文件：

/etc/supervisord.d/webui.ini

[program:webui]
command = /root/anaconda3/envs/sbird/bin/python /root/git/pyspider/run.py -c /pyspider/config.json webui
directory = /root/git/pyspider
user = root
process_name = %(program_name)s
autostart = true
autorestart = true
startsecs = 3
redirect_stderr = true
stdout_logfile_maxbytes = 500MB
stdout_logfile_backups = 10
stdout_logfile = /pyspider/supervisor/webui.log

# 重载
supervisorctl reload
# 查看状态
supervisorctl status
# output
fetcher RUNNING pid 2724, uptime 0:00:07
processor RUNNING pid 2725, uptime 0:00:07
result_worker RUNNING pid 2723, uptime 0:00:07
webui RUNNING pid 2726, uptime 0:00:07

3.总结

访问 http://10.211.55.24:5000 即可，尽情爬取吧。

李尚福、魏凤和双双被拿下，与美国一份报告是否有关？

“家属和记者取得联系”：记者的退场意味深长

这位副市长，跨省升正厅

女主播性感斗舞，直播间惨遭拿下！知名团播整大活，邀女嘉宾家人做节目

现在有什么副业可以让人快速上岸? 可以试试这个行业，上岸其实不难!!

CentOS7分布式部署pyspider

1.搭建环境：

1.1.搭建python3环境：

1.1.1.编译

1.1.2.集成环境anaconda

1.2.安装mariaDB

1.3.安装pyspider

1.4.安装Supervisor

1.4.1.Supervisor用法

1.5.安装redis

1.6.关于自启动

2.分布式部署

2.1.centos01

2.2.centos02

2.3.centos03

3.总结

您可能也对以下帖子感兴趣

李尚福、魏凤和双双被拿下，与美国一份报告是否有关？

“家属和记者取得联系”：记者的退场意味深长

这位副市长，跨省升正厅

女主播性感斗舞，直播间惨遭拿下！知名团播整大活，邀女嘉宾家人做节目

现在有什么副业可以让人快速上岸? 可以试试这个行业，上岸其实不难!!

生成图片，分享到微信朋友圈

CentOS7分布式部署pyspider

1.搭建环境：

1.1.搭建python3环境：

1.1.1.编译

1.1.2.集成环境anaconda

1.2.安装mariaDB

1.3.安装pyspider

1.4.安装Supervisor

1.4.1.Supervisor用法

1.5.安装redis

1.6.关于自启动

2.分布式部署

2.1.centos01

2.2.centos02

2.3.centos03

3.总结

您可能也对以下帖子感兴趣