冰序小站

基于 Spark 的中文新闻分类全流程实战(CentOS7 虚拟机)

发布时间:2026-05-16 18:38:48

本篇博客为零基础可复刻的实操教程,从启动 Hadoop 集群开始,覆盖环境配置、数据上传、代码编写、任务运行全流程,所有命令 / 代码可直接复制使用,完美适配 CentOS7 虚拟机环境。

实验环境

  • 操作系统:CentOS 7(虚拟机)

  • 集群环境:Hadoop 3.x + Spark 3.x(已部署基础环境)

  • 数据集:中文新闻 CSV 数据集(news\_data\.csv)

  • 运行模式:Spark 本地低内存模式(适配虚拟机硬件)


完整实验步骤(从零复刻)

步骤 1:启动 Hadoop + Spark 集群

打开虚拟机终端,首先启动集群基础服务,这是所有操作的前提:

# 启动 HDFS 分布式文件系统
start-dfs.sh

# 启动 YARN 资源调度系统
start-yarn.sh

# 验证集群服务是否正常运行(出现 NameNode、DataNode 等进程即为成功)
jps

步骤 2:本地数据集上传至虚拟机

在本地电脑 PowerShell 中执行,将桌面的数据集上传到虚拟机 hadoop 用户家目录:

# 切换到本地文件所在目录
cd D:\desktop

# SCP 命令上传数据(IP 替换为你的虚拟机 IP)
scp news_data.csv hadoop@192.168.181.10:/home/hadoop/

执行后输入虚拟机密码,等待上传完成。


步骤 3:HDFS 目录创建 + 数据上传

回到虚拟机终端,将本地数据上传到 HDFS 存储:

# 创建实验专属 HDFS 目录
hdfs dfs -mkdir -p /user/spark/news_classification/data/

# 上传本地数据集到 HDFS
hdfs dfs -put news_data.csv /user/spark/news_classification/data/

# 验证文件上传结果(显示文件信息即为成功)
hdfs dfs -ls -h /user/spark/news_classification/data/

步骤 4:CentOS7 配置 Python3 环境(核心缺失步骤)

CentOS7 默认无 Python3,需修复 YUM 源并安装,直接复制执行:

# 1. 备份原有 YUM 源
sudo mv /etc/yum.repos.d/CentOS-Base.repo /etc/yum.repos.d/CentOS-Base.repo.bak

# 2. 配置 CentOS7 官方归档源
sudo tee /etc/yum.repos.d/CentOS-Base.repo <<-'EOF'
[base]
name=CentOS-$releasever - Base
baseurl=http://vault.centos.org/centos/$releasever/os/$basearch/
gpgcheck=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-7

[updates]
name=CentOS-$releasever - Updates
baseurl=http://vault.centos.org/centos/$releasever/updates/$basearch/
gpgcheck=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-7

[extras]
name=CentOS-$releasever - Extras
baseurl=http://vault.centos.org/centos/$releasever/extras/$basearch/
gpgcheck=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-7
EOF

# 3. 重建 YUM 缓存
sudo yum clean all
sudo yum makecache

# 4. 安装 Python3.6 与 pip
sudo yum install -y epel-release
sudo yum install -y python36 python36-pip

# 5. 创建软链接(方便直接使用 python3 命令)
sudo ln -s /usr/bin/python3.6 /usr/bin/python3
sudo ln -s /usr/bin/pip3.6 /usr/bin/pip3

# 6. 验证 Python3 安装成功
python3 --version
pip3 --version

步骤 5:安装 Spark 机器学习依赖库

Spark MLlib 依赖科学计算库,执行安装:

# 安装 numpy(Spark 机器学习核心依赖)
pip3 install numpy

步骤 6:编写 Spark 中文新闻分类代码

创建轻量化代码文件,适配虚拟机低内存环境:

# 创建代码文件
nano news_classification.py

粘贴完整可运行代码(无冗余逻辑,纯核心功能):

from pyspark.sql import SparkSession
from pyspark.ml.feature import StringIndexer, Tokenizer, HashingTF, IDF
from pyspark.ml.classification import NaiveBayes
from pyspark.ml.evaluation import MulticlassClassificationEvaluator
from pyspark.ml import Pipeline

# 初始化 Spark 会话
spark = SparkSession.builder.appName("MiniNews").getOrCreate()

# 读取 HDFS 数据(10% 采样,降低虚拟机内存占用)
df = spark.read.csv(
    "/user/spark/news_classification/data/news_data.csv",
    header=True,
    encoding="utf-8"
).sample(fraction=0.1, seed=42)

print("已读取数据总行数:", df.count())

# 构建数据预处理流水线
indexer = StringIndexer(inputCol="category", outputCol="label")
tokenizer = Tokenizer(inputCol="content", outputCol="words")
hashingTF = HashingTF(inputCol="words", outputCol="rawFeatures", numFeatures=1000)
idf = IDF(inputCol="rawFeatures", outputCol="features")

# 训练预处理模型并转换数据
pipeline = Pipeline(stages=[indexer, tokenizer, hashingTF, idf])
processed_df = pipeline.fit(df).transform(df)

# 划分训练集(80%)和测试集(20%)
train, test = processed_df.randomSplit([0.8, 0.2], seed=42)

# 训练朴素贝叶斯分类模型
print("开始训练模型...")
model = NaiveBayes(featuresCol="features", labelCol="label").fit(train)
predictions = model.transform(test)

# 评估模型准确率
evaluator = MulticlassClassificationEvaluator(labelCol="label", predictionCol="prediction", metricName="accuracy")
accuracy = evaluator.evaluate(predictions)
print("模型准确率:", round(accuracy, 4))

# 关闭 Spark 会话
spark.stop()

保存退出:Ctrl \+ O → 回车确认 → Ctrl \+ X


步骤 7:低内存模式运行 Spark 任务

执行命令,适配虚拟机硬件,避免内存溢出:

spark-submit --master local[1] \
  --driver-memory 512m \
  --conf spark.executor.memory=512m \
  --conf spark.sql.shuffle.partitions=1 \
  --conf spark.driver.maxResultSize=256m \
  news_classification.py

实验成功输出结果

运行完成后,终端会打印核心结果:

Plain Text 已读取数据总行数: 8365 开始训练模型... 模型准确率: 0.1006

最终程序退出码为 exitCode 0,代表任务 100% 运行成功。


实验核心总结

  1. 全流程覆盖:集群启动 → 数据上传 → Python3 环境配置 → Spark 代码编写 → 模型训练评估;

  2. 虚拟机适配:采用数据采样 + 低内存配置,完美解决虚拟机资源不足问题;

  3. 核心技术:基于 Spark MLlib 实现 TF-IDF 文本特征提取 + 朴素贝叶斯新闻分类;

  4. 可复刻性:所有命令、代码无冗余、无错误,新手可直接复制完成实验。

评论区

1 2026-11-16 16:11:26

555

1 2026-11-16 16:10:43

(select 198766*667891)

1 2026-11-16 16:10:43

(select 198766*667891 from DUAL)

1 2026-11-16 16:10:42

555

1 2026-11-16 16:10:42

555'"

1 2026-11-16 16:10:42

555%C0%A7%C0%A2%2527%2522\'\"

1 2026-11-16 16:10:42

@@NAw8g

1 2026-11-16 16:10:41

555'||DBMS_PIPE.RECEIVE_MESSAGE(CHR(98)||CHR(98)||CHR(98),15)||'

1 2026-11-16 16:10:38

555*DBMS_PIPE.RECEIVE_MESSAGE(CHR(99)||CHR(99)||CHR(99),15)

1 2026-11-16 16:10:34

555CEJc0d9A')) OR 36=(SELECT 36 FROM PG_SLEEP(15))--

1 2026-11-16 16:10:30

555s5vW1RPV') OR 491=(SELECT 491 FROM PG_SLEEP(15))--

1 2026-11-16 16:10:27

555MwX7mnJ4' OR 281=(SELECT 281 FROM PG_SLEEP(15))--

1 2026-11-16 16:10:23

555-1)) OR 942=(SELECT 942 FROM PG_SLEEP(15))--

1 2026-11-16 16:10:20

555-1) OR 856=(SELECT 856 FROM PG_SLEEP(15))--

1 2026-11-16 16:10:15

555-1 OR 928=(SELECT 928 FROM PG_SLEEP(15))--

1 2026-11-16 16:10:11

555ghWxH8Px'; waitfor delay '0:0:15' --

1 2026-11-16 16:10:07

555-1 waitfor delay '0:0:15' --

1 2026-11-16 16:10:03

555-1); waitfor delay '0:0:15' --

1 2026-11-16 16:09:59

555-1; waitfor delay '0:0:15' --

1 2026-11-16 16:09:55

(select(0)from(select(sleep(15)))v)/*'+(select(0)from(select(sleep(15)))v)+'"+(select(0)from(select(sleep(15)))v)+"*/

1 2026-11-16 16:09:49

5550"XOR(555*if(now()=sysdate(),sleep(15),0))XOR"Z

1 2026-11-16 16:09:44

5550'XOR(555*if(now()=sysdate(),sleep(15),0))XOR'Z

1 2026-11-16 16:09:39

555*if(now()=sysdate(),sleep(15),0)

1 2026-11-16 16:09:34

-1" OR 5*5=25 or "vZ4xJ6e8"="

1 2026-11-16 16:09:33

555

1 2026-11-16 16:09:33

-1 OR 5*5=25 --

1 2026-11-16 16:09:33

-1 OR 5*5=25

1 2026-11-16 16:09:33

-1' OR 5*5=25 --

1 2026-11-16 16:09:33

-1" OR 5*5=25 --

1 2026-11-16 16:09:33

-1' OR 5*5=25 or 'XvJriUSZ'='

1 2026-11-16 16:09:28

555tTHSWTgY

1 2026-11-16 16:09:25

555

1 2026-11-16 16:09:20

555

(select 198766*667891) 2026-11-16 16:09:16

555

(select 198766*667891 from DUAL) 2026-11-16 16:09:16

555

1 2026-11-16 16:09:15

555

1'" 2026-11-16 16:09:15

555

1%C0%A7%C0%A2%2527%2522\'\" 2026-11-16 16:09:15

555

@@NrBVd 2026-11-16 16:09:15

555

1'||DBMS_PIPE.RECEIVE_MESSAGE(CHR(98)||CHR(98)||CHR(98),15)||' 2026-11-16 16:09:14

555

1*DBMS_PIPE.RECEIVE_MESSAGE(CHR(99)||CHR(99)||CHR(99),15) 2026-11-16 16:09:10

555

1rusExPx9')) OR 514=(SELECT 514 FROM PG_SLEEP(15))-- 2026-11-16 16:09:05

555

1wphoqBEl') OR 245=(SELECT 245 FROM PG_SLEEP(15))-- 2026-11-16 16:09:00

555

1luZmaOWf' OR 315=(SELECT 315 FROM PG_SLEEP(15))-- 2026-11-16 16:08:56

555

1-1)) OR 502=(SELECT 502 FROM PG_SLEEP(15))-- 2026-11-16 16:08:50

555

1-1) OR 961=(SELECT 961 FROM PG_SLEEP(15))-- 2026-11-16 16:08:46

555

1-1 OR 226=(SELECT 226 FROM PG_SLEEP(15))-- 2026-11-16 16:08:41

555

1FzQG809O'; waitfor delay '0:0:15' -- 2026-11-16 16:08:37

555

1-1 waitfor delay '0:0:15' -- 2026-11-16 16:08:30

555

1-1); waitfor delay '0:0:15' -- 2026-11-16 16:08:26

555

1-1; waitfor delay '0:0:15' -- 2026-11-16 16:08:18

555

(select(0)from(select(sleep(15)))v)/*'+(select(0)from(select(sleep(15)))v)+'"+(select(0)from(select(sleep(15)))v)+"*/ 2026-11-16 16:08:12

555

10"XOR(1*if(now()=sysdate(),sleep(15),0))XOR"Z 2026-11-16 16:08:08

555

10'XOR(1*if(now()=sysdate(),sleep(15),0))XOR'Z 2026-11-16 16:08:04

555

1*if(now()=sysdate(),sleep(15),0) 2026-11-16 16:07:59

555

-1' OR 5*5=25 or 'Gia4BLFs'=' 2026-11-16 16:07:57

555

-1" OR 5*5=25 or "nlZNZgkp"=" 2026-11-16 16:07:57

555

-1 OR 5*5=25 -- 2026-11-16 16:07:56

555

-1 OR 5*5=25 2026-11-16 16:07:56

555

-1' OR 5*5=25 -- 2026-11-16 16:07:56

555

-1" OR 5*5=25 -- 2026-11-16 16:07:56

555

1 2026-11-16 16:07:55

555

183rvCoqu 2026-11-16 16:07:52

555

1 2026-11-16 16:07:47

555

1 2026-11-16 16:07:40

555

1 2026-11-16 16:07:35

555