TT Bigdata
JaneTTR
2022-12-18
目录

虚拟环境思路

# 02-hadoop编译需要的虚拟环境

# 1、问题剖析

报错内容：

执行 mvn site 命令时出现 python3: No such file or directory 错误。

报错原因：

CentOS 7.9 默认的 Python 版本为 2.7，而 Hadoop 编译需要 Python 3 版本，因此导致执行过程中找不到 Python3。

解决办法：

双版本共存方案。

How to do:

请点我：使用多版本共存方案

 Toolchains are ignored, 'executable' parameter is set to /opt/modules/bigtop/build/hadoop/rpm/BUILD/hadoop-3.3.4-src/hadoop-common-project/hadoop-common/../../dev-support/bin/shelldocs
[DEBUG] Executing command line: [/opt/modules/bigtop/build/hadoop/rpm/BUILD/hadoop-3.3.4-src/hadoop-common-project/hadoop-common/../../dev-support/bin/shelldocs, --skipprnorep, --output, /opt/modules/bigtop/build/hadoop/rpm/BUILD/hadoop-3.3.4-src/hadoop-common-project/hadoop-common/src/site/markdown/UnixShellAPI.md, --input, /opt/modules/bigtop/build/hadoop/rpm/BUILD/hadoop-3.3.4-src/hadoop-common-project/hadoop-common/src/main/bin/hadoop-functions.sh]
/usr/bin/env: python3: No such file or directory
[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary for Apache Hadoop Main 3.3.4:
[INFO]
[INFO] Apache Hadoop Main ................................. SUCCESS [25:20 min]
[INFO] Apache Hadoop Build Tools .......................... SUCCESS [  0.005 s]
[INFO] Apache Hadoop Project POM .......................... SUCCESS [  6.812 s]
[INFO] Apache Hadoop Annotations .......................... SUCCESS [  1.180 s]
[INFO] Apache Hadoop Project Dist POM ..................... SUCCESS [  1.104 s]
[INFO] Apache Hadoop Assemblies ........................... SUCCESS [  1.158 s]
[INFO] Apache Hadoop Maven Plugins ........................ SUCCESS [  8.255 s]
[INFO] Apache Hadoop MiniKDC .............................. SUCCESS [  3.304 s]
[INFO] Apache Hadoop Auth ................................. SUCCESS [ 18.566 s]
[INFO] Apache Hadoop Auth Examples ........................ SUCCESS [  4.183 s]
[INFO] Apache Hadoop Common ............................... FAILURE [ 13.290 s]
[INFO] Apache Hadoop NFS .................................. SKIPPED
[INFO] Apache Hadoop KMS .................................. SKIPPED
[INFO] Apache Hadoop Registry ............................. SKIPPED
[INFO] Apache Hadoop Common Project ....................... SKIPPED
[INFO] Apache Hadoop HDFS Client .......................... SKIPPED
[INFO] Apache Hadoop HDFS ................................. SKIPPED
[INFO] Apache Hadoop HDFS Native Client ................... SKIPPED
[INFO] Apache Hadoop HttpFS ............................... SKIPPED
[INFO] Apache Hadoop HDFS-NFS ............................. SKIPPED
[INFO] Apache Hadoop HDFS-RBF ............................. SKIPPED
[INFO] Apache Hadoop HDFS Project ......................... SKIPPED
[INFO] Apache Hadoop YARN ................................. SKIPPED
[INFO] Apache Hadoop YARN API ............................. SKIPPED
[INFO] Apache Hadoop YARN Common .......................... SKIPPED
[INFO] Apache Hadoop YARN Server .......................... SKIPPED
[INFO] Apache Hadoop YARN Server Common ................... SKIPPED
[INFO] Apache Hadoop YARN NodeManager ..................... SKIPPED
[INFO] Apache Hadoop YARN Web Proxy ....................... SKIPPED
[INFO] Apache Hadoop YARN ApplicationHistoryService ....... SKIPPED
[INFO] Apache Hadoop YARN Timeline Service ................ SKIPPED
[INFO] Apache Hadoop YARN ResourceManager ................. SKIPPED
[INFO] Apache Hadoop YARN Server Tests .................... SKIPPED
[INFO] Apache Hadoop YARN Client .......................... SKIPPED
[INFO] Apache Hadoop YARN SharedCacheManager .............. SKIPPED
[INFO] Apache Hadoop YARN Timeline Plugin Storage ......... SKIPPED
[INFO] Apache Hadoop YARN TimelineService HBase Backend ... SKIPPED
[INFO] Apache Hadoop YARN TimelineService HBase Common .... SKIPPED
[INFO] Apache Hadoop YARN TimelineService HBase Client .... SKIPPED
[INFO] Apache Hadoop YARN TimelineService HBase Servers ... SKIPPED
[INFO] Apache Hadoop YARN TimelineService HBase Server 1.2  SKIPPED
[INFO] Apache Hadoop YARN TimelineService HBase tests ..... SKIPPED
[INFO] Apache Hadoop YARN Router .......................... SKIPPED
[INFO] Apache Hadoop YARN TimelineService DocumentStore ... SKIPPED
[INFO] Apache Hadoop YARN Applications .................... SKIPPED
[INFO] Apache Hadoop YARN DistributedShell ................ SKIPPED
[INFO] Apache Hadoop YARN Unmanaged Am Launcher ........... SKIPPED
[INFO] Apache Hadoop MapReduce Client ..................... SKIPPED
[INFO] Apache Hadoop MapReduce Core ....................... SKIPPED
[INFO] Apache Hadoop MapReduce Common ..................... SKIPPED
[INFO] Apache Hadoop MapReduce Shuffle .................... SKIPPED
[INFO] Apache Hadoop MapReduce App ........................ SKIPPED
[INFO] Apache Hadoop MapReduce HistoryServer .............. SKIPPED
[INFO] Apache Hadoop MapReduce JobClient .................. SKIPPED
[INFO] Apache Hadoop Mini-Cluster ......................... SKIPPED
[INFO] Apache Hadoop YARN Services ........................ SKIPPED
[INFO] Apache Hadoop YARN Services Core ................... SKIPPED
[INFO] Apache Hadoop YARN Services API .................... SKIPPED
[INFO] Apache Hadoop YARN Application Catalog ............. SKIPPED
[INFO] Apache Hadoop YARN Application Catalog Webapp ...... SKIPPED
[INFO] Apache Hadoop YARN Application Catalog Docker Image  SKIPPED
[INFO] Apache Hadoop YARN Application MaWo ................ SKIPPED
[INFO] Apache Hadoop YARN Application MaWo Core ........... SKIPPED
[INFO] Apache Hadoop YARN Site ............................ SKIPPED
[INFO] Apache Hadoop YARN Registry ........................ SKIPPED
[INFO] Apache Hadoop YARN UI .............................. SKIPPED
[INFO] Apache Hadoop YARN CSI ............................. SKIPPED
[INFO] Apache Hadoop YARN Project ......................... SKIPPED
[INFO] Apache Hadoop MapReduce HistoryServer Plugins ...... SKIPPED
[INFO] Apache Hadoop MapReduce NativeTask ................. SKIPPED
[INFO] Apache Hadoop MapReduce Uploader ................... SKIPPED
[INFO] Apache Hadoop MapReduce Examples ................... SKIPPED
[INFO] Apache Hadoop MapReduce ............................ SKIPPED
[INFO] Apache Hadoop MapReduce Streaming .................. SKIPPED
[INFO] Apache Hadoop Distributed Copy ..................... SKIPPED
[INFO] Apache Hadoop Client Aggregator .................... SKIPPED
[INFO] Apache Hadoop Dynamometer Workload Simulator ....... SKIPPED
[INFO] Apache Hadoop Dynamometer Cluster Simulator ........ SKIPPED
[INFO] Apache Hadoop Dynamometer Block Listing Generator .. SKIPPED
[INFO] Apache Hadoop Dynamometer Dist ..................... SKIPPED
[INFO] Apache Hadoop Dynamometer .......................... SKIPPED
[INFO] Apache Hadoop Archives ............................. SKIPPED
[INFO] Apache Hadoop Archive Logs ......................... SKIPPED
[INFO] Apache Hadoop Rumen ................................ SKIPPED
[INFO] Apache Hadoop Gridmix .............................. SKIPPED
[INFO] Apache Hadoop Data Join ............................ SKIPPED
[INFO] Apache Hadoop Extras ............................... SKIPPED
[INFO] Apache Hadoop Pipes ................................ SKIPPED
[INFO] Apache Hadoop OpenStack support .................... SKIPPED
[INFO] Apache Hadoop Amazon Web Services support .......... SKIPPED
[INFO] Apache Hadoop Kafka Library support ................ SKIPPED
[INFO] Apache Hadoop Azure support ........................ SKIPPED
[INFO] Apache Hadoop Aliyun OSS support ................... SKIPPED
[INFO] Apache Hadoop Scheduler Load Simulator ............. SKIPPED
[INFO] Apache Hadoop Resource Estimator Service ........... SKIPPED
[INFO] Apache Hadoop Azure Data Lake support .............. SKIPPED
[INFO] Apache Hadoop Image Generation Tool ................ SKIPPED
[INFO] Apache Hadoop Tools Dist ........................... SKIPPED
[INFO] Apache Hadoop Tools ................................ SKIPPED
[INFO] Apache Hadoop Client API ........................... SKIPPED
[INFO] Apache Hadoop Client Runtime ....................... SKIPPED
[INFO] Apache Hadoop Client Packaging Invariants .......... SKIPPED
[INFO] Apache Hadoop Client Test Minicluster .............. SKIPPED
[INFO] Apache Hadoop Client Packaging Invariants for Test . SKIPPED
[INFO] Apache Hadoop Client Packaging Integration Tests ... SKIPPED
[INFO] Apache Hadoop Distribution ......................... SKIPPED
[INFO] Apache Hadoop Client Modules ....................... SKIPPED
[INFO] Apache Hadoop Cloud Storage ........................ SKIPPED
[INFO] Apache Hadoop Tencent COS Support .................. SKIPPED
[INFO] Apache Hadoop Cloud Storage Project ................ SKIPPED
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  26:35 min
[INFO] Finished at: 2024-07-18T15:41:44Z
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal org.codehaus.mojo:exec-maven-plugin:1.3.1:exec (shelldocs) on project hadoop-common: Command execution failed.: Process exited with an error: 127 (Exit value: 127) -> [Help 1]
org.apache.maven.lifecycle.LifecycleExecutionException: Failed to execute goal org.codehaus.mojo:exec-maven-plugin:1.3.1:exec (shelldocs) on project hadoop-common: Command execution failed.
    at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:215)
    at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:156)
    at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:148)
    at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject (LifecycleModuleBuilder.java:117)
    at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject (LifecycleModuleBuilder.java:81)
    at org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThreadedBuilder.build (SingleThreadedBuilder.java:56)
    at org.apache.maven.lifecycle.internal.LifecycleStarter.execute (LifecycleStarter.java:128)
    at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:305)
    at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:192)
    at org.apache.maven.DefaultMaven.execute (DefaultMaven.java:105)
    at org.apache.maven.cli.MavenCli.execute (MavenCli.java:972)
    at org.apache.maven.cli.MavenCli.doMain (MavenCli.java:293)
    at org.apache.maven.cli.MavenCli.main (MavenCli.java:196)
    at sun.reflect.NativeMethodAccessorImpl.invoke0 (Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke (NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke (DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke (Method.java:498)
    at org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced (Launcher.java:282)
    at org.codehaus.plexus.classworlds.launcher.Launcher.launch (Launcher.java:225)
    at org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode (Launcher.java:406)
    at org.codehaus.plexus.classworlds.launcher.Launcher.main (Launcher.java:347)
Caused by: org.apache.maven.plugin.MojoExecutionException: Command execution failed.
    at org.codehaus.mojo.exec.ExecMojo.execute (ExecMojo.java:303)
    at org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo (DefaultBuildPluginManager.java:137)
    at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:210)
    at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:156)
    at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:148)
    at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject (LifecycleModuleBuilder.java:117)
    at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject (LifecycleModuleBuilder.java:81)
    at org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThreadedBuilder.build (SingleThreadedBuilder.java:56)
    at org.apache.maven.lifecycle.internal.LifecycleStarter.execute (LifecycleStarter.java:128)
    at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:305)
    at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:192)
    at org.apache.maven.DefaultMaven.execute (DefaultMaven.java:105)
    at org.apache.maven.cli.MavenCli.execute (MavenCli.java:972)
    at org.apache.maven.cli.MavenCli.doMain (MavenCli.java:293)
    at org.apache.maven.cli.MavenCli.main (MavenCli.java:196)
    at sun.reflect.NativeMethodAccessorImpl.invoke0 (Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke (NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke (DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke (Method.java:498)
    at org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced (Launcher.java:282)
    at org.codehaus.plexus.classworlds.launcher.Launcher.launch (Launcher.java:225)
    at org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode (Launcher.java:406)
    at org.codehaus.plexus.classworlds.launcher.Launcher.main (Launcher.java:347)
Caused by: org.apache.commons.exec.ExecuteException: Process exited with an error: 127 (Exit value: 127)
    at org.apache.commons.exec.DefaultExecutor.executeInternal (DefaultExecutor.java:402)
    at org.apache.commons.exec.DefaultExecutor.execute (DefaultExecutor.java:164)
    at org.codehaus.mojo.exec.ExecMojo.executeCommandLine (ExecMojo.java:750)
    at org.codehaus.mojo.exec.ExecMojo.execute (ExecMojo.java:292)
    at org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo (DefaultBuildPluginManager.java:137)
    at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:210)
    at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:156)
    at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:148)
    at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject (LifecycleModuleBuilder.java:117)
    at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject (LifecycleModuleBuilder.java:81)
    at org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThreadedBuilder.build (SingleThreadedBuilder.java:56)
    at org.apache.maven.lifecycle.internal.LifecycleStarter.execute (LifecycleStarter.java:128)
    at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:305)
    at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:192)
    at org.apache.maven.DefaultMaven.execute (DefaultMaven.java:105)
    at org.apache.maven.cli.MavenCli.execute (MavenCli.java:972)
    at org.apache.maven.cli.MavenCli.doMain (MavenCli.java:293)
    at org.apache.maven.cli.MavenCli.main (MavenCli.java:196)
    at sun.reflect.NativeMethodAccessorImpl.invoke0 (Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke (NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke (DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke (Method.java:498)
    at org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced (Launcher.java:282)
    at org.codehaus.plexus.classworlds.launcher.Launcher.launch (Launcher.java:225)
    at org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode (Launcher.java:406)
    at org.codehaus.plexus.classworlds.launcher.Launcher.main (Launcher.java:347)
[ERROR]
[ERROR]
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoExecutionException
[ERROR]
[ERROR] After correcting the problems, you can resume the build with the command
[ERROR]   mvn <args> -rf :hadoop-common
error: Bad exit status from /var/tmp/rpm-tmp.TNsQdj (%build)

    Bad exit status from /var/tmp/rpm-tmp.TNsQdj (%build)

RPM build errors:

> Task :hadoop-rpm FAILED

FAILURE: Build failed with an exception.

* Where:
Script '/opt/modules/bigtop/packages.gradle' line: 529

* What went wrong:
Execution failed for task ':hadoop-rpm'.
> Process 'command 'rpmbuild'' finished with non-zero exit value 1

* Try:
Run with --stacktrace option to get the stack trace. Run with --info or --debug option to get more log output. Run with --scan to get full insights.

* Get more help at https://help.gradle.org

BUILD FAILED in 1h 25m 59s
30 actionable tasks: 30 executed
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228

# 2、解决办法

# 2.1 使用多版本共存方案

在同一台计算机上实现 Python 多版本共存，常见的方式有 Conda 和 Virtualenv。它们各自有不同的优缺点，适用于不同的场景。

# 2.1.1 Conda vs Virtualenv

特性 Conda Virtualenv
包管理 集成包管理和环境管理，支持 Python 和非 Python 包 仅管理 Python 包，依赖 pip
跨平台支持 Windows, macOS, Linux Windows, macOS, Linux
环境隔离 可以创建独立的环境，支持不同版本的 Python 创建隔离的 Python 环境，依赖系统 Python 版本
依赖管理 自动处理依赖，适合数据科学和机器学习 需要手动管理依赖
安装复杂度 需要安装 Anaconda 或 Miniconda 内置于 Python 3.3+，安装简单

# 2.1.2 Conda 的优势

  • 强大的包管理：不仅支持 Python 包，还支持其他语言的包。
  • 环境独立：可以在同一环境中使用不同版本的 Python。
  • 自动依赖管理：适合复杂的依赖管理，尤其是数据科学和机器学习领域。

# 2.1.3 Virtualenv 的优势

  • 轻量级：不需要安装额外的软件包，内置于 Python 3.3+。
  • 简单易用：适合简单的 Python 项目，不需要复杂的依赖管理。

# 2.2 使用 Conda 管理多版本 Python

# 2.2.1 安装 Conda

下载并安装 Miniconda（推荐轻量级版本）：

使用以下命令下载并安装 Miniconda：

wget https://mirrors.tuna.tsinghua.edu.cn/anaconda/miniconda/Miniconda3-latest-Linux-x86_64.sh
bash Miniconda3-latest-Linux-x86_64.sh
1
2

配置清华大学镜像源：

conda config --add channels https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/free/
conda config --add channels https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main/
conda config --set show_channel_urls yes
1
2
3

# 2.2.2 创建和管理虚拟环境

创建虚拟环境：

conda create -n myenv python=3.8
1

激活虚拟环境：

conda activate myenv
1

安装包：

conda install numpy pandas
1

退出虚拟环境：

conda deactivate
1

删除虚拟环境：

conda remove -n myenv --all
1

# 2.3 使用 Virtualenv 管理多版本 Python

# 2.3.1 安装 pip

如果你的 Python 环境默认不带 pip，可以使用以下方法安装 pip：

使用 ensurepip 模块：

python -m ensurepip --upgrade
1

使用 get-pip.py 脚本：

从阿里云下载 get-pip.py：

wget https://mirrors.aliyun.com/pypi/packages/py2.py3/g/get-pip/get-pip.py
1

运行脚本：

python get-pip.py
1

# 2.3.2 安装 Virtualenv 和配置 pip 使用国内镜像

安装 Virtualenv：

pip install virtualenv
1

配置 pip 使用国内镜像：

创建 pip 配置文件：

在 ~/.pip/pip.conf 中添加以下内容：

[global]
index-url = https://pypi.tuna.tsinghua.edu.cn/simple
1
2

# 2.3.3 创建和管理虚拟环境

创建虚拟环境：

virtualenv -p python3.8 myenv
1

激活虚拟环境：

source myenv/bin/activate
1

安装包：

pip install numpy pandas
1

退出虚拟环境：

deactivate
1

删除虚拟环境：

rm -rf myenv
1

# 2.4 懒人脚本 - Virtualenv

#!/bin/bash
# Licensed to the Apache Software Foundation (ASF) under one or more
# contributor license agreements.  See the NOTICE file distributed with
# this work for additional information regarding copyright ownership.
# The ASF licenses this file to You under the Apache License, Version 2.0
# (the "License"); you may not use this file except in compliance with
# the License.  You may obtain a copy of the License at
#
#     http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License

set -ex

echo "############## SETUP PYTHON_VIRTUAL_ENV start #############"

# 设置变量
BASE_DIR="/opt/modules/virtual_env"
PYTHON_VERSION=3.7.12
PYTHON_TAR=Python-$PYTHON_VERSION.tgz
PYTHON_SRC_DIR=Python-$PYTHON_VERSION
DOWNLOAD_URL=https://repo.huaweicloud.com/python/$PYTHON_VERSION/$PYTHON_TAR
PIP_URL="https://mirrors.aliyun.com/pypi/get-pip.py"
PIP_PATH="$BASE_DIR/get-pip.py"
VIRTUAL_ENV_PATH="$BASE_DIR/python3.7/bin/python3.7"
VIRTUALENV_DIR="$BASE_DIR/hadoop_py37"

# 创建基础目录
mkdir -p $BASE_DIR
# 安装必要的依赖
yum groupinstall -y "Development Tools"
yum install -y \
    zlib-devel \
    bzip2-devel \
    openssl-devel \
    ncurses-devel \
    sqlite-devel \
    readline-devel \
    tk-devel \
    gdbm-devel \
    db4-devel \
    libpcap-devel \
    xz-devel \
    curl \
    libffi-devel

# 下载 Python 3.7 源码
if [ ! -f $BASE_DIR/$PYTHON_TAR ]; then
    echo "Downloading Python $PYTHON_VERSION..."
    curl -o $BASE_DIR/$PYTHON_TAR $DOWNLOAD_URL
else
    echo "Python $PYTHON_TAR already downloaded."
fi

# 解压源码包
if [ ! -d $BASE_DIR/$PYTHON_SRC_DIR ]; then
    echo "Extracting Python $PYTHON_TAR..."
    tar -xf $BASE_DIR/$PYTHON_TAR -C $BASE_DIR
else
    echo "Python source directory already exists."
fi

# 进入源码目录并安装 Python 3.7
cd $BASE_DIR/$PYTHON_SRC_DIR

if ! command -v python3.7 &> /dev/null; then
    echo "Installing Python $PYTHON_VERSION..."
    ./configure --enable-optimizations
    make -j$(nproc)
    sudo make altinstall
else
    echo "Python 3.7 already installed."
fi

# 下载并安装 pip
if [ ! -f "$PIP_PATH" ]; then
    echo "Downloading get-pip.py..."
    curl -o "$PIP_PATH" "$PIP_URL"
else
    echo "get-pip file exists: $PIP_PATH"
fi

if ! command -v pip3.7 &> /dev/null; then
    echo "Installing pip..."
    sudo python3.7 "$PIP_PATH"
    pip3.7 config set global.index-url https://pypi.tuna.tsinghua.edu.cn/simple
    sudo python3.7 -m pip install --upgrade pip
    echo "pip 安装完毕"
else
    echo "pip already installed."
fi

# 安装 virtualenv
pip3.7 install virtualenv

# 创建虚拟环境
if [ ! -d $VIRTUALENV_DIR ]; then
    echo "Creating virtual environment..."
    python3.7 -m venv $VIRTUALENV_DIR
else
    echo "Virtual environment already exists."
fi

# 激活虚拟环境
source $VIRTUALENV_DIR/bin/activate

# 验证虚拟环境中的 Python 版本
PYTHON_VERSION_ACTUAL=$(python --version)
echo "虚拟环境中的 Python 版本是: $PYTHON_VERSION_ACTUAL"

# 提示用户虚拟环境已激活
#echo "虚拟环境已激活。要退出虚拟环境，请运行 'deactivate' 命令。"

# 保持虚拟环境激活状态
#$SHELL

echo "############## SETUP PYTHON_VIRTUAL_ENV end #############"
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
#虚拟环境#Hadoop#Conda#Virtualenv#Python
Bower镜像加速
R环境安装+一键安装脚本

← R环境安装+一键安装脚本→

最近更新
01
Cyrus SASL/GSASL 缺失解决
06-07
02
Hadoop_3.3.4 编译实战 1.0.0+
06-06
03
Hbase_2.4.13 编译实战 1.0.0+
06-06
更多文章>
Theme by Vdoing | Copyright © 2017-2025 JaneTTR | MIT License