sqoop2 使用实例

启动Client

/usr/hadoop/sqoop-1.99.5-cdh5.5.1-bin-hadoop200/bin/sqoop.sh client

 

为客户端配置服务器：set server --host node3--port 12000 --webapp sqoop 【红色部分为 本机hostname 主机名和端口号】

sqoop:000> set server --host node3--port 12000 --webapp sqoop

 

使用 show version --all 查看服务器配置

sqoop:000> show version --all

 

1、 使用 show connector  查看sqoop的所有连接，sqoop1.99.5 所有的连接固定为四个，如下：

sqoop:000> show connector 
+----+------------------------+-----------------+------------------------------------------------------+----------------------+
| Id |          Name          |     Version     |                        Class                         | Supported Directions |
+----+------------------------+-----------------+------------------------------------------------------+----------------------+
| 1  | kite-connector         | 1.99.5-cdh5.5.1 | org.apache.sqoop.connector.kite.KiteConnector        | FROM/TO              |
| 2  | kafka-connector        | 1.99.5-cdh5.5.1 | org.apache.sqoop.connector.kafka.KafkaConnector      | TO                   |
| 3  | hdfs-connector         | 1.99.5-cdh5.5.1 | org.apache.sqoop.connector.hdfs.HdfsConnector        | FROM/TO              |
| 4  | generic-jdbc-connector | 1.99.5-cdh5.5.1 | org.apache.sqoop.connector.jdbc.GenericJdbcConnector | FROM/TO              |
+----+------------------------+-----------------+------------------------------------------------------+----------------------+

 2、 在向hdfs导入导出数据时，需要依赖以上四个连接创建link （在1.99.4版本之后 用户不需要再创建连接）

首先 查看 当前的所有link

sqoop:000> show link
+----+---------------------+--------------+------------------------+---------+
| Id |        Name         | Connector Id |     Connector Name     | Enabled |
+----+---------------------+--------------+------------------------+---------+
| 1  | hdfs_link           | 3            | hdfs-connector         | true    |
| 2  | mysql_openfire_link | 4            | generic-jdbc-connector | true    |
+----+---------------------+--------------+------------------------+---------+

 查看job

sqoop:000> show job
+----+-------------------------+----------------+--------------+---------+
| Id |          Name           | From Connector | To Connector | Enabled |
+----+-------------------------+----------------+--------------+---------+
| 1  | openfire_properties_job | 4              | 3            | true    |
+----+-------------------------+----------------+--------------+---------+

 

3、 创建 link

   创建hdfs连接

sqoop:000> create link --cid 3
Creating link for connector with id 1
Please fill following values to create new link object
Name: hdfs_link --设置连接名称
Link configuration
HDFS URI: hdfs://ns1/ --HDFS访问地址
New link was successfully created with validation status OK and persistent id 2

 创建 mysql 连接

sqoop:000> create link --cid 4
Creating link for connector with id 2
Please fill following values to create new link object
Name: mysql_openfire_link --设置连接名称
Link configuration
JDBC Driver Class: com.mysql.jdbc.Driver --驱动
JDBC Connection String: jdbc:mysql://mysql.server/database --JDBC连接
Username: sqoop --数据库用户
Password: ***** --数据库密码
JDBC Connection Properties:
There are currently 0 values in the map:
entry#protocol=tcp --之后按回车
New link was successfully created with validation status OK and persistent id 1

 

5、 创建job

sqoop:000> create job -f 2 -t 1
Creating job for links with from id 1 and to id 6
Please fill following values to create new job object
Name: mysql_openfire--设置 任务名称
FromJob configuration
Schema name:(Required)sqoop --库名：必填
Table name:(Required)sqoop --表名：必填
Table SQL statement:(Optional) --选填
Table column names:(Optional) --选填
Partition column name:(Optional) id --选填
Null value allowed for the partition column:(Optional) --选填
Boundary query:(Optional) --选填
ToJob configuration
Output format:
0 : TEXT_FILE
1 : SEQUENCE_FILE
Output format:
0 : TEXT_FILE
1 : SEQUENCE_FILE
Choose: 0 --选择文件压缩格式
Compression format:
0 : NONE
1 : DEFAULT
2 : DEFLATE
3 : GZIP
4 : BZIP2
5 : LZO
6 : LZ4
7 : SNAPPY
8 : CUSTOM
Choose: 0 --选择压缩类型
Custom compression format:(Optional) --选填
Output directory:hdfs:/ns1/sqoop --HDFS存储目录（目的地）
Driver Config
Extractors: 2 --提取器
Loaders: 2 --加载器
New job was successfully created with validation status OK and persistent id 1

 

查看job

sqoop:000> show job
+----+-------------------------+----------------+--------------+---------+
| Id |          Name           | From Connector | To Connector | Enabled |
+----+-------------------------+----------------+--------------+---------+
| 1  | openfire_properties_job | 4              | 3            | true    |
+----+-------------------------+----------------+--------------+---------+

 

 

 常用命令列表

sqoop:001> show link 显示所有链接

sqoop:001> carete link --cid 1床架连接

sqoop:000> delete link --lid 1 删除link
sqoop:001> show job 显示所有job

sqoop:001> create job --f 2 --t 1 创建job ( 从link 2 向link 1导入数据)
sqoop:000> start job --jid 1 启动job
sqoop:000> status job --jid 1 查看导入状态

sqoop:000> delete job --jid 1 删除job