spark

Converting a user program into tasks

The Spark driver is responsible for converting a user program into units of physical
execution called tasks. At a high level, all Spark programs follow the same
structure: they create RDDs from some input, derive new RDDs from those
using transformations, and perform actions to collect or save data. A Spark program
implicitly creates a logical directed acyclic graph (DAG) of operations.
When the driver runs, it converts this logical graph into a physical execution
plan.

Spark performs several optimizations, such as “pipelining” map transformations
together to merge them, and converts the execution graph into a set of stages.
Each stage, in turn, consists of multiple tasks. The tasks are bundled up and prepared
to be sent to the cluster. Tasks are the smallest unit of work in Spark; a
typical user program can launch hundreds or thousands of individual tasks.

Scheduling tasks on executors

Given a physical execution plan, a Spark driver must coordinate the scheduling
of individual tasks on executors. When executors are started they register themselves
with the driver, so it has a complete view of the application’s executors at
all times. Each executor represents a process capable of running tasks and storing
RDD data.
The Spark driver will look at the current set of executors and try to schedule each
task in an appropriate location, based on data placement. When tasks execute,
they may have a side effect of storing cached data. The driver also tracks the location
of cached data and uses it to schedule future tasks that access that data.
The driver exposes information about the running Spark application through a
web interface, which by default is available at port 4040. For instance, in local
mode, this UI is available at http://localhost:4040.

File Compression

Choosing an output compression codec can have a big impact on future users of the
data. With distributed systems such as Spark, we normally try to read our data in
from multiple different machines. To make this possible, each worker needs to be
able to find the start of a new record. Some compression formats make this impossible,
which requires a single node to read in all of the data and thus can easily lead to a
bottleneck. Formats that can be easily read from multiple machines are called “splittable.”
Table 5-3 lists the available compression options

docker vlan with socketplane on bare-metal node

multi-host networking between Dockerhosts using Open vSwitch and Consul as core enablers by building VXLAN tunnels between hosts to connect docker containers on the same virtual(logical) network with no remote/external SDN controller needed

it will install OVS and Docker and start a socketplane container that runs Consul for managing network state

install

1: git clone https://github.com/socketplane/socketplane

2: chmod u+x scripts/install.sh

3：first node

sudo BOOTSTRAP=true ./scripts/install.sh 

Subsequent Nodes:

3: sudo ./scripts/install.sh

4: sudo socketplane run -itd ubuntu

5： sudo socketplane network list

UNinstall

sudo socketplane uninstall

 cgroups

The first step, on Ubuntu, is to install the cgroups packages:

sudo apt-get install cgroup-bin cgroup-lite libcgroup1

This creates a few files for you. Notably:

• /sys/fs/cgroup is created, which has a bunch of folders in a sort of virtual filesystem that "represents" your cgroup.
• /etc/init/cgroup-lite.conf. cgroup-lite.conf creates the /sys/fs/cgroup directory, based on the contents of
• /proc/cgroups, which specifies what groups cgroup-lite should create.

 cgroup created on boot

Here's what I ended up putting in my /etc/cgconfig.conf:

group limitgroup {
  perm {
    admin {
      uid = root;
      gid = root;
    }
    task {
      uid = 1001;
      gid = 1001;
    }  
  }
  cpu {
    cpu.shares = "768";
  }
  memory {
    memory.limit_in_bytes = "3G";
  }            
}

cgconfigparser -l /etc/cgconfig.conf

This parses and then "executes" your cgconfig file; that is, if it parses correctly, it'll create your groups. I fought with creating mount directives in this file for quite a few reboots, but finally I just modified my /etc/init/cgroup-lite.conf and added the cgconfigparser -l /etc/cgconfig.conf line below the /bin/cgroups-mount line. So the modified contents of the pre-start section in /etc/init/cgroup-lite.conf file looked like

pre-start script
        test -x /bin/cgroups-mount || { stop; exit 0; }
        test -d /sys/fs/cgroup || { stop; exit 0; }
        /bin/cgroups-mount
        cgconfigparser -l /etc/cgconfig.conf
end script

[1] http://thenewstack.io/sdn-docker-real-changes-ahead/

[2] https://aucouranton.com/2015/01/16/docker-virtual-networking-with-socketplane-io/
[3] https://www.devinhoward.ca/technology/2015/feb/implementing-cgroups-ubuntu-or-debian

人体30亿个碱基对的基因组

作者：GentleYang 杨振涛
链接：https://www.zhihu.com/question/21672220/answer/19590442
来源：知乎
著作权归作者所有。商业转载请联系作者获得授权，非商业转载请注明出处。

1. 测序的时候，理论上(注意是理论上，实践中不同目的不同方法做法都不相同)是不需要测出一对碱基的，既然一个碱基对是互补配对(A-T,C-G)，那知道一个就可以了嘛，记录时也是记录一个，序列数据文件经常描述单链加上方向(如3-5)。

2 你说的多少M，并不是对容量的描述，是对序列长度(字符串长度)的描述！！！ 区分这个很重要，容量因为有多次折叠，远远超过M级别的，我在这里把容量近似地理解为信息量。

3 我们一般说的人类基因组大小3G，是指对每个染色体长度总和的描述，注意是长度；这也就解释了为什么你说的是60亿个碱基， 双链， 3G X 2 = 6G。

4 为什么你看到的保存一个人类基因组的数据文件大小是750M ？ 3G的长度，一个位置有4种或5种可能性(A,T,C,G,N)，也就是说如果你用碱基的字母代码来存储它，而不另外编码，那么一个碱基占用的是一个字节(一个字符长度)，而不是一个bit； 但实践中，我们大多采用编码的形式存储，即将 ATCG编码为00,01,10,11，这样一个字节可以存储4个碱基，4 X 750M = 3G

关于DNA序列数据编码，有很多人在研究，一些非主流的方法确实可以减少数据文件大小，但目前并未有本质性的突破，而且大多数方法的目的是解决永久存储的序列文件，也就是说你编码和解码都需要耗时很久，带来的性价比并不高，只适合于大量永久存储的序列数据；如果你天天都用的参考序列，实践中都是将比如人的3G的数据直接放在大型机的内存中， 谁要谁访问。 这是一个时间和空间的置换问题，目前并不存在任何一种可以同时获得短时间和低空间的方法

update GCC from 4.8 to 4.9 for FileZilla3.16 Client on Ubuntu14.04

$gcc -v

sudo add-apt-repository ppa:ubuntu-toolchain-r/test
sudo apt-get update
sudo apt-get install gcc-4.9 g++-4.9
sudo update-alternatives --install /usr/bin/gcc gcc /usr/bin/gcc-4.9 60 --slave /usr/bin/g++ g++ /usr/bin/g++-4.9

ano@ano-svr1:~/Downloads/FileZilla3/bin$ ./filezilla
./filezilla: /usr/lib/x86_64-linux-gnu/libstdc++.so.6: version `CXXABI_1.3.8' not found (required by ./filezilla)
./filezilla: /usr/lib/x86_64-linux-gnu/libstdc++.so.6: version `GLIBCXX_3.4.20' not found (required by ./filezilla)

[1]http://askubuntu.com/questions/466651/how-do-i-use-the-latest-gcc-on-ubuntu-14-04

High Performance SSH/SCP

http://mirror.internode.on.net/pub/OpenBSD/OpenSSH/portable/

http://www.psc.edu/index.php/hpn-ssh-patches/hpn-14-kitchen-sink-patches/viewcategory/24

Extract OpenSSH:







1



tar -xzvf openssh-6.6p1.tar.gz








Change directory in extracted folder and apply patch:







1

2



cd openssh-6.6p1

zcat /usr/src/openssh-6.6p1-hpnssh14v5.diff.gz | patch

Configure OpenSSH:







1



./configure –prefix=/usr –sysconfdir=/etc/ssh –with-pam








Remove old config files to prevent any conflicts:







1

2



rm /etc/ssh/ssh_config

rm /etc/ssh/sshd_config








Compile and install:







1

2



make

make install

Now we have the newest version of OpenSSH installed and patched with the improvements from HPN-SSH; however we still need to make some changes to the /etc/ssh/sshd_config to take advantage of them. Near the bottom of your config file you will see a section for HPN related options; I used the following options from other guides I found:







1

2

3

4

5

6

7

8

9

10

11

12



# the following are HPN related configuration options

# tcp receive buffer polling. disable in non autotuning kernels

TcpRcvBufPoll yes

 

# allow the use of the none cipher

#NoneEnabled no

 

# disable hpn performance boosts.

#HPNDisabled no

 

# buffer size for hpn to non-hpn connections

HPNBufferSize 8388608

Linux supports both /proc and sysctl (using alternate forms of the variable names - e.g. net.core.rmem_max) for inspecting and adjusting network tuning parameters. The following is a useful shortcut for inspecting all tcp parameters:
sysctl -a | fgrep tcp


For additional information on kernel variables, look at the documentation included with your kernel source, typically in some location such as /usr/src/linux-<version>/Documentation/networking/ip-sysctl.txt. There is a very good (but slightly out of date) tutorial on network sysctl's at http://ipsysctl-tutorial.frozentux.net/ipsysctl-tutorial.html.

If you would like to have these changes to be preserved across reboots, you can add the tuning commands to your the file /etc/rc.d/rc.local .

echo 1 > /proc/sys/net/ipv4/tcp_moderate_rcvbuf
           echo "8388608"> /proc/sys/net/core/wmem_max 
           echo "8388608"> /proc/sys/net/core/rmem_max 
           echo "4096 87380 8388608" > /proc/sys/net/ipv4/tcp_rmem
           echo "4096 87380 8388608" > /proc/sys/net/ipv4/tcp_wmem

# optimization start
# increase TCP max buffer size setable using setsockopt()
net.ipv4.tcp_rmem = 4096 87380 8388608
net.ipv4.tcp_wmem = 4096 87380 8388608
# increase Linux auto tuning TCP buffer limits
# min, default, and max number of bytes to use
# set max to at least 4MB, or higher if you use very high BDP paths
net.core.rmem_max = 8388608
net.core.wmem_max = 8388608

net.core.netdev_max_backlog = 5000
net.ipv4.tcp_window_scaling = 1
# optimization end

[1] http://www.psc.edu/index.php/hpn-ssh

[2]http://stackoverflow.com/questions/8849240/why-when-i-transfer-a-file-through-sftp-it-takes-longer-than-ftp

[3]http://www.cyberciti.biz/tips/sshd-server-optimization.html

aliyun client

容器集群api访问

curl --insecure --cert ~/.docker/aliyun/ClusterName/cert.pem --key ~/.docker/aliyun/ClusterName/key.pem https://123.123.123.123:1234/projects/

or

ano@node1:~$ cat .curlrc

cert = /home/ano/cert/cert.pem

key = /home/ano/cert/key.pem

insecure =

curl https://123.123.123.123:1234/projects/

服务器安全(安骑士)Agent如何卸载

#!/bin/bash

#check linux Gentoo os 

var=`lsb_release -a | grep Gentoo`

if [ -z "${var}" ]; then 

var=`cat /etc/issue | grep Gentoo`

fi

if [ -d "/etc/runlevels/default" -a -n "${var}" ]; then

LINUX_RELEASE="GENTOO"

else

LINUX_RELEASE="OTHER"

fi

stop_aegis(){

killall -9 aegis_cli >/dev/null 2>&1

killall -9 aegis_update >/dev/null 2>&1

killall -9 aegis_cli >/dev/null 2>&1

killall -9 AliYunDun >/dev/null 2>&1

killall -9 AliHids >/dev/null 2>&1

killall -9 AliYunDunUpdate >/dev/null 2>&1

    printf "%-40s %40s\n" "Stopping aegis" "[  OK  ]"

}

remove_aegis(){

if [ -d /usr/local/aegis ];then

    rm -rf /usr/local/aegis/aegis_client

    rm -rf /usr/local/aegis/aegis_update

rm -rf /usr/local/aegis/alihids

fi

}

uninstall_service() {

   

   if [ -f "/etc/init.d/aegis" ]; then

/etc/init.d/aegis stop  >/dev/null 2>&1

rm -f /etc/init.d/aegis 

   fi

if [ $LINUX_RELEASE = "GENTOO" ]; then

rc-update del aegis default 2>/dev/null

if [ -f "/etc/runlevels/default/aegis" ]; then

rm -f "/etc/runlevels/default/aegis" >/dev/null 2>&1;

fi

    elif [ -f /etc/init.d/aegis ]; then

         /etc/init.d/aegis  uninstall

   for ((var=2; var<=5; var++)) do

if [ -d "/etc/rc${var}.d/" ];then

rm -f "/etc/rc${var}.d/S80aegis"

   elif [ -d "/etc/rc.d/rc${var}.d" ];then

rm -f "/etc/rc.d/rc${var}.d/S80aegis"

fi

done

    fi

}

stop_aegis

uninstall_service

remove_aegis

printf "%-40s %40s\n" "Uninstalling aegis"  "[  OK  ]"

如何从源码直接运行:

$ git clone https://github.com/aliyun/aliyun-cli.git
$ cd aliyuncli/aliyuncli
$ python aliyuncli.py ecs DescribeRegions --output json

源码下载后, 可以不安装直接运行, 前提是要安装阿里云python版SDK.


$ pip install virtualenv

$ virtualenv venv

$ . venv/bin/activate
#for ecs

$ pip install aliyun-python-sdk-ecs
#for oss
$ pip install aliyun-python-sdk-oss

$ pip install aliyuncli
#命令行自动补全

$complete -C '~/venv/bin/aliyun_completer' aliyuncli

$echo "complete -C '~/venv/bin/aliyun_completer' aliyuncli">>~/.bash_profile


$ aliyuncli configure(aliyuncli ecs ConfigVersion)
$ cat ~/.aliyuncli/configure
[default]
region = cn-beijing

$aliyuncli configure list

创建实例

$aliyuncli ecs ImportInstance --filename ali-ecs.json
$cat ali-ecs.json
{
    "RegionId": "cn-beijing",
"ZoneId": "cn-beijing-a",
 
    "IoOptimized": "none",
    "InternetChargeType": "PayByTraffic",
    "Description": "",
    "InstanceId": "i-t2",
    "HostName": "t2",
    "ClusterId": "",
    "ImageId": "ubuntu1404_64_40G_aliaegis_20160222.vhd",
    "InstanceChargeType": "PostPaid",
    "InstanceNetworkType": "vpc",
    "VSwitchId": "vsw-25hjbr6b7",
    "PrivateIpAddress": "192.168.1.5",
    "InstanceType": "ecs.s1.small",
     
    "InstanceName": "t2",
    "Cpu": 1,
 
    "SecurityGroupId": "sg-25i8ahfb7",
    "Memory": 2048
}

$aliyuncli ecs AllocateEipAddress
$aliyuncli ecs AssociateEipAddress --InstanceId i-25nl69m89 --AllocationId eip-25dhrq5tr

删除实例

(venv) ano@ano-svr1:~$ aliyuncli ecs StopInstance --InstanceId i-<id>

(venv) ano@ano-svr1:~$ aliyuncli ecs DeleteInstance --InstanceId i-<id>

通过API更换ECS的系统盘的步骤

相较于在控制台上一键更换系统，通过API来更换系统盘需要执行更精细的步骤，具体的步骤为停机、更换系统盘、重置密码、启动ECS，具体的步骤如下（其中API格式以适用于AliyunCLI的命令为例）： 

1. 停止ECS，可以使用StopInstance方法来停止，如：

aliyuncli ecs StopInstance --InstanceId i-xxxxx

2. 使用ReplaceSystemDisk来更换系统盘，ImageId参数中指定需要更换的镜像ID（该ID可以使用DescribeImages方法来查询可用的镜像ID），：

aliyuncli ecs ReplaceSystemDisk --InstanceId i-xxxxxxxx --ImageId xxxxxxx.vhd

目前系统盘已经支持扩容，若需要在更换时同时扩容系统盘，可以加上SystemDiskSize参数，如：

aliyuncli ecs ReplaceSystemDisk --InstanceId i-xxxxxxxx --ImageId xxxxxxx.vhd --SystemDiskSize 50G

3. 重置密码，使用ModifyInstanceAttribute，指定Password参数来设置新的密码，因为涉及到机密信息，建议使用HTTPS来传输这个API请求

4. 最后使用StartInstance方法启动ECS：

aliyuncli ecs StartInstance --InstanceId i-xxxxxx

aliyuncli oss支持

OSS SDK安装

使用OSS的功能，需要与OSS SDK配合使用，因此需要您安装OSS SDK（默认CLI会自动安装OSS SDK）.

推荐采用pip方式

pip 安装SDK会自动帮助您处理好各种依赖还有升级, 因此强烈建议您采用pip方式安装SDK.

您可以直接采用下面的命令安装OSS SDK:

$ sudo pip install aliyun-python-sdk-oss

SDK升级：

sudo pip install --upgrade aliyun-python-sdk-oss

aliyuncli oss 命令说明

下面文档将通过命令说明和使用示范进行oss 命令的展示：

Config

• 命令说明：
  Config --host oss.aliyuncs.com --accessid accessid --accesskey accesskey --sts_token token 配置aliyuncli 使用的默认host，ID 和KEY。默认的host 为oss.aliyuncs.com如果需要访问oss-internal.aliyuncs.com 可以加上--host oss-internal.aliyuncs.com。
• 使用示范：
  aliyuncli oss Config --host oss.aliyuncs.com --accessid accessid --accesskey accesskey --sts_token token

GetAllBucket

• 命令说明：
  GetAllBucket 用来显示用户创建的bucket
• 使用示范：
  aliyuncli oss GetAllBucket

CreateBucket

• 命令说明：
  CreateBucket oss://bucket --acl [acl] oss://bucket 表示bucket。--acl 参数可以传入，也可以不传入。
• 使用示范：
  aliyuncli oss CreateBucket oss://myfirstbucket --acl public-read
  aliyuncli oss CreateBucket oss://mysecondbucket --acl private
  aliyuncli oss CreateBucket oss://mythirdbucket

DeleteBucket

• 命令说明：
  DeleteBucket oss://bucket 删除bucket 的命令
• 使用示范：
  aliyuncli oss DeleteBucket oss://mybucket
  aliyuncli oss DeleteBucket oss://myfirstbucket

DeleteWholeBucket

• 注意：
  该命令十分危险，将会删除所有的数据，并且不可恢复。请慎重使用。
• 命令说明：
  DeleteWholeBucket oss://bucket 删除bucket 及其内部object 以及multipart 相关的内容。
• 使用示范：
  aliyuncli oss DeleteWholeBucket oss://mybucket

GetAcl

• 命令说明：
  GetAcl oss://bucket 获取bucket 的访问控制权限
• 使用示范：
  aliyuncli oss GetAcl oss://mybucket

SetAcl

• 命令说明：
  SetAcl oss://bucket –acl [acl] 修改bucket 的访问控制权限。acl 只允许为private ， public-read ，public-read-write 三个当中的一个。
• 使用示范：
  aliyuncli oss SetAcl oss://mybucket --acl private

List

• 命令说明：
  List oss://bucket/[prefix] [marker] [delimiter] [maxkeys] 列出bucket 中的object。
• 使用示范：
  aliyuncli oss List oss://mybucket/folder1/folder2
  aliyuncli oss List oss://mybucket/folder1/folder2 maker1

MkDir

• 命令说明：
  MkDir oss://bucket/dirname 创建一个以“/”结尾的object，并且size 为0。
• 使用示范：
  aliyuncli oss MkDir oss://mybucket/folder

ListAllObject

• 命令说明：
  ListAllObject oss://bucket/[prefix] 显示bucket 下所有的object，可以指定prefix 来显示。
• 使用示范：
  aliyuncli oss ListAllObject oss://mybucket
  aliyuncli oss ListAllObject oss://mybucket/testfolder/

DeleteAllObject

• 命令说明：
  DeleteAllObject oss://bucket/[prefix] 删除bucket 下所有的object，可以指定特定的prefix 来删除。
• 使用示范：
  aliyuncli oss DeleteAllObject oss://mybucket
  aliyuncli oss DeleteAllObject oss://mybucket/testfolder/

DownloadAllObject

• 命令说明：
  DownloadAllObject oss://bucket/[prefix] localdir --replace false将bucket 下的object 下载到本地目录，并且保持目录结构。可以指定prefix下载。--replace false 表示如果下载时，本地已经存在同名文件，不会覆盖。true 则会覆盖。
• 使用示范：
  aliyuncli oss DownloadAllObject oss://mybucket /tmp/folder
  aliyuncli oss DownloadAllObject oss://mybucket /tmp/folder --replace=false
  aliyuncli oss DownloadAllObject oss://mybucket /tmp/folder --replace=true

DownloadToDir

• 命令说明：
  DownloadToDir oss://bucket/[prefix] localdir --replace=false 将bucket 下的object 下载到本地目录，并且保持目录结构。可以指定prefix下载。--replace false 表示如果下载时，本地已经存在同名文件，不会覆盖。true 则会覆盖。同downloadallobject 效果一样。
• 使用示范：
  aliyuncli oss DownloadToDir oss://mybucket /tmp/folder
  aliyuncli oss DownloadToDir oss://mybucket /tmp/folder --replace false
  aliyuncli oss DownloadToDir oss://mybucket /tmp/folder --replace true

UploadObjectFromLocalDir

• 命令说明：
  UploadObjectFromLocalDir localdir oss://bucket/[prefix] --check_point check_point_file --replace false --check_md5 false --thread_num 5将本地目录里的文件上传到bucket 中。例如localdir 为/tmp/里面有a/b，a/c，a 三个文件，则上传到OSS 中为oss://bucket/a/b，oss://bucket/a/c，oss://bucket/a。如果指定了prefix 为mytest，则上传到OSS 中为oss://bucket/mytest/a/b，oss://bucket/mytest/a/c，oss://bucket/mytest/a。--check_point check_point_file 是指定文件。指定文件后，osscmd 会将已经上传的本地文件以时间戳的方式放到check_point_file 中，uploadfromdir命令会将正在上传的文件的时间戳和check_point_file 记录的时间戳进行比较。如果有变化则会重新上传，否则跳过。默认情况下是没有check_point_file 的。
• 注意：
  由于check_point_file 文件中记录的是上传的所有文件的。所以当上传文件特别多的时候，check_point_file 会特别巨大。
• 使用示范：
  aliyuncli oss UploadObjectFromLocalDir /mytemp/folder oss://mybucket
  aliyuncli oss UploadObjectFromLocalDir /mytemp/folder oss://mybucket --check_point_file /tmp/mytemp_record.txt

Put

• 命令说明：
  Put localfile oss://bucket/object --content-type [content_type]--headers "key1:value1, key2:value2" 上传一个本地的文件到bucket 中，可以指定object 的content-type，或则指定自定义的headers。
• 使用示范：
  aliyuncli oss Put myfile.txt oss://mybucket
  aliyuncli oss Put myfile.txt oss://mybucket/myobject.txt
  aliyuncli oss Put myfile.txt oss://mybucket/test.txt --content-type plain/text --headers “x-oss-meta-des:test,x-oss-meta-location:CN”
  aliyuncli oss Put myfile.txt oss://mybucket/test.txt --content-type plain/text

Get

• 命令说明：
  Get oss://bucket/object localfile 将object 下载到本地文件。
• 使用示范：
  aliyuncli oss Get oss://mybucket/myobject /tmp/localfile

MultiGet

• 命令说明：
  MultiGet oss://bucket/object localfile --thread_num 5 将object 以多线程的方式下载到本地文件。
• 使用示范：
  aliyuncli oss MultiGet oss://mybucket/myobject /tmp/localfile
  aliyuncli oss MultiGet oss://mybucket/myobject /tmp/localfile

Cat

• 命令说明：
  Cat oss://bucket/object 读取object 的内容，直接打印出来。在object 内容比较大的时候请不要使用。
• 使用示范：
  aliyuncli oss Cat oss://mybucket/myobject

Meta

• 命令说明：
  Meta oss://bucket/object读取object 的meta 信息，打印出来。meta 信息包括content-type，文件长度，自定义meta 等内容。
• 使用示范：
  aliyuncli oss Meta oss://mybucket/myobject

Copy

• 命令说明：
  Copy oss://source_bucket/source_object oss://target_bucket/target_object --headers="key1:value1,key2:value2"将源bucket 的源object 复制到目的bucket 中的目的object。
• 使用示范：
  aliyuncli oss Copy oss://bucket1/object1 oss://bucket2/object2

Delete

• 命令说明：
  Delete oss://bucket/object 删除object。
• 使用示范：
  aliyuncli oss Delete oss://mybucket/myobject

SignUrl

• 命令说明：
  SignUrl oss://bucket/object --timeout [timeout_seconds] 生成一个包含签名的URL，并指定超时的时间。适用于bucket 为私有时将特定的object 提供给他人访问。
• 使用示范：
  aliyuncli oss SignUrl oss://mybucket/myobject

Init

• 命令说明：
  Init oss://bucket/object 初始化生成一个Upload ID。这个Upload ID 可以配合后面的MultiUpload命令来使用。
• 使用示范：
  aliyuncli oss Init oss://mybucket/myobject

ListPart

• 命令说明：
  ListPart oss://bucket/object --upload_id xxx显示指定object 的Upload ID 下已经上传的Parts。相关概念见OSS API文档。必须要指定Upload ID。
• 使用示范：
  aliyuncli oss ListPart oss://mybucket/myobject --upload_id 75835E389EA648C0B93571B6A46023F3

ListParts

• 命令说明：
  ListPart s oss://bucket显示bucket 中未完成的multipart Upload ID 和object。一般在删除bucket提示bucket 非空的情况下可以用这个命令查看是否有multipart 相关的内容。
• 使用示范：
  aliyuncli oss ListParts oss://mybucket

GetAllPartSize

• 命令说明：
  GetAllPartSize oss://bucket 显示bucket 中还存在的Upload ID 已经上传的Parts 的总大小。
• 使用示范：
  aliyuncli oss GetAllPartSize oss://mybucket

Cancel

• 命令说明：
  Cancel oss://bucket/object --upload_id xxx 终止Upload ID 对应的Multipart Upload 事件。
• 使用示范：
  aliyuncli oss Cancel oss://mybucket/myobject --upload_id D9D278DB6F8845E9AFE797DD235DC576

MultiUpload

• 命令说明一：
  MultiUpload localfile oss://bucket/object 将本地文件以multipart 的方式上传到OSS。
• 使用示范：
  aliyuncli oss MultiUpload /tmp/localfile.txt oss://mybucket/object
• 命令说明二：
  MultiUpload localfile oss://bucket/object --upload_id xxx --thread_num 10 --max_part_num 1000将本地文件以multipart 的方式上传到OSS。本地文件划分的块数由max_part_num 来指定。这个命令在实现的时候，会先去判断Upload ID对应的Parts 的ETag 是否和本地文件的MD5 值是否相等，相等则跳过上传。所以如果在使用之前生成一个Upload ID，作为参数传进来。即使上传没有成功，重复执行相同的multiupload 命令可以达到一个断点续传的效果。
• 使用示范：
  aliyuncli oss MultiUpload /tmp/localfile.txt oss://mybucket/object --upload_id D9D278DB6F8845E9AFE797DD235DC576
  aliyuncli oss MultiUpload /tmp/localfile.txt oss://mybucket/object --thread_num 5
  aliyuncli oss MultiUpload /tmp/localfile.txt oss://mybucket/object --max_part_num 100

UploadPartFromFile

• 命令说明：
  UploadPartFromFile localfile oss://bucket/object --upload_id xx --part_number xxx 主要用于测试，不推荐使用。

UploadPartFromString

• 命令说明：
  UploadPartFromString oss://bucket/object --upload_id xxx --part_number xxx --data xxx 主要用于测试，不推荐使用。

docker

配置Docker加速器

您可以使用如下的脚本将mirror的配置添加到docker daemon的启动参数中。

echo "DOCKER_OPTS=\"\$DOCKER_OPTS --registry-mirror=https://<your_mir>.mirrror.aliyuncs.com\"" | sudo tee -a /etc/default/docker

https://cr.console.aliyun.com/ 查看专属加速器地址

[1] https://github.com/aliyun/aliyun-cli