インタークラウド環境構築システムによるアプリケーション環境構築支援のための機能拡張

全文

(1)Vol.2018-OS-144 No.10 2018/7/31. IPSJ SIG Technical Report. 1,a). NII. 1. 1. 2. 1. SINET VCP VCP (1). VCP (2). (3). Jupyter. (4) (5)VCP. GPU. VCP. GPU VCP. SINET5. Enhancement of an On-demand Configuration System for Inter-Cloud Applications. 1. [1] NII. SINET5[2], [3], [4]. 800 100Gbps NII. SINET 30. SINET5. 10. [5]. VPN. SINET. LMS. 1. National Institute of Informatics 2. a). Gunma University [email protected]. c 2018 Information Processing Society of Japan ⃝. 1.

(2) Vol.2018-OS-144 No.10 2018/7/31. IPSJ SIG Technical Report Cloud Datacenters. VC Controllers Jupyter Notebook / Python/REST I/F. app bins/libs Container. VCP Manager. VCP Base Container. VC Controller Core. OS (kernel) Hardware / VM. VCP User Virtual Router. ü Machine allocation ü Software deployment. On-demand Configuration Service. L2VPN. L2VPN. On-demand Configuration Service. Template. app bins/libs Container. SINET VPN. VCP System in Datacenter A. SINET VPN. University or Research Institute. Public/Academic Cloud. Template. 2 VCP Universities or Research Institutes. 1. 1 Virtual Cloud Provider VCP [6], [7]. SINET. e. L2VPN. 1. Moodle [8] VCP Docker[9]. VCP [6] SINET5 2.1. [7] Jupyter. VCP. VCP. SINET. 1. Notebook[10] VCP. NII Docker. Literate Computing for Reproducible Infrastructure LC4RI. Jupyter. Notebook API. [11], [12], [13] VCP VC. 2.1.1 VCP. GPU. 2. VCP. VCP. VCP. VCP Manager VCP. (1). (3). Virtual Router. VCP. (4) VCP. (5)VCP GPU. VC VCP. GPU. VPN. VCP VCP. VC. VC Controller. (2). VCP. VC VC. GPU VCP. VCP SINET5 BGP. 2.. SINET5 SINET. c 2018 Information Processing Society of Japan ⃝. L2VPN. L2VPN 2.

(3) Vol.2018-OS-144 No.10 2018/7/31. IPSJ SIG Technical Report. Jupyter Notebook. #. VCP SDK (Python). vc = VcpSDK("vcp_accesskey", "my_vc_name") #. VC Controller Core. flaver"small". VC. nodes = vc.unit.create("instance_name", spec) # vc. VC. # vc.cleanup(). Terraform Azure. spec. # spec. VCP REST API. AWS. VC. spec = vc.spec.find("aws", "small"). Configuration file. Open Stack. 4 VCP SDK. .. ### VCC. ###. vcc: host: VC Controller IP. Public/Academic Clouds. name: VCC Name ### AWS. 3 VCP. ###. aws: access_key: "AWS. ". secret_key: "AWS. ". private_network: "default". IPSec. ###. VPN. ###. sakura:. Open vSwitch OVS [14]. token: ". ". secret: ". Quagga[15]. ". private_network: "default" :. 5 VCP SDK. Docker. raform Terraform. 2. VM. VCP VC. Provider. VM/BM. AWS Amazon Web Services. Google Cloud Platform. GCP. Microsoft Azure OpenStack. Provider Docker-in-Docker. Ter-. raform. Terraform API. Provider. VC. Python VCP SDK. 4 4. VM. BM. 1. VC. AWS. 1 2 4. VC. VC 6. VC 2.2 VCP. aws. VCP. API. spec 3 spec Azure. azure. VC. VC. small medium large. Terraform[16] REST API. VCP SDK small. VCP. Python. VCP SDK. 3 VCP REST API. YAML Configuration file. cleanup() POST Ter-. c 2018 Information Processing Society of Japan ⃝. AWS 3.

(4) Vol.2018-OS-144 No.10 2018/7/31. IPSJ SIG Technical Report. 5. YAML SINET. VCP SDK REST API. VPN. VCP. VC VPN. 2.3. Jupyter Notebook VCP. Terraform. AWS Azure GCP OpenStack HPC. OpenHPC[17]. VCP Terraform REST API. LMS. VCP. VCP SDK. moodle[18]. BlueGreenDeployment[19]. GPU. CPU. [8] VDI. Guacamole[20]. GPU. VDI. FPGA. Galaxy. [21]. Docker VCP. 3. VCP. 2.4. (1) VC (2) VC. (3). VC. (4). VPN. (5) VCP. VC. GPU. VC 3.1 VC VC. VCC. VCC Web. Web UI 6. VC. VCC. Web UI VC. GakuNin. [22] NII. c 2018 Information Processing Society of Japan ⃝. 4.

(5) Vol.2018-OS-144 No.10 2018/7/31. IPSJ SIG Technical Report. cAdvisor Shibboleth. VCC Manager UI. VC Controller. VCC info. VC Controller Core. Token mngmt DB. VC. Notebook / CLI VCP SDK based code User A's Token. Vault. Prometheus. pull cAdvisor. audit log. VCC documents. VCC. User A's Token. VC. Prometheus User A's Token. cAdvisor. Prometheus Prometheus. VCC User. VCC Manager. GUI 6 VC. Grafana. Grafana. Prometheus ID 3.3 VCP 3.1. VCC. VC. Vault. VCC. KVS VCP VCC. Web UI Vault. VCC VCC. VCP 5. HashiCorp Vault Vault [23]. REST API. Vault Vault. Key Value Store KVS. VCC Vault. VCC. VCP VCC. KVS. VCC. Cubbyhole. VCC. HashiCorp Vault. ssh. Vault VCP. VC. 3.2 VC a). Jupyter Notebook VCP. VC. VCC. b)VCC Vault. c) a) Prometheus[25] c). VCP. cAdvisor[24] b) 2.3. Grafana[26]. Moodle. LMS. Moodle API LUKS Linux Unified Key Setup cAdvisor. Google. Docker VCC. c 2018 Information Processing Society of Japan ⃝. 5.

(6) Vol.2018-OS-144 No.10 2018/7/31. IPSJ SIG Technical Report. VC 1. VC 2. VM(Ubuntu 16.04; ami=niivcp-gpu-20180620). VC 3. Nvidia driver(nvidia-384). Region. Region A. Region B. Cloud Provider. Private Cloud A. Private Cloud B. Cloud API. Region C. OpenStack Newton API. 7. Region D. …. Region E. Region F. AWS. Azure. AWS API. Azure API. Base Container(Ubuntu 16.04; image=gpu-base:nvidia-384). …. nvidia-docker2. docker-ce(18.03.1-ce). libcuda1-384. Nvidia driver(nvidia-384). …. API. Application Container. Application Container. nvidia/cuda. chainer/chainer. Application Container tensorflow/tensorflow:latest-gpu. 3.4 8 GPU. VCP. Docker-in-Docker. VCP 7. API. 2. Cloud API API. Cloud Provider. 3.5 VCP. GPU. Cloud API. Docker-in-Docker. GPU. Region VCP VC. GPU API. NVIDIA. NVIDIA Docker nvidia-docker [27] GPU. VCP 7. Cloud API. Open Con-. Cloud Provider. tainer Initiative OCI [28] nvidia-docker2 swarm. docker-compose. Docker NVIDIA Docker. Cloud API. VCP. SPI Service. Docker-in-Docker. Provider Interface Terraform. VCP. VC. Cloud Provider. GPU YAML. GPU. YAML. GPU. small/medium/large 8 Cloud API VCP SDK. Docker-in-Docker VM. 7. VCP SDK Cloud Provider setter/getter. nvidia-docker2 Nvidia. GCP. OpenStack GCP. driver. VM. OS. VM. 3 Cloud API. c 2018 Information Processing Society of Japan ⃝. Cloud API. Chainer[29]. TensorFlow[30]. nvidia-doker2 6.

(7) Vol.2018-OS-144 No.10 2018/7/31. IPSJ SIG Technical Report 350. 300 302.28. 299.11. 300. 239.98. 250. 250. 240.24. 200. 200. 150 150. 100. 100. 65.64. 63.35. 63.22. Chainer/MNIST. Chainer/MNIST w/VCP. 65.37. 50. 50 0. 0 Chainer/MNIST. Chainer/MNIST w/VCP. TensorFlow/CIFAR10 TensorFlow/CIFAR10 w/VCP. 9 AWS. TensorFlow/CIFAR10 TensorFlow/CIFAR10 w/VCP. 10 Azure Galaxy Template (Jupyter Notebook). AWS. Azure. Galaxy Users. VCP User. VCP. (VC Administrator). AWS. Azure. VCP Manager. Unit: galaxy+nfs. VC Controller GW. App: Galaxy +Slurmctld NFS Server. VCP. 4. GPU. VC. nvidia-docker2. AWS VPC. VCP system. GPU VC. OpenStack. BC BM. node #1 App: Slurmd BC BM. disk. Docker-in-Docker Chainer. TensorFlow. AWS AWS AWS. Azure. Azure. Unit: compute. node #2 App: Slurmd BC NFS Client BM. Scale-out. node #n App: Slurmd BC VM. node #n+1 App: Slurmd BC VM. SINET5. Persistent Data • Galaxy DB • Tool Config • Workflow • Reference Data etc …. NII Cloud: @chiba. VCP. Unit: galaxy-compute. Amazon: @ap-northeast-1 BM. Bare-metal Machine. App. Application Container. VM. Virtual Machine. BC. VCP Base Container. 11 Galaxy. GPU. Azure. w/VCP. VC. Docker-in-Docker Chainer. AWS Azure. VC. MNIST. VC. VCP TensorFlow VC 1. CIFAR10. AWS. 1.06% Azure. 0.11%. 2 VCP. 5. Galaxy. Chainer TensorFlow Chainer MNIST[31] TensorFlow. VCP. Galaxy. CIFAR10[32]. NII AWS. MNIST. 28×28 Chainer. CIFAR10. NII. OpenStack. ×20. 600. SINET5. SC17 VCP. 32×32. 10. VCP. TensorFlow. 11. VCP. 12. Jupyter Notebook. 1. Galaxy 9. 10. AWS. Azure 10. 1.0. 1.. VCP VC. AWS Azure. 1. VC. 2. NFS. GPU. c 2018 Information Processing Society of Japan ⃝. 7.

(8) Vol.2018-OS-144 No.10 2018/7/31. IPSJ SIG Technical Report. 1 OS CPU CPU. GPU NVIDIA. AWS, AWS-VCP. Azure, Azure-VCP. Ubuntu 16.04.4 LTS. Ubuntu 16.04.4 LTS. v. 4.4.0-128-generic. v. 4.13.0-1018-azure. Intel(R) Xeon(R) CPU E5-2686 v4 @ 2.30GHz. Intel(R) Xeon(R) CPU E5-2690 v3 @ 2.60GHz. 4. 6. 59 GiB. 55 GiB. p2.xlarge. Standard NV6. Tesla K80 × 1. Tesla M60 × 1. v. 384.130. v. 384.130. 2 Chainer. TensorFlow. chainer/chainer. tensorflow/tensorflow. :v4.2.0-python3. :1.8.0-gpu-py3. Ubuntu 16.04.4 LTS. Ubuntu 16.04.4 LTS. Python. v. 3.5.2. v. 3.5.2. CUDA. v. 8.0.61. v. 9.0.176. cuDNN. v. 6.0.21. –. OS. 14. 12 Galaxy. Jupyter Notebook. 15. AWS. SLURM SLURM Galaxy 3.. 13. Galaxy. Galaxy. 14. 2. 13 Galaxy. VC 2.. 4. SINET5 Galaxy. c 2018 Information Processing Society of Japan ⃝. Virtual Private Cloud. AWS. VPC. VCP 8.

(9) Vol.2018-OS-144 No.10 2018/7/31. IPSJ SIG Technical Report. VC 5.. 2. Linux. 2.. AWS. VC. 7.. NFS 6.. 3.. Galaxy AWS. CYCLONE. VM. VCP. 4. Jupyter. 4 VCP. 15. VCP. 4 (1). AWS. (2). (3). (4). NFS. (5)VCP. 1 GPU. SLURM. VCP. GPU. 6.. VCP. InterCloud[33]. Contrail[34]. SINET5 VC. SLA. FPGA. AWS. JST CREST. CloudFormation[35], OpenStack Heat[36] Apache Libcloud[37]. JPMJCR1501. API. API. Terraform VCP REST API. [1]. VCP SDK RightScale[38]. [2] [3]. Jupyter Notebook. [4]. GTS[39] Skyport[40] CYCLONE[41] GTS ´ GEANT Testbed Service OpenStack SDN. [5] [6]. Skyport AWE/Shock. Docker CYCLONE. c 2018 Information Processing Society of Japan ⃝. [7]. GICTF: GICTF White Paper, pp. 1–31 (2010). SINET, https://www.sinet.ad.jp/ (accessed on 0107-2018). S. Urushidani and S. Abe and K. Yamanaka and K. Aida and S. Yokoyama and H. Yamada and M. Nakamura and K. Fukuda and M. Koibuchi and S. Yamada : New Directions for a Japanese Academic Backbone Network, IEICE Transactions on Information and Systems, E98.D (3), pp. 546–556 (2015). Kurimoto, T., Urushidani, S., Yamada, H., Yamanaka, K., Nakamura, M., Abe, S., Fukuda, K., Koibuchi, M., Ji, Y., Takakura, H., and Yamada, S.: A fully meshed backbone network for data-intensive sciences and SDN services, Proc. ICUFN2016, pp. 909–911 (2016). https://cloud.gakunin.jp/ (accessed on 01-07-2018). S. Yokoyama and Y. Masatani and T. Ohta and O. Ogasawara and N. Yoshioka and K. Liu and K. Aida: Reproducible Scientific Computing Environment with Overlay Cloud Architecture, Proc. 9th IEEE Cloud, pp. 774–781 (2016). Takefusa, A., Yokoyama, S., Masatani, Y., Tanjo, T., Saga, K., Nagaku, M. and Aida, K.: Virtual Cloud Service System for Building Eﬀective Inter-Cloud Applica-. 9.

(10) Vol.2018-OS-144 No.10 2018/7/31. IPSJ SIG Technical Report. tions, Proc. IEEE CloudCom2017, pp. 296–303 (2017).. [34]. [8]. [9] [10] [11]. Moodle Jupyter Notebook MoodleMoot Japan 2018 (2018). Docker, https://www.docker.com/ (accessed on 01-072018). Jupyter Notebook, http://jupyter.org/ (accessed on 01-07-2018). Masatani, Y.: Collaboration and automated operation as literate computing for reproducible infrastructure, https://conferences.oreilly.com/jupyter/ jup-ny/public/schedule/detail/59995 (accessed on 01-07-2018) (2017).. [12]. [13] [14] [15] [16] [17] [18] [19]. [20] [21] [22] [23] [24] [25] [26] [27] [28] [29]. [30] [31]. [32]. [33]. [35] [36] [37] [38] [39]. IPython notebook , SC2015-6 pp. 27–32 (2015). NII Cloud Operation Team, https://github.com/ NII-cloud-operation/ (accessed on 01-07-2018). Open vSwitch, http://openvswitch.org/ (accessed on 01-07-2018). Quagga Routing Suite, https://www.quagga.net/ (accessed on 01-07-2018). Terraform, https://www.terraform.io/ (accessed on 01-07-2018). OpenHPC, https://openhpc.community/ (accessed on 01-07-2018). Moodle, https://moodle.org/ (accessed on 01-072018). BlueGreenDeployment, https://martinfowler.com/ bliki/BlueGreenDeployment.html (accessed on 01-072018). Apache Guacamole, https://guacamole.incubator. apache.org/ (accessed on 01-07-2018). Galaxy, Data intensive biology for everyone, https: //galaxyproject.org/ (accessed on 01-07-2018). https://www.gakunin.jp/ (accessed on 01-07-2018). HashiCorp Vault, https://www.vaultproject.io/ (accessed on 01-07-2018). cAdvisor, https://github.com/google/cadvisor (accessed on 01-07-2018). Prometheus, https://prometheus.io/ (accessed on 0107-2018). Grafana, https://grafana.com/ (accessed on 01-072018). NVIDIA Docker, https://github.com/NVIDIA/ nvidia-docker (accessed on 01-07-2018). Open Container Initiative, https://www. opencontainers.org/ (accessed on 01-07-2018). Tokui, S., Oono, K., Hido, S. and Clayton, J.: Chainer: a Next-Generation Open Source Framework for Deep Learning, Proc. Workshop on Machine Learning Systems (LearningSys) in NIPS, pp. 1–6 (2015). TensorFlow, https://www.tensorflow.org/ (accessed on 01-07-2018). LeCun, Y., Cortes, C. and Burges, C. J.: The MNIST Database of handwritten digits, http://yann.lecun. com/exdb/mnist/ (accessed on 01-07-2018). Krizhevsky, A.: The CIFAR-10 dataset, https://www. cs.toronto.edu/~kriz/cifar.html (accessed on 0107-2018). Buyya, R., Ranjan, R. and Calheiros, R. N.: InterCloud: Utility-oriented Federation of Cloud Computing Environments for Scaling of Application Services, Proc. ICA3PP’10, Springer-Verlag, pp. 13–31 (2010).. c 2018 Information Processing Society of Japan ⃝. [40]. [41]. M. Coppola and P. Dazzi and A. Lazouski and J. Jensen and I. Johnson and P. Kershaw and P. Mori and F. Martinelli: The CONTRAIL approach to Cloud Federations, Proc. Science - International Symposium on Grids and Clouds 2012 (019), pp. 1–14. AWS CloudFormation, https://aws.amazon.com/ cloudformation/ (accessed on 01-07-2018). OpenStack Heat, https://wiki.openstack.org/wiki/ Heat (accessed on 01-07-2018). Apache Libcloud, https://libcloud.apache.org/ (accessed on 01-07-2018). RightScale, https://www.rightscale.com/ (accessed on 01-07-2018). ´ GTS (GEANT Testbed Service), https://www.geant. org/Services/Connectivity_and_network/GTS/ (accessed on 01-07-2018). W. Gerlach and W. Tang and K. Keegan and T. Harrison and A. Wilke and J. Bischof and M. D’Souza and S. Devoid and D. Murphy-Olson and N. Desai and F. Meyer: Skyport - Container-Based Execution Environment Management for Multi-Cloud Scientific Workflows, Proc. IEEE DataCloud ’14, pp. 25–32 (2014). D. Gallico and M. Biancani and C. Blanchet and M. Bedri and J. F. Gibrat and J. I. A. Baranda and D. Hacker and M. Kourkouli: CYCLONE: A Multi-cloud Federation Platform for Complex Bioinformatics and Energy Applications (Short Paper), Proc. IEEE CloudNet, pp. 146–149 (2016).. 10.

(11)