Building machine learning systems with Python (779436), страница 45
Текст из файла (страница 45)
Then check Create a new keypair. Name the key pair file awskeys.pem. Download and save this file somewheresafe! This is the SSH (Secure Shell) key that will enable you to log in to your cloudmachine. Accept the remaining defaults and your instance will launch.[ 280 ]Chapter 12You will now need to wait a few minutes for your instance to come up. Eventually,the instance will be shown in green with the status as running:In the preceding screenshot, you should see the Public IP which can be used to log into the instance as follows:$ ssh -i awskeys.pem ec2-user@54.93.165.5Therefore, we will be calling the ssh command and passing it the key files that wedownloaded earlier as the identity (using the -i option).
We are logging in as userec2-user at the machine with the IP address as 54.93.165.5. This address will, ofcourse, be different in your case. If you choose another distribution for your instance,the username may also change.
In this case, try logging in as root, ubuntu (forUbuntu distribution), or fedora (for fedora distribution).[ 281 ]Bigger DataFinally, if you are running a Unix-style operating system (including Mac OS), youmay have to tweak its permissions by calling the following command:$ chmod 600 awskeys.pemThis sets the read/write permission for the current user only. SSH will otherwisegive you an ugly warning.Now you should be able to log in to your machine.
If everything is okay, you shouldsee the banner, as shown in the following screenshot:This is a regular Linux box where you have sudo permission: you can run anycommand as the superuser by prefixing it with sudo. You can run the updatecommand it recommends to get your machine up to speed.Installing Python packages on Amazon LinuxIf you prefer another distribution, you can use your knowledge of that distributionto install Python, NumPy, and others.
Here, we will do it on the standard Amazondistribution. We start by installing several basic Python packages as follows:$ sudo yum -y install python-devel \python-pip numpy scipy python-matplotlibTo compile mahotas, we will also need a C++ compiler:$ sudo yum -y install gcc-c++Finally, we install git to make sure that we can get the latest version of the code forthe book:$ sudo yum -y install gitIn this system, pip is installed as python-pip. For convenience, we will use pip toupgrade itself. We will then use pip to install the necessary packages as follows:$ sudo pip-python install -U pip$ sudo pip install scikit-learn jug mahotas[ 282 ]Chapter 12At this point, you can install any other package you wish using pip.Running jug on our cloud machineWe can now download the data and code for the book using this sequenceof commands:$ git clone \https://github.com/luispedro/BuildingMachineLearningSystemsWithPython$ cd BuildingMachineLearningSystemsWithPython$ cd ch12Finally, we run this following command:$ jug executeThis would work just fine, but we would have to wait a long time for the results.
Ourfree tier machine (of type t2.micro) is not very fast and only has a single processor.So, we will upgrade our machine!We go back to the EC2 console, and right-click on the running instance to getthe pop-up menu. We need to first stop the instance.
This is the virtual machineequivalent to powering off. You can stop your machines at any time. At this point,you stop paying for them. Note that you are still using disk space, which also has acost, billed separately. You can terminate the instance, which will also destroy thedisk. This loses any information saved on the machine.Once the machine is stopped, the Change instance type option becomes available.Now, we can select a more powerful instance, for example, a c1.xlarge instancewith eight cores.
The machine is still off, so you need to start it again (the virtualequivalent to booting up).AWS offers several instance types at different price points. As thisinformation is constantly being revised as more powerful options areintroduced and prices change (generally, getting cheaper), we cannotgive you many details in the book, but you can find the most up-to-dateinformation on Amazon's website.We need to wait for the instance to come back up. Once it has, look up its IP addressin the same fashion as we did before.
When you change instance types, your instancewill get a new address assigned to it.[ 283 ]Bigger DataYou can assign a fixed IP to an instance using Amazon.com's ElasticIPs functionality, which you will find on the left-hand side of the EC2console. This is useful if you find yourself creating and modifyinginstances very often.
There is a small cost associated with this feature.With eight cores, you can run eight jug processes simultaneously, as illustrated in thefollowing code:$ # the loop below runs 8 times$ for counter in $(seq 8); do>jug execute &> doneUse jug status to check whether these eight jobs are, in fact, running. After yourjobs are finished (which should now happen pretty fast), you can stop the machineand downgrade it again to a t2.micro instance to save money. The micro instancecan be used for free (within certain limits), while the c1.xlarge one we used costs0.064 US dollars per hour (as of February 2015—check the AWS website forup-to-date information).Automating the generation of clusters withStarClusterAs we just learned, we can spawn machines using the web interface, but it quicklybecomes tedious and error prone.
Fortunately, Amazon has an API. This meansthat we can write scripts, which perform all the operations we discussed earlier,automatically. Even better, others have already developed tools that can be used tomechanize and automate many of the processes you want to perform with AWS.A group at MIT developed exactly such a tool called StarCluster. It happens to be aPython package, so you can install it with Python tools as follows:$ sudo pip install starclusterYou can run this from an Amazon machine or from your local machine. Either optionwill work.We will need to specify what our cluster will look like.
We do so by editing aconfiguration file. We generate a template configuration file by running thefollowing command:$ starcluster help[ 284 ]Chapter 12Then pick the option of generating the configuration file in ~/.starcluster/config. Once this is done, we will manually edit it.Keys, keys, and more keysThere are three completely different types of keys that are importantwhen dealing with AWS. First, there is a standard username/passwordcombination, which you use to log in to the website. Second, there isthe SSH key system, which is a public/private key system implementedwith files; with your public key file, you can log in to remote machines.Third, there is the AWS access key/secret key system, which is just aform of username/password that allows you to have multiple users onthe same account (including adding different permissions to each one,but we will not cover these advanced features in this book).To look up our access/secret keys, we go back to the AWS Console, clickon our name on the top-right, and select Security Credentials.
Now atthe bottom of the screen, there should be our access key, which maylook something like AAKIIT7HHF6IUSN3OCAA, which we will use asan example in this chapter.Now, edit the configuration file. This is a standard .ini file: a text file where sectionsstart by having their names in brackets and options are specified in the name=valueformat.
The first section is the aws info section and you should copy and paste yourkeys here:[aws info]AWS_ACCESS_KEY_ID = AAKIIT7HHF6IUSN3OCAAAWS_SECRET_ACCESS_KEY = <your secret key>Now we come to the fun part, that is, defining a cluster. StarCluster allows youto define as many different clusters as you wish. The starting file has one calledsmallcluster. It's defined in the cluster smallcluster section.
We will edit itto read as follows:[cluster smallcluster]KEYNAME = mykeyCLUSTER_SIZE = 16This changes the number of nodes to 16 instead of the default of two. We canadditionally specify which type of instance each node will be and what the initialimage is (remember, an image is used to initialized the virtual hard disk, whichdefines what operating system you will be running and what software is installed).StarCluster has a few predefined images, but you can also build your own.[ 285 ]Bigger DataWe need to create a new SSH key with the following command:$ starcluster createkey mykey -o ~/.ssh/mykey.rsaNow that we have configured a sixteen node cluster and set up the keys, let's tryit out:$ starcluster start smallclusterThis may take a few minutes as it allocates seventeen new machines.
Why seventeenwhen our cluster is only sixteen nodes? StarCluster always creates a master node.All of these nodes have the same filesystem, so anything we create on the masternode will also be seen by the worker nodes. This also means that we can use jugon these clusters.These clusters can be used as you wish, but they come pre-equipped with a job queueengine, which makes it ideal for batch processing.
The process of using them is simple:1. You log in to the master node.2. You prepare your scripts on the master (or better yet, have them preparedbefore hand).3. You submit jobs to the queue. A job can be any Unix command.The scheduler will find free nodes and run your job.4. You wait for the jobs to finish.5. You read the results on the master node. You can also now kill all the slavenodes to save money. In any case, do not leave your system running when youdo not need it anymore! Otherwise, this will cost you (in dollars-and-cents).Before logging in to the cluster, we will copy our data to it (remember we had earliercloned the repository onto BuildingMachineLearningSystemsWithPython):$ dir=BuildingMachineLearningSystemsWithPython$ starcluster put smallcluster $dir $dirWe used the $dir variable to make the command line fit in a single line. We can login to the master node with a single command:$ starcluster sshmaster smallclusterWe could also have looked up the address of the machine that was generated andused an ssh command as we did earlier, but using the preceding command, it doesnot matter what the address was, as StarCluster takes care of it behind the scenesfor us.[ 286 ]Chapter 12As we said earlier, StarCluster provides a batch queuing system for its clusters; youwrite a script to perform your actions, put it on the queue, and it will run in anyavailable node.At this point, we need to install some packages again.
![](https://s.studizba.com/z.php?f=/templates/si/i/noavatar.png&w=100&h=100&t=1"){kind=avatar}
