Building machine learning systems with Python (779436), страница 44
Текст из файла (страница 44)
This is a smalloptimization. This way, sklearn is only imported when it's really needed:scores_base = accuracy(haralicks, labels)scores_chist = accuracy(chists, labels)combined = hstack([chists, haralicks])scores_combined= accuracy(combined, labels)Finally, we write and call a function to print out all results. It expects its argument tobe a list of pairs with the name of the algorithm and the results:@TaskGeneratordef print_results(scores):with open('results.image.txt', 'w') as output:for k,v in scores:output.write('Accuracy [{}]: {:.1%}\n'.format(k, v.mean()))print_results([('base', scores_base),('chists', scores_chist),('combined' , scores_combined),])That's it. Now, on the shell, run the following command to run this pipeline using jug:$ jug execute image-classification.pyReusing partial resultsFor example, let's say you want to add a new feature (or even a set of features).
Aswe saw in Chapter 10, Computer Vision, this is easy to do by changing the featurecomputation code. However, this would imply recomputing all the features again,which is wasteful, particularly, if you want to test new features and techniques quickly.[ 272 ]Chapter 12We now add a set of features, that is, another type of texture feature called linearbinary patterns. This is implemented in mahotas; we just need to call a function,but we wrap it in TaskGenerator:@TaskGeneratordef compute_lbp(fname):from mahotas.features import lbpimc = mh.imread(fname)im = mh.colors.rgb2grey(imc)# The parameters 'radius' and 'points' are set to typical values# check the documentation for their exact meaningreturn lbp(im, radius=8, points=6)We replace the previous loop to have an extra function call:lbps = []for fname in sorted(images):# the rest of the loop as beforelbps.append(compute_lbp(fname))lbps = to_array(lbps)We call accuracy with these newer features:scores_lbps = accuracy(lbps, labels)combined_all = hstack([chists, haralicks, lbps])scores_combined_all = accuracy(combined_all, labels)print_results([('base', scores_base),('chists', scores_chist),('lbps', scores_lbps),('combined' , scores_combined),('combined_all' , scores_combined_all),])[ 273 ]Bigger DataNow, when you run jug execute again, the new features will be computed,but the old features will be loaded from the cache.
This is when jug can be verypowerful. It ensures that you always get the results you want while saving you fromunnecessarily recomputing cached results. You will also see that adding this featureset improves on the previous methods.Not all features of jug could be mentioned in this chapter, but here is a summary ofthe most potentially interesting ones we didn't cover in the main text:• jug invalidate: This declares that all results from a given function shouldbe considered invalid and in need of recomputation. This will also recomputeany downstream computation, which depended (even indirectly) on theinvalidated results.• jug status --cache: If jug status takes too long, you can use the--cache flag to cache the status and speed it up. Note that this will notdetect any changes to the jugfile, but you can always use --cache --clearto remove the cache and start again.• jug cleanup: This removes any extra files in the memoization cache.
This isa garbage collection operation.There are other, more advanced features, which allow you tolook at values that have been computed inside the jugfile. Readup on features such as barriers in the jug documentation onlineat http://jug.rtfd.org.Using Amazon Web ServicesWhen you have a lot of data and a lot of computation to be performed, you might startto crave more computing power. Amazon (http://aws.amazon.com) allows you torent computing power by the hour. Thus, you can access a large amount of computingpower without having to precommit by purchasing a large number of machines(including the costs of managing the infrastructure).
There are other competitors in thismarket, but Amazon is the largest player, so we briefly cover it here.Amazon Web Services (AWS) is a large set of services. We will focus only on theElastic Compute Cloud (EC2) service. This service offers you virtual machines anddisk space, which can be allocated and deallocated quickly.There are three modes of use. First is a reserved mode, whereby you prepay to havecheaper per-hour access, a fixed per-hour rate, and a variable rate, which depends onthe overall compute market (when there is less demand, the costs are lower; whenthere is more demand, the prices go up).[ 274 ]Chapter 12On top of this general system, there are several types of machines available withvarying costs, from a single core to a multicore system with a lot of RAM or evengraphical processing units (GPUs).
We will later see that you can also get several ofthe cheaper machines and build yourself a virtual cluster. You can also choose to geta Linux or Windows server (with Linux being slightly cheaper). In this chapter, wewill work on our examples on Linux, but most of this information will be valid forWindows machines as well.For testing, you can use a single machine in the free tier. This allows you to playaround with the system, get used to the interface, and so on. Note, though, that thismachine contains a slow CPU.The resources can be managed through a web interface.
However, it's also possibleto do so programmatically and to write scripts that allocate virtual machines,format hard disks, and perform all operations that are possible through the webinterface. In fact, while the web interface changes very frequently (and some of thescreenshots we show in the book may be out of date by the time it goes to press),the programmatic interface is more stable and the general architecture has remainedstable since the service was introduced.Access to AWS services is performed through a traditional username/passwordsystem, although Amazon calls the username an access key and the password a secretkey. They probably do so to keep it separate from the username/password youuse to access the web interface. In fact, you can create as many access/secret keypairs as you wish and give them different permissions. This is helpful for a largerteam where a senior user with access to the full web panel can create other keys fordevelopers with fewer privileges.Amazon.com has several regions.
These correspond to physicalregions of the world: West coast US, East coast US, several Asianlocations, a South American one, and two European ones. If you willbe transferring data, it's best to keep it close to where you will betransferring to and from. Additionally, keep in mind that if you arehandling user information, there may be regulatory issues regulatingtheir transfer to another jurisdiction. In this case, do check with aninformed counsel on the implications of transferring data aboutEuropean customers to the US or any other similar transfer.Amazon Web Services is a very large topic and there are various booksexclusively available to cover AWS.
The purpose of this chapter is to give youan overall impression of what is available and what is possible with AWS. In thepractical spirit of this book, we do this by working through examples, but we willnot exhaust all possibilities.[ 275 ]Bigger DataCreating your first virtual machinesThe first step is to go to http://aws.amazon.com/ and create an account. Thesesteps are similar to any other online service.
A single machine is free, but to get more,you will need a credit card. In this example, we will use a few machines, so it maycost you a few dollars if you want to run through it. If you are not ready to take outa credit card just yet, you can certainly read the chapter to learn what AWS provideswithout going through the examples. Then you can make a more informed decisionon whether to sign up.Once you sign up for AWS and log in, you will be taken to the console. Here, you willsee the many services that AWS provides, as depicted in the following screenshot:[ 276 ]Chapter 12We pick and click on EC2 (the top element on the leftmost column—this is the panelshown as it was when this book was written. Amazon regularly makes minor changes,so you may see something slightly different from what we present in the book).
Wenow see the EC2 management console, as shown in the following screenshot:In the top-right corner, you can pick your region (see the Amazon regionsinformation box). Note that you will only see information about the region that youhave selected at the moment. Thus, if you mistakenly select the wrong region (or havemachines running in multiple regions), your machines may not appear (this seems tobe a common pitfall of using the EC2 web management console).[ 277 ]Bigger DataIn EC2 parlance, a running server is called an instance. We select Launch Instance,which leads to the following screen asking us to select the operating system to use:[ 278 ]Chapter 12Select the Amazon Linux option (if you are familiar with one of the other offeredLinux distributions, such as Red Hat, SUSE, or Ubuntu, you can also select one ofthese, but the configurations will be slightly different).
Now that you have selectedthe software, you will need to select the hardware. In the next screen, you will beasked to select which type of machine to use:[ 279 ]Bigger DataWe will start with one instance of the t2.micro type (the t1.micro type was an older,even less powerful machine). This is the smallest possible machine and it's free.Keep clicking on Next and accept all of the defaults until you come to the screenmentioning a key pair:We will pick the name awskeys for the key pair.
![](https://s.studizba.com/z.php?f=/templates/si/i/noavatar.png&w=100&h=100&t=1"){kind=avatar}
