As a practical matter, there are two basic problems involved. First is the problem of looking at a photo and interpreting what is in it - i.e. whether a blob is a leaf or a pile of leaves, tracing the outline of a leaf as a curve, etc. If you rely on human beings to do this, it would be very labor intensive. So it would be nice to have computer programs to do the job. However, the mathematics needed to program computers to recognize what is in a picture is not perfected although there are many methods that work in special situations. Efforts to solve the problem are currently topics in Computer Science rather than topics in Mathematics. The general topic is "Pattern Recognition". Specific methods would be "Neural Nets", "Fuzzy Classification", "Attention Selection".
Let's assume you have gotten data about types of leaves or curves that outline leaves. Telling whether two curves are "similar" is a special case of the first problem, which was recognizing what is in a picture. I think I can find Computer Science papers written about this problem, but it is not a separate branch of Mathematics. If you have data that does not represent a shape (such as leaf length, width, area,... etc.) then the mathematics called "Statistical Pattern Recognition" is applicable. (Don't restrict your thinking to an x-y graph in two dimensions.) Statistical Pattern Recognition may not do a job. Whether it works depends on whether your data really has enough information to solve our problem. (The mathematical field called "Statistics" is not the same as "Statistical Pattern Recognition".)