Dealing With the Deluge of Duplicates in the FamilySearch Family Tree

In the last few weeks, many users of the FamilySearch.org Family Tree have seen a dramatic increase in the number of duplicate entries found by the program. In some instances while working with some research on a particular family, I have been faced with as many as a hundred or more duplicates that must be merged before I can adequately proceed with research. These duplicate entries are not immediately apparent. They only appear as you do research and add information from records found from FamilySearch.org.  Here is a screenshot of part of the list of changes I made to one family in England recently that developed as a result of the numerous duplicates that appeared.

The original entry was a single individual with very little information and no discoverable duplicate entries. As I added information and each time I corrected the entry, as many as five or more duplicate entries would appear. The screenshot shows less than half of the list of changes that had to be made because of the duplicates that appeared each time information was added. Many of the merges produced even more occurrences of duplicates that had to be merged. I estimate that I had to resolve more than a hundred duplicates to clean up this one family.

I can speculate about what is happening. It appears from consulting with others who are experiencing the same issues, that FamilySearch has added a significant number of additional individuals to the database. Perhaps this comes about as a result of opening up more links to the original data in moving away from the new.FamilySearch.org program? In one case, a duplicate record appeared that could only have been added by the user many years ago.

There are some important issues here. One is that an entry that appears to be without any duplicates, could have many duplicates if some additional research is done and information added. So, for example you find a relative where the entry shows Temple work needed but there is very limited information available. If you do some very basic research, such as adding a birthplace or date or adding a spouse, the Family Tree program will immediately begin finding duplicates that were previously not discoverable. As more information is added, duplicates will continue to appear.

In one case, we assumed that this phenomena was caused by having huge IOUS or multiply submitted entries. However, I have seen instances where duplicates start appearing in lines where there has been very little research and a very limited number of submissions.

What is the solution? Keep carefully merging entries until the duplicates run out. There is apparently no other solution. It is significant that these duplicates have now begun to appear. They have been in the program all along but not found when searching. Now they are being found. This must mean we are closer to separating new.FamilySearch.org and the end of the ultimate problems with the data may be in sight.

Reflections on Duplicates and Merging in the FamilySearch Family Tree

As I have been working on specific family lines in the FamilySearch.org Family Tree recently, I have discovered some interesting issues that accompany the plethora of duplicate entries inherited from the combining of over a hundred years' research by a multitude of researchers.

One very interesting phenomena involves the appearances of duplicates as information is added to the entries through research and attached as sources. For example, I began searching for the wife of a relative that was shown a unmarried and who was born and died in England. My relative has a very common name: John Briant (also spelled Bryant). I checked to see if there were any duplicates for him in the Family Tree and none were found. I decided to see if he ever married and began doing research. Almost immediately, I found a marriage record that identified his wife's name as "Susan."

I did some additional research and found more records. It seems that adding a wife's name (or spouse's name) assists the search engine in finding additional information. I then found a marriage record that identified the wife's father, thereby providing her with a surname. I continued to do research and now found even more records. What was interesting is that I also began to find duplicate records in the system. Every time I added more researched information about the family, I found additional duplicates. At any point from when I first identified the wife's name. I could have printed ordinance cards. But I decided to add additional information on the hope that there were some children in the newly identified family.

But as I continued to add information and sources, I got even more duplicates. There seems to be a direct correlation between adding source information and detecting more duplicates in the Family Tree. So, it would also appear, that researchers who stop short of discovering additional source information about the people they identify run the real risk of duplicating ordinances. When I first identified the wife, I found no duplicates. It was only as I added specific information from several sources that the duplicates appeared. I am guessing that this is why I found several duplicate entries for this same couple with almost no information, just the bare names and dates with incomplete places. Apparently, the program does not find duplicates without a certain level of information about the individuals.

I realized that I had been seeing this phenomena before without realizing exactly what was happening. I suggest that the number of duplicate entries in the program will continue to increase if the contributors fail to add sources to the information they supply and then add the information from the sources to the entries and keep searching multiple times for duplicates entries.

More important than the difficulty

I have written recently about some of the challenges in working with all of the inherited data in the FamilySearch.org Family Tree. However, I should not fail to mention the benefits. After spending many hours untangling the duplicate entries and working through the unsupported additions to families, I was able to find more than a few new people to add to the Family Tree and then, in the evening, go to the Temple and again take a family name. All of the effort and overcoming problems is worth it. There is a reason why I would go through all of the frustration and spend the time to work out the difficult relationships and in the end, we can have the satisfaction of progressing a little further in our family history.

The most interesting part of this whole experience is the fact that the tools provided by FamilySearch are now sufficient to resolve some of these most difficult problems. Quoting from the LDS.org website:

Heavenly Father’s plan of happiness enables family relationships to continue throughout eternity. Through family history work, we can learn more about our ancestors, identify and prepare the names of those who need gospel ordinances, and perform ordinance work for them in holy temples. The Church provides many resources to help us learn about our family history and participate in temple work for the dead.

This is most certainly true. I only feel badly that more members of the Church cannot understand the huge opportunity we have to finally overcome some of these challenges.

Overcoming the Fire Swamp of the Family Tree

With my last post, I was working my way out of the "fire swamp" of the FamilySearch.org Family Tree. After several long hours of work, I was finally able to see some light and get that particular family under control. This process took somewhere close to between fifty and a hundred meges. Here is what that section of the Family Tree looked like when I started:

Here is what the same family line looks like now that I spent hours merging and cleaning up the entries.

Also, very interestingly, this particular family no longer connects to my own family lines. My Bryant line has Jane Bryant (b. 1789, d. 1874) married to Thomas Shoobridge (b. 1784, d. 1863) but there are no known parents for Thomas.

Now, it was time to see if there is some connection for the Thomas Shoobridge or whether I have more merges to do. I did find that his mother's name was Anne.

There is apparently only one Anne married to a Shoebridge in Rolvenden, Kent, England or any of the very small places around that town. But Richard and Anne Shoebridge (Richard had two wives, both with the name Ann or Anne) did not have a child named Thomas in 1784. But more research shows that this Thomas's mother was Anne Shoobridge and he was "base born" or illegitimate. \

So, my Thomas Shoobridge was not part of the fire swamp at all and we don't know his father's name. That seems to be the end of the story.

Stepping Off into the Family Tree Fire Swamp

My reference to the "Fire Swamp" obviously comes from The Princess Bride and I have used this analogy in previous posts. This conversation from the movie provides the backdrop for my comments on my recent experiences with the FamilySearch.org Family Tree.

[after Westley rescues her from the lightning quicksand] 

Buttercup: We'll never succeed. We may as well die here. 

Westley: No, no. We have already succeeded. I mean, what are the three terrors of the Fire Swamp? One, the flame spurt - no problem. There's a popping sound preceding each; we can avoid that. Two, the lightning sand, which you were clever enough to discover what that looks like, so in the future we can avoid that too. 

Buttercup: Westley, what about the R.O.U.S.'s? 

Westley: Rodents Of Unusual Size? I don't think they exist.
[Immediately, an R.O.U.S. attacks him]

I do not think that there is any coincidence in the fact that multiple copies of individuals in the Family Tree are called Individuals of Unusual Size or IOUSs. Here is a screenshot of my present fire swamp situation:

I was examining one of the Family Tree branches that go well beyond my original research. I am reasonably sure I have never seen this surname in any of my own research before. I got to this section by following back into my supposed ancestors. Here is a more graphic example of the problems with this one family:

This is all in addition to the fact that there are multiple duplicates in layers. After making several merges, I noticed that the number of potential merges had increased rather than decreased.

Now, I tend to look at this mess more as an opportunity than a problem. It is evident that very little effort has gone into the research of this family when there are people designated as "Childrens Shoebridge."

This whole problem became evident when I went out my Bryant family line and started to look at some of the children (cousins). The first indication was a Jane Bryant (b. 1788, d. 1874) married to Thomas Shoobridge (b. 1784, d. 1863). He shows up as an only child in the list above with a mother named "Anne." The above list shows up as a result of looking at the descendancy of his father (maybe) Richard Shoebridge (b. 1737, d. 1820).

The Thomas Shoofridge (b. 1784, d. 1863) has a duplicate entry.

After merging these duplicate records, there were three more possible duplicates for his wife.

One of these was not a match, so I looked at the other two and merged them leaving the wife, Jane Bryant, with a birth date after her christening date. The merge also left her with duplicate children. At this point the duplicates just kept multiplying and spreading further and further.

The rule here is to start at the first place in the Family Tree where there is no controversy and work your way backward to the person of interest. What I did see is that despite the fact that this person could have been an IOUS, the mergers continued to world and made good progress in straightening out this problem. The key here is patience and knowing the rules of the fire swamp.

Searching for your Relatives?

In our highly mobile society, I have found that many genealogists are really interested in discovering their living relatives. Motivation for this can come through a desire to find out information about family lines, but it may also arise as a result of a need to belong to a family. As a result of this general interest in living relatives, there are two apps in the FamilySearch.org App Gallery that are aimed at the processs of discovering living relatives.

The first of these is called "Are we related?" and is an Android app. Since I do not have any Android devices, I could not test the program. So I will have to leave that up to you to comment on. As usual, I suggest a visit to the GenSoftReviews.com website but this app does not yet have any reviews. Perhaps you can add one?

RelativeFinder.org has been around for a while and has developed quite a following. Developed by the Family History Technology Lab at Brigham Young University, it is an easy to use, free, collaborative program that shows relationships using the Family Tree.

Relative Finder also has yet to garner any reviews in GenSoftReviews.com.

The status of genealogy in LDS culture

The Doctrine and Covenants, Section 128: 17-18 states:

17 And again, in connection with this quotation I will give you a quotation from one of the prophets, who had his eye fixed on the restoration of the priesthood, the glories to be revealed in the last days, and in an especial manner this most glorious of all subjects belonging to the everlasting gospel, namely, the baptism for the dead; for Malachi says,last chapter, verses 5th and 6th: Behold, I will send you Elijah the prophet before the coming of the great and dreadful day of the Lord: And he shall turn the heart of the fathers to the children, and the heart of the children to their fathers, lest I come and smite the earth with a curse. 

18 I might have rendered a plainer translation to this, but it is sufficiently plain to suit my purpose as it stands. It is sufficient to know, in this case, that the earth will be smitten with a curse unless there is a welding link of some kind or other between the fathers and the children, upon some subject or other—and behold what is that subject? It is the baptism for the dead. For we without them cannot be made perfect; neither can they without us be made perfect. Neither can they nor we be made perfect without those who have died in the gospel also; for it is necessary in the ushering in of the dispensation of the fulness of times, which dispensation is now beginning to usher in, that a whole and complete and perfect union, and welding together of dispensations, and keys, and powers, and glories should take place, and be revealed from the days of Adam even to the present time. And not only this, but those things which never have been revealed from the foundation of the world, but have been kept hid from the wise and prudent, shall be revealed unto babes and sucklings in this, the dispensation of the fulness of times.

The extension of these basic concepts has culminated in the building of over 150 Temples worldwide and the development of an extensive international genealogical support systems consisting of over 4,900 Family History Centers and other related libraries and archives. It is apparent that a significant portion of the resources of The Church of Jesus Christ of Latter-day Saints goes into promoting and supporting the "work for the dead" as it is often referred to. Quoting from the Mormon Newsroom topic on "Genealogy:"

Genealogy, the study of one’s ancestors or family history, is one of the most popular hobbies in the world. People of all faiths and nationalities enjoy discovering where they come from. For members of The Church of Jesus Christ of Latter-day Saints, however, learning about one’s family history is more than just a casual endeavor. Latter-day Saints believe families can be together after this life. Therefore, it is essential to strengthen relationships with all family members, both those who are alive and those who have died. 

Latter-day Saints believe that the eternal joining of families is possible through sacred sealing ceremonies that take place in temples. These temple rites may also be performed by proxy for those who have died. Consequently, for Mormons, genealogical research or family history is the essential forerunner for temple work for the dead. In Latter-day Saint belief, the dead have the choice to accept or reject the services performed for them. 

Since 1894, The Church of Jesus Christ of Latter-day Saints has dedicated time and resources to collecting and sharing records of genealogical importance. Due to cooperation from government archives, churches, and libraries, the Church has created the largest collection of family records in the world, with information on more than 3 billion deceased people. This effort was originally facilitated through the Genealogical Society of Utah and now through FamilySearch, a non-profit organization sponsored by the Church. 

FamilySearch provides access to information from 100 countries, including birth, marriage, and death records, censuses, probates and wills, land records, and more. These records are made available to the public free of charge through the FamilySearch.org website, the world-renowned Family History Library in Salt Lake City, and through a network of 4,600 local family history centers in 126 countries.

As noted, the Church has been collecting and sharing genealogical records since 1894. Now, you would expect that a significant percentage of the Church membership would be either directly or indirectly involved in genealogical research into their families. The actual numbers of active participants are very low. Why, given the obvious importance of the work, does this condition exist?

This is one of the most serious issues I have thought about and researched for some time and I have isolated some of the main factors. Here is my list, not in any particular order, of some of the factors that I see that influence this disparity.

The members of the Church around the world reflect the societies in which they live. The statement above about genealogy being "one of the most popular hobbies in the world" is commonly quoted but unsupported. Interest in families is almost universal, but genealogy is a specific activity that involves research into ancestral lines and involvement in this type of research is highly specialized and challenging. Using the term "hobby" when talking about genealogy is inappropriate. It is true that people have different levels of interest but genealogical research increases in complexity and difficulty exponentially as the researcher goes back in time. There are really very few people with the interest and the skills to do intensive genealogical research. 

With regard to the status of their family history, members of the Church fall into three general categories: those who have little or no information about their ancestral families, those who have some general information and those who have extensively researched compiled genealogies, usually inherited from ancestors who were intensely involved in genealogical research. Considering the entire membership of the Church, most of the members fall within the first two categories. 

Even though the number of members who have extensive genealogies is declining as a percentage of the overall membership, these members are concentrated in the historically predominantly LDS geographic areas such as Utah, Arizona and other western states in the United States and they still have a prominent position in the activity and leadership of the Church. With respect to these "legacy" members of the Church, the "easy" part of their genealogy has been done. Most of the work left is undone because of the scarcity of records or the difficulty of doing the research. 

This overburden of "completed" genealogy creates a conflict in the minds of the members with respect to "doing their family history." This conflict arises because of the doctrinal admonitions and the reality of actually making progress where the activity has moved well beyond adding the "first four generations" to a family tree. 

Many of these legacy members of the Church are the descendants of immigrants from Europe or other countries around the world. For some, access to the records of their ancestors is either extremely limited or involves skills regarding reading languages other than English. These challenges can be overwhelming and prevent even the beginnings of interest in doing the research necessary.

Because of the specialized nature of research beyond a the very basic, first four generations, level, information and promotional materials for many years have been directed at "beginners." This focus is also very appropriate given that the overall numbers of the Church membership indicate that there are more such beginners than legacy members. However, the continued emphasis on beginning research and the ease of doing so has created unrealistic expectations on the part of those members who have the overburden of "completed" ancestral lines. 

There is, in fact, a huge opportunity for those who have the skills and interest to do family history research. In the past year or so, the programs, data and methods of advancing genealogical research have changed dramatically and opportunities are available for legacy members to do substantial additional research. However, the cultural burden imposed by the past history and perception that the work is complete as far as is possible, has discouraged all but the most dedicated researchers. Hence, there is almost no leadership support for the currently outlined activities aimed at increasing membership activity among the legacy members and this, by association, extends to those who could do some basic, four generation, work also. 

We are at the threshold of having a significant advance in the our ability to push back ancestral research into the past. But we are still struggling with the cultural overburden of our historical involvement. It may take a metaphorical "40 years in the wilderness" for this to happen however.