errors

To use pajek with the datafiles read by the flor.exe program (last updated 5/26/99), it will be helpful to read Analyzing Large Kinship and Marriage Networks by drw, Vladimir Batagelj and Andrej Mrvar. 1999 (forthcoming). Social Science Computer Review you will need a password from drwhite@uci.edu if you do not remember it.

Final Version

The final version of the algorithm is completed and a preliminary analysis done.
The Alberti graph is partitioned out of the whole ball of wax (50,000 nodes). Just looking at the top 4 generations of the lowermost big family segment, the program found 29 of 30 links in the genealogy (97% accuracy) and the database was missing only 4 significant links in the genealogy. One case was not matched because of a spelling variant that has now been corrected (Attaviano=>Ottaviano). Only in once case (Antonio Doffo Alberti) would we have made a match for a name missing a marriage date where the tax year would have made the match correctly. I hesitate to do this for now because it might introduce more errors.
Pajek will now read the ENTIRE GRAPH of 57,505 nodes and partitions for families are made automatically by adding their names to the fam-list file. Single families or their unions can be examined. Graphs such as Alberti by decade are now a matter of course for any of the 1,000 or more families. The vertical axis is the actual date of marriage by decade, starting in 1200. The color coding gives the "accuracy" measure, the number of potential fathers found (green=1, red=2, blue=3).
The total graph has 86,187 lines, 28,680 arcs, 30,959 components, and 2,134 independent marriage cycles, of which 2,127 (99.6%) are in the giant component (#59 in the partition) of 19,030 vertices and 21,426 arcs. Of the total of 2,134 independent marriage cycles, 2,113 (98.6%) are in the giant bicomponent of 5,653 vertices and 7,765 arcs. This is a very hefty bicomponent. The remaining cycles are generally created by blood marriages outside the bicomponent. Of the vertices in the bicomponent:

3,568 have degree 2

1,027 have degree 3

538 have degree 4

250 have degree 5

125 have degree 6

68 have degree 7

40 have degree 8

16 have degree 9

14 have degree 10

4 have degree 11

2 have degree 12

1 has degree 13

4 are at generational depth 1 (latest)

60 are at depth 2

207 at depth 3

512 at depth 4

766 at depth 5

852 at depth 6

797 at depth 7

707 at depth 8

572 at depth 9

383 at depth 10

306 at depth 11

204 at depth 12

127 at depth 13

91 at depth 14

44 at depth 15

16 at depth 16

4 at depth 17 (earliest)

Instructions

I sent the *.dat and the fam-list files (you can edit the latter) by email. Run the program using options 1,1,1. The program will construct the master best.net file and a series of *.clu and *.cls files for each family that you add to the list in fam-list. Right now there are alberti and medici followed by stop, and the rest of the names are ignored. Option 1,1,1 will allow you to construct separate graphs for each family and option 1,1,2 will concatenate all the families in your list (you use the fam1nam.clu, fam2acc.clu, fam3dec.clu in this case for the instructions below). You dont need to use option 1,2, it was just for program development with a reduced database.

Run Pajek and select best.net.

Then read the family partition such as albe1nam.clu. Click Partitions/First Partition then Partitions/Second Partition and then Partitions/Extract Second from First. Save this NAMES partition, e.g., as Albe1nam411.clu where 411 is the number of nodes in the NAMES reduction (including the 7 timeline nodes if you add dates).

Then read albe2acc.clu. Click Partitions/First Partition thenPartitions/Extract Second from First. Save this Partition, e.g., as Albe2acc411.clu where 411 is the number of nodes in the ACCURACY reduction.

Then read albe3dec.clu. Click Partitions/First Partition thenPartitions/Extract Second from First. Save this Partition, e.g., as Albe3dec411.clu where 411 is the number of nodes in the DECADES reduction. albe1nam.clu

Now again read (1) tne total network best.net and (2) the family partition albe1nam.clu. To extract the alberti subnetwork, click Operations/Extract from Network/Partition [from 1 to 1 is automatic: the partition is 0/1 where the 1s are the Alberti nodes].

To construct the graph with decades read the DECADES partition albe3dec411.clu that you saved. Now click Draw/Draw Partition. In the Draw screen click Layes/in y direction followed by Layers/Optimze layers in x direction/Forward. Your graph will appear. If you did this correctly and you chose the 1 option at the start that adds timelines, then the seven nodes to the right of the image will be the timeline nodes with appropriate edge labels for the periods. To see the timeline and firstname labels click Options/Lines/Mark lines. To see the last name labels (which clutters the page) click Options/Mark vertices using/Labels. To see the difference between male and female lines the click Options/Mark vertices using/No Labels.

Thus you can do a graph for ANY family or ANY COMBINED SET of families (like the Alberti-Medici graph) by adding to the fam-list and options 1,1,1 or 1,1,2. However, a better way of doing this is just to run the FLOR.EXE program once, and later run FLOR-ADD.EXE giving the first four letters of each family name you want to concatenate. If you pick albe and medi for example, then the albemedi2acc.clu and albemedi3dec.clu files will be created for you as partitions, and you can extract in this way the union of any set of families (the output file names will always those of the first two families).

For checking the data family by family, print relevant sections of the "scratch" file created bu the program, in which each family is taken up alphabetically and within each family the sibling sets are taken up alphabetically by the name of the father. The format will become evident if you stare at it for a while. The last pair of numbers on the right are masterID numbers for a couple corresponding to the individual, and the parents of that individual. These numbers are repeated in the "Spss" file which can be inserted directly as variables in the master file but some of the nunbers will refer to contructed parents. There is no funny stuff anymore with constructed parents. All assumptions are STRICTLY CONSERVATIVE. If there is anything funny in the scratch file let me know immediately.

To see the size of the bicomponent (which can generate a 0/1 partition that can go into the MASTER file for cross tabulation with other variables) the quickest way is to read tne total network best.net and click Partition/Components/Bi-Components ... and choose a large minimum size such as 100 or 1000. Then click Hierarchy/Extract Cluster (you can alway choose cluster 1 when there is only one giant bicomponent). You will be amazed at how quickly the bicomponent is computed (in a few seconds). The click Operations/Extract from Network/Cluster and the total network will reduce to 5726. This is to big to draw all at once I believe but it can become an important analytic variable, and its relinking index is an important fact. You can also reduce the total network for a specific range of decades, and then recalculate the bi-component for that specific historical era. This opens the way to my kind of analyis. Have fun!

Name Matching Notes: Algorithm tested for Alberti

14albberjac - perfect match of father except for wife's name (computer: sandra; genealogy: Lisa)
75% predicted with 100% accuracy for Alberti genealogy I-VII (missing VIII)
15% probably right 67% later than genealogy
10% probable 60% from alternatives given dates of marriages
total accuracy probably 91% but rated by certain / likely / best guess
some misses due to longer fa-son lag in marriage dates: 48,60,61,64,66 years, often due to second marriages

Second Test of First Algorithm

Includes a numeric quality rank for predicting potential matches
TRUE ANCESTORS occur at the following rank frequencies:
57 were rank 1 (predicted perfectly
1 tied for quality rank 1
5 tied for inferior rank 1 (both some possibility)
7 rank 2
2 rank 3
1 rank 4
1 rank 5
2 tied among six with inferior rank
2 not found at all, although six with inferior rank
PREDICTIONS NOT IN THE GENEALOGY FALL AS FOLLOWS
55 have only one possible match
6 have 2 possible matches
1 have 5 possible matches
1 have 6 possible matches
1 have 8 possible matches
LOOKED AT FROM VIEWPOINT OF HOW MANY POSSIBLE CHOICES:
-ONE: 100% in 75 predictions, 30 verified
-TWO-: 67% in 17 predictions, 8/12 verified
THREE: 59% in 17 predictions, 10/17 verified
FOUR+: 40% in 23 predictions 6/15 verified
FOR PAJEK FILES, stripping away the arcs from nodes with more than 1, 2, 3 etc. parents of a given type with successively improve the accuracy of the genealogy
OVERALL ACCURACY IF YOU TAKE ALL PREDICTIONS is 90% with no contradictions in genealogy
among those found in the genealogies, the confirmed hit rate is 73%
almost 100% of the ancestors in the genealogies who do have offspring are found by the algorithm as possible alternatives, even if not the top prediction
Graphic Image of ALL predictions (men only), even if erroneous in many places
KEY:
Yellow: 1 possibility only (100% accurate)
Green: 2 possilities (67% accurate)
Red: 3 possibilities (59%)
Other: 40% accurate, 4-9 possibilites (many those lacking dates from the tax records); the algorithm can be improved by upgrading predictions where there is one CLEAR LEADER because of marriage dates even if the name of the father is a common one. This procedure will also improve accuracy.

First Program

fa-son.for
fa-son.exe
flor1.for with documentation lines and output pgraph explanation
flor1.exe
marriage dates assumption: Son later (10 to 40) years (increase to 66?)
marriage dates assumption: Son under 10/over 40: not the father (some errors!)
tax rolls assumption: Son same or up to 60 years later, if no marriage data
first names matchup: strip everything after / . , m.
adjacent lines with same master ID: same ego

Errors: Changes in the SPSS data

14albberant1 and 14albberant1 should be SEPARATE MASTER IDS IV:4 and IV:4 in GENEALOGIES
The Spss data is fairly clean as regards standardization of spellings BUT SOME MORE NAMES WERE STANDARDIZED.
problems encountered: Niccolo vs. Niccolo/aio m. etc.
188buobadghe Badella -> Badello should be male not female
367fretegner Teglia -> Teglio should be male not female
720sconerber Nera -> Nero should be male not female
14albmargia1 'F' should be female not male
9aglguemas marr 378 -> 1378 (error in date)
404gindomgiu taxyear 48 -> 480 (just a guess)
if you transform/calculate a new Spss variable taxlessm=taxyear+1000-marr and do a univariate frequency count you will see that the difference runs from -65 to +64 with a mean of 2, and two wild outliers of +75 and +99 as follows:
292corlucgio taxed: 1427 Married: 1328 obviously one of these is wrong
62balaldagn taxed: 1427 Married: 1352 [[and of course there are many more where the dates are 50 or more years discrepant]
18aldmatbon fpart 366 -> 1366

Second Program

flor.for with documentation lines and output pgraph explanation
flor.exe
Graphic Image of BEST predictions (men AND women), colored yellow=one choice, green=2, red=3 and blue=4.

For my info

tax years are 351 378 403 427 458 480
ngh years are 351 378 403 427 458 480 - can get date from taxyear
pair480 1-over 334
ngh351 11-44
ngh378 11-44
qt403 1-4 quarters of town?
ngh403 11-44
ngh427 11-44
ngh458 11-44
ngh480 11-44