To use pajek with the datafiles read by the flor.exe
program (last updated 5/26/99), it will be helpful to read
Analyzing Large Kinship and Marriage Networks
by drw, Vladimir Batagelj and Andrej Mrvar. 1999 (forthcoming).
Social Science Computer Review you will need a password from firstname.lastname@example.org if you do not remember it.
The final version of the algorithm is completed and a preliminary analysis done.
The Alberti graph is partitioned out of the whole ball of wax (50,000 nodes).
Just looking at the top 4 generations of the lowermost big family segment, the program found
29 of 30 links in the genealogy (97% accuracy) and the database was missing only 4 significant links in the genealogy.
One case was not matched because of a spelling variant that has now been corrected (Attaviano=>Ottaviano).
Only in once case (Antonio Doffo Alberti) would we have made a match for a name missing a marriage date
where the tax year would have made the match correctly. I hesitate to do this for now because it might
introduce more errors.
Pajek will now read the ENTIRE GRAPH of 57,505 nodes and partitions for families are
made automatically by adding their names to the fam-list file. Single families or their unions can be
examined. Graphs such as Alberti by decade are now a matter of course
for any of the 1,000 or more families. The vertical axis is the actual date of marriage by decade, starting in 1200.
The color coding gives the "accuracy" measure, the number of potential fathers found (green=1, red=2, blue=3).
The total graph has 86,187 lines, 28,680 arcs, 30,959 components,
and 2,134 independent marriage cycles, of which 2,127 (99.6%) are in the giant component
(#59 in the partition) of 19,030 vertices and 21,426 arcs. Of the total of 2,134 independent
marriage cycles, 2,113 (98.6%) are in the giant bicomponent of 5,653 vertices and 7,765 arcs. This is a very hefty bicomponent.
The remaining cycles are generally created by blood marriages outside the bicomponent. Of the vertices
in the bicomponent:
3,568 have degree 2
1,027 have degree 3
538 have degree 4
250 have degree 5
125 have degree 6
68 have degree 7
40 have degree 8
16 have degree 9
14 have degree 10
4 have degree 11
2 have degree 12
1 has degree 13
4 are at generational depth 1 (latest)
60 are at depth 2
207 at depth 3
512 at depth 4
766 at depth 5
852 at depth 6
797 at depth 7
707 at depth 8
572 at depth 9
383 at depth 10
306 at depth 11
204 at depth 12
127 at depth 13
91 at depth 14
44 at depth 15
16 at depth 16
4 at depth 17 (earliest)
I sent the *.dat and the fam-list files (you can edit the latter) by email.
Run the program using options 1,1,1.
The program will construct the master best.net file and a series of *.clu and *.cls files for each family that you add
to the list in fam-list. Right now there are alberti and medici followed by stop, and the rest of the names are ignored.
Option 1,1,1 will allow you to construct separate graphs for each family and option 1,1,2 will concatenate all the families in
your list (you use the fam1nam.clu, fam2acc.clu, fam3dec.clu in this case for the instructions below). You dont need to use
option 1,2, it was just for program development with a reduced database.
Run Pajek and select best.net.
Then read the family partition such as albe1nam.clu. Click Partitions/First Partition
then Partitions/Second Partition and then Partitions/Extract Second from First. Save this NAMES partition, e.g., as Albe1nam411.clu
where 411 is the number of nodes in the NAMES reduction (including the 7 timeline nodes if you add dates).
Then read albe2acc.clu. Click Partitions/First Partition
thenPartitions/Extract Second from First. Save this Partition, e.g., as Albe2acc411.clu
where 411 is the number of nodes in the ACCURACY reduction.
Then read albe3dec.clu. Click Partitions/First Partition
thenPartitions/Extract Second from First. Save this Partition, e.g., as Albe3dec411.clu
where 411 is the number of nodes in the DECADES reduction.
Now again read (1) tne total network best.net and (2) the family partition albe1nam.clu.
To extract the alberti subnetwork, click
Operations/Extract from Network/Partition [from 1 to 1 is automatic: the partition is 0/1 where the 1s are the Alberti nodes].
To construct the graph with decades read the DECADES partition albe3dec411.clu that you saved. Now click Draw/Draw
Partition. In the Draw screen click Layes/in y direction followed by Layers/Optimze layers in x direction/Forward.
Your graph will appear. If you did this correctly and you chose the 1 option at the start that adds timelines, then the seven nodes to the right
of the image will be the timeline nodes with appropriate edge labels for the periods. To see the timeline and firstname labels click
Options/Lines/Mark lines. To see the last name labels (which clutters the page)
click Options/Mark vertices using/Labels. To see the difference between male and female lines the click
Options/Mark vertices using/No Labels.
Thus you can do a graph for ANY family or ANY COMBINED SET of families (like the Alberti-Medici graph) by adding to the fam-list and options 1,1,1 or 1,1,2. However,
a better way of doing this is just to run the FLOR.EXE program once, and later run FLOR-ADD.EXE
giving the first four letters of each
family name you want to concatenate. If you pick albe and medi for example, then the albemedi2acc.clu and albemedi3dec.clu
files will be created for you as partitions, and you can extract in this way the union of any set of families (the output file names
will always those of the first two families).
For checking the data family by family, print relevant sections of the "scratch" file created bu the program,
in which each family is taken up
alphabetically and within each family the sibling sets are taken up alphabetically by the name of the father. The format
will become evident if you stare at it for a while. The last pair of numbers on the right are masterID numbers for a couple
corresponding to the individual, and the parents of that individual. These numbers are repeated in the "Spss" file which can be inserted
directly as variables in the master file but some of the nunbers will refer to contructed parents. There is no funny stuff anymore with
constructed parents. All assumptions are STRICTLY CONSERVATIVE. If there is anything funny in the scratch file let me know immediately.
To see the size of the bicomponent (which can generate a 0/1 partition that can go into the MASTER file for cross tabulation with other
variables) the quickest way is to read tne total network best.net and click Partition/Components/Bi-Components ... and choose a large
minimum size such as 100 or 1000. Then click Hierarchy/Extract Cluster (you can alway choose cluster 1 when there is only one giant
bicomponent). You will be amazed at how quickly the bicomponent is computed (in a few seconds). The click Operations/Extract from Network/Cluster
and the total network will reduce to 5726. This is to big to draw all at once I believe but it can become an important analytic variable, and its
relinking index is an important fact. You can also reduce the total network for a specific range of decades, and then recalculate the bi-component
for that specific historical era. This opens the way to my kind of analyis. Have fun!
Name Matching Notes: Algorithm tested for Alberti
14albberjac - perfect match of father except for wife's name (computer: sandra; genealogy: Lisa)
75% predicted with 100% accuracy for Alberti genealogy I-VII (missing VIII)
15% probably right 67% later than genealogy
10% probable 60% from alternatives given dates of marriages
total accuracy probably 91% but rated by certain / likely / best guess
some misses due to longer fa-son lag in marriage dates: 48,60,61,64,66 years, often due to second marriages
Second Test of First Algorithm
Includes a numeric quality rank for predicting potential matches
TRUE ANCESTORS occur at the following rank frequencies:
57 were rank 1 (predicted perfectly
1 tied for quality rank 1
5 tied for inferior rank 1 (both some possibility)
7 rank 2
2 rank 3
1 rank 4
1 rank 5
2 tied among six with inferior rank
2 not found at all, although six with inferior rank
PREDICTIONS NOT IN THE GENEALOGY FALL AS FOLLOWS
55 have only one possible match
6 have 2 possible matches
1 have 5 possible matches
1 have 6 possible matches
1 have 8 possible matches
LOOKED AT FROM VIEWPOINT OF HOW MANY POSSIBLE CHOICES:
-ONE: 100% in 75 predictions, 30 verified
-TWO-: 67% in 17 predictions, 8/12 verified
THREE: 59% in 17 predictions, 10/17 verified
FOUR+: 40% in 23 predictions 6/15 verified
FOR PAJEK FILES, stripping away the arcs from nodes with more than
1, 2, 3 etc. parents of a given type with successively improve the accuracy of the genealogy
OVERALL ACCURACY IF YOU TAKE ALL PREDICTIONS is 90% with no contradictions in genealogy
among those found in the genealogies, the confirmed hit rate is 73%
almost 100% of the ancestors in the genealogies who do have offspring are found by the
algorithm as possible alternatives, even if not the top prediction
Graphic Image of ALL predictions (men only), even if erroneous in many places
Yellow: 1 possibility only (100% accurate)
Green: 2 possilities (67% accurate)
Red: 3 possibilities (59%)
Other: 40% accurate, 4-9 possibilites (many those lacking dates from the tax records); the algorithm can be improved by upgrading predictions where there is one CLEAR LEADER because of marriage dates even if the name of the father is a common one.
This procedure will also improve accuracy.
flor1.for with documentation lines and
marriage dates assumption: Son later (10 to 40) years (increase to 66?)
marriage dates assumption: Son under 10/over 40: not the father (some errors!)
tax rolls assumption: Son same or up to 60 years later, if no marriage data
first names matchup: strip everything after / . , m.
adjacent lines with same master ID: same ego
Errors: Changes in the SPSS data
14albberant1 and 14albberant1 should be SEPARATE MASTER IDS IV:4 and IV:4 in GENEALOGIES
The Spss data is fairly clean as regards standardization of spellings BUT SOME MORE NAMES WERE STANDARDIZED.
problems encountered: Niccolo vs. Niccolo/aio m. etc.
188buobadghe Badella -> Badello should be male not female
367fretegner Teglia -> Teglio should be male not female
720sconerber Nera -> Nero should be male not female
14albmargia1 'F' should be female not male
9aglguemas marr 378 -> 1378 (error in date)
404gindomgiu taxyear 48 -> 480 (just a guess)
if you transform/calculate a new Spss variable taxlessm=taxyear+1000-marr and do a univariate frequency count you will
see that the difference runs from -65 to +64 with a mean of 2, and two wild outliers of +75 and +99 as follows:
292corlucgio taxed: 1427 Married: 1328 obviously one of these is wrong
62balaldagn taxed: 1427 Married: 1352 [[and of course there are many more where the dates are 50 or more years discrepant]
18aldmatbon fpart 366 -> 1366
flor.for with documentation lines and
Graphic Image of BEST predictions (men AND women), colored
yellow=one choice, green=2, red=3 and blue=4.
I am trying to make the most of the dates for fa/son matching hence:
lana has only two entries: 353 (421 cases) and 382 (326 cases). What are these?
cambm has dates from 1330-1524 what are they?
fpart has dates from 1299-1503 what are they?
nc427 1-8, diminishing freqs what are they?
comp427 eg C999000L what are they?
patr480 0-57930 mean of 73 what are they?
For my info
tax years are 351 378 403 427 458 480
ngh years are 351 378 403 427 458 480 - can get date from taxyear
pair480 1-over 334
qt403 1-4 quarters of town?