CG in flux, redux deluxe

<CG>, bless them, has a very liberal attitude about its database, and readily allows improvements and corrections to the data it contains.

Which is a mixed blessing, because, like any double-edged sword, it cuts both ways. One obvious problem is unvetted changes that must get backed out. There’s also the hypothetical possibility of edit vortexes being created, i.e. where a change is made, then unchanged, then changed back, etc. Wikipedia has seen similar issues, and has implemented a policy to avoid this. I’ve yet to see an example of this on <CG>, but the possibility exists, given that there are many editors, all of whom can independently make changes.

In fact, the freedom to change the data is very attractive. But, there is always the problem of tracking such changes, to do so consistently, and to have a global view of how many changes are being made, and by who.

Until recently, <CG> didn’t even track changes, at least not so that other editors could publicly view the log. And, even today, non-editors like myself, cannot view this information.

Next, the question is does anybody even measure and track the changes to the <CG> database, beyond the basic mysql tools?

Let’s ask a simple question as an example. Between August 23, 2014 and May 15, 2015, how many new players have been added to the <CG> database?

It turns out not to be a simple question. One of the reasons is that players get merged, when <CG> realizes that two sets of games belong to the same player. So one <CG> player (i.e. <CG> pid) disappears. That effect must be corrected for, before determining how many players have been added.

But there is also the case of a player being deleted only to have the same player reappear with a different name. One (name,pid) in one snapshot, and a different (name,pid) in another. So, it not really the same as if one player was deleted from the old snapshot, and a new, different, player added to the new snapshot.

There are several categories like these to track, in order to truly have an in-depth understanding of how the database is evolving.

Having two snapshots, taken about 9 months apart (~3/4 of a year), allows me to examine many of the changes to the <CG> database, at least from the player’s information.

Two CG data structures, C0 and C1, are loaded in. C0 is the earlier snapshot, and it is then examined for <CG> pid’s which are missing from C1, the later version. Call this list of id’s M0. Similarly, one can long for pid’s in C1 which are missing from C0, put inside M1 (M as in missing, 1  as in from C1).

The idea is that M0 should be the players who get deleted from the database, while M1 are supposed to be those who were added. One would expect M1 to be far larger than M0, as it reflects all the new players from new games added to the <CG> database.

The surprise is the there are 1009 entries in M0, and 1272 entries in M1.

What is going on?

Well, the first thing to do is to sort these lists by number of games in the database. We expect a lot of churn for players who number few games, but players with a substantial number of games can’t just disappear from the database, can they?

The other thing we can do is restrict our selection further to only those players with FIDE id’s. That allows us to search the other data set for a correspondence, in turn allowing us to track the <CG> changes with better understanding (remember, we are doing this all forensically from the data sets themselves).

So, with the understanding that fide0 and fide1 are dictionaries allowing  to lookup a player via the FIDE id, the following code and output should now make sense:

Note: Updated with correct gender fields thanks to <chessgames>

>>> for m in M0:
        if m.fide_id and m.games >= 3:
                print m
                try:
                        print fide1[m.fide_id]
                except:
                        pass
                print

     50280 / $   1210246  Alina Motoc                          (1985.06.01)  RO --      Y  F  2446 / 161
    127402 / $   1210246  Alina L'Ami                          (1985.06.01)  RO --      Y  F  2446 / 356

     41887 / $   4103564  Alexander Vladimirovich Zakharov     (1943.10.07)  RU --      Y  M  2437 / 114
    106617 / $   4103564  Alexander Zakharov                   (1943.10.07)  RU --      Y  M  2395 / 152

     50881 / $  14900866  Zuzana Hagarova                      (1977.11.25)  SK --      Y  F  2430 / 77
    143722 / $  14900866  Zuzana Stockova                      ()            -- --      Y  F  2374 / 79

    127442 / $  14507323  Kristina Solic                       (1986.12.19)  HR --      Y  F  2237 / 77
    133694 / $  14507323  Kristina Saric                       (1986.12.19)  HR --      Y  F  2261 / 94

     36786 / $   4636546  Jessica Nill                         (1979.10.11)  DE --      Y  F  2339 / 70
    146132 / $   4636546  Jessica Schmidt                      ()            -- --      Y  F  2339 / 71

     36097 / $    500755  Niina Koskela                        (1971.08.08)  FI NO      Y  F  2250 / 51
     89067 / $    500755  Niina (Koskela) Sammalvuo            ()            -- --      Y  F  2282 / 86

     53086 / $   7300115  Carolina Mazariego                   (1961.04.04)  GT --      Y  F  2086 / 50
     89048 / $   7300115  Silvia Carolina (Mazariegos) Kummerfeld  (1961.04.04)  GT --      Y  F  2012 / 66

     57096 / $  14102757  Natalia Kiseleva                     (1978.05.27)  UA DE      Y  F  2382 / 47
    123913 / $  14102757  Natalia Straub                       (1978.05.27)  UA DE      Y  F  2226 / 55

     50550 / $   1107453  Barbara Grabarska                    (1977.00.00)  PL --      Y  F  2253 / 44
     83238 / $   1107453  Barbara Jaracz                       (1977.06.08)  PL --      Y  F  2319 / 63

     53196 / $   4152662  Ekaterina Kineva                     (1986.02.27)  RU --      Y  F  2210 / 39
    150275 / $   4152662  Ekaterina Averchenko                 (1986.02.27)  RU --      Y  F  2199 / 48

     49644 / $    916528  Marija Stankovic                     (1975.08.10)  YU SB      Y  F  2259 / 37
    131910 / $    916528  Marija Marcetic                      (1975.08.10)  YU SB      Y  F  2180 / 39

     54430 / $   1114158  Edyta Andrzejewska                   (1981.03.03)  PL --      Y  F  2244 / 37
    133029 / $   1114158  Edyta Jakubiec                       (1981.03.03)  PL --      Y  F  2237 / 42

     84018 / $   9301232  Abdulla Faisal                       (1984.02.03)  AE --      Y  M  2316 / 35
     55764 / $   9301232  Abdulla Kashwani Faisal              (1985.00.00)  AE --      Y  M  2316 / 53

    118538 / $   4109961  Swietlana Miednikova                 (1956.04.09)  RU --      Y  F  2196 / 29
     64230 / $   4109961  Svetlana (Kruchonok) Mednikova       (1956.00.00)  RU --      Y  F  2205 / 40

    136505 / $   4193172  Ekaterina Timofeeva                  (1989.01.25)  RU --      Y  F  2331 / 28
    103233 / $   4193172  Ekaterina Timofeeva                  (1989.01.25)  RU --      Y  F  2299 / 69

     56166 / $   2905108  Svetla Yordanova                     (1976.05.08)  BU --      Y  F  2248 / 27
     58685 / $   2905108  Svetla Yordanova                     ()            -- --      Y  F  2158 / 39

     57013 / $   2401207  Carey A Wilman                       (1962.02.13)  UK --      Y  F  2051 / 27
    150905 / $   2401207  Carey A Groves                       (1962.02.13)  UK --      Y  F  2027 / 28

     52134 / $  14111268  Petr Lichman                         (1991.00.00)  UA DE      Y  M  2332 / 26
    114609 / $  14111268  Peter Lichmann                       (1991.03.26)  UA DE      N  M  2389 / 52

     58120 / $   1203347  Andreea Sasu-Ducsoara                (1975.00.00)  RO FR      Y  F  2244 / 26
    140258 / $   1203347  Andreea Bollengier                   (1975.05.17)  FR --      Y  F  2257 / 33

     48927 / $  14600609  Anita Licina                         (1972.09.27)  SI --      Y  F  2195 / 24
    147268 / $  14600609  Anita Vrabic                         (1972.09.27)  SI --      Y  F  2112 / 35

     58333 / $   1503260  Torill Hagesaether                   ()            NO --      Y  F  2013 / 24
    127930 / $   1503260  Torill Skytte                        (1982.05.11)  NO --      Y  F  1991 / 42

    122324 / $    913618  Maja Kostic                          (1973.06.06)  YU SB      Y  F  2046 / 20
    147269 / $    913618  Maja Velickovski-Nejkovic            (1973.06.06)  YU SB      Y  F  2083 / 31

     22019 / $   1106619  Monika Bobrowska                     ()            -- --      N  M  2470 / 19
     51452 / $   1106619  Monika (Bobrowska) Socko             (1978.03.24)  PL --      Y  F  2505 / 561

    103740 / $  14300800  Anzel Laubscher                      (1978.00.00)  SA --      Y  F  1856 / 19
    115927 / $  14300800  Anzel Solomons                       (1978.01.06)  SA --      Y  F  1908 / 56

     71274 / $  14400073  Vesna Misanovic                      (1964.11.27)  BA --      Y  F  2384 / 18
     37917 / $  14400073  Vesna Misanovic                      (1964.11.27)  YU BA      Y  F  2301 / 39

    141274 / $   1001949  Francis Lessmann                     (1953.00.00)  NL --      N  M  2109 / 18
    133490 / $   1001949  Francis Lesman                       (1953.00.00)  NL --      N  M  2111 / 23

     50560 / $   2906554  M Andreeva                           ()            -- --      Y  F  2218 / 17
    127951 / $   2906554  Mariya Janeva                        (1977.05.29)  BU --      Y  F  2042 / 24

     51876 / $  12800864  Zivile Ciuksyte                      ()            -- --      Y  F  2150 / 14
     94536 / $  12800864  Zivile Sarakauskiene                 (1978.11.06)  LT --      Y  F  2224 / 93

     55594 / $    933171  Radmila Kovacevic                    (1983.00.00)  YU SB      Y  F  2082 / 13
    135866 / $    933171  Radmila Zivic                        (1983.06.25)  YU SB      Y  F  2058 / 25

     80235 / $   4680790  Rena Mamedova                        ()            -- --      Y  F     0 / 13
     57079 / $   4680790  Rena Graf                            (1966.04.26)  UZ DE      Y  F  2314 / 26

    127919 / $   4001486  Marie Boyard                         (1989.04.17)  LU --      Y  F  2035 / 13
    103636 / $   4001486  Marie Boyarchenko                    ()            -- --      N  F  2028 / 45

    145792 / $   4113110  Lidia Tomnikova                      (1995.05.27)  RU --      Y  F  2216 / 13
    151902 / $   4113110  Lidia Tomashevskaya                  ()            RU --      Y  F  2210 / 22

     24981 / $    910589  Sanja Vuksanovic                     (1967.03.04)  YU SB      Y  F  2265 / 12
     49651 / $    910589  Sanja Vuksanovic                     (1967.03.04)  YU SB      Y  F  2274 / 52

     67237 / $   4152280  Olga Guseva                          ()            -- --      Y  F  2300 / 11
    121629 / $   4152280  Olga Iljushina                       (1981.11.02)  RU --      Y  F  2367 / 82

     87921 / $  14101807  Svetlana Ivanova                     (1976.02.19)  UA --      Y  F  2302 / 11
    141428 / $  14101807  Svetlana Fomichenko                  (1976.02.19)  UA RU      Y  F  2256 / 20

     16376 / $   2006162  Vadim Tsemekhman                     (1974.10.02)  US --      N  M  2355 / 10

    150854 / $   7101244  Dewi Ardhiani Anastasia Citra        (1994.00.00)  ID --      Y  F  2128 / 10
    119108 / $   7101244  Dewi Ardhiani Anastasia Citra        (1994.01.15)  ID --      Y  F  2180 / 65

     27645 / $   2802910  Aleksandar Serebrjanik               ()            RU --      Y  M  2444 / 9
     51139 / $   2802910  Alexander Kaspi                      (1964.05.21)  RU IL      Y  M  2491 / 58

     37756 / $  15000010  Rolando Redzepagic                   ()            -- --      Y  M     0 / 9
     17033 / $  15000010  Rolando Kutirov                      (1962.01.23)  YU MK      Y  M  2390 / 53

     72442 / $  15002241  Dritan Suaibi                        (1986.09.26)  MK --      Y  M  2065 / 9
     88259 / $  15002241  Dritan Suajibi                       (1986.00.00)  MK --      N  M     0 / 11

    150637 / $   1407635  Thomas Christensen                   (1976.00.00)  DK --      N  M  2139 / 9

     34972 / $    304271  Martina Pallova                      (1973.00.00)  CS --      Y  F  2203 / 8
    134396 / $    304271  Martina Folkova                      (1973.05.04)  CS --      Y  F  2203 / 28

     55634 / $  14507250  R Relic Radakovic                    ()            -- --      N  M  2085 / 8
     88982 / $  14507250  Rajna (Radakovic) Sargac             (1974.06.27)  HR --      Y  F  2243 / 61

     74920 / $   4675436  Maksim Iounoussov                    ()            -- --      Y  M  2184 / 8
    112061 / $   4675436  Maximilian Weimann                   (1987.02.18)  DE --      N  M  2188 / 17

     99164 / $  14100878  Viktoriya Schweitzer                 (1959.12.19)  FR --      Y  F  2286 / 8
     12250 / $  14100878  Viktoria Artamonova                  (1959.00.00)  UA --      Y  F  2177 / 16

    101861 / $   3204677  Robin Shaw                           ()            AU --      N  M  1849 / 7
    152179 / $   3204677  Robin A Shaw                         ()            AU --      N  M  1849 / 9

    133833 / $   2614502  Qi You Wu                            (1995.00.00)  CA --      Y  M  1982 / 7

     65535 / $   4616227  Veronika Werner                      (1976.06.23)  DE --      Y  F  2245 / 6
    146135 / $   4616227  Veronika Kiefhaber                   (1976.06.23)  DE --      Y  F  2229 / 7

    127972 / $   4617304  Jordanka Belic                       (1964.01.12)  YU SB      Y  F  2298 / 6
     60294 / $   4617304  Jordanka Belic                       (1964.01.12)  YU SB      Y  F  2378 / 46

     57151 / $   1002740  Adinda Koster                        (1971.09.02)  NL --      Y  F  2060 / 5
    147211 / $   1002740  Adinda Serdijn                       (1971.09.02)  NL --      Y  F  2048 / 13

     68980 / $   4165810  Alexandra Nazarova                   ()            -- --      Y  F  2286 / 5
    109124 / $   4165810  Alexandra Shumilina                  (1986.11.04)  RU --      Y  F  2298 / 14

     88731 / $  16000110  Rugabira Aciel Nzamwita              ()            RW --      N  M     0 / 5
    103451 / $  16000110  Aciel Nzamwitar                      ()            RW --      N  M     0 / 18

     98846 / $  13701010  Indira Babataeva                     ()            -- --      N  M  2172 / 5
    127395 / $  13701010  Indira (Babataeva) Bajt              (1980.00.00)  KZ SI      Y  F  2223 / 13

     61385 / $  13501445  Tatiana Martyniouk                   ()            -- --      Y  F  2134 / 4
    103659 / $  13501445  Tatiana Berlin                       (1977.04.09)  BY --      Y  F  2207 / 25

     87003 / $  14109190  Myroslava Yakivchik                  ()            -- --      Y  F  2234 / 4
    102249 / $  14109190  Myroslava Hrabinska                  (1984.05.22)  UA --      Y  F  2278 / 14

     36306 / $   9800026  Amelie Payet                         ()            SC --      N  F  2018 / 3

     52490 / $   5702348  Abang Azwan Draup                    (1983.00.00)  MY --      N  M  1648 / 3
    118899 / $   5702348  Hj Abang Azman Draup                 (1983.00.00)  MY --      N  M  1708 / 4

    107789 / $    913545  Sava Kizova                          ()            -- --      Y  M  2134 / 3
     49660 / $    913545  Sava Stoisavljevic                   (1974.07.05)  YU SB      Y  F  2125 / 5

    133470 / $   1210785  Corina Moldovan                      (1983.03.05)  RO --      Y  F  2234 / 3
     54790 / $   1210785  Corina Moldovan                      ()            -- --      Y  F  2234 / 21

    135964 / $   4170164  Oksana Molkova                       (1986.10.05)  RU --      Y  F  2154 / 3
    147237 / $   4170164  Oksana Golubeva                      (1986.10.05)  RU --      Y  F  2154 / 14

    140281 / $   4801067  Iman Hasan Mohammed Al-Rufaye        (1982.00.00)  IQ --      Y  F  2171 / 3
     58380 / $   4801067  Iman Hasan Mohammed Al-Rufaye        (1982.02.16)  IQ --      Y  F  2118 / 18

===========

c1-w-fide-c0.txt

>>> for m in M1:
        if not m.fide_id: continue
        try:
                print fide0[m.fide_id]
                print m, "\n"
        except:
                pass

# C0 comes first again, then C1.                
                
    145792 / $   4113110  Lidia Tomnikova                      (1995.05.27)  RU --      Y  F  2216 / 13
    151902 / $   4113110  Lidia Tomashevskaya                  ()            RU --      Y  F  2210 / 22

    101861 / $   3204677  Robin Shaw                           ()            AU --      N  M  1849 / 7
    152179 / $   3204677  Robin A Shaw                         ()            AU --      N  M  1849 / 9

     53522 / $  14901579  Jaroslav Fabian                      (1958.01.08)  SK --      N  M  2275 / 29
    152153 / $  14901579  Jaroslav Fabian                      (1958.01.08)  SK --      N  M  2169 / 4

    114336 / $    623520  Nikolas Jactel                       (1980.06.13)  FR --      N  M  2109 / 1
    151274 / $    623520  Nicolas Jactel                       (1980.06.13)  FR --      N  M  2126 / 2

 

zz

Advertisements

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s