Jump to content

Discussion of voting process


Recommended Posts

Because there are a few other folks with an interest in the data, I'm planning to post the tabulated spreadsheet of raw and weighted averages without the columns from individual voters. I promise to not look at individual votes either, other than to do some random checks to make sure my formulas are correct.

Link to post
Share on other sites
  • Replies 214
  • Created
  • Last Reply

Top Posters In This Topic

20 minutes ago, Christian said:

And for the sake of transparency on my end, and during the call announcing the results, can you confirm that the individual votes won't be visible to anyone but you (and maybe Ken)? I know most of us aren't shy about our opinions and have already shared our Top 25 lists, but with some rejiggering to follow the initial rollout of the "final" results (if I've kept up with the evolving conversation), I'd rather not be the object of some sort of lobbying campaign - during the results call or afterward - by proponents of a certain film I've rated lower than most. 

In this matter, I probably trust Darren more than I trust myself, but I think it is probably a best practice that at least two people have access to/copies of the votes themselves. It's been tough for me in the NCFCA or the EJ (before Joel took some of that over) to see some results I didn't always agree with and not hear a little voice that said, "...nobody would know if..." 

Again, not that I don't trust D., just that it's a bad look. 

So maybe Darren could pick one other person (or someone could volunteer) to just spot-check some of the ballots. Doesn't need to be me (I'd kinda prefer it isn't, but I'll do it if nobody else is willing). 

Link to post
Share on other sites
1 hour ago, Darren H said:

Because there are a few other folks with an interest in the data, I'm planning to post the tabulated spreadsheet of raw and weighted averages without the columns from individual voters. I promise to not look at individual votes either, other than to do some random checks to make sure my formulas are correct.

Thanks, Darren. I'm reassured by your and Ken's comments. 

"What matters are movies, not awards; experiences, not celebrations; the subjective power of individual critical points of view, not the declamatory compromises of consensus." - Richard Brody, "Godard's Surprise Win Is a Victory for Independent Cinema," The New Yorker

Link to post
Share on other sites
2 hours ago, Christian said:

And for the sake of transparency on my end, and during the call announcing the results, can you confirm that the individual votes won't be visible to anyone but you (and maybe Ken)? I know most of us aren't shy about our opinions and have already shared our Top 25 lists, but with some rejiggering to follow the initial rollout of the "final" results (if I've kept up with the evolving conversation), I'd rather not be the object of some sort of lobbying campaign - during the results call or afterward - by proponents of a certain film I've rated lower than most. 

I can't speak for anyone else, but I feel that no one should lobby you to change your mind. And I doubt anyone would...your voice and perspective is valuable to the project, as your voice, not as an amplification of someone else!
But I do like to hear people advocating for their choices, particularly if it's not something I appreciated. I like to hear people's passion in support of things they found meaningful, because maybe there's something I missed that I could better appreciate through someones advocacy. I mayor may not change my own level of appreciation, but at least I can understand the work better and why it is well regarded.
But maybe that approach isn't helpful to you when you are forming your own judgments, I don't know. At any rate, I will remember not to lobby you, out of respect for your opinions.

Link to post
Share on other sites

Thanks, Joshua. I don’t mind people advocating for their picks. I just didn’t want it known to all who had ranked which films lower. If someone wants to publicly take a stand against a certain film during the call or at the board, that’s their prerogative. What I was seeking to avoid was a visible display of how each voter ranked each film, which could - could - lead to a discussion putting me or other voters on the defensive during the call. “C’mon, Christian? You gave a 6 to Movie X but only a 2 to a movie that everyone else wants in the Top 25! What‘ll it take to get you to change your mind??”

Edited by Christian

"What matters are movies, not awards; experiences, not celebrations; the subjective power of individual critical points of view, not the declamatory compromises of consensus." - Richard Brody, "Godard's Surprise Win Is a Victory for Independent Cinema," The New Yorker

Link to post
Share on other sites

> Again, not that I don't trust D., just that it's a bad look. 

The only way I will be exercising any influence over the list is in determining how many votes are necessary to qualify for the list. I'm hoping the standard deviation will give us a number that also feels intuitively right.

Instead, we could decide on the cutoff point now. Someone suggested earlier that we go with only two votes because it would be the equivalent of the old procedure, one nomination and one second. Three votes? Four? I'm also happy to look at the data tomorrow, offer a recommendation here before announcing the list, and see if there are any major concerns or recommendations before finalizing it.

Link to post
Share on other sites
2 minutes ago, Darren H said:

Instead, we could decide on the cutoff point now. Someone suggested earlier that we go with only two votes because it would be the equivalent of the old procedure, one nomination and one second. Three votes? Four? I'm also happy to look at the data tomorrow, offer a recommendation here before announcing the list, and see if there are any major concerns or recommendations before finalizing it.

I think I threw out 2 votes as a cutoff, just as an easy number to demonstrate how the formula worked. I'd probably advocate a little higher than that.

"Anyway, in general I love tragic artists, especially classical ones."

"Even the forms for expressing truth can be multiform, and this is indeed necessary for the transmission of the Gospel in its timeless meaning."

- Pope Francis, August 2013 interview with Antonio Spadaro

Link to post
Share on other sites

If I understand correctly, you are speaking of votes that are not "haven't seen," correct? Because the formulas mentioned (similar to IMDB) already have a frequency adjustment built in...?

I too think the standard deviation provides insight. We've always tried to decide that number beforehand in Ecumenical Jury and it hasn't always worked. (Smaller jury pool, fewer films, more discrepancy with access to films.) In at least two years we adjusted that number after the fact based on data..

Would it make any sense to have two numbers? One for inclusion in the list at all and one for inclusion in the Top 25? The hardest ones to deal with in the Ecumenical Jury would be films (like Silence one year) where it was on the wrong end of the cusp as far inclusion but had numbers that would have put it in Top 5. Personally (and this is me spitballing, not being directive), I'd like to a number in the 3-5 range for being in the Top 100 and maybe a little higher (8-10) for inclusion in the Top 25. So if a film had the rating to be in Top 10 but not the frequency, we could downgrade it slightly but still include it.

EDIT: Darren I know you wouldn't exert any untoward influence. I also know in this day and age that appearances are sometimes as important as the actual practice. I trust myself and you to not put thumbs on the scale, but I can't say that has been true of every person who has collected and tabulated results, and it seems like a relatively easy gesture at transparency to share the data with at least one other person. 

 

Link to post
Share on other sites

Yeah, the weighting formula includes a minimum vote count. Once that's applied, it's unlikely that a film with fewer than 8 votes will make it into the top 25. I can't think of any film that, say, seven of us would give a 6 but that no one else has seen.

Link to post
Share on other sites
5 minutes ago, Darren H said:

Yeah, the weighting formula includes a minimum vote count. Once that's applied, it's unlikely that a film with fewer than 8 votes will make it into the top 25. I can't think of any film that, say, seven of us would give a 6 but that no one else has seen.

True, I hadn't thought about that, but it doesn't seem very likely.

 

Link to post
Share on other sites

Quick update. (I really should be doing work for my day job, but I couldn't not look this morning.)

On average, each film received 14.7 votes, and we ended up with a very low standard deviation (5.2). If we extend to the second deviation, the cutoff point would be 4.3, so we'd eliminate films that received fewer than five votes. That would trim only eight films. By comparison, we have 31 films that all 22 voters scored.

This seems about right to me. Are we good with "at least five votes" as our rule?

Link to post
Share on other sites

Five vote cutoff sounds good to me.

"Anyway, in general I love tragic artists, especially classical ones."

"Even the forms for expressing truth can be multiform, and this is indeed necessary for the transmission of the Gospel in its timeless meaning."

- Pope Francis, August 2013 interview with Antonio Spadaro

Link to post
Share on other sites

So, this is a delicate matter, but we need to address it before distributing the results.

I *thought* we had agreed that voting would be limited those people that submitted nominations, but we did a lot of discussion and can't find that agreement. We had at least two people who voted who did not submit nominations. Also, if Darren says 22 people voted in total and we had 23 nominators, that means at least three people submitted nominations and then did not vote. 

We can proceed in one of a couple ways:

1) Tally based just on the votes of those who nominated.

2) Tally including all the votes.

I'm inclined to...well, I'm not sure what I'm inclined to do, other than it's one of my personal buttons when people in charge of anything announce one set of rules and procedures and then follow another. That being said, not electing Supreme Court Justices here, so I'm willing to roll with it if that's what others want. (incidentally, if you have strong feelings either way and don't want to post publicly, feel free to PM me.)

Link to post
Share on other sites

Should we reach out to the one person who didn't vote and offer them a chance to vote today, just in case an emergency came up for them?

"Anyway, in general I love tragic artists, especially classical ones."

"Even the forms for expressing truth can be multiform, and this is indeed necessary for the transmission of the Gospel in its timeless meaning."

- Pope Francis, August 2013 interview with Antonio Spadaro

Link to post
Share on other sites
6 minutes ago, Evan C said:

Should we reach out to the one person who didn't vote and offer them a chance to vote today, just in case an emergency came up for them?

Darren already sent several reminders, so I would expect that if something happened to prevent someone from voting who wanted to vote, it should be up to that person to reach out to Darren (or me) and indicate as much.. If the nature of an emergency or an impediment is that the person literally could not vote or inform D that he/she was not voting, then it seems unlikely that an additional 24 hours would help. That's not meant to impugn anyone. There could be all sorts of reasons why someone misses a vote deadline that are unavoidable. 

Link to post
Share on other sites

I just weighted the scores, and we definitely have some issues to address. I've always known Ordet and The Passion of Joan of Arc had a chance of landing in the top 10, but they placed #1 and #2 -- because more than a third of voters gave both films a 6! We also have two Malick films and two Tarkovsky films in the top 11, again because several people gave them both a 6.

I'd prefer to not have two films by one director in the top 25, but I don't know if that's a popular opinion.

Link to post
Share on other sites

Maybe extend the ranking of the top 25 to ranking the top 30 (still with only 1-25 points), with a director's second film ranking 26-28 depending on the order in which they finish?

"Anyway, in general I love tragic artists, especially classical ones."

"Even the forms for expressing truth can be multiform, and this is indeed necessary for the transmission of the Gospel in its timeless meaning."

- Pope Francis, August 2013 interview with Antonio Spadaro

Link to post
Share on other sites
9 minutes ago, Darren H said:

I just weighted the scores, and we definitely have some issues to address. I've always known Ordet and The Passion of Joan of Arc had a chance of landing in the top 10, but they placed #1 and #2 -- because more than a third of voters gave both films a 6! We also have two Malick films and two Tarkovsky films in the top 11, again because several people gave them both a 6.

I'd prefer to not have two films by one director in the top 25, but I don't know if that's a popular opinion.

That's very much my opinion as well, even though I love both Dreyer films. (Malick and Tarkovsky not so much, but I can't say I am surprised by that result.) My recommendation would be to leave the higher of the two ranked films and put the second at 26, 27, and 28. I think we should also note that in introduction and blurbs (and/or book chapters).

Of course we could also have a run-off: Ordet vs. Joan in the showdown of the century!

Link to post
Share on other sites

If we apply that rule it will also affect Bresson and the Dardennes -- again, to absolutely no one's surprise. Although I am a bit surprised by which Bresson films made the cut. And, yes, I'm going to keep teasing results throughout the day!

Link to post
Share on other sites

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.

Guest
Reply to this topic...

×   Pasted as rich text.   Paste as plain text instead

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.


×
×
  • Create New...