In Defense of Peer Review

hi everybody I recently got back to Colorado and I'm gonna be here for a little bit sometimes I look through the reddits and I find some interesting threads and occasionally they encourage me to make a statement or say something and I came across the thread that was a bit critical of the entire peer review process and the article the the thread cited several different issues with the overall peer review process in particular in the American academic system for example the grievant Studies affair and then some issues with some Israeli research making some claims about medicine and perhaps they weren't true and so what I wanted to do was make a video because I see this as a common thread throughout the cryptocurrency space this questioning a constant questioning of why is peer-review necessary or does peer review actually produce good results or our scientists just all biased and they all have agendas and apparently just say whatever you want get it published and it's not a legitimate way of resolving debate so it's incredibly important first to separate social studies or even studies the liberal arts from the STEM fields if you look at the the social studies the liberal arts these fields what we've noticed over the last 50 years in particular has been a degradation of rigor and a replacement of opinion as if it's fact and the grievin Studies affair and numerous other examples have have really shown that it's not okay to be able to take mine Kampf and replace Jews with white men and suddenly that's publishable because certain circles politically think that's fine in a way this has badly damaged in my view and many people the integrity of the overall university system and academic affairs and that's certainly a concern and it's a big topic and it's it's something that we as a society have to work our way through and I would agree that if you've replaced the scientific method rigor critical thinking with just a political agenda and just a subjective opinion whatever happens to be the latest buzzword if you use that either published and if you don't well you're excommunicated or if there are forbidden topics to talk about like intelligence studies or other things that's certainly an issue but when you look at the STEM fields particularly mathematics computer science physics these fields their integrity is mostly intact I and for the most part if you're arguing around a particular topic like if something provably secure it is this mathematical proof correct this has not been in any way compromised by a political agenda I don't even know what the political agenda would be if a particular block cipher is right or wrong is this there's no such thing as a cisgender or transgender symmetric crypto scheme it's it's a statement of this is your adversary this is your mathematical model this is this is the rationale behind why it's secure and it's argument it's a mathematical proof it's something that anyone regardless of their language their culture their their socioeconomic status may be anywhere in the world can read that and either say you're right or you're wrong and if you're wrong provide a counter example or some provide some sort of hole and the way that the proof was constructed and you also have to understand that research is only as good as the process upon which the research is analyzed so if you look at cryptography in particular this is an incredibly fertile rich field that is very adversarial when you publish a paper it always regardless of who you are is instantly viewed with some degree of skepticism and there are many people whom have made careers out of doing nothing but trying to find ways to rip apart your papers and the whole point of the process that cryptographers have created with conferences in ewa Sariel peer review and a very fast-paced system and the fact that governments militaries intelligence agencies rely upon the outputs of this process to secure classified information military secrets nation secrets has created a fertile environment where bad ideas are usually meticulously and mercilessly picked apart very rapidly so when you look at the grand scheme of things you look at grand security claims Intel SGX is is a great example of this where it has a lot of merit and in some cases trusted hardware is magical but then scientists can make a career doing nothing but finding holes in that scheme when I was at CCS in Toronto right after we presented or Boris Genesis just down the hall there was a group of scientists who they've published several papers showing where Intel's claims about security are wrong and they're making their careers basically forcing Intel to own up to the fact that maybe their system isn't as good as it should be the other thing is that unlike journals which takes sometimes years for research to actually end up getting published fully peer-reviewed and get through the system conferences are very frequent if you look at the cryptographic world you have euro crypt CCS real world crypto financial crypto in dozens of other conferences every year almost every month there's some form of conference that's going on so it really doesn't slow you down to write a paper in a very structured thoughtful way get it into a conference and then get some review from the community and all of a sudden now you have some of the brightest people in the world waking up trying to find a way to destroy your argument because they know that it benefits their academic career if they can find a flaw in your paper furthermore marrying formal methods with peer review basically forces the scientists to be honest because what happens is you extract the specification from the paper and all these hand waving areas of the paper where they say well ideal functionality just assume the network works perfectly and there's no latency in a blockchain can instantly be transported anywhere in the world to any user regardless of their internet connection the formal methods guy says well that's a great fantasy but if we're actually going to build this and implement this and I'm going to write a specification for this we actually have to be very specific about what you mean when you say you can do X or Y and then when the engineer gets involved the engineer can further force practicality out of these schemes so if you utilize formal methods in peer review you're not guaranteed to get a perfect secure system but what you are guaranteed to get is an adversarial system where people have a reputational and career benefit of finding a flaw in your ideas and you have somebody just as smart as the scientist who's in a different field of computer science using very sophisticated techniques in some cases computer-aided techniques like model checking and other such things to try to tease out whether these ideas are actually practicable or actually work and you can do all of these things in weeks to months not years to decades that's the magic of building a good streamlined process and what does it mean it means by the time we get to implementing the actual protocol it's already been rigorously rigorously debated and argued and people have great incentives to say that what we have done is wrong so it's not just a spray and pray it's not just a release and say well let's see if some guys or gals somewhere in the world can now figure out if the system is broken or not that's perfectly fine in the open-source ethos when it's BitTorrent when it's OpenOffice when the consequences of a bug or poor design means that you have suboptimal product it crashes it may have a security flaw maybe in Bobby's essay gets stolen or corrupted but when you're talking about a world financial system that will event we have billions to trillions of dollars of value behind it why the hell wouldn't you want some third-party assurance from people who aren't shills people who have no financial incentive to say it's good that is good why wouldn't you want to utilize a four hundred plus year old system that's given us modern-day physics modern-day mathematics modern-day medicine modern-day computers and all the things we take for granted these LCDs semiconductors all these things why wouldn't you want the same process that train the minds that built these things and kept the minds that built these things accountable to be applied to your money system especially if the trade-off is it slows you down by a factor of several months not years but several months so I take great umbrage when I see reddit threads or I see people attempt to over generalize and say because of the grievant Studies affair or because somebody decides that they want to abandon logic in their particular discipline of study that somehow computer science has been corrupted the system that this field uses is fundamentally good it works very well it's very fast it's very responsive and it allows you to go from lab to industry quickly why because the primary subsidy for this system comes from industry Microsoft IBM Apple Google Intel AMD Nvidia are massive proprietors their benefactors they give money to the research arms and if you look at many of the papers in computer science they come from scientists who are in and out of industry all the time what are the big pioneers of AI is Andrew Ning and another one's Peter Norvig another Sebastian Thrun and these guys wrote all the text books they they've done amazing research and AI and they just so happened to be working on things like self-driving cars and Coursera and Udacity and the Google Translation system so that while they were doing scientific research they're building production systems consumer products that we use every single day this is because the field of computer science is set up and built from the ground up to be able to facilitate that seamless movement between the needs of Industry and maintaining scientific rigor so at AI ohk what we've done is we've tried to be very pragmatic you can go crazy with formal methods the SEL for project or concert and other projects are great examples of that there are major milestones wonderful research but they took years to decades to get done that would not make Cardno or any product i which kate competitive if we went down that road but using things like TLA using things like using things like just writing a latex back in and getting the mathematics down this does not take years this takes weeks to months and then we now have an ambiguity free mathematically precise way to talk about what the hell are we doing if you look at recent bugs like for example coin desk had an article about a flaw and z cash the z cash team I know personally we sometimes publish papers with a Z cash team these people are wonderful professionals they're great engineers they have phenomenal processes and they have great standards and I would count them among the top five development teams in all of cryptocurrency despite the fact that they have that talent and those capabilities there was a bug that if exploited would have allowed the infinite production of money in that system effectively making the entire cryptocurrency useless especially given the fact it's hard to know who owns what now the use of formal methods actually allows you to catch these types of bugs allows you to discover these types of things not after they've happened after you shipped the code to customers but before you ship code to customers so this is why you use formal methods z cash is based on peer-reviewed research and it's only one of the core reasons i think a lot of people trust it and trust the anonymity guarantees because the fact that great science has got together and thought very carefully about this and they're using very complicated cryptographic primitives that normal engineers normal people really don't understand so in defense of peer review I think that what we need to do is take a step back and think about things very rationally and we have to ask what are the incentives for people to say you're right what are the incentives for people to try to find a way to prove you wrong you have a very good productive system when people have brand reputational career or financial incentives to try to find a way to prove the work you're doing is broken and if they can only do that through a structured process where they have to be honest they're not shilling they're not writing papers and just making things up but they actually have to provide a counter argument a counter example they have to actually demonstrate wherein the proof your wrong then that's the best of both worlds because in the process of rebutting you in destroying your argument they actually teach you something and you're having a constructive productive dialogue so I think that this is frankly the standard our industry as a whole has to embrace if theory agrees with us they've given 5 million with others to Stanford and set up a lab and they're starting to move in the peer review direction Algren is doing the same thing unity is doing the same thing z cash is writing papers many other people are writing papers so this is not an eye on which K thing anymore this is quickly becoming an industry standard and all of you everyone you really have to think carefully if a person is making a claim you have to train yourself to ask a question well has that claim been vetted if so by whom what incentives did that person or organization have to either agree or disagree with the claim and by what process have we established trust that that claim is credible the only way I know how to vet scientific research is by utilizing what science has done for four hundred years I fully admit and accept the university is not a perfect ecosystem I fully accept and admit that it has some real big problems especially on the social sciences in the liberal arts area and it's losing credibility there but that is not our problem in our space in our field computer science is led by industry and the science is sculpted by industry so it means it has to be done quickly it has to be done with a product oriented and it has to be done in an objective way because at the end of the day the free market doesn't care what gender you are doesn't care what your political values are your your personal philosophy or whatever grievances you may have had a hundred years ago free market care is doesn't work does it work well and is it the most efficient way of solving this particular problem and can we distribute it can we get it out there is it easy to do so when you have free market competitive nature behind things the research you do the science you do has to work it has to be pure and when the military has to get it right when the intelligence agencies has to get it right then that really means you have to have a good effective system so that's a brief video but I just wanted to really rebut this this claim that somehow because one part of the system is not working the whole system should be thrown away I really do believe the processes we were following are necessary they're not sufficient you have to do more you have to have formal methods you have to give it time you have to build the systems and it doesn't matter if you have peer review done that doesn't promise you that it's competitive it's practical or it's going to work for the problems your customer have so the ultimate test of any product you build is shipping that product to the customer to the marketplace and seeing what the marketplace does over the long run not in the short term is you can ship trash and trash can win in the short term but over the long term are three to five year to 10 year horizon the market will inevitably tell you if these ideas are meaningful so that's basically it and I really I just I I want to make sure that everybody understands that these processes exist for a reason these people exist for a reason and this is your money and at the end of the day if we end up being pulled into this industry and cryptocurrencies work these protocols will be used for years to decades to centuries and if they're Mis designed the poor decisions that I make and other designers make you're going to have to live with for the next 20 30 40 50 years your kids your grandchildren may have to live just we have to live with the poor decisions that government's made in the past whatever at Bretton Woods was decided whatever at the World Economic Forum was decided thirty years ago that is now impacting the markets the banks your your cash in your pocket today the national debt of the United States or wherever you happen to live the good decisions the bad decisions you have to live with so my core argument is we have a moral obligation to use the best tools the best techniques and the best processes possible to ensure that the system we construct ends up being used wisely responsibly and securely and fairly as we move into the 21st century at our company the only way we know how to do this is to marry good engineering peer review and formal methods together as a bundle process and get a customer oriented culture where we get the product out in front of people as quickly as possible even if that means sometimes it's not an optimal product it's a little slow or crashes or doesn't have the right performance when you go through the process of peer review and third-party security auditing in formal methods you can at least guarantee things can't happen like people can't lose their money or there aren't going to be foundational problems in the protocol which can be exploited to create coins out of thin air or other such things and then you have to just work hard to make the product better and more competitive and have a better user experience so anyway that's that's that so thank you guys for your time and talk to you again later