Systems performance analysis is an important skill for all computer users, whether you're trying to understand why your laptop is slow, or optimizing the performance of a large-scale production environment. It is the study of both operating system (kernel) and application performance, but can also lead to more specialized performance topics, for specific languages or applications.
Operating systems based on two different kernels are used as examples in this book: Linux-based: Ubuntu, Fedora, and CentOS; and illumos-based (a fork of OpenSolaris): SmartOS, and OmniOS. You may be interested in only one of these, but covering others provides additional perspective, helping you better understand the design choices, and performance results, of each.
This book is primarily for system administrators, support staff, operators, and devops in enterprise and cloud environments. It is also a useful reference for developers, database administrators, and web server administrators who would like to understand operating system and application performance.
While it covers performance tools and the background for understanding them, what makes this book different is the inclusion of many performance methodologies, including those covered quickly in my USENIX 2012 talk. I've been teaching and developing systems performance classes on and off for ten years, and have found methodologies to be crucial for giving students a starting point, and then guiding them through performance activities. The USE Method is one example I developed for this purpose.
Educational Data Mining (EDM) in the research field will constitute an application in major techniques like Data Mining, Machine Learning and Statistical Techniques in the education and organization sector. It aims at defining better mechanisms to analyze student performance by use of sophisticated predictive techniques. The insights from the analysis based on their previous performances can be used for future performance prediction, counselling students for university enrollment and to help them select electives for their undergraduate courses. This also helps students decide on their career path and the colleges to monitor student performance at any given time, so that it can be used as a record for their placements. To gain insights from the data one can choose their own metrics. The major challenge in this is to capture and clean the data and also to find an appropriate technique to carry on with the analysis. This paper introduces best suited methods to capture data by choosing the right metrics and performance indicators. In EDM, a right metric is the one that is unbiased and which considers all the aspects whether academic or non-academic. Through this paper we also intend to help analysts decide on choosing the right approach, techniques and algorithms to start with EDM.
Common mistakes in performance eval no goals ignore input errors biased goals omitting assumptions and limitations unsystematic approach overlook important parameters inappropriate experiment design assuming no change in the future ignore significant factors unrepresentative workload(s) ignoring social aspects
Common mistakes in performance eval no analysis analyse without understanding the problem too complex analysis erroneous analysis wrong evaluation technique ignoring variability incorrect performance metric improper treatment of outliers improper presentation of results no sensitivity analysis
As a teacher of computer networking, I appreciate the layering in the network architecture as it helps me organize the course content, structure the lectures and have focused discussions with the students. Layering has many advantages (reduced complexity through separation of concerns and ease of maintenance through modularity). It has also disadvantages mainly in terms of performance (sometimes redundant functionality) or lack of optimized operations (particularly stresses in wireless communications where cross layer interactions/design may be useful, e.g., exploiting properties of physical channel at network or application layers). However, the advantages by far surpass the disadvantages.
NOTE: Due to the switch from Mondays to Tuesdays weare currently short one meeting but we may leave thingsas they are due to the extra meeting added after the lastofficial day of classes.UNDER CONSTRUCTION: I'm adding more information andmany of the links on this page will not work yetIf you have suggestions for papers you would liketo read or topics you would like to cover please let me know. Paper Summaries *** NEW *** There may be two different types of summaries for differenttypes of papers. Research papers will require research papersummaries. Other papers (e.g., survey papers) will requirea survey paper summary. The format of these can be foundby following the links below. UNLESS OTHERWISE STATED SUMMARIES WILL BE DUE ONMondays at 11:59 am (i.e., midday / noon eastern time) CLICK ON THIS LINK AND READ AND FOLLOW THEINSTRUCTIONS CAREFULLY FOR HOW TO DO SUMMARIESInformation and examples Recorded Class Videos The recorded videos of the course meetings from eachweek can be accessed using the link below.The required username and password have been posted to Piazza.Class Videos Week 1: Monday Sept. 14, 2020 (Topic: Introduction)No readingsIntroductionsCourse overview and a bit of backgroundInformation about the mini assignment (due next week) Week 2: Tuesday Sept. 22, 2020 (Topic: Measuring WiFi Throughput) Mini Assignment Due prior to Monday Sept 21 at noon. Mini Assignment InstructionsMini Assignment Presentations (limited to X minutes)CHECKLIST Read the assignment carefully. Conduct the experiments. Check back here to find out how long your have for your presentation. Think about the message / story for your presentation. Read the portion of the assignment "Some Tips for Giving a Presentation" Create the presentation (be sure to use the points described in the assignment). Generate a PDF of your presentation. Submit the presentation using the naming scheme provided before the deadline. Practice your presentation out loud ideally using an online video tool. Week 3: Tuesday Sept. 29, 2020 (Topic: Variability) Due Monday Sept 28 at 11:59 am (just before midday / noon) NOTE: Papers 2a - 2d are each only about 1-2 pages(so there isn't as much reading as it may seem)Some Statistics / Background (watch some Khan Academy Videos) [NO SUMMARY REQUIRED]How to read a paper[NO SUMMARIES REQUIRED]Paper Reading Check List,Sugih Jamin, firstname.lastname@example.org.How to Read a Research Paper,by Spencer RugaberEfficient Reading of Papers in Science and TechnologyPrepared by Michael J. Hanson, Updated by Dylan J. McNamee, January 6, 2000.How to Read a PaperS. Keshav, ACM SIGCOMM Computer Communication Review, Volume 37, Number 3, July 2007.Conducting Repeatable Experiments and FairComparisons using 802.11n MIMO Network[RESEARCH SUMMARY REQUIRED]Ali Abedi, Andrew Heard and Tim BrechtOperating System Review, 2015.CHECKLIST (in order) Create a PDF file that contains a simple bit of text like "Hello World" When you get email from Crowdmark (I will set this upsoon) with instructions for submitting yoursurvey (it will likely be called an assignment),test that you can submit that file to Crowdmark as your solution.You will be replacing this submission later but you don't want to discoverproblems at the last minute. Watch the Khan Academy Videos (for the topics you don't already know) Read the information provided about Bootstrapping Read papers 2a - 2d above Read the instructions for producing a RESEARCH PAPER SUMMARY. Read the example research paper summary Read paper 3 Write a research paper summary for paper 3 Proof read your research paper summary Edit your research paper summary Proof read your research paper summary Edit your research paper summary Proof read your research paper summary Edit your research paper summary Generate a PDF of your research paper summary Submit the summary before the deadline. Week 4: Tuesday Oct. 6, 2020 (Topic: Background and Motivation) *** NEW ***Due Monday Oct. 5 at 11:59 am (JUST BEFORE MIDDAY / NOON)From Repeatability to Reproducibility and Corroboration(Survey Paper Summary required -- skip the last question (i.e., just cover questions 1-3).Dror FeitelsonOperating Systems Review, 2015.Producing Wrong Data Without Doing Anything Obviously Wrong!(Research Paper Summary required)Todd Mytkowicz, Amer Diwan, Matthias Hauswirth, and Peter F. SweeneyASPLOS, 2009.Come up with a potential group and potential project. See more detailsin the CHECKLIST below.CHECKLIST (in order) Read the information above carefully regarding due dates and times. Read the instructions for producing a SURVEY PAPER SUMMARY. Read paper 1. Write the summary for paper 1 Read paper 2. Write the summary for paper 2. Proof read your summaries multiple times, editing them and improving them each time. Generate a PDF of both summaries. Submit the summaries before the deadline. Read the information on the course web page about what is required for the project. NOTE THAT THIS HAS BEEN UPDATED SO PLEASE READ IT AGAIN. Reach out to people in the class about forming groups via Piazza. Arrange to meet with potential group members via MS Teams, Zoom or whatever to discuss possible project ideas. Choose a potential project idea. Have only ONE member of the group click on the link below tosend email containing the following information: Full names of the group members (using the names the university knows about and uses :-) The title of the project A short one pargraph description of the idea behind the project. Note that this isn't the "Project Proposal" (that will come later). This will be used for approval before you actually do a project proposal. Click me to send email for Potential Group for CS854 Reading Week : Tuesday Oct. 13, 2020 (TBA) No class (due to reading week) Week 5: Tuesday Oct. 20, 2020 (Topic: Workloads and Benchmarks) Due Monday Oct. 19 at 11:59 am eastern (JUST BEFORE MIDDAY / NOON)Except for the Project Proposal (see below for details).Read these papers in the order shown belowNote thatthe first two are relatively short and the 4th doesn'tneed to be read in excruciating detail.Workloads (Creation and Use),(No Summary required) Alan Jay Smith,CACM, November, 2007.httperf -- tool for measuring web server performance(No Summary required),David Mosberger and Tai Jin ,ACM SIGMETRICS Performance Evaluation Review (PER), Volume 26 Issue 3, Dec. 1998.Methodologies for Generating HTTP Streaming Video Workloads to Evaluate Web Server Performance,(Research Paper Summary required)Jim Summers, Tim Brecht, Derek Eager, and Bernard Wong,5th Annual International Systems and Storage Conference (SYSTOR), 2012.Characterizing the Workload of a Netflix Streaming Video Server,(No Summary required)Jim Summers, Tim Brecht, Derek Eager and Alex Gutarin,IEEE International Symposium on Workload Characterization (IISWC), 2016.You are not required to do a reallydetailed reading of this paper and don'tneed to understand all of the results.Focus on what is being done, why, and how.Read the questions below first and ensure that youread the paper well enough to answer all of them.Answer the following questions about this week's papers. Why is it important to understand workloads? Do you think httperf is a good tool or not and justify/explain your answer? What is the difference between a workload generator, aworkload and a benchmark? Explain why you think you've been asked you to read (in order)papers 2, 3, and 4. Describe *** exactly two *** (no more and no less)ideas for research thatcould be conducted as followupwork to the "Characterizing the Workload of a Netflix Streaming Video Server"paper.The answers to all questions must all fiton a single page (I expect most will be much shorter).Your solution must be submitted to Crowdmark as a PDF file.Course Project Proposal:Submit a PDF slide presentation for your course projectproposal. See the information on the course project pagefor details and please follow the instructions there.**** NOTE: This is due Wed. Oct 21 11:59 pm eastern (just before midnight)CHECKLIST (in order) Remember to post your group members, project workingtitle and short description to the Piazza thread.Piazza Group Project List Read the information provided carefully regarding whatyou need to do and the due dates and times. Start by reading the list of questions in point number 5 aboveto help guide some of your reading.Keep these questions in mind as you read the papers. Read Paper 1. Read Paper 2. Read Paper 3. Write the summary for Paper 3. Proof read your summary multiple times, edit it and improve it each time. Generate a PDF of your summary then proof read the PDF and edit if needed. Submit your summary for Paper 3 using Crowdmark(this will be labeled Q3). Read paper 4. Create a document that answers the questions aboveabout this week's readings. Proofread the answers and produce a PDF for submission Submit your answers to the questions using Crowdmark(this will be labelled Q5) Submit your course project proposal. Your file name must contain only the Waterloo email/userid (NOT student number) followed by .pdf This will be the group member that Iwill email feedback too (they will forward itto the other group members).For example if I'm the group member that email shouldbe sent to, my file would be named brecht.pdf(because brecht is my Waterloo userid/email name).Following these instructionsis important because I will have a scriptthat looks at the filename and sends a copy of the markedup file to that Waterloo userid. Send the file as an attachment by clicking on this link: Click me to email your Project Proposal Week 6: Tuesday Oct. 27, 2020 (Topic: Conducting Evaluations and Common Mistakes) Due Monday Oct. 26 at 9:00 am **** NOTE THE CHANGE IN TIME ****I know that I said I would have two readings but instead of coveringtwo research papers this week we are reading one paper and coveringtwo other very short documents / videos.The Truth, The Whole Truth, and Nothing But the Truth: A PragmaticGuide to Assessing Empirical Evaluations(Survey Paper Summary required -- skip the last question (i.e., just cover questions 1-3).Blackburn et al.,Transactions on Programming Languages and Operating Systems, 2016.Systems Benchmarking Crimes(No survey required)Gernot HeiserWatch the this video until the 19:19 mark (feel free to watch more if you wish).Common Mistakes and How to Avoid Themby Raj Jain.(No survey required)Here is a copy of the slides are used in the video (approximately the slides)Common Mistakes and How to Avoid ThemThe lecture and slides are based on (from) Raj Jain's textbook.The Art of Computer Systems Performance Analysis:Techniques for Experimental Design, Measurement, Simulation,and Modeling, Wiley- Interscience, New York, NY, April 1991,CHECKLIST (in order) Reread the post in Piazza "Writing issues / tips" (I've added some notes this week)and be sure to follow those tips when writing your surveys.Read the information provided about whatyou need to do and the due dates and times carefully. Read Paper 1. Write the summary for Paper 1. Proof read your summary multiple times, edit it and improve it each time. Generate a PDF of your summary then proof read the PDF and edit if needed. Submit your summary for Paper 1 using Crowdmark(this will be labeled Q1). Read document 2 above. Watch the video in 3 above and review the slides. Week 7: Tuesday Nov. 3, 2020 (Topic: Dynamic Instrumentation) Due Monday Nov. 2 at 9:00 am**** NOTE THAT THERE WILL BE A TIME CHANGE ON NOV 1 ****I suggest that you check the times carefully for the time zone thatWaterloo, ON is inand your time zone and remember that you can always submit early.Dynamic Program Instrumentation for Scalable Performance Tools,(Research Paper Summary required)Jeffrey K. Hollingsworth Barton P. Miller Jon Cargille,HPCC 1994.Dynamic Instrumentation of Production Systems,(Research Paper Summary required)Bryan M. Cantrill, Michael W. Shapiro and Adam H. LeventhalUSENIX ATC 2004.OPTIONAL (BUT HIGHLY RECOMMENDED) READING/LISTENING:A Wardrobe for the Emperor: Stitching Practical Bias intoSystems Software ResearchBrian Cantrill, Keynote Talk at USENIX ATC 2016.Follow the link to the slides and/or the audio of the talk.This talk presents some interesting viewsfrom one of the authors of Dtrace on USENIX,conferences, the review process, program commitee meetingsand publishing.I would recommend listening to the audio while following along with the slides(the slides are nteresting but the audio is much more illuminating).CHECKLIST (in order) Reread the post in Piazza "Writing issues / tips" and be sure to follow those tips when writing your surveys.Read the information provided about whatyou need to do and the due dates and times carefully. Read Paper 1. Write the summary for Paper 1. Proof read your summary multiple times, edit it and improve it each time. Generate a PDF of your summary then proof read the PDF and edit if needed. Submit your summary for Paper 1 using Crowdmark(this will be labeled Q1). Read Paper 2 above. Write the summary for Paper 2. Proof read your summary multiple times, edit it and improve it each time. Generate a PDF of your summary then proof read the PDF and edit if needed. Submit your summary for Paper 2 using Crowdmark(this will be labeled Q2). Week 8: Tuesday Nov. 10, 2020 (Topic: Miscellaneous) NOTE THAT THERE ARE MULTIPLE DIFFERENT DUE DATES THIS WEEK AND NEXT.SEE THE CHECKLIST THIS WEEK AND NEXT FOR DETAILS.YOU CAN CERTAINLY COMPLETE THINGS BEFORE THE DEADLINE(ESPECIALLY PICKING A TOOL).Learning in situ: a randomized experiment in video streaming(Answer the questions provided below -- instead of a paper summary)Francis Y. Yan, Hudson Ayers, Chenzhi Zhu, Sadjad Fouladi,James Hong, Keyi Zhang, Philip Levis, and Keith Winstein,NSDI 2020.If you follow the link below you can also get access to the talk slidesand a video of the talk.Link to the NSDI page that contains talk slides and the video.I don't know but I suspect that to answer the questionsit would be best to be sure that you read the paper.Watching the video is not required but you may findit useful to help understand that paper.You might want to watch the talk first and then read the paper. How would you rate the performance evaluations in this paper?Be sure to state what is good and bad about the evaluationsand be clear which evaluations you are referring to. How would you rate the paper in terms of the claims it is making?Describe if you think the claim(s) made are sound or not and explainyour reasoning. Be sure to describe which claim(s) you arereferring to.Your answers must fit on one page (single sided).Use the same font size and margin restrictions as for summaries.Produce a PDF file to be uploaded via Crowdmark.Using point form is fine.An introduction to Docker for reproducible research(No Summary Required)Carl BoettigerOperating Systems Review, 2015.Choose the tool you will describe for Week 9.This is being done via Piazza. Working together as a group choose a tool that is usedto help measure and/or understand performance and create a slide deck(presentation) describing that tool and what it can be used for.If possible include a small example of how it can be usedand an example of the output. Examples of some possible tools include but are not limited to:gprof, Instruments, vmstat, iostat, oprofile, vtune, TAU, JMeter, iperf NVIDIA seems to have severalGPU profiling tools. Here is a list of performance analysis tools This seems to refer to sometools for profiling python programs.CHECKLIST (in order) Choose a few tools your group would like to cover.Pick a few in case the one you want is taken.Check that those choices have not already been pickedby other groups. Due by Wednesday Nov. 4. 11:59 pm (just before midnight).One group member post something to the Piazza thread I've created statingwhich tool your group would like to cover and the names of the peoplein your group (you will be using the same groups as for the course project).Wait for me to approve the tool and its assignment to your group.I'll do this via the same Piazza thread.See Week 9 for further instructionsand deadlines related to creating your and readingother groups' slide decks. Read Paper 1. Write the answers to the questions provided for Paper 1. Proofread your answers, check the spelling, iterate,produce the PDF and then check it. If required, fix the PDF.Due by Monday Nov. 9. 9:00 am. Submit the answers to the questions via Crowdmark.Read paper 2. Be prepared to discuss Papers 1 and 2 duringthe online class discussion. Week 9: Tuesday Nov. 17, 2020 (Topic: Performance Tools) NOTE THAT THERE ARE MULTIPLE DIFFERENT DUE DATES THIS WEEK.SEE THE CHECKLIST FOR DETAILS. As a group, create a slide deck describing the toolyou have chosen to present and upload them to the Piazzathread about the tools. As individuals, pick and read two slide decksand be prepared to discuss and ask questions about those tools in class.CHECKLIST (in order)As a group, find different sources of information about the tooland learn about the tool.As a group, try to use the tool to understand how you create a small simple example of how to use the tool.As a group, create a slide deck (presentation) about the tool.It should take someone about 10 to 15 minutes to read throughand understand the slide deck.Due by Sunday Nov. 15, 11:59 pm (just before midnight).One member of the group submit the slide deckby posting to the Piazza thread (the same one used above).The slide deck must be a powerpoint fileor a link to a Google slides presentation. If you use a link, after posting to Piazza,please have another group member check the link to bebe certainthat it can be access by anyone who hasthe link(without requiring a Google account or having to log in).Feel free to use speaker notes to help the reader,if you like.Due Tuesday Nov. 17, 9:30.As individuals, choose and read through the slide decks from at least TWO groups.Attend the online class discussion and be preparedto discuss the different tools, ask questions abouttools other groups describe and answer questionsabout the tool your group covered. Week 10: Tuesday Nov. 24, 2020 (Topic: Reproducibility in Machine Learning) Due Monday Nov. 23 at 9:00 amImproving Reproducibility in Machine Learning Research (A Report from the NeurIPS 2019 Reproducibility Program)(Answer the questions below about this week's paper)Joelle Pineau,Philippe Vincent-Lamarre,Koustuv Sinha,Vincent Lariviere,Alina Beygelzimer,Florence d'Alche-Buc,Emily Fox,Hugo LarochelleAnswer the following questions about this week's paper. In your own words briefly (in one paragraph) summarize the paper. What if anything new did you learn from this paper? Is there anything from this paper that you think you can applyto your course project? If so briefly describe what applies and whyit is relevant to your project. Provide any other points that you would like to make about the paper (optional).OPTIONAL reading, if you are interested in finding out more aboutthe "NIPS" experiments described on page 6 of this week's paper.This is the experiment conducted toexamine the consistency of their reviewing process.You can find a starting point for reading below.Many people have interpreted that experimentsas showing that getting your paper accepted to thatconference (and possibly others) was/is fairly random. The NIPS experimentEric Price, Dec 15, 2014, If you know of or find a better source (starting point) please let me know.CHECKLIST (in order)Read the paper.Create a PDF document that answers the questions for this paper.This document must be at most 1 page and follow the formattingguidelines used throughout the course.Proofread and edit your answers to the questions.Submit the answers to the questions via Crowdmark. Week 11: Tuesday Dec. 1, 2020 (Topic: Variability in Clouds)Due Monday Nov. 30 at 9:00 amIs Big Data Performance Reproducible in Modern Cloud Networks?(Write a Research Paper Summary)Alexandru Uta, Alexandru Custura, Dmitry Duplyakin, Ivo Jimenez,Jan Rellermeyer, Carlos Maltzahn, Robert Ricci, and Alexandru Iosup,NSDI, 2020.If you follow the link below you can also get access to the talk slidesand a video of the talk.Link to the NSDI page that contains talk slides and the video.OPTIONAL reading, if you are interested this is ourpaper suggesting that RMIT should be used for cloud experiments.Conducting Repeatable Experiments in Highly Variable Cloud Computing Environments,Ali Abedi and Tim Brecht,8th ACM/SPEC International Conference on Performance Engineering (ICPE), 2017. CHECKLIST (in order) If you use a Zoom app to attend class, please update it to thelatest release. If you use a browser, I think you should be fine. Read paper 1. Create a PDF version of your summary for paper 1 Proofread and edit your summary. Submit your summary via Crowdmark. If you like read paper 2. Friday Dec. 11, 2020 (Topic: Project Paper Submission) Friday Dec. 4 11:59 pm eastern (just before midnight): Abstract submission deadline Friday Dec. 11 11:59 pm eastern (just before midnight): Paper submission deadlineNote that you can continue to submit new versions of your paperand abstract right up to the deadlines. I suggest that you submit early andupdate your paper as you near the deadline.Late submissions will not be accepted!!! Even a few minutes late.This simulates a real conference submission experience.Submissions will be done using HotCRPUse the link provided in the CHECKLIST below for submissions.If you don't already have an account on you will first need to create and account and register your paperand all authors. If you already have an account, log in using yourexisting HotCRP account (a link is provided for this below).All members of the group should create an account nowas you will all eventually need one for reviewing.This will also allow any member of the group to submit andupdate the paper.Be sure to create your account early and to submit earlyto leave time to fix problems if you do not pass the automatedPDF checks, etc.CHECKLIST (in order) If you don't already have one create a hotcrp.com account. You can use this link. Submit the abstract for your paper. You can change this later butit must be relatively complete and accurate representation ofyour project. -cs854-perf20.hotcrp.com Note that this is required to allow me to start planningreviewer assignments well in advance of the paper deadline.Once you have submitted your abstract I believe that HotCRP willassign a number to your paper (e.g., 42).Paper submissions will be anonymous, instead where the authorlist would appear, add your paper number (e.g., Paper #42). Create a PDF of a draft of your paper that you wouldn't be embarassedto submit. Submit a draft of your paper using this link -cs854-perf20.hotcrp.com If there is a PDF checker on HotCRP for your submission,run it and fix any problems that might exist (e.g., typicalissues are fonts and/or margins that are too small, etc.)Continue working on your paper, improve the writing,proofread it and edit it. Submit the final version of your paper using the same link -cs854-perf20.hotcrp.com Wednesday Dec. 16, 2020 (Topic: Paper Reviews)Due Wednesday Dec. 16 11:59 pm eastern (just before midnight)Reviews will also be done via the conference web page(once the paper assignments have been madeyou will receive email explaining how to access the papersyou need to review).Reviews will be anonymous.Background reading (not required but might be helpful)Writing reviews for systems conferences,by Timothy Roscoe, 2007.CHECKLIST (in order) Log into hotcrp, read through and understand the review formyou will be completing for each paper anddownload the papers you have been assigned. -cs854-perf20.hotcrp.com Read your assigned papers.Complete the reviews for each paper.Check if your review is complete, sufficiently long,detailed and accurate (your reviews will be ratedby other reviewers).Submit your reviews. You will not be able to seeother the other reviews for the papers you havebeen assigned until you have completed your review. -cs854-perf20.hotcrp.com Friday Dec. 18, 2020 (Topic: Program Committee Meeting) Friday Dec. 18 at 9:00 am eastern (NOTE THE SLIGHTLY DIFFERENT TIME!) NOTE THIS IS AN EXTRA CLASS MEETING AND IT WOULD BE GREAT IF YOU CAN ATTTEND.Past students have said that this meeting was very beneficial.CHECKLIST (in order) Log into hotcrp and read the reviews for all papers you've been assigned. -cs854-perf20.hotcrp.com Note: you will not be able to view reviews for your paper submissionuntil after some time after the program committee meeting has ended.You will recieve an email notification of the decision for your paperalong with the reviews (or a link to the reviews). Important: use the HotCRP system to rateeach of the other reviews for each paper you have reviewed(i.e., rate other people's reviews). Conduct an on-line discussion with other reviewers of the same paper. Try to understand the different reviews and try to reach an agreementabout whether or not each paper should be accepted. Attend class where we will hold the conference program committee meeting.      HomeVideoLogisticsScheduleProjectsResourcesIntegrity      Last modified: Fri Dec 4 14:51:28 EST 2020 2b1af7f3a8