Exams suck

ivan

TL;DR: assessment methods should be adapted to students’ level

I just read an article about two girls who got perfect scores on the SAT test: 2400/2400. It got me thinking that the standard grades-based methods for measuring student’s progress are outdated. Grades function OK for the admissions process, but normal tests are nearly useless as feedback mechanism for students who fall “outside of the mean” in terms of ability:

  1. students who are very weak (30-60%) essentially get as feedback “You suck!”
  2. students who are very strong (85%+) get the constant feedback “You’re the best!”

In both of these cases the feedback will not inspire the student to study. In this post I’ll try to make the case that we need to build student metrics which are adapted to the student. I’ll also propose a metric that would incentivize students to learn. IMHO, learning — not ranking — should be the ultimate goal of the educational system so we better setup the incentives right.

Measuring students

I’m sure we can all agree that the purpose of quizzes and tests is to capture some “signal” about what students know and what they don’t know. Let us put aside for the moment the discussions about what you want to be measuring (skill? fact knowledge? integration tests? “which equation to use” skill?) and why you are measuring it (for self assessment, grades, rankings, firing of teachers). Let’s focus on the problem in an abstract sense in terms of information theory. Let’s assume that we want to measure something about the students — hell let’s go all-out-engineering on this and say the student is the signal.

The distribution of student abilities has a Gaussian shape. A Gaussian or “Bell” curve has most of its mass concentrated near the average value (~68% of the mass of a Gaussian is situated within one standard deviation (a.k.a. one sigma)  around the average value, called µ). However, the Gaussian distribution has infinite support. Even though unlikely, it’s still possible that a student comes around who is really good. Like, out-of-this-world good. Well no, actually they are in this world, but our measurement methods are inadequate. There is a cutoff at 2400.  Tina and Marie Vachovsky fall outside of the dynamic range of the SAT test.

Observations:

  • Information is being lost! A good testing system should adapt to the level of the testee so that it always reports useful information. The grade 2400 is good enough for the university admissions office to do the right thing with their applications, but other than that it is useless.
  • The only way  Tina and Marie could have useful feedback about their studies is if they are presented with challenging questions. With regular tests they just get “You are the best” every time, which is nearly useless feedback and only serves to feed the ego. As someone who used to get good grades (for some time), I can tell you that the first B I got was quite a hit. I had learned to depend on my “grades” for some part of my self esteem so suddenly “You’re no good at differential equations in 2D using sneaky tricks from complex analysis” turned into “You’re no good, generally.”
  • FACT: We need to throw out the notion of exams. Group assessments, in particular the summative kind, don’t make any sense what so ever. The teacher is forced to produce a custom exam adapted for the level of the students he is teaching, then students “write” the exam in order to get good grades. The grade will be average on average, the good students will get good grades and the “weak ones” will be singled out so that the teacher can start to worry about them.
  • The “weak ones” could be students who are slow learners, students who are missing some prerequisites, or students who are not interested in that subject right now. For them, this exam scenario is a nightmare. YOU ARE NO GOOD. YOU GOT A 40 OUT OF 100 ON THE EXAM. Perhaps the student didn’t know how to solve the quadratic equation in the second step of a seven part question, but the exam won’t care and give him a zero on that question: “You should have known that! You’re no good!”

All this got me thinking that grades should report your current learning effort (how many concepts did you learn this month) and not how far you are on the overall progress. Sure thee could be a “progress report” as well to show how much you’ve learned, but that shouldn’t be what matters. In my school system (Montreal), we used to get two grades. An achievement grade and an “Effort” grade. All I’m saying is that the achievement grade should not matter so much. Let’s reward kids for the “Effort” column regardless of their achievement scores.

Proposal

Assume we standardize a taxonomy of concepts, each concept being like a “stage” in a computer game. You can think of the planets in the khan-academy galaxy. I can’t find the link right now, but I know of a company that had a complete knowledge graph and always scheduled the quiz questions so that you would be practicing on topics which you didn’t know but you had all the prerequisites for. So assume we have this bunch of “stages” to clear and to clear a stage you have to pass a bunch of difficult exercises which require the use of that concept.

The student profile should show the grade as a triple (w, m, y), where w is the number of stages I cleared in the last seven days, m is how many stages I cleared in the last 30 days, and y is how many I cleared in the last year. This is analogous to how the UNIX command top reports CPU load averages.

In this new system, it wouldn’t matter how much you know so long as you are making good progress. For example, “Grade: (3,10,340)” means the student learned 3 new concepts last week, 10 this month, and 340 this year. If you were a 12 year old kid, wouldn’t you show off with this “Hey look I have (7,30,365) — I learned a new concept each day during the last year!” The best thing is that it works for adults too.

To summarize what I said in many words that could have been said in few, I think that we need to start thinking about new assessment methods adapted to the knowledge of the student. This way students will always be adequately challenged and  be in the moment, present, in the zone, on a roll, wired in, in the groove, on fire, in tune, centered, or singularly focused. Now that’s learning! Enough with this fear-based motivation to get good grades on exams. The focus on rankings is a vestige of the old become-a-good-robot-for-the-system-days of education. In the XXIst century, let’s focus on learning.

Starting a company in Quebec

ivan

Today I was looking into administrative matters for Minireference Co. and I want to collect some notes here which might be helpful to other entrepreneurs starting their ventures in Quebec.

The first thing to do is to register (en) with Revenu Quebec and obtain you “tax numbers”. Your tax numbers allow other businesses to track the taxes they pay to you. You can use this form to verify a QST number for any company. You can find more info about here and here.

Registering is important because all the GST and QST taxes you will pay as a company are refundable. The GST is refunded as input tax credits (ITCs), the QST as input tax refunds (ITRs).  Be sure to register BEFORE you incur any major expenses because to be entitled to ITCs or ITRs, you must have been a registrant during the reporting period in respect of which you paid the taxes. At the end of the year, you will be able to get back the tax paid on pretty much all your operating expenses and even home office expenses.

The second thing you have to do is collect GST and QST on all the sales you do.  In the case of Minireference Co., I am dealing in  books, more specifically printed books which are zero rated articles in Quebec so I only have to collect GST.

There is a nice guide to all of this, which I have not read yet but which I am going to print. The guide doesn’t include the changes  which take effect Jan 1st (new way to calculate QST:  now calculated at 9.975% on the selling price not including GST). You can get in touch with RQ by phone. The waiting time was very reasonable.

To be continued …

Network protocols discussion

ivan

There is an interesting discussion about network protocols going on at hacker news. In just a few posts some very knowledgeable people stepped in to explain what is going on. I saved the URL in the Links section of /miniref/comp/network_programming but it made me think about how much better informal explanation is to formal explanations.

Here you have hackers talking to other hackers. Whether it is a javascript-wielding young blood or an old dude who writes server-side stuff in C, all these people need to send data on the network and know of some some protocols for doing that: TCP:HTTP for web dev mostly while systems people probably think more in terms of (TCP:*).

Everyone jumps in to check what is going on on the HN discussion. And then suddenly learning happens. The discussion is a bit disorganized (shown as discussion tree) but let me give you the walk through.

First chetanahuja tells it like it is:

IP layer is for addressing nodes on the internet and routing packets to them in a stateless manner (more or less… I’m not counting routing table caches and such as “state”). TCP […] are built on top of the IP layer to provide reliable end-to-end data transfers in some sort of session based mechanism.

The key thing to know is that IP is a best-effort protocol. If you send an IP packet from computer A to computer B, the network will try to deliver it but there are not guarantees that it will succeed. No problem though, we can build a reliable protocol (the Transmission Control Protocol) on top of the unreliable one. This is why the Internet is usually referred to as TCP/IP, not just IP even thought IP stands for Internet Protocol. TCP/IP is the internet made reliable.

TCP is important because it allows for reliable communication: When A sends some data to B, B will reply with an ACKnowledge packet to tell A when he received the data. Reliability comes from the fact that A will retransmit all the packets for which no ACKs are received (the sender assumes these packets got lost). The other thing the TCP protocol gives you is the notion of a port — a multiplexing mechanism for running multiple networks services on the same machine. Port 80 is the HTTP port (web browsing). When you type in 11.22.33.44 into the browser, your browser will send a TCP packet to port 80 on 11.22.33.44:80, where the TCP port is separated by a colon. Another important port is port 25 (SMTP) which is used for email. When an email to user@11.22.33.44 is to be delivered, a connection will made to 11.22.33.44:25.

Sometimes you don’t want to have so much transmission control overhead. Imagine that you are sending some voice data so you want to send as many packets (maybe use forward error correcting codes) but you don’t want to bother with retransmission of lost packets. The voice data simply isn’t useful if it is not delivered on time. In such cases we would prefer a more basic protocol which doesn’t isolate us from the unreliability of IP.

This protocol is called UDP and it is really barebones. UDP is basically IP with some added port numbers (and error detection checksums). From now on we can’t simply talk about “port 80″ on host but we must say whether we mean port 80 in the tcp protocol (TCP:80) or in the udp protocol (UDP#80).
Speaking of UDP ports, let me tell you about a really important one. The (206.190.36.45, 98.138.253.109, 98.139.183.24).” My browser will connect to one of these IPs (over HTTP = TCP:80) chosen at random.

Another useful UDP service is DHCP (dynamic host configuration protocol). This is the magical process by which you are automatically assigned an IP address when you join a network. DHCP bootstraps the communication (you joined a new network, ok, but what is the network number for this network? who should you talk to? What IP address should you respond to?). Either you know this information (someone gave it to you on a piece of paper [sysadmin]) or you can make a DHCP request (which is UDP broadcast) and a DHCP server will respond to you and assign you an IP address, tell you what the network number is and tell you which router to talk to to go towards the Internet (the route).

Ok so everyone knows the basics now, but HN doesn’t just give you the basics — it gives you the advanced stuff too. What are the current problems with TCP/IP?

advm says:

Maybe TCP’s issues aren’t apparent when you’re using it to download page assets from AWS over your home Internet connection, but they become apparent when you’re doing large file transfers between systems whose bandwidth-delay products (BDPs) greatly exceed the upper limit of the TCP buffers on the end systems.
This may not be an issue for users of consumer grade Internet service, but it is an issue to organizations who have private, dedicated, high-bandwidth links and need to move a lot of data over large distances (equating to high latency) very quickly and often; CDNs, data centers, research institutions, or, I dunno, maybe someone like Google.
The BDP and the TCP send buffer size impose an upper limit on the window size for the connection. Ideally, in a file transfer scenario, the BDP and the socket’s send buffer size should be equal. If your send buffer size is lower than the BDP, you cannot ever transfer at a greater throughput than buffer_size / link_latency, and thus you cannot ever attain maximum bandwidth. I can explain in more detail why that’s true if you want, but otherwise here’s this: http://www.psc.edu/index.php/networking/641-tcp-tune
Unfortunately for end systems with a high BDP between them, most of the time the maximum send buffer size for a socket is capped by the system to something much lower than the BDP. This is a result of the socket implementation of these systems, not an inherent limitation of TCP.
An accepted user-level solution to this issue is to use multiple sockets in parallel, but that has its own issues, such as breaking fairness and not working well with the stream model. I can explain this more if you want, too, just let me know.

There are other problems with TCP, such as
slow start being, well, slow to converge on high-BDP networks,
bad performance in the face of random packet loss (e.g., TCP over Wi-Fi),
congestion control algorithms being too conservative (IMO, not everyone needs to agree on the same congestion control protocol for it to work well, it just needs to converge to network conditions faster, better differentiate types of loss, and yield to fairness more quickly),
TCP features such as selective ACKs not being widely used,
default TCP socket settings sucking and requiring a lot of tuning to get right,
crap with NAT that can’t be circumvented at the user level (UDP-based stream protocols can do rendezvous connections to get around NAT), and more.

People write whole papers on all these things. Problem is most of the public research exists as a shitty academic papers you wouldn’t probably bother reading anyway, and most of the people actually studying this stuff in-depth and coming up with solutions are private researchers and engineers working for companies like Google.

The last paragraph is a good reason why there should be a “No BS guide to computer systems”. I bet I can show services at all layers of the OSI stack and really go into details. The sockets are pretty cool.

pjscott describes the crypto stack in just one sentence:

Their crypto stuff looks pretty reasonable. Key exchange uses ECDH with either the P-256 or curve25519 polynomials. Once the session key is established, it’s encrypted with AES-128 and authenticated with either GCM or HMAC-SHA256. None of this is implemented yet, but it’s at least cause for hope.

latitude also gives other considerations about doing crypto over the Internet.

I’ll tell you a dirty little secret of the protocol design.
Say, you want to design a protocol with reliable delivery and/or loss detection. You will then have ACKs, send window and retransmissions. Guess what? If you don’t follow windowing semantics of TCP, then one of two things will happen on saturated links – either TCP will end up with all the bandwidth or you will.
So – surprise! – you have no choice but to design a TCP clone.

That said, there is a fundamental problem with TCP, when it’s used for carrying secure connections. Since TCP acts as a pure transport protocol, it has no per-packet authentication and so any connection can be trivially DoS’d with a single fake FIN or RST packet. There are ways to solve this, e.g. by reversing the TCP and security layers and running TCP over ESP-over-UDP or TLS-over-UDP (OpenVPN protocol). This requires either writing a user-space TCP library or doing some nasty tunneling at the kernel level, but even as cumbersome as this is, it’s still not a reason enough to re-invent the wheel. Also, if you want compression, it’s readily available in TLS or as a part of IPsec stack (IPcomp). If you want FEC, same thing – just add a custom transform to TLS and let the client and server negotiate if to use it or not.

I mean, every network programmer invents a protocol or two in his lifetime. It’s like a right of passage and it’s really not a big deal.

I learned stuff today. I hope you guys learned something too.

Techzing interview

ivan

Earlier this year I launched my book on hacker news which resonated very positively with the hacker crowd.  This HN exposure landed me an  interview on the TechZing podcast  to discussed my textbook project. Even though it was an hour and a half long interview, there were a some things that we didn’t get to discuss. I want to take the moment now to write down my observations about the textbook business and the educational market.

This blog post is organized with the best stuff at the top so feel free to trail off at any point.

Insights

The most important things I’ve learned about the textbook business:

  1. Writing is tough, but writing down lecture notes after a lecture is easy.
  2. Teaching students is gold. By interacting with your students 1-on-1 you get feedback on your explanations.
    If you are lucky you will get a “Sorry, I didn’t get that”, which allows you to iterate.
  3. People still appreciate the printed book. Some people are willing to pay good money for a PDF.

Opportunities

Print-on-demand and eBook technology allow for everyone to publish and sell books. This is a revolution on a Gutenberg scale. One of the forefathers of the Internet/WWW, when asked about the motivation behind his inventions said he did it “so people will be able to earn a living from the fruits of their intellectual labour.”  We have now finally reached this moment where this idea is practical.  Could books be the missing monetization strategy for the Internet?

What have been traditionally two markets—the general audience and the educational market—are now becoming a single market of people who want to learn. Lord knows there are things to learn out there so there is an opportunity for knowledge products for people who want to learn. The key monetization routes will be through selling organized knowledge as textbooks, ebooks, or apps.

I used the term revolution above and I stand by this choice of wording because this is what we call it when a value chain collapses from six-plus levels to three levels. The value chain in the “book business” previously looked like this:

author
  __editor
       __copy-editor
            __typesetter
                 __printer
                     __distributor[1..]
                           __book store 
                                __client

With print-on-demand the new book business will look like this:

 
author -- printer -- shipping -- client
^^^^^^                           ^^^^^^

Let us call this “author centered” publishing. From now on, authors can expect to get up to 50% of the profits instead of 10% (which could be as low as 5% of the list price). Good times for authors. Incentive-giving-to-move-to-a-new-publisher times.

Even amazon looks like a dinosaur in this context:

author
  __editor
       __copy-editor
            __typesetter
                 __printer
                      __amazon
                          __shipping
                               __client

Why do you need the warehouse to store all the books? Why not ship from the printer?

There is one element in the traditional publishing value chain that we must keep. Copy editing is actually very important because you really want someone to go through
your writing and fix mistakes in it. You can use your target audience (crowdsource copy-editing), but nothing beats professional services.

OK, so you want to see the future of publishing? Here it is:

 author -- (1) pub.srvc. -- (2) printer  -- shipping -- client
                         _ (3) booksite -- client

 

The opportunities are (1) for small publishing houses (copy editor + creative person for covers + latex guy) to really come-in and take over the entire market within a couple of years. You could also have larger publishers who focus on marketing the book to certain audiences etc.

Opportunity (2) is for new print-on-demand shoppes to come up (compete with lulu.com and lightning source). These giants have as their main advantage the established processes they have in place, but how difficult would it be to build an “Espresso Book Machine”-like system based on a quality BW laser printer (think buying toner in gallon tubes at costco 😉 and some automation. The competitive advantage of a small print shop would be that they offer pick up service (0\$ shipping). Currently lulu charges you 6\$ for shipping to Canada (9\$ for 2 books, 12\$ for three books, …, 3+3n.) Shipping within the states is \$5 which is better, but still not free. In particular for printing small books (100-200pp) it would not make sense to order from lulu. They would charge you 5\$ for the printing and another 6\$ for the shipping. Your cost 11\$. If you go to a local print shoppe, they will charge you 7\$ for printing. Same product, half price.

The third opportunity is for high-level editorial services (think curation of content) which would collect book recommendations and let authors and readers interact. Ideally there should be independent “book blogs” for discovery of new content — not marketplaces. Something must be done about the current appstore monopoly. Every app you develop relying on Apple for your distribution is feeding the monster at 30%. Every web app you develop based on Google or FB apis could stop working tomorrow if the API is retired. Go get hosting somewhere and build your own website. Don’t depend on anyone. Okay sorry I got a little off the topic of textbooks. Let’s get back on topic.

I was telling you guys about the book and stuff from the interview. One thing which we talked a lot about was the hacker news launch.

The HN launch

I told Jason how surprized I was when I got 30 000 visitors in one day and how I didn’t get up from my chair for one day. There were roughly 7000 people who clicked on one of the modals. Of these 300 people bought the book in print. By the evening of Jan 1st and into Jan 2nd there were also 100 PDFs purchased from gumroad.

 

I still working out the numbers (conversion rates) and I don’t want to get too hyped up about them (ok ok, 7k –> 300 = 4.3%) because the HN audience is really VERY sympathetic to the product. I am not sure if everyone else on the internet will like it as much. 

(SIDENOTE: I am finding it hard to get the analytics I want for the book page. GA reports analytic en masse so I cannot see what individual visitors did when they came to  the site. I have basic questions I need answers for and it seems like the current state of analytics is very unimpressive (relative to my expectations). Here is what I would like to know:

  1. Which modal my visitors looked at before deciding to continue onto lulu.com/shop or gumroad?
  2. Which of the 800 people who clicked through to lulu.com/shop are the 300 that actually ended up buying the book?
  3. Which sections did they read (scroll to and stay for 4secs+)?
  4. What “path” did each visitor follow through the modals? (subquestion: did anyone see the apg-get install mechanics? did anyone see the integral calculus modal? )

Are there solutions for these? I think the only way I can have end-to-end information is if I run the whole show. If I want to have information about converstions I must build my own shopping cart. Wait, we are on the Internet — I can just submit a feature request to lulu.com support and gumroad support. I am working on the full writeup of the launch experience here which will have more graphs and numbers. (/SIDENOTE)

I got a lot of feedback from the discussion on hacker news. People really like the idea. The tech crowd of Hacker News is precisely the kind of crowd is interested in learning about advanced math and physics. Many programmers learn the about calculus in mechanics at University but never actually understood these subjects. This is way when the no bullshit guide to mass in physics the really wanted and the 29 dollars price range is definitely not an obstacle for them. Several people also asked for a PG 13 version cleaned up with out of cities in the references to park and alcohol. This is definitely something I will look into it because no told jokes need to be about these subjects. We can stick to the political stuff and the joke about the investment banker being dropped off a building.

What is the goal of the book?

The goal of the book and more generally of Minireference Co. is to teach. Teach students how to get rid of the exam stress when they’re doing their studies. If you know the material really well, then there is nothing tricky that the teacher
can do on the final. Understanding trumps memorization any day of the week. A secondary goal is to teach math to adults, grown ups, so they can let go of their math complexes. There is no reason why a forty year old person should avoid conversations about math and feel uncomfortable when their teenage daughter or son asks them about the solutions to a quadratic equation.

The third goal is to prevent the next generation of analytic reminded youth from going into the defence, pharmaceutical and finance sectors, which I consider to be detrimental to society. I grew up listening to Rage Against the Machine and I feel it is my duty to continue their work in educating the next generations about the system. By situating analytical knowledge in the context of the current world geopolitical situation, it is my hope that the next generation of Einsteins, Gates, Pages, and Zuckerbergs will make informed and moral choices. With knowledge comes responsibility, and I don’t want my students to think about the numbers without understanding what the numbers represent in the real world.

Textbook market

There are a couple of intrenched companies in the publishing world (the big five). Mainstream publishers in the educational market produce textbook that are so expensive, that we can talk about a textbook racket. The readers, subject to their teachers authority, are forced to buy specific textbooks, often at an exorbitant prices > \$100. Mainstream textbooks are also too long and full of fluff like full-page photos designed to pad the pages and impress the student with the “high endness” of the 1000-page publication.  Mainstream textbooks are the kind of product which is the signed by committee. They’re thick and boring.

On the other hand there are several positive things about textbooks.  Irrespective of the widened usage of electronic formats, the “book format” remains the primary medium of intellectual discourse, of which textbooks are a subset. Textbooks are old technology, but good technology. Textbook, as a mean for acquiring knowledge, are better than most educational resources produced for the web.  And it’s not just eBooks, print is here to stay because students don’t like the idea of ebooks replacing textbooks.  Having a PDF to go along with your printed textbook is definitely a feature, but not as a replacement.

 

Business model

The business model for Minireference Publishing Co. is quite simple: we sell math and science textbooks and PDFs. The specifics of the book “container” are not important. What is important and of value is that we offer an “information distillation” service: complicated science subjects are presented and explained in a concise coherent narrative, including all prerequisites. Instead of reading 100 wikipedia pages to learn about calculus in a month, students can read one chapter in the No BS guide and pick up the same material in a week. 

Backstory

During the interview, I had a chance to give the full story about the genesis of the book. At 7min40sec in the interview, I say how I started from a collection of notes on advanced physics subjects and that at some point decided to make those notes into a book. Jason replies to this jokingly “Wow that is a big jump!” but I totally missed his joke and just kept on blabbing.

Pivot 1: TOO ADVANCED. There are not that many physicists. We need to go for something more mainstream. New product will be a mini-reference book of formulas for all of science.

Pivot 2: FORMULAS ARE NOT ENOUGH to learn. Let’s have the formulas, but add enough context and explanations to explain where the formulas come from and how they are used.

Once you have the idea… It took two years and 200 commits. It wasn’t high intensity work: I just wrote down lecture notes and my favourite explanations after teaching. During the summer of 2012, I worked intensely to tie together and organize all the material into a coherent story with a beginning (solving equations), a middle (use equations to predict the motion of objects in physics), and an end (learn where the equations of physics arise from calculus).

Product

What is special about this book is the deed forms contains a complete dependency graph of topics. Each subject is explain along with all the prerequisite material. 

Another thing special about the book is its conversational tone. The narration in the book switches from serious to joke mode and back to serious again, and is intended to keep the reader engaged.  Everyone needs a little brake after learning pages and pages of formulas… 

Technology used

During the interview, I didn’t get a chance to discuss the technology stack I used to generate the book. The book started as a bunch of text file in dokuwiki. I then used the dokutexit plugin to export the book as LaTeX. 

Another important tool for the production of the book has been to use the text-to-speech tool in Mac OS X for proofreading. It allowed me to catch lots of mistakes and quickly. 

I use lulu.com for print-on-demand and gumroad.com for the PDF distribution.

Future

Some future directions for the development of the book are:

  • Finish the linear algebra textbook
  • Write Tome II on electricity and magnetism and vectors calculus
  • Future plans: Write a book about probability and stats 
  • Future plans: Make a No BS guide to Python and JavaScript

Speaking of JavaScript I’m currently exploring the using khan-exercises framework so I could offer practice problems on the site.

The main challenges we face right now is marketing the book to a wide audience.

UPDATE: Since the publication of this post, the No Bullshit guide to math and physics has been improved and revised several times. Sales going Okay. Need more word of mouth…