Inherit a mess, build a miracle
class Miracle extends Mess { override def screwUp = winBig }
Not exhaustive, just list a few.
I happened to come across this recently published paper with the titled Face classification using electronic synapses. Since I did similar research at school I spent some time reading it. As I scrolled down the web page, there was a section called “Peer Review File” which greatly aroused my interest. I clicked linked and quickly found this Review #3 (shown below), though lengthy, it is very read-worthy. It reminds me of those perplexing days struggling with a research topic that was born with defects.
To give you a little bit background in case you are not familiar with machine learning based neural networks – you can deem them (detailed definition can be found here) as a black box which you can feed it human images and it will tell you who it is. Of course, in order to give neural networks such capabilities, you must “train” it first, which is traditionally done digitally on a CPU based computer. In recent years a new type of analog device called “memristor” emerged as a potentially “promising” component in many applications. Loosely speaking, a memristor is a variable resistor that can be modulated by external electric stimuli. Subsequently, some researchers claim that they might find spots in machine applications where low power feature is desired. Many works tackle the charming features of memristors by showing a trending machine learning based example, but most of them are so vulnerable in turns of the following points
If these comments appear in your paper review, it’s very difficult to respond, because they shake the very foundation your research is built upon. It’s not like there are side experiments you have missed or too broad claims that you need to narrow down. It embodies a sad truth – not all research topics are created equal. This paper is lucky to be accepted for publication finally, but I can feel the pain the authors have felt during the revision process.
Appendix – Review #3 to the above-mentioned paper.
Reviewer #3 (Remarks to the Author):
This manuscript describes an experimental implementation of the on-chip training of a very small, single-layer perceptron neural network on a toy-size problem. The in-situ training is described, generalization and noisy inference is tested, and network accuracy is shown. The authors do share a large amount of data illustrating the high variability of their devices, which is good. Authors also focus overly much on the power advantages of such an online array, which of course are large — yet do not provide useful information to other experts in the field, vis-a-vis: If we or they built a large array of these devices and used it to train a multi-layer neural network on a non-toy problem, would there be any chance of it actually working (e.g., converging to acceptably-interesting accuracies within any reasonable time)? I recommend the authors go back and attempt to answer the question above using simulation of neural networks as informed by their extensive data on device-to-device variability and conductance response and provide the answer in their revised paper. Also, since I have very good reason to believe the answer to my question above is “No, it won’t work with these devices,” then authors will furthermore need to describe for the reader how much these devices need to be improved (in order to make non-toy DNN work), whether this is feasible, and how this could potentially be done. I recommend reject and mandatory revision before resubmission. Specific comments:
1) The authors include the usual hand-waving rhetoric about the glories of neuromorphic computing, but of course low energy computation is completely useless if you are not obtaining a result of relevance. In this context, future neuromorphic systems built around such devices MUST be able to train the exact same size (or bigger) networks as are currently trained by GPUs, MUST obtain similar classification accuracies by the end of training, and most likely MUST provide a speedup advantage IN ADDITION TO lower power. Yes, there is a niche market for solutions that train DNN at lower power to sufficient accuracy but more slowly than GPUs, but it is a fairly small niche. There is NO niche for solutions that fail to achieve similar classification accuracies on non-toy-size DNN. There is zero mention or acknowledgement of this reality in the current intro to this paper, and this must be fixed.
2) Authors claim “this is the first experimental demonstration of such a significant cognitive task.” I get tired of papers that are explicitly built so that one can claim they are “the first,” leaving all the actual hard work of taking something from first demonstration to usefulness to someone else. In this case, I cannot agree that the network shown here can be considered “significant.” This network and dataset is far too small to allow any predictive (or even indicative) power for larger non-toy networks. One can say that MNIST itself is also far smaller in size and easier in difficulty than CIFAR or ImageNet, but if a memristor-based solution were able to deliver identical-accuracy-performance on MNIST, I would have to claim that that accomplishment would be a major step towards the demonstration of NVM-based neuromorphic systems. The present network is far far smaller and much easier — to the extent that success on this toy problem is indicative of nothing.
3) There is a major bait-and-switch between the aspect ratio of the experimental array (described as having 128 rows and 8 columns) and the logical array needed for the network of interest (described as having 3 rows and 320 columns). Thus statements like “A fully parallel read operation” is clearly a bald-faced lie, since there is absolutely no way the authors can experimentally be inputting all 320 columns at the same time given the dimensions of the physical array. Statements like this will need to be fixed to reflect the actual realities of the experiment as it was actually performed.
4) Ways in which this experiment is a toy experiment – It is not that the images are too small – 20×20 is not far off in size from MNIST. The problem is that there are so few classes, which appear to be cherry-picked from the dataset in order to maximize the distance between the classes and make the problem easier. – Batch-based processing (e.g., going through the whole dataset and THEN updating all the weights) has long been abandoned by the AI community. IF you wanted to do this for a large dataset, you would take a large hit in convergence rate, plus you must describe how this data will be stored locally (or include the power for aggregating this data offline in your power and latency assessment). – The convergence of this learning appears to be so rapid that the authors are able to almost completely avoid having to use any SET pulses during the experiment. This will most definitely not be the case for any non-toy experiment. (Authors should show the histogram of desired conductance changes during training, so readers will know what the balance between requests-for-lower-conductances and requests-for-higher-conductances). – Because the dataset is toy-size, authors probably terminated training too early, thus losing the opportunity for further improvements in generalization performance or classification of noisy inputs. – The hundreds of write-read-verify cycles required for these devices are not going to scale well for a non-toy problem, without some trick which allows you to do this extremely infrequently and/or efficiently
5) Other comments: – If authors are going to focus on power to this extent, they must include the power required to implement their tanh() function, to compute the delta values after each example, and to aggregate and store the weight updates during their batch-based programming.
6) The RRAM devices do seem to show a nice gentle behavior. This reviewer is impressed at the slick usage of a logarithmic horizontal axis to make the conductance response “look” linear, when in actuality it is quite far from linear. That said, the supplementary data in S3 S4 and S5 (showing just how bad the variability between these devices really is) is greatly appreciated by the reviewer. After seeing this data, I was amazed that even this toy experiment worked at all, until I realized that the toy database only required RESET pulses, and that in the verify version of your experiment, you were willing to wait as long as it took, even if that meant many hundreds of programming pulses. One comment on Figure 3B,C — it is disconcerting that the vertical axes of B and C are not the same scale. You should either start your RESET characteristic at the same peak value that the SET characteristic terminates at, OR at least make the two plots have the same vertical scale so that the reader can immediately observe that the SET and RESET characteristics are NOT completing each other. Assuming that there were even any SET pulses needed during the “training” of this toy problem, it would be good to show the “SET” versions of Figures 3F,G.
7) The results with classification of noisy training data and with the test data (non-training data) are appreciated, and help improve the quality of the paper. Two points here:
– since the 9000 noisy images are noise added to the training images, they should NOT be referred to EVER as “test patterns” (see line 269). It is VERY important that it be 100% clear this is noise on already-seen training images, NOT on never-before-seen test data.
– to differentiate the statistically relevant percentage numbers in Figure 5b (out of 9000 noisy images) from the generalization “percentages” in Figure 5c, please label the green bars as generalization accuracy and include the exact fraction of correct generalized images (e.g., 22/24, etc.) as well as the percentage (to make it clear these “percentages” are accumulated over a very small number of instances).
Authors take great pains to laud the great power of experimental results, and similarly spend significant effort disparaging simulation-based efforts (including Ref 36 and the relevant portions of Ref 19, which also included a MUCH larger “first ever” experiment than the present manuscript). However, this reviewer would claim that simulations — especially ones that are done well enough that they can project accurately for the field whether the devices at hand will succeed or will fail at tasks of actual interest — are much more useful for the field than experimental demonstrations at toy scale without such simulations. In most fields, toy experiments have little forward predictive power for tasks of actual interest — for neural networks, they have zero forward predictive power.
That said, it would not take much for the authors of this manuscript to take their extensive experimental data and to project through simulation whether these devices as they stand (nonlinear conductance response AND massive device-to-device variability) would suffice for even easy non-toy problems like MNIST. Such simulations would also determine whether the nonlinear conductance response itself were invalidating — e.g., if every single device in a large array functioned exactly like the device described by Figure 3B,C with ZERO variability, even in that ideal case would one have a shot at training MNIST successfully? This is the most important and relevant question, which remains unanswered (or even acknowledged) by the present manuscript. This is why the manuscript should not be accepted as it currently stands. Without this kind of information, the reviewer cannot agree that “these experimental results pave the way” towards the desired energy-efficient neuromorphic systems. Instead, this manuscript is merely yet another “first to show yet more useless results” paper.
One of the work items that really plagued me was a chargeback service. The story is – our team needs to maintain the record of how many queries are serviced by us for each of our customers, so by the end of each month, the data can be utilized to generate imaginary bills (not real bills that incur money, but only for the purpose of tracking how resources within the company are used).
The inner workings of the chargeback service are simple: we setup a daily cron job that after each day ends, it issues some queries to the monitoring service asking about the relevant query usage in the previous day, and gets back a large piece of text as the response. Then the response is parsed and goes through stages of grouping, sorting and the final results are written to databases owned by the billing team – that’s the whole workflow. It sounds so straightforward that it’s almost like a hit-button-and-forget thing.
But after about half a year without major problems, this chargeback service started to fall apart. The billing team started to ping me about the bad data, like 3 or 4 times in a week. Most of the causes were flaky monitoring service that laid downstream to our service – since a time point I had no clue of, the monitoring service quite often did not return meaningful data to the cron job’s request, so the job thought there was nothing to do and did not write anything to billing team’s database. Re-running the chargeback job manually thus became my daily routine. Although it took only a few minutes, it was so repetitive and frustrating.
I was sure no human could bear with it, so I tried to fix. The first thing that came to my mind was scheduling the cron job to run at a different time of day, which was built on the assumption that the monitoring service was under high workload during some interval of the day so it rejected queries. So I moved the cron job from 1:00 AM UTC to 4:00 AM UTC. It seemed to mitigate the problem a little bit but did not cure it. Later on, I learned that monitoring service could be flaky at any time of the day.
Eventually, I realized that there must be some mechanism for the cron job to understand if the monitoring service was returning a meaningful response or not. Interesting enough, when the monitoring service was behaving badly, it did not throw exceptions or send warning signals, instead, it merely returned a text response that almost contained no data. Making use of this characteristic, I modified the cron job to check if the response contains data fields we are interested in, and if not, the job sleeps for 1 hour and retry.
This fix seemed to be effective, until one day I got paged again by the billing team again. I checked the data and found that though there was something there, it was only half of the normal daily volume. It turned out that one of the symptoms of the monitoring service being flaky was it only returned a partial dataset, which means my cron job not only needs to distinguish between “there is data” and “there is no data”, it also needs to judge if the data looks good. Fortunately, in our company’s scenario there is one viable way to achieve this – pull the previous day’s data and compare with today’s, if the discrepancy is too large (say, 30% increase or decrease), then see it as a failure and wait for 1 hour and retry.
I made the fix and deployed the code. It ran quite well ever since. Now it’s been 3 months and I have not been paged again once because of it. Sometimes a simple solution can be very effective as long as it can address your scenario well.