CS262B Advanced Topics in Computer Systems
Spring 2009

David E. Culler


A new OS architecture for heterogeneous multicore systems analysis

Provide a short summary of the paper:
This paper advocates for a redesign of operating systems to reflect the
distributed nature of the underlying hardware. Future generations of
hardware will offer larger numbers of cores that are also heterogeneous. At
the same time, the interconnect between these cores will be subject to many
of the same problems currently experienced in networks. Furthermore, the
larger number of cores will make cache coherency much more complicated and
make it more difficult to provide the shared-memory abstraction. So, the
authors propose that new operating systems be composed of multiple kernels,
one for each core, and that those kernels communicate strictly through
well-defined message-passing interfaces. Then, operating systems will look
very similar to distributed systems and will be able to leverage all of the
advances in that field.
What is the strength of this paper:
The authors of this paper provide an actual operating system implementation
that they benchmark on a variety of machines to substantiate their claims.
As such, they are able to demonstrate very good scaling characteristics.

What is the weakness of this paper:

The authors are unable to ensure that the performance gains witnessed as
compared to Linux and Windows are due specifically to the multi-kernel model
advocated for in this paper. It is possible that some of these differences
are due to implementation artifacts or other factors.
Your qualifications to review this paper:
I know the material, but am not an expert.


Writing quality:

Good.
Relevance to SOSP:
People would read it before conference and attend session.
Experimental Methodology:
Good.
Novelty of Paper:
This is a new contribution to an established area.
Overall paper merit:
Strong accept.
Additional comments:
Would it have been possible to show the incremental improvements of each
aspect advocated for in the paper? For example, could blocking and
non-blocking operations be compared for a workload in AnonOS? Similarly,
could shared memory and message-passing costs be compared within AnonOS?
This would ensure that all of the gains witnessed are due specifically to
the functions advocated for in this paper.



Attribute

Value
   
Provide a short summary of the paper  The paper describes multikernal, a new os design, that uses message passing for communication
What is the strength of the paper? (1-3 sentences) They have an actual implementation of an OS that is very different than conventional OSs in an area that is interesting to many people.  They have a decent evaulation.
What is the weakness of the paper? (1-3 sentences) I didn't like the graphs earlier in the paper with no explanation of where they came from.  I also didn't find the paper to really draw me in.  There's a lot of details without strong motivation.
Your qualifications to review this paper
  1. I know the material, but am not an expert
Writing Quality
  1. Average
Relevance to SOSP?
  1. People would read it before conference and attend session
Experimental Methodology Good
Novelty of paper This is a new contribution to an established area/This is very novel
Overall paper merit
   
   
   
   
  Strong accept - This is of interest to Sensys, and a novel or new contribution to existing area with good/average methodology, or an incremental contribution paper that has excellent methodology.
   
   
Provide additional detailed comments to the author The paper is in a good area with good motivation for a new os but the writing could be a little better.  There's also a jump from the high level to the details and it would be nice to see more of the OS design.
Additional comments to PC
(not seen by author)


 


Provide a short summary of the paper The authors present a new OS structure (multikernel) that treats a multicore machine as a network of independent cores, and makes no assumptions of underlying shared resources. By borrowing techniques from the distributed systems world, the multikernel model focuses explicitly on throughput for individual cores and opens avenues for in situ optimizations that would not have been possible in a conventional operating system. A basic implementation of this model shows the potential for improvement and outperforms conventional operating systems in a number of ways.

What is the strength of the paper? (1-3 sentences) This is an extremely novel and interesting idea. The design and implementation are very well-reasoned and convincing. The evaluation appears to be grounded in real-world workloads and evidence.

What is the weakness of the paper? (1-3 sentences) The evaluation section leaves something to be desired, with incomplete looks at a number of different “case studies” (a term that doesn't particularly fit the depth of data presented). It is somewhat unclear how many of the optimizations presented in this paper could also be achieved in a conventional OS and how difficult that process would be.

Your qualifications to review this paper 3: I know the material, but am not an expert

Writing Quality 4:

Good Relevance to SOSP? 5: People would read it before conference and attend session

Experimental Methodology 4: Good

Novelty of paper 5: This is very novel.

Overall paper merit Strong accept - This is of interest to SOSP, and a novel or new contribution to existing area with good/average methodology, or an incremental contribution paper that has excellent methodology.

Provide a short summary of the paper This paper advocates a distributed, shared-nothing approach to operating systems design termed multikernel. Every core runs an independent kernel node with its own copy of replicated OS state, and these nodes communicate via explicit, asynchronous messages. The authors argue that this approach is a more appropriate match to ongoing hardware design trends than approaches based on multhreading on top of implicit shared memory. The quantitative analysis is based on microbenchmark case studies and other benchmarks on a specific implementation of a multikernel designed to run on commodity shared memory hardware. The authors make the argument that the mutlikernel OS they implemented in top of a shared memory system performs comparably to a heavily optimized macrokernel OS, and is extremely scalable going forward.

What is the strength of the paper? (1-3 sentences) Compelling qualitative argument for distributed systems approach, within many examples of proven techniques from a variety of subfields of computer science which could be leveraged given the adoption of such an approach. Does a good job of delineating the differences between the multikernel approach and the specific design choices made in AnonOS, and providing a discussion of alternatives that were also evaluated. The microbenchmark analysis demonstrates the authors' understanding of the many complexities inherent to building hardware-specific CPU drivers and communication frameworks for OS tasks.

What is the weakness of the paper? (1-3 sentences) Little actual discussion of heterogeneity - while multikernels have some clear scalability and partitioning benefits, more detail about how multikernel implementations will deal with, take advantage of, and abstract or expose hardware heterogeneity to applications would have been appreciated. More detail or specific examples of allowing applications to use underlying shared memory hardware capabilities in future systems would have been interesting. Your qualifications to review this paper 4. I know a lot about this area

Writing Quality 4. Good Relevance to SOSP? 5. People would read it before conference and attend session

Experimental Methodology 4. Good Novelty of paper 4. This is a new contribution to an established area

Overall paper merit 5. Strong accept - This is of interest to Sensys, and a novel or new contribution to existing area with good/average methodology, or an incremental contribution paper that has excellent methodology. Provide additional detailed comments to the author Since the relative performance on the shared memory machine was not great, I wonder if a more compelling approach would have been to provide an implementation on top of a distributed SMP, and then argue that the approach would make even more sense as latencies shrunk. This argument was mentioned but not evaluated, whereas the evaluation left doubt as to the point at which the added shared memory overheads would actually cause the performance on real benchmarks to cross over in AnonOS's favor.


Short summary: The authors argue that an operating system should be treated like a distributed message passing system where shared state is made explict rather then implicit.  They contend that this would allow a machine to consist of very hetrogeneous cores running different systems software and yet still communicating, and scaling to very large numbers of these cores.  The design has a preliminary implementation which reveals no significant performance cost on current hardware for this approach.

Paper strengths: the design is of current interest, and using current shared-memory machines to implement a message passing system is easy and convincing because the architectures are fundamentally message based, under the hood.  There are several strong arguments made in the paper for this architecture such as dealing with hetrogeneity, especially citing network and graphics cards as "too smart" to be accesed as dumb peripherals like current techniques require as well as adding the ability to batch updates for more efficient bus utilization-- the typical message system arguments. 

Paper weaknesses: The exact design of their operating system seems strangely vague to me considering they have actually implemented it.  Processes are run using something like scheduler activations, but how memory sharing is coordinated between these processes (they say it's possible) doesn't seem clear; they really have two designs going on, one which would be on an ideal architecture which is natively message based, and one which is actually implemented on shared hardware.  How exactly do mechanism like vm work in the implementation, and does this differ from how it would on the shared architecture?  Where does the networking stack go?  Some of this is burried in there, but could be much better organized.  Furthermore, what are the implications of the split phase nature?  I think a few examples of how you would implement kernel subsystems would really help.  In general, what hardware architecture are they proposing?  It's worth noting that message systems (microkernels) did not originally take off more or less because the constant was too big, and this appears in their charts, for a single core.  They're arguing that this has changed now that we have lots of cores and what we care about is further out on that graph.  The fundamental arguement of this paper is that systems should be based on message passing.  Good arguement.  However, by introducing an actual system, they add a huge number of questions which can't really be answered in the same paper as a position paper arguing for messages.  I'm not sure they quite manage to finesse these two tasks.

Qualifications: 3?

Writing quality: excellent.

Relevence to SOSP: 4 or 5?

Experimental methodology: the preliminary implementation is an important step towards evaluating the proposed architectual change, and seems sufficient for a first step.  This feels like a very reasonable thing to do and furthermore is well motivated, so the burden of proof may be somewhat lower.  The mix of macro- and micro-benchmarks serve to illuminate the strengths while arguing that the architecture has no inherrent inefficiencies.  It would have been nice to see a carefully engineered implementation of one of the numerical tasks achieve an optimial speedup.  The loopback tests and 9(c) especially leave me somewhat curious since it's not clear that, in the case of 9(c) the performance of the systems "do not differ significantly."

Novelty: they cite a lot of work, although interestingly I wish there was actually more discussion about the difference between this system and microkernels; they do mention exokernels and two true microkernel references, but I would like some more discussion about this.  [I think] the most novel element is advocating exposing message passing among hetrogeneous hardware, although there certainly are things like this in the supercomputer world-- maybe this is an indication that this is a good idea, since those systems are definitly scalable.

Overall Paper merit: Accept

I agree with many of their arguements, and think that the implementation does a reasonable job of making the point that there is not inherent bottleneck.  However, the real coup would be demonstrating a system where some significant benefit is attained or else integrates hetrogeneity much better then is possible today.  Their mention of an ARM port makes me think they agree with this point.  There are several very good papers in all this work, and the question is how to divide it.  I'm not sure this piece exactly hits the nail on the head but it's a nice piece of work, with a lot of nice engineering techniques.


Summary: This paper focuses on the problem that hardware is changing and diversifying faster than operating system software. The internal organization of a general-purpose computer increasingly resembles a network rather than a single system; therefore it is no longer useful to tune a general purpose OS design for a particular hardware model. The approach that this paper takes is to model a new OS structure, the multikernel, that treats the machine as a network of independent cores, assumes no inter-core sharing at the lowest level, and moves traditional OS functionality to a distributed system of processes that communicate via message-passing. The authors provide the main design principles of replication, message passing and split-phase invocation and implement the multikernel model (AnonOS). They show that AnonOS is scalable and adapts to a diverse set of hardware characteristics, while providing competitive performance.

What is the strength of the paper? (1-3 sentences) The main contribution of this paper is to treat the machine as a network of independent cores and then to make use of previous ideas in distributed systems. I believe that this core idea has a lot of potential for expansion. Furthermore, this paper is very well structured with a clear problem statement, the identification of the design principles for their model and a detailed justification for these choices, and finally an implementation and evaluation of the model.

What is the weakness of the paper? (1-3 sentences) One weakness of the paper is not to present the negative aspects of their work and to state some issues that they noticed while implementing AnonOS.

Your qualifications to review this paper: 2
Writing quality: 5
Comment: The writing was clear and I liked how the authors motivated each part and the design decision of the paper.
Relevance to SOSP? 5
Comment: I think it is a very interesting paper and tackles a problem relevant to the community.
Experimental methodology: 4
Comment: There is a nice parallel between section 5 and 3.4 where the authors evaluate all the goals stated previously, but in some cases the presentation could have been a little clearer.
Novelty of paper: 5
Comment: Their idea of treating the machine as a network of independent cores with no directly shared memory is, to the best of my knowledge, very novel.
Overall paper merit: 5
Provide additional detailed comments to the author: I have written the comments with each question instead.


Provide a short summary of the paper The emergence of heterogeneous multicore architectures necessitates the need for a complete redesign of the underlying OS structure used to manage them.  Rather than looking at the OS from a traditional point of view as one based on shared memory data structures and lock-based contention mechanisms, the OS of the future should be viewed as a distributed system of interacting components that communicate via message passing.  Viewing the OS as a distributed system allows us to customize the lowest levels of an OS to take advantage of the particular features of a given hardware platform, while maintaining a good layer of abstraction for any necessary communication between core OS services.
What is the strength of the paper? (1-3 sentences)  The case is made pretty well as to why you might like to have an OS designed from the ground up as a distributed system.  The design, and implementation of AnonOS is clear, and the experiments ran are pretty convincing for the criteria they layed out for evaluation i.e. comparable performacne to OSs running on todays hardware, but with much better scalability properties for future hardware.
What is the weakness of the paper? (1-3 sentences)

The paper seems to be very repetitive in places.  Every new section seems to re-provide motivation for designing the OS as a distributed system.  The case is made well in sections 1/2, leave it at that. 

The evaluation could have been done on more advanced architectures, such as the Intel Core i7, or the AMD barcelona.  Additionally, it would have been interesting to see how well it scaled in a virtual machine setting that was set up for 256+ cores to see how it scaled, even if the performance wasn't too great. 

Your qualifications to review this paper
  1. I know a lot about this area
Writing Quality
  1. Good
Relevance to SOSP?
  1. People would read it before conference and attend session
Experimental Methodology

 The authors verify their design through a set of experiments designed to demonstrate that it (1) has good baseline performance, (2) scales with the number fo cores, (3) is portable with minimal refactoring, (4) expoits message passing throughout, and (5) is sufficiently modular to amke use of topology-awareness.

Novelty of paper  OS as a distributed system.....
Overall paper merit
   
   
   
   
X Strong accept - This is of interest to SOSP, and a novel or new contribution to existing area with good/average methodology, or an incremental contribution paper that has excellent methodology.
   
   
Provide additional detailed comments to the author  



Provide a short summary of the paper  The paper describes a new operating system design, the multikernel, for many-core processors. The idea is to treat each core as a semi-independent device that communicates with other cores through message passing. The system design is similar to microkernels in that the OS consists of userspace services (running on seperate cores) but differs in that even the kernel does not have shared-memmory.
What is the strength of the paper? (1-3 sentences) The paper presents an interesting OS design that takes microkernels to a logical conclusion of removing all implicit sharing within the kernel. The design is well motivated: sharing must be removed to scale to many cores.
What is the weakness of the paper? (1-3 sentences)  The paper does not explain how this work is different from prior microkernel work until the middle of the paper. The experiments do not consider heterogeneous processors/cores as was discussed in the motivation. Finally, the experiments dont seem to show a fundemental inprovement in scalability.
Your qualifications to review this paper
  1. I know nothing about this area
  2. I have passing familiarity
  3. I know the material, but am not an expert
  4. I know a lot about this area
  5. I am an expert on this topic
Writing Quality
  1. Unacceptable
  2. Poor
  3. Average
  4. Good
  5. Excellent
Relevance to SOSP?
  1. I do not think people would read the paper nor attend the session
  2. I doubt anyone would attend session
  3. People would attend session
  4. People would attend session, but not read it beforehand
  5. People would read it before conference and attend session
Experimental Methodology
Novelty of paper
Overall paper merit
Reject - Paper is not of interest to Sensys, it has already been done, or has serious methodology flaws.
Weak Reject - Paper is potentially of interest to Sensys, and has some value in the ideas presented, but it has either potential methodology issues or it is a borderline incremental/already been done contribution.
Weak Accept - Paper is of interest to Sensys, and is an incremental contribution paper with average/weak methodology.
Accept - This is of interest to Sensys, a novel or new contribution with average/weak methodology, or incremental contribution paper that has good methodology.
Strong accept - This is of interest to Sensys, and a novel or new contribution to existing area with good/average methodology, or an incremental contribution paper that has excellent methodology.
Definite accept - This is of interest to Sensys, and a novel or new contribution to existing area with excellent/good methodology.
   
Provide additional detailed comments to the author

I recommend describing the novelty of the design (compared to microkernels, Tornado, etc.) early in the paper.

In the evaluation section, heterogenious cores/processors are not mentioned despite this being a core motivation for that paper. Adding this analysis would strengthen the paper.

 

Additional comments to PC
(not seen by author)
 


Summary:

This paper proposes the re-evaluation of the design of multicore operating systems as a distributed networked system.  By viewing each core as a separate processing unit with its own state and using message-passing and various distributed-system protocols to manage consensus and sharing, you can generalize the operating system for the ever changing underlying hardware.  A general message-passing interface and split-phase programming can allow for pipelining and other optimizations that decouple the techniques and mechanisms from the policies for dealing with sharing across core.  Furthermore, many of the contributions and work in the distributed system theoretical and systems literature can be pulled into future OS design for multicore systems.

Strengths:

The authors bring up many convincing arguments and motivate the problem quite well.  Furthermore, the AnonOS implementation of the multikernel design is a nice demonstration that this is more than just an academic discussion.  The fact that it was able to perform comparably to current systems is encouraging.


Weaknesses:

It's not clear to me how much of the performance overhead seen in the evaluation is inherent to their implementation of the design and how much is the actual design of a multikernel system.  To the authors' benefit, however, they do state that the evaluation is not thorough and that instead what should be focused on is the fact that the performance numbers are comparable which motivates the design choices.

 
Your qualifications to review this paper: 3

Writing Quality: 5

Relevance to SOSP?: 5

Experimental Methodology: 4

Novelty of paper: 4

Overall paper merit: 5

Provide additional detailed comments to the author:

This is a nice piece of work that focuses the community on thinking about new designs for multicore system and I think that going about it like this paper has will allow operating system designers to more easily keep up with the rate of change of hardware.  That said my only concern is that if this does become a viable design point, that more distributed system problems will be introduced at the cost of performance that could/would have been solved more easily with current techniques.  In other words, if you choose to introduce the benefits of distributed system design into OS design you also introduce the inherent set of problems that that community is also trying to tackle.  Small problems in more distributed higher-latency network settings might be amplified at the scale of a multicore OS.

Additional comments to PC:

I think this is a fresh look at multicore OS design.  It's important, i think, for the community to consider this work in moving forward and keeping up with the hardware trends and to move away from incremental improvements and tweaks made to new systems based on specific design decisions made for the older versions.




Attribute

Value
   
Provide a short summary of the paper  To make an operating system that scales to heterogenous multicore systems, the authors develop an OS where the cores are loosely coupled as in a cluster: Each core communicates with other cores only uses an asynchronous messages abstraction (which, on the systems the authors test, is implemented using shared memory); most operating system state is replicated between cores instead of shared. Performance results show that this approach can scale to tens of cores better than the approaches used by commodity operating systems.
What is the strength of the paper? (1-3 sentences) The "multikernel" criteria are clearly stated and likely widely applicable. The design choices are backed up by experiments on real systems that suggest their apporach is likely to scale to many-way multicore systems.
What is the weakness of the paper? (1-3 sentences)

Heterogenity, depsite playing such a prominent role in the paper, is not actually addressed, not even in simulation or a model.

The paper does not do a good job at quickly distinguishing itself from prior work.

Your qualifications to review this paper I know the material, but am not an expert.
Writing Quality Good
Relevance to SOSP People would attend session

Experimental Methodology Excellent
Novelty of paper Incremental improvements
Overall paper merit Strong accept
Provide additional detailed comments to the author

The paper talks about heterogenity as a motivating factor, but does not attempt to evaluate how the approach deals with heterogenity with any rigor; the paper would be fine without mentioning heterogenity.

How this approach differs from microkernel and K42-like approaches should be addressed more prominantly than a couple of paragraphs near the end of the paper.

The experiments are generally good, however, some things about them were wanting:

- Given that the interconnect network requires more hops between different pairs of processors in some cases, it would be helpful to see worst case RPC timings.

- Comparing with Apache (a webserver usually chosen for configurability that typically operates using many kernel threads) with some custom webserver seems unfair. The lack of description of the architecture of the custom web server (does it use a thread/process poll model like Apache?) makes it difficult to gauge how realistic the comparision is.

Additional comments to PC
(not seen by author)
 


Short summary of the paper       
- Build an OS where the default case is no-sharing / full replication,
instead of removing sharing when it is a bottleneck.  This will help
with scalability, as well as heterogeneity.  Different ISA-d cores as
well as possibly non-cache-coherent cores may become the norm. 
Treating the OS as a distributed system of cores will help will all of
this.

What is the strength of the paper? (1-3 sentences)      
- Well thought out both in regards to different facets of their systems
and with respect to its place in the literature.

What is the weakness of the paper? (1-3 sentences)      
- A lot of the value of this depends on future non-cache coherence, but
they go back and take advantage of sharing for their performance gains.
 In some sense, they design their system against the "common case" of
sharing by adding the layer of indirection, but never really make the
case that this will be valuable.  Their experiments do not specifically
target these claims, but instead just show that their OS is okay in
some cases.  It is also not clear that their gains cannot be applied to
current monolithic OSs.

Your qualifications to review this paper    

   5. I am an expert on this topic.  It's arrogant to say that.

Writing Quality    

   3. Average - Some parts were a little difficult to read or not
clearly explained.

Relevance to SOSP?    

   5. People would read it before conference and attend session

Experimental Methodology      Average

Novelty of paper      This is a new contribution to an established
area

Overall paper merit    
     Accept - This is of interest to SOSP, a novel or new
contribution with average/weak methodology, or incremental contribution
paper that has good methodology.
       
Provide additional detailed comments to the author      
    Are you sure about the IPI's being serial in Linux?  That may
be the case, but I'm under the impression that they use broadcast IPIs.
 Regardless of whether or not they use broadcast, I think that
broadcast is available.  That should only be one message on the
APIC/system bus that all cores can respond to at once.  This makes me
wonder what else is involved that makes the Windows/Linux cases slower

Additional comments to PC
    Am I the only one who gets irritated when people make claims
based on lines of code or on time to implement something?



Attribute

Value
   
Provide a short summary of the paper The author argues that the shared-memory model for multicore systems is inappropriate in the face of their rapidly-increasing complexity and heterogeneity; instead, he says that we should shift to the shared-nothing distributed system paradigm.  The author proposes the multikernel model, similar to an exokernel, where the simple kernel is shared across all cores and a per-core user-level monitor coordinates core-specific state.  The author provides a multikernel implementation called AnonOS, which he demonstrates to have similar scalability to Windows and Linux and better performance in some cases.
What is the strength of the paper? (1-3 sentences) The author provides a good motivation for the paradigm shift and builds upon many good past results to develop his multikernel model.
What is the weakness of the paper? (1-3 sentences) The author is a bit inconsistent about his claims; eg, in Figure 7, he shows that the unmap latency achieved by AnonOS is lower than that of either Windows or Linux, but later on, he says that the OSes should not really be compared since the latter are so much more mature.
Your qualifications to review this paper
  1. I know nothing about this area
  2. I have passing familiarity
  3. I know the material, but am not an expert -- X
  4. I know a lot about this area
  5. I am an expert on this topic
Writing Quality
  1. Unacceptable
  2. Poor
  3. Average
  4. Good
  5. Excellent -- X
Relevance to SOSP?
  1. I do not think people would read the paper nor attend the session
  2. I doubt anyone would attend session
  3. People would attend session
  4. People would attend session, but not read it beforehand
  5. People would read it before conference and attend session -- X
Experimental Methodology -- 5
Novelty of paper -- 4
Overall paper merit -- 6
Reject - Paper is not of interest to Sensys, it has already been done, or has serious methodology flaws.
Weak Reject - Paper is potentially of interest to Sensys, and has some value in the ideas presented, but it has either potential methodology issues or it is a borderline incremental/already been done contribution.
Weak Accept - Paper is of interest to Sensys, and is an incremental contribution paper with average/weak methodology.
Accept - This is of interest to Sensys, a novel or new contribution with average/weak methodology, or incremental contribution paper that has good methodology.
Strong accept - This is of interest to Sensys, and a novel or new contribution to existing area with good/average methodology, or an incremental contribution paper that has excellent methodology.
Definite accept - This is of interest to Sensys, and a novel or new contribution to existing area with excellent/good methodology. -- X
   
Provide additional detailed comments to the author  I would like to have the claims clarified -- you say that AnonOS should not be compared with Linux and Windows yet present superior performance in some cases.  Why is it that they should not be compared?  Also, you say that there is a significant difference between multikernels and VMMs; I would have liked to see more justification for why this is true, especially since in the related work section, you mention that the distributed systems paradigm is also used in VM work like Disco.
Additional comments to PC
(not seen by author)
Work is very interesting.  I would have liked to see the author mention that the distributed system paradigm has been applied before to manycore systems.  As is, it sounds like he is the first (until I reach the related work section).



Provide a short summary of the paper
The paper argues that with the increasing number of cores on modern processors and the likelihood of heterogeneity across these cores, current shared-memory OS architectures, which rely on processor-specific optimizations, will no longer be able to scale. The paper proposes recasting the operating system as a distributed system where communication between cores is achieved through message passing and consensus protocols are used to synchronize state. The paper goes further than previous message passing approaches in advocating that the kernel itself be split amongst cores and in the degree of separation between components.

What is the strength of the paper? (1-3 sentences)

I think the motivation for moving operating systems towards architectures traditionally used in distributed systems was cogent, especially the discussion regarding heterogeneity. The necessity of dealing with multiple cores is well understood, but the effects of dealing with heterogeneity seem to garner less attention.

What is the weakness of the paper? (1-3 sentences)

The paper presents the proposal and motivation for a highly-decoupled operating system using message passing as highly original work. They do not cite previous work until late in the paper and even then, do not, in my opinion, give it the full credit due. Disco, for example, they cite as "further evidence that the shared-memory model is not a complete solution." Most of the motivations for and advantages to message passing presented in the paper come directly from previous work in this area and while this paper has a contribution, it is weakened by it's failure to adequately credit earlier work.


Your qualifications to review this paper?
3. I know the material, but am not an expert

Writing Quality
4. Good

Relevance to SOSP?
3. People would attend session

Experimental Methodology:
Average

Novelty of paper:
Incremental improvement

Overall paper merit
Accept - This is of interest to Sensys, a novel or new contribution with average/weak methodology, or incremental contribution paper that has good methodology.



Provide a short summary of the paper: This paper proposes structuring multicore OSes as shared-nothing distributed systems with message passing between the kernels running on each core. It also evaluates a prototype called AnonOS that uses this architecture and is shown to perform reasonably well and be reasonably easy to extend. What is the strength of the paper? Good writeup of motivation and detailed discussion about why this distributed system design might be a good idea. What is the weakness of the paper? Evaluation is somewhat limited and doesn't show very impressive scaling yet. There is little explanation for this behavior. There is also little discussion into the disadvantages of structuring an OS as a distributed system and the tradeoffs involved.

Your qualifications to review this paper: 3 (I know the material, but am not an expert)

Writing Quality: 5 (Excellent)

Relevance to SOSP: 4 (People would attend session but not read it beforehand)

Experimental Methodology: 3 (Average) The evaluation contains a number of benchmarks, but some of them are artificial (e.g. local loopback is not representative of many apps), and some of them show unexplained behavior (e.g. why does scalability decrease on systems with 28+ cores in the CPU-bound OpenMP benchmark?). The graph in motivation is also not very informative (we see number of contended cache lines, but not how this affects performance). In general one concern I have with this kind of paper is whether a simpler solution would get you 90% of the way there. If you propose a revamp of OS architecture, you should have good arguments about why something like Tornado, Corey, etc can't do nearly as well as your solution.

Novelty: 4 (This is a new contribution to an established area)

Overall Merit: 4 Accept (Average/weak methodology but somewhat novel contribution)

Additional Comments: The idea of exposing messages as the communication mechanism is not novel; highly scalable distributed systems platforms such as Erlang use it and have been using it for 15-20 years. However, applying this in an OS setting is interesting, and the paper makes some good arguments about why future hardware trends will necessitate this. The main thing I'm worried about though is a) what are the downsides of this approach (people use shared memory for a reason: it's pretty easy to think about) and b) where is the proof that we need this big a change in architecture to achieve good results. The paper presents a new point in the design space but doesn't justify that some simpler solution won't work well. A second criticism I have is that there's very little talk about how to integrate this with multi-threaded programming platforms like TBB and OpenMP. The OpenMP graph doesn't show great scalability.. why is this? Is something missing from the design or is it just an artifact of being a prototype?



Provide a short summary of the paper    
This paper presents an operating systems designed as a distributed system on multicore systems. It treats each core as a completely different kernel with different scheduling decisions. It argues that through three main mechanism: message passing instead of shared memory, replication, and asynchronous calls we can achieve better scalability on multicore systems. 

What is the strength of the paper? (1-3 sentences)
Problem well motivated with data from existing operating systems
Looking into the future hardware while demonstrating that this design is also competitive on current hardware systems
Evaluation includes both micro and macro level benchmark demonstrating scalability of this system.

What is the weakness of the paper? (1-3 sentences)      
Unfortunately, due to the limitation of current hardware, there are a limited number of cores on the system being evaluated.
Local (intra core) communication involves two context switch and is potentially expensive. Frequent operations like this can have significant effect eventhough it is only constant time factor.

Your qualifications to review this paper    
I know a lot about this area... hopefully.

Writing Quality    
   5. Excellent

Relevance to SOSP?    

   5. People would read it before conference and attend session



Provide a short summary of the paper:

This paper starts from the premise that hardware is evolving too rapidly for OS software to keep up with it.  Furthermore, hardware is becoming less uniform over time and the current strategy of treating any unusual processing units as off chip devices doesn't scale well.  The solution proposed by the paper is to treat the processor more like a distributed system and have the OS be programmed like a distributed system.  Each core is its own processing node and everything communicates via message passing.  This allows an OS to be easily adapted to different architectures by simply optimizing the message passing code.  The rest of the OS can remain unchanged.  Also described is an implementation of an OS called AnonOS using the ideas presented in the paper.  It is shown that AnonOS scales better than existing OSes on some workloads, and as well on others even though AnonOS has not been optimized much at all.

What is the strength of the paper? (1-3 sentences) 

The paper makes a strong argument for why message passing is a preferable paradigm for programming an OS in.  Furthermore, they show that existing OSes employ techniques that simply don't scale well in multi-core NUMA environments.

 

What is the weakness of the paper? (1-3 sentences) 

The paper uses one example to argue why existing OSes will not be able to adapt to new architectures well.  This is unconvincing for two reasons.  Firstly, perhaps it was simply bad design in Windows that made it so much work to remove the dispatcher lock, and secondly, implementing a whole new OS is a huge amount of work too.

Your qualifications to review this paper

Writing Quality

Relevance to SOSP?

 

Experimental Methodology:
Average

Novelty of paper:
This is a new contribution to an established area

Overall paper merit:
Strong accept - This is of interest to Sensys, and a novel or new contribution to existing area with good/average methodology, or an incremental contribution paper that has excellent methodology.
      
Provide additional detailed comments to the author:
The presentation is good and the ideas are certainly interesting.  I would like to see some absolute performance comparisons to existing systems.

     
Additional comments to PC:
Similar work has been done before (Tornado/K42), but this is still an interesting paper and worth reading.


Experimental Methodology     Good
Novelty of paper     This is a new contribution to an established area
Overall paper merit     Strong accept - This is of interest to Sensys, and a novel or new contribution to existing area with good/average methodology, or an incremental contribution paper that has excellent methodology.

Provide additional detailed comments to the author
Additional comments to PC
(not seen by author)



Attribute

Value
   
Provide a short summary of the paper The authors propose a new operating system design, a multikernel, for emerging heterogenous multi/manycore architectures. Their design uses the distributed systems techniques of replication, message-passing, and split-phase (i.e. asynchronous) communication. They discuss an implementation of their operating sytem design, and evaluate their design using a series of microbenchmarks as well as a web server and relational database.
What is the strength of the paper? (1-3 sentences) The authors include a detailed discussion of how system diversity, core heterogeneity, interconnect topology, and cache coherency are motivating factors for redesigning operating system structure. The discussion of these is valuable to the community at large to consider.
What is the weakness of the paper? (1-3 sentences) The authors motivate heterogenous systems as being a major influence in the design of their operating system, but they fail to perform any evaluation on a heterogenous system. Granted, there are not many (if any) heterogenous systems other than GPU + CPU, but they failed to even show their system on this configuration.
Your qualifications to review this paper I know a lot about this area
Writing Quality Good
Relevance to SOSP? People would attend session, but not read it beforehand
Experimental Methodology Good
Novelty of paper This is a new contribution to an established area
Overall paper merit Accept - This is of interest to SOSP, a novel or new contribution with average/weak methodology, or incremental contribution paper that has good methodology.
   
Provide additional detailed comments to the author
Additional comments to PC
(not seen by author)
We are seeing a lot of operating systems papers that have proposed novel techniques, and shown how much better applications perform using their operating system. However, many/most of the performance benefits seem to come from eliminating bottlenecks such as kernel crossings, that contemporary operating systems could employ themselves. Its harder to evaluate these papers when it is unclear how many of these novel techniques are the true cause of the performance increases, and how many of the techniques have artifacts that can be applied to existing operating systems that benefit performance.