Retrospective: 2015 as a Tech and Career Mentor

2015 was a pretty great year mentoring for me. After several years of offering advice to others, I now have a small, consistent group of individuals that I work with. Each in a different way, important to them, but probably more important to me.

My Background and Thoughts On Mentoring

I generally mentor people at Microsoft who are experiencing some issue in their career growth. Sometimes the issues are things they can change, sometimes the environment needs to change, sometimes it is technical, sometimes it relates to missing skill-sets and sometimes its personal issues. No matter what the cause, as an external observer, I can either directly help them or I can find someone else who can. I can be supportive, uplifting or give them the hard talk about something they need to change. Whatever they need. As a mentor I'm adaptable to their situation.

Most of my mentoring sessions consist of drawing parallels between road-blocks that I've faced and how I overcame them. By explaining through a real scenario it usually becomes much clearer to others that they have options. More increasingly though I find that my experiences aren't the same as theirs. This was a great mentoring realization for me, people have unique problems, they have unique strengths and weaknesses. My "solution" isn't their "solution". Instead I need to offer many viewpoints and using the combined experiences of all of my mentees, not just my own experiences. Intelligent people tend to solve their own problems once they start to realize how many options they have. Persistent problems tend to remain so long as your options are limited.

Helping someone find their options isn't limited to those with wisdom and experience either. This is my pitch to those who think they are too young mentor others. Mentoring is just as much about listening, learning, problem solving and being supportive as it is about learning purely from experience. If you are giving someone the solutions to their problems, it isn't really a mentoring session anymore. You are trying to make them better, help them to find their own solutions, not demonstrate how easily and intelligently you could resolve their issue.

Mentoring doesn't have to be formal. It could be one friend to another. It doesn't have to be long or short. A single meeting or many years worth of them, both outcomes are acceptable. It could be a regular weekly meeting or it could be a couple of random days in the year. It doesn't have to be scheduled at all, for instance, getting feedback on a potentially career limiting email from your mentor can often be the difference between success and failure.

We need more mentors thinking about diversity. Rico Mariani recently started an inspiring series of thoughts and threads (well more than that, he started a scholarship) focused on women in tech. He immediately reached out to everyone on his team and made it very clear, minorities in tech, regardless of which minority, need equal opportunities and access to strong mentors. So I'm going to do some more thinking on this. I think it is very hard for someone to reach out and accept mentoring and the bigger the differences which exist (level, race, gender, social status, personalities) the harder it is for someone to take that first step. Its probably time to reach out to potential mentees, a lot more often, and accept the rejections in favor of helping the diversity imbalances.

My Mentees for 2015

This won't focus solely on 2015, since one of my most important roles in mentoring occurred during 2014 just following the large Microsoft lay-offs. My list of mentees (anonymous) and how we spend our time.

1. The Entrepreneur - I once owned my own company and was independently successful for a few years during my 20's. This experience taught me quite a bit. One of my mentees is at a point in his life where he wants to develop a small company for himself and so we have sessions on how to transition from your day job to being a founder.
2. The Classic - Before the Microsoft mentoring site went down I got my last mentee referral for what would become my most normal mentoring setup. We meet on his schedule and the topics are around how to manage his career development, communications, timing and positioning. Sorry, I'll expand more on that in a bit ;-)
3. The Switcher - Within the past few months I've started working with someone who is unhappy in their position on their current team. They had already made a decision that improving their situation was not the right move, that they needed a change. For this person, I'm helping them improve their technical interviewing skills.
4. The LayOffs - When the large scale Microsoft lay-offs happened I realized quickly that many intelligent and capable people were going to start having a very hard time in the job market. It is easy to have your skills atrophy when you work in a mature company like Microsoft. For this group, I got to learn the challenges of mentoring in a group setting.
5. The Boxer - Everyone needs change and growth opportunities. This person had been working in the same area for too long and was feeling constrained, as if they had been put in a box for safe keeping. When in this position, having options and knowing what you can and should be able to ask for to improve your situation is key.

With the exception of the group training I did when the lay-offs occurred, my diversity scorecard has me gender biased (zero female mentees) but doing okay otherwise (I have mentees of many races, though they are all male). During the lay-offs, I was able to help multiple female engineers reset their skills and I'm happy that a majority of them were able to quickly find a position that was as good or better than what they left.

You Get More than you Give

My mentees are consistently conscious of my time. They don't want to waste it, since they view my time as important, and they view me as someone who maximizes the value of my own time. Guess what, they are right! I do value my own time and I can say hands down the time I spend mentoring is far more important than anything else I could be doing. So the secret to mentoring is that you spend time doing it, its that you can help yourself and others at the same time, in ways that will help you in the future.

After all, how do you become better at anything? You PRACTICE it. Mentoring gives you the opportunity to be a great listener and a strong communicator. It helps you to organize your experiences and arguments through self-reflection that you are able to share with someone else. Your mentee can sometimes reverse the experience on you and potentially point out something you missed when you dealt with a situation that you are using as an example. These moments, for me, have always led to improvements in my own career.

Your mentees will also have experiences that differ from your own. This will help you increase your own understanding of diversity and help provide you with insights to improve how you integrate and communicate with your co-workers. Perhaps a certain cultural bias prevents someone from speaking up, and you are able to experience this through your mentee, then it can make you more sensitive to this issue.

The last bit for me is around building your moral standards. If I'm not helping people, I'm not happy. There is a certain amount of human interaction and empathy that goes into the entire process that leaves me feeling like I've done something, bigger than myself. I've reached out and helped someone else. You can get this many ways, through teaching, mentoring or volunteering, so I challenge you to do all 3 to maximize your own benefits ;-)

Reaching Out

You've read everything above and you are thinking, "Sign me up!" So how do you start? I don't have 40 years of experience doing this, but I do have enough to know one thing. Not everyone realizes they need a mentor and pointing it out often isn't enough to convince them. Mentoring is a two-way street and even though I have time to give my advice it doesn't mean that others have the time to hear it. It is an exceptional case when someone walks up to you and ASKS you to mentor them. This almost never happens. 

So you have to put yourself out there and let people know you are ready to help. When approaching a new mentee with an offer, do so privately. Not everyone sees having mentors as a strength and some see it as a sign of weakness. A powerful mentor relationship can also be seen as a threat or even favoritism. This is normally irrational and unhealthy thoughts by others but it is something to be aware of.

By the numbers, the amount of people that I will approach is quite small. I'm evaluating a lot of different outcomes and consequences. As a mentor I should do a MUCH better job of approaching people. 

Once approached, the acceptance rate is probably 50/50. So be prepared to be rejected. This is key, never take someone else's rejection of your mentoring offer as anything against you. There are a lot of factors they are evaluating as well, including whether or not they even feel comfortable talking to you. They may not see you as a mentor at all. This is normal. When you get a rejection, kindly let them know that you are available in the future if things change, and move on.

Even out of those who accept, there is a likelihood you will never have your first mentoring session. If you do, there is a high likelihood it will be your last or next to last. Being mentored is not for everyone and maybe the session made them feel awkward. I bet psychologists have a similar track record with new clients. Also, not every problem is one to be fixed. Your mentee may find that it isn't the right time for them to think about and correct the issues impacting them. This is especially true in long term career mentoring.

Once you've gotten started be prepared to adapt yourself to many different types of people and expectations. If they want you to set up a schedule and manage it, then do that. Otherwise, if they want to set it up instead, then let them do that. The majority of the improvement they make will be when they are not interacting directly with you, so figure out how to follow up with them. If they like goals, set them up. The more you mentor the more you'll figure this out, but being prepared will hopefully your first few mentoring encounters much more fruitful.

If in doubt, get a mentor yourself and ask them about mentoring. Its the ultimate mentoring meta ;-)

Diversifying

Diversifying your mentees can be extremely difficult. First, people are more comfortable around those more like them. For this reason, in a work environment, people of similar ethnic backgrounds are likely to stick together. When a team has good diversity this doesn't happen as much, but then the team itself can stick together and becomes a blocker for technical diversity. Since minorities, by definition, are fewer within the groups, trying to find a mentor that matches your minority can be hard.

This leaves our minority populations in tech with a hard problem. First, they can't find high level mentors within their own minority. Second, they may find it hard or challenging to find high level mentors at all. This all assumes that they reach out. If they don't then its perhaps more unlikely that an appropriate mentor would not find them.

This type of diversity bias is what we have to overcome. It isn't really fixed by taking a "fair" approach to the problem. You can't sit back, and expect the odds to play themselves out here. While you may think that by accepting anyone and everyone you'll achieve diversity, you won't. You have to break the "fairness" and make a point of being unfair in your selection of mentees and how you reach out. Make a point of prioritizing for ethnic and gender diversity when you fill up your time initially and then worry about filling out the majority slots later. Trust me, they won't be nearly as hard to fill.

For my part, I'm going to commit to filling out my diversity card in the upcoming year of 2016. I've already tweeted that I have some mentoring capacity. I'm reserving that space for women and other minorities in tech. I will be proactive and start reaching out to potential mentees and work with other mentors in my network in case they know of people actively looking. I am going to increase my efforts towards STEM education for minorities though I don't yet have a plan for how I'm going to do that. If anyone has ideas let me know ;-) I'll make sure to be more vocal about my efforts as well in case it can inspire or inform others. Bringing awareness is a key step in solving any problem.

Finally a challenge to all you would be mentors. If you aren't sure, and you want to talk to someone about becoming a mentor, I hear you can call anyone on this thing called the Internet. I'd be happy to talk with you about my experiences and answer any questions you might have before you get started.

A Collection of Principles for Fail-Fast

Previously I blogged about how EdgeHTML has adopted a model of fail-fast by identifying hard to recover from situations and faulting, rather than trying to roll back or otherwise proceed. There we covered a lot of details on the what and even the how. At that time I didn't establish principles and since writing that article I've received a lot of questions from my own developers around when to use fail-fast. So here it is, the principles of fail-fast.

Principle #1 All memory allocations shall be checked and the process fail-fast on failure.

Follow the KISS principle and just assume that all memory conditions (including stack overflows) are leading to a situation in which even if recovered the first, second or third party code will not run correctly.

Exceptions:

Exploratory allocations may be recoverable. Textures are a commonly used resource and are limited in availability. So some systems may have a recovery story when they can't allocated. However, even these systems likely have some required memory, such as the primary texture, and that should be demanded.

Principle #2 Flow control is ONLY for known conditions. Fail-fast on the unknown.

When writing new code favor fail-fast over continuing on unexpected conditions. You can always use failure telemetry to find common conditions and fix them. Telemetry will not tell you about logic bugs caused by continuing on the unexpected path.

A prime example of this is when using enumerations in a switch. Its common practice to put a non-functional default with an Assert. This is way too nice and doesn't do anything in retail code. Instead fail-fast on all unexpected flow control situations. If the default case handles a set of conditions, then put in some code to validate that ONLY those conditions are being handled.

Third party code is not an excuse. It is even more important that you use fail-fast to help you establish contracts with your third party code. An example is a COM component that returns E_OUTOFMEMORY. This is not a SUCCESS or S_OK condition. Its NOT expected. Using fail-fast on this boundary will provide the same value as using fail-fast in your own memory allocator.

Exceptions:

None. If there is a condition that should be recovered then it is a KNOWN condition and you should have a test case for it. For example, if you are writing code for the tree mutations caused by JavaScript operations on the Browser DOM, then there are known error recovery models that MUST be followed. No fail-fast there because the behavior is spec'ed and failure must leave the tree in a valid state. Maybe not an expected state for the developer using the API, but at least spec'ed and consistent.

Principle #3 Use fail-fast to enforce contracts and invariants consistently

Contracts are about your public/protected code. If you expect a non-null input, then enforce that with a fail-fast check (not much different from the allocation check). Or as before with enumerations, if you expect a certain range then fail-fast in the out of bounds conditions as well. When transitioning from your public to your private code you can use a more judicious approach since often times parameters have been fully vetted through your public interface. Still, obey the control flow principles.

For variable manipulation within your component, rely on checks for your invariants. For instance, if your component cannot store a value larger than a short, then ensure that down casts aren't truncating and fail if they do. This classically becomes a problem between 32 and 64-bit code when all of a sudden arbitrary code can manipulate values larger than originally designed for.

While a sprinkling of fail-fast around your code will eventually catch even missed invariant checks, the more consistently you use them, the closer your telemetry will be able to point you to the sources of failure.

Exceptions:

None. Again, if you find a condition hits too often, then you'll be forced to understand and supply a fix for it. Most likely a localized fix that has little or no impact on propagating errors to other surrounding code. For instance, truncation or clamping can be a designed (and perfectly acceptable) part of the component depending on its use case.

Principle #4 If you are unsure whether or not to use fail-fast, use fail-fast

This is the back-stop principle. If you find yourself not able to determine how a component will behave or what it might return (this can happen with black box APIs or even well documented, but closed APIs) then resort to fail-fast until you get positive confirmation of the possibilities.

As an example some COM APIs will return a plethora of COM error codes and you should not arbitrarily try to recover from the various failures or figure out which codes can and can't be returned. By using fail-fast and your telemetry pipeline you'll be able to find and resolve the sets of conditions that are important to your application and you'll have confidence that your solutions fix real world problems seen by your users.

Oddly, this is even more critical when working on pre-release operating systems, services or APIs. Often the introduction of a new error code or the increase in a specific set of error codes is indicative of an OS level bug. By tightening the expectations of your application on a specific API surface area you become one of the pinning tests for that API. While APIs do change, having unexpected or new behavior propagate through your application in unexpected and unintended ways is a bug waiting to happen. Better to crash and fix than to proceed incorrectly.

Exceptions:

Yes, of the fail-fast variety please ;-)

From State of the Art to the State of Decay

I'm constantly in meetings where we are discussing 10+ year old code that "smart people wrote" so it must be fairly good. It seems the people take offense when you call 10 year old code bad names especially if they were somehow connected.

Let me start with myself. I've been working on the Internet Explorer code base for over 10 years, having started just before IE 7 Beta 1 was shipped. Instead of referring to other people's terrible code, I'll refer to my own terrible code. I'll also talk about our thinking on "State of the Art" at the time and how that leads to now where the same code which was the bees knees is now just a skeleton of its former self having suffered through the "State of Decay".

State of the Art in 2005

We have a legacy code base that has been dusted off after many years of neglect (IE 6 had been shuttered and IE 7 was the rebirth). The ultimate decay, many years of advancements in computer science, none of which were applied to one of the largest and most complex code bases you could conceive of. We had some new tools at the time, static analysis tools, which were very good at finding null pointer dereferences alongside actual security bugs. We had tens of thousands (maybe even hundreds of thousands) across all of Windows that needed to be investigated, understood and fixed.

Our ideas of state of the art at this time were that the code shouldn't crash. Ever. Reliability in our minds was error recovery and checking null's all over the place and protecting against the most malicious of external code since after being shuttered we had no idea who or how many consumers of our code there was. Fixing any single line of code presented us with a challenge. Did we break someone? In this state every line of code could take hours to days to understand the ramifications of adding even a single null check and bail-out. And spend hours and days we did.

To the extent that I personally introduced hundreds, perhaps thousands of null pointer checks in the code. After all, this was the state of the art at the time. We wanted stability, reliability and we wanted most of all to shut down those rare cases where the tool was pointing out an actual security hole as well. Of course I thought I was doing the right thing. My code reviewers did too. We all slept well at night getting paid to put in null pointer checks to work around the fact that our invariants were being broken constantly. By doing all of this work we were forever changing our ability to use those crashes we were fixing to find actual product bugs. "State of the Art" indeed. So how should those conversations about 10+ year old code be going? Should I really be attached to those decisions I made 10 years ago or should I evolve to fix the state of decay and improve the state of the art moving forward?

State of Decay in 2015

Those decisions 10 years ago have led to a current state of decay in the code. Of those thousands of null pointer checks, how many remain? When new developers look at the code, what should they glean from the many if-checks for conditions that may or may not be possible? The cognitive load it took me to put one in was hours and even sometimes days. What about the cognitive load to take one out?

It is now obvious that the code is no longer state of the art, but bringing the code up to similar quality as new code is an extreme amount of work. It is technical debt. To validate the technical debt we talk about how long that code has been executing for and "not causing problems" and we bring up those "smart people" from the past. Yes we were smart at the time and we made a local optimum decision to address a business need. It doesn't mean that maintaining that decision is the best course of action to resolve future business needs (in fact it is rarely the case that leaving anything in a state of disrepair is a good decision as the cost of deferring the repair of decay is non-linear).

I gave a simple example of null pointer checks to address reliability. There were many more decisions made that increased our technical debt versus the current industry state of the art. Things like increasing our hard dependencies on our underlying operating system to the extent that we failed to provide good abstractions to give us agility in our future technology decisions (I'm being nice, since we also failed to provide abstractions good enough to even allow for proper unit testing). We executed on a set of design decisions that ensured only new code or heavily rewritten code would even try to establish better abstractions and patterns for the future. This further left behind our oldest code and it meant that our worst decisions in the longest state of decay were the ones that continued accruing debt in the system.

Now there are real business reasons for some of this. Older code is more risky to update. Its more costly to update. It requires some of your most expert developers to deal with, developers you'd like to put on the new shiny. Not everyone's name starts with Mike and ends with Rowe and so you can risk losing top talent by deploying them on these dirtiest of jobs. Those are valid reasons. Invalid reasons are that smart people wrote the code 10+ years ago or that when you wrote that code it was on state of the art thinking. As we've noted, the state of the art changes, very quickly, and the new code has to adapt.

Start of the Art in 2015

Its useful then to look at how many "bad decisions" we made in the past since if we could have told the future, we could have made the 2015 decision and been done with all of this talk. Our code could be state of the art and our legacy could be eliminated. So what are some of those major changes that we didn't foresee and therefore have to pay a cost now and adapt?

Well, reliability went from trying to never crash, to crash as soon as something comes up that you don't have an answer for. Don't recover from memory failures. Don't recover from stack overflow exceptions. Don't recover from broken invariants. Crash instead. I recently posted on our adoption of the fail-fast model in favor of our error recovery model and the immense value we've already gotten from this approach. You can read about that here, "Improving Reliability by Crashing".

Security has seen an uptick in the number of exploits driven by use after free type conditions. Now, years ago a model was introduced to fix this problem, and it was called Smart Pointers. At least in the Windows code base where COM was prevalent this was the case. Even now in the C++ world you can get some automatic memory management using unique_ptr and shared_ptr. So how did that play out? Well, a majority of the code in the district of decay was still written using raw pointers and raw AddRef/Release semantics. So even when the state of the art became Smart Pointers, it wasn't good enough. It wasn't deployed to improve the decay, the decay and rot remained. After all, having a half and half system just means you use the broken 50% of the code to exploit the working 50%.

So basically "smart pointers" written by "smart people" turned out to not be the final conclusion of the state of the art when it came to memory lifetime and use after free. We have much, much better models if we just upgrade all of the code to remove the existing decay and adopt them. You can read about our evolution in thinking as we advanced through "Memory Protector" and on to "MemGC". If you read through both of those articles one commonality you'll find is that they are defense in depth approaches. They do completely eliminate some classes of issues, but they don't provide the same guarantees as an explicitly reference counted system that is perfect in its reference counting. They instead rely on the GC to control only the lifetime of the memory itself but not the lifetime of its objects.

So there is yet another evolution in the state of the art still to come and that is to move the run-time to an actual garbage collector while removing the reference counting semantics that still remain to control object lifetime. Now you can see how rapidly the state of the art evolves.

We've also evolved our state of the art with regard to web interoperability and compatibility. In IE 8 for the first time we introduced document modes after were created a huge mess for the web in IE 7 where our CSS strict mode changed so much from IE 6 that we literally "broke the web". Document modes allowed us to evolve more quickly while providing legacy sites on legacy modes the ability to continue running in our most modern platform. This persisted all the way until IE 11 where we then had 5 different versions of the web platform all living within the same binary and the number of versioning decisions was reaching into the tens of thousands. One aspect of this model was our Browser OM which I've recently documented some aspects of, "JavaScript Type System Evolution", if you are interested in going deeper.

So the state of the art now is EdgeHTML and an Evergreen web platform. This in turn allows us to delete some of the decay, kind of like extracting a cavity, but even this carries so much risk that the process is slow and tedious. The business value of removing inaccessible features (or hard to access features) becomes dependent on what we achieve as a result. So even the evergreen platform can suffer from decay if we aren't careful and still there are systems that undergo much head scratching as we determine whether or not they are necessary. But at least the future here is bright and this leads me to my final state of the art design change.

The industry has moved to a much more agile place of components, open source code, portable code and platform independent code. Components that don't have proper abstractions lack agility and are unable to keep pace with the rest of the industry. There are many state of the art improvements that could apply to this space, but I'm going to call out one that I believe in more than others, and that is replacing OS dependencies, abstractions and interfaces with the principles of Capabilities. And I mean Capabilities in terms of how Joe Duffy refers to them on his recent blogging on the Midori project. I plan on doing a deep session on Capabilities and applying them to the Web Platform in a future article, so I won't dive in further here. This is also a space where, unlike the above 3 state of the art advancements, I don't have much progress I can point to. So I hope to provide insights on how this will evolve the platform moving forward rather than how the platform has already adopted state of the art.

Accepting the Future

When it comes to legacy code we really have to stop digging in our heels. I often find people with the mind set that a given bit of code is correct until it is proven incorrect. They take the current code as gospel and often copy/paste it all over the place, using the existing code as a template for every bit of new code that they write. They often ignore the state of the art, since there aren't any examples in the decaying code they are working on. It is often easier to simply accept the current state of that code than it is to try and upgrade it.

I can't blame them. The costs are high and the team has to be willing to accept the cost. The cost to make the changes. The cost to take the risks. The costs to code review a much larger change due to the transformation. The costs to add or update testing. They have to be supported in this process and not told how smart developers 10 years ago wrote that so it must have been for a reason. If those developers were so smart, they might have thought about writing a comment explaining why they did something that looks absolutely crazy or that a future developer can't understand by simply reading it.

You could talk about how they were following patterns and at the time everyone understood those patterns. Well, then I should have documents on those patterns and I should be able to review those every year to determine if they are still state of the art. But the decay has already kicked in. The documents are gone, the developers who wrote the code don't remember themselves what they wrote or what those patterns mean anymore. Everyone has moved on, but the code hasn't been given the same option. Your costs for that code are increasing exponentially and there is no better time than now to understand it and evolve it to avoid larger costs in the future.

We should also all agree that we are now better than ourselves 10 years ago and its time to let the past go. Assume that we made a mistake back then and see if it needs correcting. If so, correct it. If not, then good job, that bit of code has lived another year.

If you are afraid of the code, then that is even more reason to go after it. I often here things like, "We don't know what might break." Really? That is an acceptable reason? Lack of understanding is something we accept in our industry as a reason to put yellow tape around an area and declare it off limits. Nope. Not a valid reason.

The next argument in this series is that the code is "going away". If its "going away", then who is doing that? Because code doesn't go anywhere by itself. And deleting some code is still an action that a developer has to perform. This is almost always another manifestation of fear towards the code. By noting it will "go away" you defer the decision of what to do with that code. And in software, since teams change all the time, maybe you won't even own the code when the decision is has to be made.

Conclusion

When it comes to evolving your code to the state of the art don't be a blocker. Accept that there are better ways to do whatever you did in the past and always try to be truthful when computing the value proposition when you decide not to upgrade a piece of code. Document that decision well and include a reasoning around what additional conditions would have swung your decision (you owe it to your future self). Schedule a reevaluation of the decision for some time in the future and see if decisions have changed. The worst case is you lose the analysis and the cost of analysis becomes exponential as well. If that happens you will surely cement that decay into place for a long time.

Try to avoid having your ego or the ego's of other make decisions about the quality of code from 10, 5 or even 2 years ago. Tech is one of the fastest evolving industries and it is a self reinforcing process. As individuals it is hard to keep up with this evolution and so we let our own biases and our own history slow advancement when the best thing we can do is getting out of the way of progress. Our experience is still extremely valuable, but we have to apply it towards the right problems. Ensuring that old code stays old is not one of them.

Cleaning up the decay can be part of the culture of your team, it can be part of the process, it can be rewarded or it can just be a dirty job that nobody wants to do. How much decay you have in your system will be entirely dependent on how well you articulate the importance of cleaning it up. The distance between state of the art and state of decay can be measured in months so it can't be ignored. You probably have more decay than you think and if you don't have a process in place to identify it and allocate resources to fix it, it is probably costing you far more than you realize.

Improving Web Debugging with insights from the Native Debugger

Modern web browsers provide some very capable debugging tools for JavaScript. The ease of use of many of the features matches the ease of use of the language. Most features are accessible and controllable with intuitive UI. However, for the power user coming from native languages, there are some very useful yet missing features. Given the features of the JavaScript language there is the possibility of making even better hybrid use cases, some of which I would like to explore in this post.

My focus will be on breakpoints. Breakpoints in JavaScript debuggers come from either the debugger statement or by selecting lines of code in your script from the UI. Since JavaScript is complicated in that it can perform many operations in a line of code it has taken a while for debuggers to be able to break at the proper time in the evaluation of a statement. Usually it breaks at the beginning and you can kind of step through to get to the point you want. But if you are evaluating many hits of the same breakpoint this can be tedious. An advanced user might use conditional breakpoints, but depending on when the break happens you might find you are too early or too late to evaluate your condition appropriately. All of this aside, we find our way through the muck and manage to make it work.

In native, breakpoints are about code addresses coming from the symbolic debug information emitted with the program. Your debugger will either allow you to set breakpoints based on a direct address, or you can try to evaluate a symbol instead and obtain an address. Many debugging sessions in C++ start with a script or two to set up a bunch of breakpoints for a set of key functions. This is woefully missing from the current set of JavaScript based debuggers.

Another type of native breakpoint that is very handy is the memory breakpoint. You can set a breakpoint for when a value of code is read from or written to or both. This is also based on addresses and so these breakpoints are per instance. When you have thousands of instances this can be extremely handy.

So that is our list. We want symbolic breakpoints and memory breakpoints. We probably also want an easier way to run "debugger" scripts than typing into the console. Now, how native works and how JavaScript works is quite different so we can improve on some of these facilities as well as we describe them. Each section following will be broken down into a basic description, a set of challenges and a set of solutions/improvements. Not every challenge will have a solution provided unless I'm exceptionally on point today.

Symbolic Breakpoints

We want the ability to, given a name of a function and potentially the name of a script or HTML page (module), set a breakpoint on that function when we enter it. This is the same as in the native debugger where I might use a command like:

bp foobarbaz!MyClass::MyFunction

The native debugger also allows for things like deferred breakpoints (bu) which match modules not yet loaded. We can set multiple breakpoints using a wildcard match and the symbolic breakpoint command (bm). Additionally, with some clever scripting we can choose to debug the call in or the return value by using some debugger commands to step out of my function and consult, generally, the @eax/@rax registers.

bp foobarbaz!MyClass::MyFunction "gu; r @rax;"

Challenges

That is a pretty cool feature list. So what are some of the challenges? Well, JavaScript doesn't have symbols. It has function objects and function objects have code. Many function objects don't even have names. So how do you set breakpoints on those things? Interesting problem, we'll get to that in solutions.

Forgetting that many things aren't named, many things are. Every function has a "name" property and so you could imagine the debugger could set a breakpoint on anything with a name. It could also act like the native debugger if more than one exists and give you some sort of handle that you can set the breakpoint on in the case of say 15 functions with exactly the same name.

Modules are rough as well. For instance, you have multiple scripts in a single HTML file or you have dynamically generated names on the server etc... So how to match modules will be an interesting issue.

Speaking of modules, what about script contexts? Isn't that like multiple processes? Have you ever done multi-process debugging in something like WinDBG or VS? Not fun I can say. You set breakpoints in only the current process and when you break in and out of the debugger you have to constantly figure out your context to know what is going on and whether or not you are setting breakpoints in the right space. With things like iframes kind of acting like processes (or maybe physically being in another process in the case of cross-domain iframe isolation) this problem is basically something that has to be solved.

Solutions

Thankfully most of these have some immediate solutions. I'm going to tackle the symbolic issues first. For this to work you have to treat functions like objects. Functions are themselves breakpoint handles so I should be able to set a breakpoint based on a function object itself and this will be a function instance breakpoint.

I shouldn't stop there though. I also want symbolic names to work and even have deferred breakpoints for functions that don't exist yet. So for that, anytime a function is evaluated I will compare its name against my list of symbol expressions. A function might even change its name (not currently, but I don't see any reason why it shouldn't be able to), so while I only have to evaluate these breakpoints against a given function once, if the name changes, I'll have to invalidate the cache and reevaluate on next execution.

This in turn gives us the equivalent of bp, bu and even bm. You could imagine that bm would take a JavaScript RegExp object or string and so I could write some quite complicated function matching breakpoints. This would be pretty cool. To extend this we allow for each command to run on either the execution or return of the function. And of course, we supply appropriate UI so that on the return you can review the return value in a special variable window in a super easy way. Also, if you want to write conditional breakpoints we'd surface the return value in a special syntax so you can query it. Maybe, you want to only break when function foo returns true, the breakpoint/conditional for that might be...

debugger.breakPointReturn(null, /^foo$/, "@retVal === true");

The solution for modules I really don't have a good answer for. What I think it means is that you want the debugger to tell you about these modules, but then you want to either have positive or negative module matching as part of the breakpoint condition. For this reason I would actually make my breakpoint API take both module and symbol as separate values. This is different from native in that it only takes the full string and uses the ! character to split on. Since JavaScript has different identifier characters, building some like this would be somewhat foolish. Also, with WebAssembly in the future, we probably can't make guesses as to what is and is not valid for identifiers.

Cool, so the last solution we need is how to handle this problem of multiple global scopes. This is where the debugger can shine. I should be able to set breakpoints constrained to a specific global scope, constrained to a specific "module" as defined by the html or script name of the document where the source came from, or even set breakpoints generally across the entire environment. I think that in most cases users would be happy with breaking on anything, anywhere given a specific name.

Earlier we allowed you to set a breakpoint on a given function object. That is highly constrained and doesn't have the multiple global scopes problem so long as you tagged the right function.

Finally we get OM function breakpoints out of this as well. Since the OM is defined as a series of JS functions in all browsers now you can use them, or their symbolic names, to match things. We could even allow the constructor name + member name syntax to be very precise, and this would alleviate problems with minified scripts where the names are encoded up to the point that the function is executed or when function objects are used instead of function names. All of that obfuscation is now "debuggable". That would be super cool.

I'm pretty much freaking out this coffee shop with all of my excitement at this point. I really want these features. Just these breakpoints alone would simplify tons and tons of different debugging experiences that I deal with on a daily basis.

Memory Breakpoints

There is already a useful bit of this in the platform today in Observer.observe. This is some ES 7(?) syntax that allows you to be notified when changes to an object are made. I believe that the callbacks run in micro-task checkpoints, but don't quote me on this I could be completely wrong.

Memory breakpoints are bit more intrusive. They break on read or write depending on how they are configured. They also occur at a time where you have both the old value and new value waiting there so you can choose to "stop" the store or "update" the store if you'd like. This can be use to say tweak out a scenario where you believe a wrong value is breaking your site. Simply memory breakpoint, set the corrected value, and let the page continue on to success.

We probably also want to allow instance based or Constructor based syntax for this. Having all constructed instances of a given type have the breakpoint on automatically would be more useful than having to pick out specific instances. This is something lacking in the native debugger unless you do something really clever with multiple breakpoints.

Thankfully the feature list for this one is shorter than the last, but we still have some key challenges to overcome.

Challenges

Memory breakpoints are going to offer mostly performance challenges, but also some challenges to JIT'ed code. While accessing storage locations can be easily dealt with in the interpreter and even some not so well optimized JIT code, once something becomes a simple mov instruction in the assembly with no other overhead, evaluating the memory breakpoint can be challenging. So then the question becomes, do you have to immediately disable a ton of optimizations the moment this is turned on? I think the answer is yes, which is unfortunate. Nothing like super slowing your code when you are trying to repro a bug.

Beyond this JavaScript has fields and properties. Where properties are a bit harder to manage. They do indicate a "slot" on the object that is accessed, but they may also represent a "slot" in your prototype chain instead. Should memory breakpoints only operate on instance fields or should they extend to property getter/setter pairs. I think since you can set function breakpoints, restricting to fields might be okay. Though this does reduce the effectiveness of the feature.

Finally, the Constructor approach can be challenging. Every Global has its own set of Constructors. So if you set a Constructor memory breakpoint it would only apply to instances in the current global. This is similar to using function objects versus symbolic names. You could overcome this by trying to set a symbolic name memory breakpoint on EVERY instance, but that overhead seems way too larger as well. Ultimately the uses cases would define the requirements for this feature and whether or not the costs on runtime are justified by the debugging efficiency gained by the web developers.

Solutions

We inlined most of the solutions, but lets start again. First, we realize that memory breakpoints would likely disable some of the JIT'ed functional execution. This is generally okay since most engines are set up for this type of fallback already. They can fallback to less optimized code or even back to interpretation if necessary. We'd have some target for execution speed, such as no slower than executing at interpreted speed.

We'd restrict to fields since properties are "functions" and not really fields anyway. A property might also end up setting or retrieving from a field so you could still see that access if you wanted to. This would have an impact on OM breakpoints where properties are the way something like Element.className works, disallowing you from watching that property. You could use OM symbolic breakpoints with conditional logic to overcome this.

I think tagging a constructor so that any object returned as a result of new on that function were automatically breakpoint'ed is just super cool. I don't think there are many challenges there once the general infrastructure is in place. I'm curious if others would find that useful though. For browser built-ins the multiple Globals could probably be overcome as well if that proved to be an issue.

Conclusion

Curious if these enhancements would be useful to web developers or not. This could be a case of me solving my own problems. I often use native debugging in order to quickly figure out website problems and reduce the causes for broken websites. In these cases I don't have access to the original source nor can I easily make changes. But also, its a reflection of how we work in native debugging in general. The tools are so powerful that we can basically rewrite the code in place, even down to the assembler, so putting in logging and other types of solutions then rebuilding to try again is completely unnecessary.

If you have any debugging scenarios or stories where these features would help leave a comment or reach me on Twitter @JustRogDigiTec. I'd love to hear that these facilities might be useful beyond my own use cases.

Fixing Web Interoperability with Testing

For an old and mature API surface area such as HTML 5 you would think that it would be relatively well tested. We've put years of effort into writing tests, either as browser vendors, or maybe just as sites or frameworks to make sure that our stuff works. But rarely do these efforts scale to the true size of the API set, its complications, its standardization, its interop across browsers and all of the quirks that exist that the web is reliant on.

However, as judged by the numerous differences between browsers, it is pretty clear there are no canonical tests. Tests for the web platform, if you assumed all browsers were correct in their own way, would have tens of acceptable results. Developers would have to be aware of these different results and try to code in a way that allowed all such possibilities. Alas, we don't have tests like this and developers only write to one given implementation.

What does Interop Look Like Today?

Currently Interop between browsers is decided by site reports and bug fixes against those site reports. It means reductions of millions of lines of JavaScript, HTML and CSS to figure out what might be causing the problem for the given site.

Depending on which browser vendor is trying to match another, those bug fixes may or may not be good for the web. Arguably implementing a bunch of webkit prefixes is great to make sites work, users might even be happy as well, but it reduces the uptake on the standards versions of those same APIs. To the extent, that a relatively well positioned webkit API might actually need to be in the standard and marked as an alias for the standard itself. I'm not picking on webkit prefixes here either, it just so happens that they were more popular and so became the mind share leaders.

So browser vendors rely on site reporting and mass per site testing, analysis of complicated scripts and reductions and potentially silent bug fixes to arrive at some sort of interoperable middle ground where most existing sites just work, most newly written sites are probably portable, and the web has multiple different ways of doing the same thing. This creates a complicated, sparsely tested API, with many developer pitfalls. Current web interop really isn't looking so good.

But, are tests really going to make a difference?

Web Testing Statistics

I am about to throw out numbers. These numbers are based on real testing, aggregation, and profiling, however, they are not precise. We have not exhaustively tested our own collection methodology to ensure that we aren't missing something. But given the size of the numbers we believe these to be fairly statistically accurate. I'll apply an assumption that I have a +/- 5% error in the data I'm about to present.

Also, these are about tests that we run against EdgeHTML. Which also means the data collected is about how these tests work in EdgeHTML. It is possible that numbers swing and change as features are detected in different browsers. Okay, you probably just want the numbers ;-)

Test Set	Unique APIs	Comments
Top 10000 WebSites	~2300	Site Crawler live data, not testing
EdgeHTML Regression Suite	~2500	Filtered, incidental usage not counted
Top WebDriver Enabled Suites	~2200	Web Driver enabled, any browser can run
CSS 2.1 Suite	~70	Not an API testing suites

But these numbers don't mean anything, how many APIs are there? I can't give the number for every browser, but I can tell you that EdgeHTML is actually on the low side due to how our type system works. We've adopted an HTML 5 compliant type system earlier than some of the other browsers. We've also worked hard to reduce vendor prefixes. For this reason, Safari, may have over 9k APIs detectable in the DOM. FireFox, which has a high level of standards compliance, will often have may moz prefixed properties and therefore their numbers will be elevated as well. 

But for EdgeHTML the statistics are that we have ~4300 APIs and ~6200 entry points. The entry points is important since these represent things that aren't necessarily visible to the web developer. The way we do callable objects (objects that behave like functions) for instance. Or the fact that read-write properties have both a getter and a setter which must be tested.

The numbers we collected on testing have been aggregated to APIs, but we also keep the data on entry-points. The entry-point data always results in a lower percentage of testing, since not testing a getter or setter can only take away from your percentage it can't add to it.

So what if we added up all of the unique coverage? If we do that across the suites, then we end up with ~3400 APIs covered. That's not bad, we are starting to show that the "used web" might heavily intersect in that juncture right? Okay, lets run a few more numbers then.

What about unique to the top 10000 web sites? Turns out there are ~600 APIs in that group. Ouch, that could be an interop nightmare. If we instead use just publicly available testing (and remove our proprietary testing) the numbers jump up significantly. We are now over 1200 APIs that are hit by live sites which are not tested by public test suites... Ugh.

Confidence in Existing Tests

So we've talked a lot about the testing that does and does not exist, but what about the confidence we apply to the existing testing. This is going to be less data driven since this data is very hard for us to mine properly. It is, in fact, a sad truth that our tracking capabilities are currently lacking in this area and it creates a nasty blind spot. I'm not aware of this state for other browser vendors, but I do know that the Blink project does a lot of bug tagging and management that I would argue is superior to our own. So kudos to them ;-)

So notice that the EdgeHTML test suite is considered a regression suite. This is an evolution of testing that started likely 20 years ago. While we have added some comprehensive testing for new features, many older features, the old DOM if you will, only has basic testing which is mostly focused on verifying the API basically works, that it most certainly doesn't crash and often contains some set of tests that were previously bugs in the API that we eventually fixed and created a "regression" test for. In IE 9, which is where this legacy DOM suite forks into a legacy and modern suite, we carry over about half of our entry points. I have numbers from this period that range between 3100 and 3400 entry points depending on the document mode.

Given we mostly have regression suites, we also find that many APIs are only hit a small number of times. In fact, our initial test numbers were quite high, around 60% entry point coverage, until we factored out incidental hits. Once the filtering was employed we were back down at 30% and that is a high amount, since there is no perfect way to filter out incidental usage.

This all combines to make my confidence in our regression suite at about 50/50. We know that many of the tests are high value because they represent actual bugs we found on the web in how websites are using our APIs. However, many of them are now more than 5, 10 or even 15 years old.

What about the public web-test suites that we covered? Well, that I'm going to also call 50/50 because we aren't really seeing that those suites touch the same APIs or test the APIs that the web actually uses. I mean, 1200 APIs not hit by those suites is pretty telling. Note, we do add more public test suites to our runs all the time, and quality of a suite is not purely about how many APIs they test. There is a lot of value, for instance, in a suite that only tests one thing extremely well. This requires that you grade the suites quality through inspection. This is a step we have not done. We rely on jQuery and other major frameworks to vet their own suites and provide us with a rough approximation of something that matches their requirements.

So in all, given that our existing suites aren't very exhaustive, they often test very old functionality, and there are a lot of missing APIs in the suites that are used by top sites, I'd subjectively rank the current test suites in my low confidence category. How can we get a medium to high confidence suite, that is cross vendor/device capable, and if we could what would it look like?

Tools for a Public Suite

We already have some public suites available. What makes a suite public? What allows it to be run against multiple browsers? Let's look at some of the tools we have available for this.

First and foremost we have WebDriver, a public standard API set for launching and manipulating a browser session. The WebDriver protocol has commands for things like synthetic input, executing script in the target and getting information back from the remote page. This is a great start for automating browsers in the most basic ways. It is not yet ready to test more complicated things that are more OS specific. For instance, synthetic input is not the same as OS level input. Even OS level synthetic input is not the same as input from a driver within the OS. Often times we have to test deeply to verify that our dependency stacks work. However, the more we do this, the less likely the test is going to work in multiple browsers anyway. Expect advancements in this area over the coming months and years as web testing gets increasingly more sophisticated and WebDriver evolves as a result.

But WebDriver can really test anything in an abstract way. In fact, it could test an application like Notepad if there were a server for Notepad that could create sessions and launch instances. Something that generic doesn't sound like a proper tool, in fact it is just part of a toolchain. To make WebDriver more valuable we need to specify tests in ways that anyone's WebDriver implementation can pick them up, run them, and report results.

For this we have another technology called TestHarness.js. This is a set of functionality that when imported into your HTML pages allows them to define suites of tests. By swapping in a reporting script, the automation can then change the reporting to suit its own needs. In this way you write to a test API by obeying the TestHarness.js contract, but then you expect that someone running your test will likely replace these scripts with their own versions so that they can capture the results.

The final bit of the puzzle is enabling tests to be broadly available and usable. Thankfully there is a great, public, open source solution to this problem in GitHub. By placing your tests in a public repository and using TestHarness.js, you can be rest assured that a browser vendor can clone, run and report on your test suites. If your test suite is valuable enough it can be run by a browser vendor against their builds, not only protecting them from breaking the web, but protecting your from being broken. For large frameworks in broad usage across the web this is probably the best protection you could have to ensure your libraries continue to work as the web evolves.

The Vendor Suite

Hopefully the previous section has replaced any despair that I might have started at the beginning of the article with some Hope instead. Because I'm not hear to bum anyone out. I'm hear to fix the problem and accelerate the convergence of inteoperability.

To this end, my proposal is what I call the vendor suite. And note, I call it this, but it doesn't matter if a vendor releases this suite or someone in the community, but it has to be informed by the vendors and use data from the actual browsers in order to achieve a level of completeness the likes of which we have never seen. Also, these are my own opinions, not those of the my entire team which increases the likelihood that a large undertaking such as this may be something driven by the community itself and in turn be supported by the vendors.

The qualities of the vendor suite are that it accomplishes the following immediate goals...

1. We define tests in 3 levels
1. API - Individual API tests, where each API has some traits and some amount of automated and manually created testing is involved. The level of testing is at a single API, and the number of tests is dependent on the complexity of the API itself.
2. Suite - A more exhaustive set of tests which focus on a specification or feature area. These suites could accompany a specification and be part of their contribution to the standards process.
3. Ad Hoc - Tests which represent obscure, but important to test behaviors. Perhaps these behaviors are only relevant to a given device or legacy browser. Potentially their goal is to "fail" on a compliant and interoperable browser.
2. Tests should be written to verify the interoperable behavior until all browsers agree to transition to the standards behavior if their is a discrepancy. Interoperability is the primary goal. Do websites work? Not, does a browser adhere to a spec for which no web developer is pining.
3. As much testing should be driven by traits as possible.
1. Attributes - At an API level the behavior of a property - attribute pair will be generically tested. Examples of common behavior are how the property - attribute reflect one another. Also, how MutationObserver's are fired.
2. Due to traits based testing we can verify down to the ES 6 level the configuration of the property. This allows auto-generation of testing for read-only, "use strict" adherence, along with other flags such as enumerability and configurability.
3. Tests should be able to be read and rewritten while maintaining any hand-coded or manually added value. Things like throwing exceptions are often edge cases and would need someone to specifically test for those behaviors.
4. No vendor will pass the suite. It is simply not possible. For example:
1. Vendor prefixed APIs will be tested for.
2. Hundreds if not thousands of vendor specific APIs, device configurations, scenario configurations, etc... may be present.
5. To this end, the baselines and cross comparisons between browsers are what is being tested for. Vendors can use the results to help drive convergence and start talks on what we want the final behavior to be. As we align to those behaviors the tests can be updated to reflect our progress.

Having such a large scale, combined effort is really how we start converging the web. This is how we all start quickly working through obscure differences with a publicly visible suite, publicly contributed to, and to be honest, prioritized by you, the community. With such obscure differences out of the way it frees up our resources to deliver new web specifications and further achieve convergence on new APIs much more quickly. 

Final Thoughts

So what do you think? Crazy idea? Tried a thousand times? Have examples of existing collateral that you think accomplishes this goal? I'm open to all form of comment and criticism. Any improvements to the web that let me get out of the business of reducing web pages and finding issues earlier is going to be a huge win for me regardless of how we accomplish it. I want out of the business of trying to understand how a site is constructed. There are far more web developers than there are browser developers so the current solutions simply don't scale in my opinion.

As another thought, what about this article is interesting to you? For instance, are the numbers surprising or about what you thought in terms of how tested the web is? Would you be interested in understanding or seeing these numbers presented in a more reliable and public way?