Better Blogs Are Harder To Find

As the web grows in popularity and power, so does the amount of data stored and transferred between systems, many of which know nothing about each other. From early on, the format that this data was transferred in mattered, and like the web, the best formats were open standards that anyone could use and contribute to. XML gained early popularity, as it looked like HTML, the foundation of the web. But it was clunky and confusing. 

That’s where JSON (JavaScript Object Notation) comes in. If you’ve consumed an API in the last five to ten years, you’ve probably seen JSON data. While the format was first developed in the early 2000s, it really took off as the major format after its first standards were published in 2013. Understanding what JSON is and how it works is a foundational skill for any web developer. 

In this article, we’ll cover the basics of what JSON looks like and how to use it in your web applications, as well as talk about serialized JSON—JST and JWT—and the competing data formats. 

What JSON looks like

JSON is a human-readable format for storing and transmitting data. As the name implies, it was originally developed for JavaScript, but can be used in any language and is very popular in web applications. The basic structure is built from one or more keys and values:

{ key: value }

  You’ll often see a collection of key:value pairs enclosed in brackets described as a JSON object. While the key is an alphanumeric string, the value can be a string, number, array, additional object, or the literals, false, true and null. For example, the following is valid JSON:

{ key: "String", Number: 1, array: [1,2,3], nested: { literals: true } }

Using  JSON in API calls

One of the most common uses for JSON is when using an API, both in requests and responses. It is much more compact than other standards and allows for easy consumption in web browsers as JavaScript can easily parse JSON strings, only requiring JSON.parse() to start using it.

JSON.parse(string) takes a string of valid JSON and returns a JavaScript object. For example, it can be called on the body of an API response to give you a usable object. The inverse of this function is JSON.stringify(object) which takes a JavaScript object and returns a string of JSON, which can then be transmitted in an API request or response. 

JSON isn’t required by REST or GraphQL, both very popular API formats. However, they are often used together, particularly with GraphQL, where it is best practice to use JSON due to it being small and mostly text. If necessary, it compresses very well with GZIP.

GraphQL’s requests aren’t made in JSON, instead using a system that resembles JSON, like this

{ foo{ bar baz } }

Which will return the relevant data, and if using JSON, it will match very closely:

{ foo:{ bar: "data", baz: "data" } }

Using JSON files in JavaScript

In some cases, you may want to load JSON from a file, such as for configuration files or mock data. Using pure JavaScript, it currently isn’t possible to import a JSON file, however a proposal has been created to allow this. In addition, it is a very common feature in bundlers and compilers, like webpack and Babel. Currently, you can get equivalent functionality by exporting a JavaScript Object the same as your desired JSON from a JavaScript file.

export const data = {foo: "bar"}

Now this object will be stored in the constant, data, and will be accessible throughout your application using import or require statements. Note that this will import a copy of the data, so modifying the object won’t write the data back to the file or allow the modified data to be used in other files. 

Accessing and modifying JavaScript objects

Once you have a variable containing your data, in this example data, to access a key’s value inside it, you could use either data.key or data[“key”]. Square brackets must be used for array indexing; for example if that value was an array, you could do data.key[0], but data.key.0 wouldn’t work.

Object modification works in the same way. You can just set data.key = "foo" and that key will now have the value “foo”. Although only the final element in the chain of objects can be replaced; for example if you tried to set data.key.foo.bar to something, it would fail as you would first have to set data.key.foo to an object.

Comparison to YAML and XML

JSON isn’t the only web-friendly data standard out there. The major competitor for JSON in APIs is XML. Instead of the following JSON:

{ "hello": "world" }

in XML, you’d instead have:

<?xml version="1.0" encoding="UTF-8" ?> <hello>world</hello>

JSON was standardized much later than XML, with the specification for XML coming in 1998, whereas Ecma International standardized JSON in 2013. XML was extremely popular and seen in standards such as AJAX (Asynchronous JavaScript and XML) and the XMLHttpRequest function in JavaScript.

XML is part of a major API standard, Simple Object Access Protocol (SOAP). This standard can be significantly more verbose than REST and GraphQL, in part due to the usage of XML and because the standard includes more information, such as describing the XML namespace as part of the envelope system. This might be a reason why SOAP usage has declined for years.

From https://ift.tt/8Jv1dlh.

Another alternative is YAML, which is much more similar in length to JSON compared to XML, with the same example being:

--- hello: world

However, unlike XML, YAML doesn’t really compete with JSON as an API data format. Instead, it’s primarily used for configuration files—Kubernetes primarily uses YAML to configure infrastructure. YAML offers features that JSON doesn’t have, such as comments. Unlike JSON and XML, browsers cannot parse YAML, so a parser would need to be added as a library if you want to use YAML for data interchange.

Serialized JSON

While many of JSONs use cases transmit it as clear text, the format can be used for secure data transfers as well. JSON web signatures (JWS) are JSON objects securely  signed using either a secret or a public/private key pair. These are composed of a header, payload, and signature. 

The header specifies the type of token and the signing algorithm being used. The only required field is alg to specify the encryption algorithm used, but many other keys can be included, such as typ for the type of signature it is. 

The payload of a JWS is the information being transmitted and doesn’t need to be formatted in JSON though commonly is. 

The signature is constructed by applying the encryption algorithm specified in the header to the base64 versions of the header and payload joined by a dot. The final JWS is then the base64 header, base64 payload, and signature joined by dots. For example:

eyJ0eXAiOiJKV1QiLA0KICJhbGciOiJIUzI1NiJ9.eyJpc3MiOiJqb2UiLA0KICJleHAiOjEzMDA4MTkzODAsDQogImh0dHA6Ly9leGFtcGxlLmNvbS9pc19yb290Ijp0cnVlfQ.dBjftJeZ4CVP-mB92K27uhbUJU1p1r_wW1gFWFOEjXk

JSON Web Tokens (JWT) are a special form of a JWS. These are particularly useful for authorization: when a user logs into a website, they will be provided with a JWT. For each subsequent request, they will include this token as a bearer token in the authorization header.

 To create a JWT from a JWS, you’ll need to configure each section specifically. In the header, ensure that the typ key is JWT. For the alg key, the options of HS256 (HMAC SHA-256) and none (unencrypted) must be supported by the authorization server in order to be a conforming JWT implementation, so can always be used. Additional algorithms are recommended but not enforced.

In the payload are a series of keys called claims, which are pieces of information about a subject, as JWTs are most commonly used for authentication, this is commonly a user, but could be anything when used for exchanging information. 

The signature is then constructed in the same way as all other JWSs.

Compared to Security Assertion Markup Language Tokens (SAML), a similar standard that uses XML, JSON allows for JWTs to be smaller than SAML tokens and is easier to parse due to the use of both tokens in the browser, where JavaScript is the primary language, and can easily parse JSON.

Conclusion

JSON has come to be one of the most popular standards for data interchange, being easy for humans to read while being lightweight to ensure small transmission size. Its success has also been caused by it being equivalent to JavaScript objects, making it simple to process in web frontends. However, JSON isn’t the solution for everything, and alternate standards like YAML are more popular for things like configuration files, so it’s important to consider your purpose before choosing. 

The post A beginner’s guide to JSON, the data format for the internet appeared first on Stack Overflow Blog.

While blockchains are huge right now, finding one to build on that doesn’t use a ton of energy, has good privacy protections, and operates efficiently is harder than it looks. The original breakout blockchain, Bitcoin, was slow to adopt any innovations coming out of research. Other blockchains use the electricity of a small country to play elaborate gambling games. For someone looking to build the future of Web3, what are your options?

On this sponsored episode of the podcast, we talk to Tezos co-founder Arthur Brietman. After finding out that the Bitcoin blockchain wouldn’t incorporate all the good ideas generated around it—proof of stake, privacy improvements, and smart contracts to name a few—he decided to build his own. 

Arthur has a background in machine learning and statistics but spent his early 20s teaching self-driving cars how to turn left and working in quantitative finance for high-frequency trading. High-frequency trading was data-driven, but there was so much noise that machine learning didn’t do very well. Self-driving cars, meanwhile, presented a more structured problem, so neural networks could yield good results. 

Around that time, Arthur got bit by the crypto bug. It lived at the intersection of a lot of his interests: Cryptography touched on computer science and math, but his time in finance got him wondering about banks and money work. The idea of individual sovereignty scratched a personal philosophical itch. 

Naturally, Arthur decided to try some mining software. It took all of his computer’s resources, so he uninstalled it. But after seeing the price of Bitcoin break a dollar and other news items about it, he looked closer. He started to think about what a company could do if it didn’t have to maintain banking relationships. He thought about possible applications, like decentralized poker. 

When Bitcoin refused to adopt the improvements developed by competing alt coins, Arthur started thinking about a new blockchain that would respond to new developments and focus on efficient processing, security, and a good smart contract system. Forking the code wasn’t enough; he needed a new ledger. 

That’s when Tezos was born. It was initially built by a small team of OCaml programmers using that language’s functional subset. Arthur was inspired by the example of WhatsApp, which was built by a small team of senior Erlang engineers. While OCaml would limit the talent he could hire, it would be a very efficient way to build an error-free transaction system. He could have built the whole thing in Java, sure, but Arthur estimates that it would have cost a whole lot more. If you’re interested in learning more about what an engineer’s blockchain ecosystem looks like, check out the Tezos home page. Discover building on Tezos: https://tezos.com/build/

The post Talking blockchain, functional programming, and the future with Tezos co-creator Arthur Breitman appeared first on Stack Overflow Blog.

Identifying causal effects is an integral part of scientific inquiry. It helps us understand everything from educational outcomes to the effects of social policies to risk factors for diseases. Questions of cause-and-effect are also critical for the design and data-driven evaluation of many technological systems we build today. 

To help data scientists better understand and deploy causal inference, Microsoft researchers built a tool that implements the process of causal inference analysis from end to end. The ensuing DoWhy library has been doing just that since 2018 and has cultivated a community devoted to applying causal inference principles in data science. To broaden access to this critical knowledge base, DoWhy is migrating to an independent open-source governance model in a new PyWhy GitHub organization. As a first step toward this model, we are announcing a collaboration with Amazon Web Services (AWS), which is contributing new technology based on structural causal models. 

What is causal inference?

The goal of conventional machine learning methods is to predict an outcome. In contrast, causal inference focuses on the effect of a decision or action—that is, the difference between the outcome if an action is completed versus not completed. For example, consider a public utility company seeking to reduce their customers’ usage of water through a marketing and rewards program. The effectiveness of a rewards program is difficult to ascertain, as any decrease in water usage by participating customers is confounded with their choice to participate in the program. If we observe that a rewards program member uses less water, how do we know whether it is the program that is incentivizing their lower water usage or if customers who were already planning to reduce water usage also chose to join the program? Given information about the drivers of customer behavior, causal methods can disentangle confounding factors and identify the effect of this rewards program. 

Figure 1: A public utility introduces a program that rewards water usage reduction. Are people who sign up using less water than they would have otherwise? 

How do we know when we have the right answer? The effect of an action like signing up for a customer loyalty program is typically not an observable value. For any given customer, we see only one of the two respective outcomes and cannot directly observe the difference the program made. This means that processes developed to validate conventional machine learning models—based on comparing predictions to observed, ground truths—cannot be used. Instead, we need new processes to gain confidence in the reliability of causal inference. Most critically, we need to capture our domain knowledge, reason about our modeling choices, then validate our core assumptions when possible and analyze the sensitivity of our results to violations of assumptions when validation is not possible. 

Four steps of causal inference analysis

Data scientists just beginning to explore causal inference are most challenged by the new modeling assumptions of causal methods. DoWhy can help them understand and implement the process. The library focuses on the four steps of an end-to-end causal inference analysis, which are discussed in detail in a previous paper, DoWhy: an End-to-End Library for Causal Inference, and related blog post: 

Modeling: Causal reasoning begins with the creation of a clear model of the causal assumptions being made. This involves documenting what is known about the data generating process and mechanisms. To get a valid answer to our cause-and-effect questions, we must be explicit about what we already know. 

Identification: Next, we use the model to decide whether the causal question can be answered, and we provide the required expression to be computed. Identification is the process of analyzing our model. 

Estimation: Once we have a strategy for identifying the causal effect, we can choose from several different statistical and machine learning-based estimation methods to answer our causal question. Estimation is the process of analyzing our data. 

Refutation: Once we have our answer, we must do everything we can to test our underlying assumptions. Is our model consistent with the data? How sensitive is the answer to the assumptions made? If the model missed an unobserved confounder, will that change our answer a little or a lot? 

This focus on the four steps of the end-to-end causal inference process differentiates the DoWhy library from prior causal inference toolkits. DoWhy complements other libraries—which focus on individual steps—and offers users the benefits of those libraries in a seamless, unified API. For example, for estimation, DoWhy offers the ability to call out to Microsoft’s EconML library for its advanced estimation methods. 

Current DoWhy deployments

Today, DoWhy has been installed over one million times. It is widely deployed in production scenarios across industry and academia—from evaluating the effects of customer loyalty and marketing programs to identifying the controllable drivers of key business metrics. DoWhy’s rich API has enabled the creation of downstream solutions such as AutoCausality from Wise.com, which automates comparison of different methods, and ShowWhy from Microsoft, which provides a no-code GUI experience for causal inference analysis. In academia, DoWhy has been used in a range of research scenarios, including sustainable building design, environmental data analyses, and health studies. At Microsoft, we continue to use DoWhy to power causal analyses and test their validity, for example, estimating who benefits most from messages to avoid overcommunicating to large groups. 

A community of more than 40 researchers and developers continually enrich the library with critical additions. Highly impactful contributions, such as customizable backdoor criterion implementation and a user-friendly Pandas integration, have come from external contributors. Instructors in courses and workshops around the world use DoWhy as a pedagogical tool to teach causal inference. 

With such broad support, DoWhy continues to improve and expand. In addition to more complete implementations of identification algorithms and new sensitivity analysis methods, DoWhy has added experimental support for causal discovery and more powerful methods for testing the validity of a causal estimate. Using the four steps as a set of fundamental operations for causal analysis, DoWhy is now expanding into other tasks, such as representation learning. 

Microsoft continues to expand the frontiers of causal learning through its research initiatives, with new approaches to robust learning, statistical advances for causal estimation, deep learning-based methods for end-to-end causal discovery and inference, and investigations into how causal learning can help with fairness, explainability and interpretability of machine learning models. As each of these technologies mature, we expect to make them available to the broader causal community through open source and product offerings. 

An independent organization for DoWhy and other open-source causal inference projects

Making causality a pillar of data science practice requires an even broader, collaborative effort to create a standardized foundation for our industry. 

To this end, we are happy to announce that we are shifting DoWhy into an independent open-source governance model, in a new PyWhy effort. 

The mission of PyWhy is to build an open-source ecosystem for causal machine learning that advances the state of the art and makes it available to practitioners and researchers. In PyWhy, we will build and host interoperable libraries, tools, and other resources spanning a variety of causal tasks and applications, connected through a common API on foundational causal operations and a focus on the end-to-end analysis process.

Our first collaborator in this initiative is AWS, which is contributing new technology for causal attribution based on a structural causal model that complements DoWhy’s current functionalities. 

We are looking forward to accelerating and broadening adoption of our open-source causal learning tools through this new Github organization. We invite data scientists, researchers, and engineers, whether you are just learning about causality or already designing new algorithms or even building your own tools, to join us on the open-source journey towards building a useful causal analysis ecosystem. 

We encourage you to explore DoWhy and invite you to contact us to learn more. We are excited by what lies ahead as we aim to transform data science practice to drive improved modeling and decision making. 

The post DoWhy evolves to independent PyWhy model to help causal inference grow appeared first on Microsoft Research.

We chat with Jason Warner, former CTO of Github, who has taken his experience in software development and engineering management to the world of venture capital.

Episode Notes

Jason is now a managing director at Redpoint Ventures and has led one investment so far, backing a company called Alchemy that is focused on infrastructure and dev tools for web3.

He describes himself as a “very average” programmer, but an excellent engineer, and explains how he parlayed his unique skill set into key roles at Heroku and Github.

Our lifeboat for the week goes to dfrib for suggestion a solution to: Error “nil requires a contextual type” using Swift

This week’s episode is sponsored by UiPath:

UiPath 2022.4 release brings automation access for all. Learn new skills, focus on critical thinking, and enjoy value-added work. We welcome Robots on Mac, semantic automation through Clipboard Aim, a new Attended Framework and more. Read more here.

The post How an average programmer became Github’s CTO appeared first on Stack Overflow Blog.

Lapses such as racially biased facial recognition or apparently sexist credit card approval algorithms have thankfully left companies asking how to build AI ethically. Many companies have released “ethical AI” guidelines, such as Microsoft’s Responsible AI principles, which requires that AI systems be fair, inclusive, reliable and safe, transparent, respect privacy and security, and be accountable. These are laudable, and will help prevent the harms listed above. But often, this isn’t enough.

When the harm is inherent in how the algorithm is used

Harm can result from what a system is used for, not from unfairness, black-boxyness, or other  implementation details. Consider an autonomous Uber: if they are able to recognize people using wheelchairs less accurately than people walking, this can be fixed by using training data reflective of the many ways people traverse a city to build a more fair system. 

But even with this inequity removed, some people may believe the intended use of the algorithm will cause harm: the system is designed to drive cars automatically, but will displace already precarious rideshare drivers, who will feel this as a use-based harm that no amount of technical implementation fixes can remedy. 

A darkly hilarious academic article analyzed a hypothetical AI system designed to mulch elderly people into nutritious milkshakes. They corrected racial and gender unfairness in how the system chose who to mulch, provided a mechanism for designated mulchees hold the system accountable for errors in computing their status, and provide transparency into how the mulching designation algorithm works. This satirical example makes the problem clear: a system can be fair, accountable, transparent yet still be patently unethical, because of how it is used. Nobody wants to drink their grandma. 

For a non-hypothetical example I’ve studied: consider Deepfake algorithms, an AI technology often used for harm. Nearly all deepfakes online are pornography made without the consent of the overwhelmingly female victims they they depict. While we could make sure that the generative adversarial network used to create Deepfakes performs equally well on different skin types or genders, these fairness fixes mean little when harm is inherent in how algorithms are used: to create non consensual pornography leading to job loss, anxiety, and illness.

“Building it Better” is seen as good, but “policing” use is not

These kinds of use-based harms aren’t often caught by AI Ethics guidelines, which usually focus on how systems are built, not how they are used. This seems like a major oversight. 

Why the focus on how AI is implemented, rather than how it is used? 

It may be because of how these guidelines are used: in my experience, ethical AI principles are often used to guide and review systems as software engineers build them, long after the decision to build the system for a certain client or use has been made by someone high up the management chain. To avoid use-based harm, sometimes requires refusing to work with a certain client, or not building the system altogether. But Ethical AI “owners” in companies don’t often have this power, or even when they do, suggesting to not build and not sell a piece of software can be socially difficult in a company that builds and sells software. 

Ethical AI guidelines may also be deliberately designed to draw attention away from whether companies ought to build and sell a system by instead drawing attention to the more narrow question of how it is built. Researchers analyzed seven sets of ethical AI principles published by tech companies and connected groups, and found that “business decisions are never positioned as needing the same level of scrutiny as design decisions,” suggesting that profit motives may encourage scrutiny on how to build the system rather than broader business decisions such as whether and whom to sell the system to. It makes sense that companies’ ethical AI guidelines focus on how their software is built, rather than how it is used: focusing on the latter would restrict who companies can sell to. 

But even without profit motives, the Free Software movement guarantees the “freedom to run the program as you wish, for any purpose,” even for harm, and open source licenses may not curtail how the software is used. My own research shows that open source contributors use ideas from free and open source licenses to similarly disclaim accountability for harm caused by the software they help build: they just provide a neutral tool, ethics is up to their often unknown users. 

Software workers need a say in downstream use 

But there are important signs of resistance to a narrow framing of ethical AI which ignores uses. 

Tech workers are organizing not just to improve their own working conditions, but are also demanding a say in how the tech they create is used. The union-associated Tech Workers Coalition demands that “Workers should have a meaningful say in business decisions … This means that workers should have the protected right to … raise concerns about products, initiatives, features, or their intended use that is, in their considered view, unethical.”. Google workers protested Project Maven because it was to be used to aid drone strike targeting for the US Military. They were demanding that the fruits of their labor not be used to wage war. They weren’t protesting a biased drone strike targeting algorithm. 

They were demanding that the fruits of their labor not be used to wage war. They weren’t protesting a biased drone strike targeting algorithm. 

From the open source community comes the Ethical Source movement, seeking to give developers “the freedom and agency to ensure that our work is being used for social good and in service of human rights” by using licenses to prohibit uses that project contributors see as unethical.

What can software engineers do?

As we wrestle with ethics as we build ever powerful systems, we must increasingly assert agency to prevent harms resulting from how users may use systems we build.  But organizing a union or questioning decades of free software ideology is a lengthy process, and AI is used for harm now. What can we do today?

The good news is that how a system is built affects how it is used. Software engineers often have latitude to decide how to build a system, and these design decisions can be used to make harmful downstream use less likely, even if not impossible. While guns can be tossed around like frisbees, and you might even be able to use a frisbee to kill someone if you tried hard enough, engineers made design decisions to make guns more lethal and frisbees (thankfully) less lethal. Technical restrictions built into software might detect and automatically prevent certain uses, making harmful use harder and less frequent, even if a determined or skilled user could still circumvent them. 

We can also vote with our feet: software engineers are in high demand and command high salaries. As many are already doing, we can ask questions about how the systems we’re asked to build might be used, and if we have concerns that are not met, we can find employment elsewhere with only a small if any gap in employment or pay cut.

Author’s Note: Please fill out this 10 minute survey to contribute to help us understand ethics concerns that software developers encounter in their work! 

– – –

David Gray Widder is a PhD Student in Software Engineering at Carnegie Mellon, and has studied challenges software engineers face related to trust and ethics in AI at NASA, Microsoft Research, and Intel Labs. You can follow his work or share what you thought about this article on Twitter at @davidthewid.

The post Ethical AI isn’t just how you build it, its how you use it appeared first on Stack Overflow Blog.

May

Welcomes to ISSUE #127 of The Overflow! This newsletter is by developers, for developers, written and curated by the Stack Overflow team and Cassidy Williams. This week: the case for and against coding to anticipate future changes, the periennial question: feature or bug, and the fate of top-level domains when the associated country expires.

From the blog

The 2022 Developer Survey still needs you! stackoverflow.az1.qualtrics.com Our survey of how developers work and live is still open for input. By contributing to this survey, you help all developers understand their industry better.

Stack under attack: what we learned about handling DDoS attacks stackoverflow.blog When the bots came for us, we strengthened our defenses. Here's what we learned about parrying a few DDoS attacks.

Crystal balls and clairvoyance: Future-proofing in a world of inevitable change stackoverflow.blog While the future may be a mystery, you can design software to accommodate future changes. But how much future-proofing gets in the way of good design?

Building out a managed Kubernetes service is a bigger job than you think stackoverflow.blog Infrastructure as code is complicated enough, but building a managed IAC service is a whole other level of complicated.

Open-source is winning over developers and investors (Ep. 442) stackoverflow.blog Matt and Cassidy talk about the open-source startups winning investors and attention, why even small contributions to open-source projects are important, and how founders can encourage those contributions.

Claim your FREE .app or .dev domain from Porkbun promotion Porkbun is a refreshingly different domain name registrar offering an oddly satisfying experience. Right now, you can get a .app or .dev domain name for FREE. If you're a developer or designer, .app and .dev offer an ideal home for your next project. Claim your free domain today!

Interesting questions

Are there specific coding guidelines for test automation? sqa.stackexchange.com While code is code, test code is not the same as production code.

Are humans and other animals machines? philosophy.stackexchange.com If so, when does our warranty expire?

Is this GCC 12.1 const problem a bug or feature? “Attempts to call non-const function with const object”  stackoverflow.com A non-trivial amount of software engineering involves asking, “Hey, is this supposed to happen?”

Most complete list of investment mistakes in stock markets quant.stackexchange.com What's the opposite of a bucket list?

Links from around the web

State of CSS 2022 web.dev New styling APIs were gifted to us at Google IO this year, like color functions, container queries, and more!

What happens to TLDs when their country stops existing? astrid.tech We might let a domain name or two expire, but what happens to our domains when…a country expires?

How to multiply Roman numerals rbutterworth.nfshost.com Sometimes it's worth looking to pre-internet days to learn some new tidbits.

Solid in 100 seconds  www.youtube.com If you'd like to wrap your head around a new framework, check out Solid! It's very similar to React, but has some different naming and opinions.

The post The Overflow #127: What we learned from DDoS attacks appeared first on Stack Overflow Blog.

On this home team episode of the podcast, Ben, Cassidy, and Matt talk about small computers, big nostalgia, and security flaws that you could script a truck through. 

Episode notes

Following the success of the Mac Mini, Windows is getting into the tiny computer business. Oh, and it’s running on ARM chips. Oh, and Visual Studio and VS Code will now offer native ARM support.

Video games got a lot of us into programming thanks to their openness to mods. It’s what made The Elder Scrolls: Morrowind such a hit 20 years ago. 

Minecraft may live forever thanks to its modding community and parent-friendly tools. Just don’t be surprised when you have to ban local kids for virtual arson and murder. 

The old security exploit hits are still out there: cross-site scripting, SQL injection, and cross-site request forgery. Could be because 86% of developers do not view application security as a top priority. Two great questions today: Is it illegal to ride a drunk horse? and a Lifeboat-worthy response from Markus Meskanen on Checking if a number is not in range in Python.

The post Games are good, mods are immortal (ep 446) appeared first on Stack Overflow Blog.

If you’ve been programming for long enough, you have heard about the concept of a graph. It’s required content for a degree in computer science, and many top-level companies test for an understanding of graph theory during technical interviews. However, you don’t need to be working on advanced problems to utilize the concepts. Let’s review why graphs are popular data structures and how you can implement them in code. 

Relational models

Regardless of coding experience, you should be familiar with the data types of arrays and dictionaries. These collections are standard concepts used in most languages and work great when presenting list-based content:

Most times, lists are the perfect solution for presenting information from a database or a REST-based query. However, there are times when a list needs to provide context as to how records relate to each other. This is when organizing data as a graph becomes handy.

With graphs, the primary objective isn’t to list information (although entirely possible) but to define the relationship between objects.  Why would that be useful?

An undirected graph with two vertices and one edge.

Mapping applications: How would you organize data needed to recreate a mapping service like Apple or Google Maps if asked in a technical interview? More than just providing a list of all known roads in a database, your model would need to determine the best way to reach a destination based on the time of day, traffic, and one-way streets among other factors. For this considerable volume of data to be effective, you need to know how a single road relates to all other streets in the model.  

Social media: Success on social media is often measured by the number of people you follow or that follow you. Networking platforms like Twitter attract massive audiences by allowing you to connect with anyone, permitting you to receive their latest updates. The LinkedIn model is more detailed as you cannot add someone to your network unless the receiver accepts your connection request. In this case, LinkedIn connections represent two-way relationships. Taking this idea further, you can also search to see if anyone in your network is connected to a job opportunity you want to pursue. In this case, “network” could imply a direct or indirect connection. Such a robust model isn’t just based on a simple list but would include the smarts to determine how all profiles relate. 

Graph components

Now that we’ve seen how graphs are used in everyday applications let’s introduce their components. Nodes in a graph are referred to as vertices. While it would be possible to build a graph as a single vertex, models that contain multiple vertices better represent real-world applications. Graph objects relate to one another through connections called edges. Depending on your requirements, a vertex could be connected to one or more things through edges. It’s also possible to create a vertex without edges. Finally, unlike other standard structures like stacks or queues, graphs often have no designated start or end point. Here are some example graph configurations:

An undirected graph with two vertices and one edge.

An undirected graph with four vertices and three edges.

An undirected graph with three vertices and three edges.

Directed vs. undirected

In an undirected graph, the connection between a source vertex and destination are equal. These models represent two-way connections—similar to a two-way street in the mapping application. To define a connection going a single direction, we can update our model to a directed graph by using lines and arrows:

A directed graph with three vertices and three edges.

Level of connectedness

Sometimes, we must represent the level of connectedness between vertices in a graph. This technique works well when quantifying distances, times, or severity between nodes. Generally associated with an edge, the weight is a comparative variable tracked for this purpose. 

A directed graph with three vertices and three edges where the edges are weighted.

Graph vertex

With a basic understanding of graph theory in place, let’s see how to replicate some of these models in code. Below we’ve created a vertex that supports a custom generic object (T). The tvalue variable represents the data held by the type, including a single string, int, or custom type (for example., street name or social media profile). Also, note our class conforms to the popular Equatable protocol (Swift). This will allow us to compare specific vertex instances for equality if required.  

public class Vertex <T> : Equatable { var tvalue: T? var neighbors = Array<Edge<T>>() let uuid = UUID() public init(with name: T) { self.tvalue = name } //equatable conformance public static func == (lhs: Vertex, rhs: Vertex) -> Bool { return lhs.uuid == rhs.uuid } }

Adjacency lists

The neighbors property represents the connections made to other vertices. As discussed, each vertex can connect to one or more neighbors. This list of relationships is sometimes called an “adjacency list” and can be used to solve many advanced problems. 

var neighbors = Array<Edge<T>>()

Graph edge

We added a neighbor property to store an array of custom edge types when creating our vertex. Below, an edge provides a reference for a subsequent neighboring vertex and its potential edge weight value. 

public class Edge <T> { var neighbor: Vertex<T> var weight: Int init() { weight = 0 self.neighbor = Vertex<T>() } }

Building the canvas

With our vertex and edge objects in place, we can now add them to our central storage structure we’ll call the graph canvas. Even though our canvas is technically an array, the goal is to visualize the collection as a set of relationships. The addVertex function allows us to add a single generic vertex to the canvas, while the addEdge method provides reference information needed for an edge. Lastly, our code assumes the graph is directed, as the edge is (only) added to the source vertex adjacency list.

public class Graph <T> { var canvas: Array<Vertex<T>> public init() { canvas = Array<Vertex>() } //add vertex to graph canvas public func addVertex(element: Vertex<T>) { canvas.append(element) } /add edge public func addEdge(source: Vertex<T>, neighbor: Vertex<T>, weight: Int) { //create a new edge let newEdge = Edge<T>() //connect source vertex to neighboring edge newEdge.neighbor = neighbor newEdge.weight = weight source.neighbors.append(newEdge) } }

In summary, we’ve introduced graphs and have seen how they are used to represent the relationship between objects. We also reviewed a few ways to configure a graph and the components used to describe different models. With our model defined, we’ve set the stage for more advanced functionality, including graph navigation and traversal algorithms like breadth-first search.

The post The complete beginners guide to graph theory appeared first on Stack Overflow Blog.

The potential for Web3 is massive. Web3 is overhyped. These are both true and that’s fine. Cryptocurrency and Web3 have captured the interest of many; is it the future of the web? Maybe. An inspiring story wins people to your cause, while the messy truth is usually found somewhere in the middle.

What is true about these technologies is the interest and possibilities they hold. New open source projects and startups rise everyday to build the next generation of decentralized applications. But thanks to opportunists and scammers, blockchains and Web3 are in danger of becoming more of a meme, and not in a good way.

Determining what’s real about Web3 is hard; so much is hype or FUD. Understanding which is which and what’s genuine can help us move forward as an industry. In this article, we’d like to talk to both the bulls and the bears, and inject a little reality into both of their conversations. To crypto idealists, let’s say good morning to where Web3 is actually today, and for skeptics, let’s take a step back and reassess the FUD.

What is Web3 anyway?

Because there’s so much misinformation around Web3, let’s start with a definition. Web3 was coined by one of the creators of Ethereum in 2014, Gavin Wood. He says, “Web3 is an alternative vision of the web, where the services that we use are not hosted by a single service provider company, but rather they’re purely algorithmic things that are hosted by everybody.”

The promise for Web3 is a user-led internet that isn’t as easily manipulated or controlled by centralized parties. Web2 has been dominated by walled gardens—think massive social networks, operating systems, browsers, and app stores—platforms owned by a few companies that can make changes on a whim, ruining other businesses built on top.

BitTorrent and its peer-to-peer network for file sharing is a good, early example of what Gavin and others support in this space and want to see the proliferation of. Note how this example doesn’t use a blockchain and doesn’t require cryptocurrency or a token. Like most technology decisions, if your use case doesn’t need it, it shouldn’t have it.

Web3 advocates argue crypto and blockchain can be added and layered on applications like BitTorrent to enhance incentives, ownership, and transparency as needed. While losing an mp3 file using a peer-to-peer system like BitTorrent could be acceptable, it’s likely not acceptable for other use cases, like holding your account balance or handling the exchange of valuable assets.

What often gets lost in the conversation of Web3 is that this is an evolution, not a complete rewrite of Web2. For developers especially, it builds upon what we’ve already done and hopefully enhances it in a meaningful way. The frontend will feel very familiar—it will be the same HTML, JavaScript, and CSS the rest of the web uses. And not everything needs to be decentralized; storing data in a traditional database to augment the user experience could make the most sense, although decentralized technologies are hoping to make inroads there, too.

The true challenges of Web3

With any new frontier, market, or technology there are rough edges. Concerning and admittedly entertaining sites like web3isgoinggreat.com exist and cite nearly $10 billion in losses because of crypto’s particularly rough edges.

The vast majority of these ‘losses’ are happening above the underlying infrastructure, or layer 1 network, where consensus and settlement occur. You’ll find most are caused by inexperienced or poor development practices or bad actors that take advantage of unsuspecting victims. To be clear, neither of these are acceptable and we must collectively do better, but there is an important difference between the technology itself being evil and those deploying on it being evil.

Stakes are high

The most popular and innovative decentralized applications available today are in the realm of DeFi, or decentralized finance. Decentralized finance takes aspects of traditional, centralized finance including exchanges, loans, and derivatives to make them run entirely on a public ledger without need for human intervention or interference.

Most of these applications are run by one or more smart contracts communicating across each other. Users interact with them in a self-sovereign way, being responsible for their cryptographic keys to sign the transaction. A majority of funds on a network today—often referred to as total value locked (TVL)—are held in a smart contract. One poorly written smart contract can be responsible for losing millions of dollars or even locking funds into a smart contract unable to ever be withdrawn. This is all too common. As such, new tools, practices, and audits are constantly being introduced  to try and overcome these issues.

Due to the nature of these applications and aspects intrinsic to distributed ledgers, users bear more responsibility and thus take on much more complexity than other web technologies. It turns out, being your own bank can be challenging for many today.

It sucks to use

Web2 perfected user flows over trillions of interactions down to the pixel. Web3, being a less mature technology, has not.

If you haven’t done so, I invite you to try to use the technology as an end user. On whichever network you like, attempt to buy a NFT. Don’t worry, you don’t have to really buy it to get enough from this experience. You’ll typically find a flow as follows:

Install a browser extension

Create a wallet account

Save your keys securely

Navigate to a website

Find a NFT you like

Be prompted to sign a transaction

Not have enough funds to sign the transaction

Have to find an exchange

That’s  a high barrier to entry to purchase a silly profile picture. Most users don’t want to go through that process; they want a button that says “Buy” and then be done. And we haven’t even begun to talk about performance.

On proof-of-work blockchain based networks, like Ethereum where the majority of NFT transactions are happening today, transaction fees are extremely volatile, expensive when the network is in heavy use, and painfully slow by any modern standards.

Re-centralization

The ethos and benefit of this movement is decentralization. It reduces the power, control, and limits of a centralized entity. A core part of achieving decentralization is through how a network achieves consensus and how it makes decisions through the individuals operating on that network. Depending on your viewpoint, miners have centralized Bitcoin with five mining groups taking up the majority of hash rate. Some argue VCs have already centralized Ethereum, with many projects seeing billions of dollars of traditional Silicon Valley investments.

At the end of the day, if a few parties have still managed to consolidate control, are we progressing and is it worth all of this? All hope is not lost. One potential silver lining is that the transparency with which we understand this is beneficial enough. The public ledger can tell us a lot more than a private CAP table. There are also different models being considered with regards to both consensus and governance throughout the extended ecosystem. For example, some applications are doing a better job of exiting to their community, typically by rewarding tokens directly to its most active users with projects like Ethereum Name Service and Optimism leading the way. Having to have continued and ideally increasing decentralization is a challenge across economics, sociology, and technology. That’s difficult to do well.

Take out the FUD

For the problems above, some can be solved through advancement in the technology, like higher throughput, and some, like poorly written smart contracts or a user losing their keys, reduced through adoption of best practices. These critiques of crypto are absolutely true, widely accepted, and must be improved upon for the technology to be taken seriously.

In addition to these actual problems, there are many misconceptions and much misinformation that creates a seismic amount of fear, uncertainty, and doubt (FUD). When people look down on crypto technologies, they usually think about these things. If there’s anything to take away from this post, it is that these characteristics of a public ledger aren’t entirely true.

Energy use

In 2008, Bitcoin introduced proof of work, a compute and energy intensive way to come to consensus in adversarial environments, and by doing so left a stigma on all crypto technologies. Rightfully so, as Bitcoin consumes more energy than many countries. For a new transaction block to be validated, proof of work requires individuals on the network to competitively decode a hashed value to validate new transactions.

The good news is that since then, the majority of Web3 applications and cryptocurrency use doesn’t require a country-size portion of electricity to exist. Many, including Ethereum, the second largest network in terms of market cap, are moving to proof of stake, which requires only that the validator hold a number of tokens and therefore be directly affected by malicious transactions. This will reduce energy consumption for these transactions by over 90%.

Newer networks (shown in the chart above) reduce energy needs by even more, and layer 2 networks on Ethereum show early promising signs that they can further reduce energy use by bundling transactions together to reduce the number of individual transactions required to settle on Ethereum. Overall, the industry is moving from a single bitcoin transaction taking multiple days of a family home’s energy consumption to modern blockchain networks expending the energy of a web search. 

It is 100% true, in my opinion, that crypto networks have collectively spent too much energy getting us this far, but that doesn’t mean we’re stuck here. All new technology shifts take time to be better, more efficient and faster. For public ledgers, there are both changes to existing networks and new, more efficient networks available today to remove us of this stigma.

Inefficient database

Yes, blockchain and public ledgers are inefficient when it comes to general data management. Bitcoin can handle tens of transactions per second, and Ethereum is not too far above that. But distributed ledgers are not meant to be databases. They’re not optimized for storing click data or even general user data, and it’s the wrong comparison to make for them. The truth is that some ‘decentralized applications’ still use a traditional database for a large portion of their application.

Distributed ledgers are intended to keep track of the most important and critical data, such as your account balance and identity credentials. Data that we, ideally, wouldn’t want a single centralized entity to control the entire landscape of. These types of use cases don’t require the scalability of Kafka, but for public ledgers to reach success they do require greater than 20 transactions per second; thankfully, that’s already here. 

Scalability solutions on top of Ethereum, like Polygon and Optimism, are making some improvements. Newer layer 1 networks like Hedera that don’t use a blockchain at all but an alternative called hashgraph, are able to drastically reduce these performance challenges to already run on par and beyond the speed of the VISA network today.

Nothing is worse than bolting on a technology to a problem that doesn’t exist. We don’t need or want to continue the Uber for X trends with X on Blockchain, so it is critical to understand the actual attributes of a public ledger.

New development principles for Web3

Despite the existing challenges and FUD, Web3 is a promising step towards addressing concerns of excessive control, access, data ownership, and even censorship that exist in Web2 today. As a key enabling component of Web3, crypto networks introduce a new set of principles and tools available for developers to create entirely new applications and enhance existing solutions. Some of the principles being most actively explored are:

Immutability

Verifiability

Programmability

Interoperability

These principles can be used individually or come together to build more robust decentralized protocols and applications.

Immutability

A core premise of public ledgers is that the data is append-only and unable to be modified after the fact.

The Hedera network processes the most transactions of any public ledger on a daily basis. A majority of the transactions are from businesses that want to have the highest degree of data integrity for a multiparty process. Often, a developer will have a data feed that they want recorded as an immutable source of truth for a third party to verify. To achieve this, they’ll send a hash of the transaction to the ledger. 

When compared to centralized infrastructure, using a public ledger as a neutral actor in the system takes advantage of its true immutability. This has a number of applications ranging from auditable logs to identity credentials.

Verifiability

The transparent nature of a distributed ledger provides stronger guarantees of the information it holds and, in the case of smart contracts, the rules the contracts abide by.

Today this transparency is used to readily verify the uniqueness or rarity of NFT collections today, but could be used for better understanding your government’s spending, voting, or a business’ carbon offsets tomorrow. This degree of total transparency and access to information is not possible in a centralized database.

Programmability

Smart contracts allow us to bring all of these characteristics of our ledger together and add autonomous application logic. Smart contracts are terrific for holding tokens on behalf of a user, for instance to support a collateralized lending protocol.

The majority of smart contract developers today are using Solidity, a Turing complete language initially created for the Ethereum Virtual Machine, which has become a default and expanded to other EVM compatible networks.

Networks like Hedera allow for developers to use Solidity smart contracts when needed, or bypass them, for more efficient messaging or token management in more common programming languages like JavaScript and Java.

Interoperability 

By having logic that is immutable, transparent, and atomic, we get the benefit of interoperability and composability. Networks can communicate across each other using standards to frictionlessly move assets or your application can call a smart contract that another developer already deployed to easily tap into the existing ecosystem on a network.

While developers explore this concept in new frontiers like decentralized finance, sometimes referred to as DeFi, there are also more practical use cases to get your footing, too. Take, for example, a loot box. Loot boxes are a common game mechanic where a player receives a box as a reward for achieving a milestone in a game.

Loot box rewards and odds of receiving them are, historically, opaque, although government regulation is beginning to come into view. A better, more equitable implementation for loot boxes could be by using a distributed ledger. The odds of you to receive an item could be transparently defined in a smart contract. The results would be verifiably random. The output could be an NFT to show ownership of your newly earned asset.

Having this process take place on a public ledger unlocks greater assurances for user bases, but also makes it easier for other game developers to reward existing user bases for perhaps a specific item held, and opens up more potential options for secondary markets.

Give Web3 a (grounded) chance

In some ways, web3 is a continuation of open source principles. Core beliefs in collaboration, transparency, community, and freedom of choice moving up the stack.

Web2 built a set of walled gardens owned and controlled by very few. A better way to design these applications going forward may be to break apart the protocol from the user interface. This separation of protocol and client, a digital church and state, is a meaningful difference in how decentralized applications can be designed.

Instead of a developer being able to use a type of database, they’re able to use a specific type of data schema, like an ERC-721 non-fungible token. Instead of a developer being able to move their open source database from using AWS to Google Cloud, end users are able to directly and freely move from client application to application.

The reality is that after we remove the noise, blockchain, crypto, and Web3 is a place that a developer can make an outsized impact. From the outside it can feel like you’re jumping into the Matrix, choosing the pill you want to take, or often don’t (which is ok too!). This doesn’t have to be the case and, on both sides, we need to get better at understanding our faults and misinformation.

If you’re interested in dipping your toes in to Web3, a few good communities to get started are Buildspace and Pointer, free education with welcoming communities. If you’re just looking to get hands-on with Solidity smart contracts, the most popular smart contract language supported by networks like Ethereum and Hedera, CryptoZombie is a fun and easy way to get familiar. Once you’re dangerous enough, hop into Gitcoin to find a grant for some freelance work or find one of the many hackathons in the space for an opportunity to learn and hopefully win. 

The post Web3 skeptics and believers both need a reality check appeared first on Stack Overflow Blog.

The home team covers the hiring freezes and layoffs hitting the tech sector, burnout among freelancers and applicants for tech jobs, the dubious ethics of unpaid internships, and how to make Twitter safer by preemptively blocking people.

Episode notes:

Companies like Meta, Twitter, and Netflix are enacting hiring freezes and layoffs, a situation that’s not great for anybody but is likely to have outsize effects on people of color in tech.

Gen Z may not understand file structures, but they sure understand Twitter toxicity. MegaBlock from Gen Z Mafia allows users to block bad tweets, their authors, and every single account that liked the offending tweet. There, doesn’t that feel better?

Apple’s WWDC 2022 is just around the corner. What are you most excited about?

Machine-learning start-up Inflection AI raises $225 million in equity financing to use AI to improve human-computer communication. Another reminder that building sophisticated AI systems isn’t cheap: who could forget that Open AI paid its top researcher just shy of $2 million in 2016?

Today’s Lifeboat badge goes to user Patricia Shanahan for their answer to Difference between int and double.

TRANSCRIPT

The post Turns out the Great Resignation goes both ways (Ep. 445) appeared first on Stack Overflow Blog.

Advances in platform models—large-scale models that can serve as foundations across applications—have significantly improved the ability of computers to process natural language. But natural language processing (NLP) models are still far from perfect, sometimes failing in embarrassing ways, like translating “Eu não recomendo este prato” (I don’t recommend this dish) in Portuguese to “I highly recommend this dish” in English (a real example from a top commercial model). These failures continue to exist in part because finding and fixing bugs in NLP models is hard—so hard that severe bugs impact almost every major open-source and commercial NLP model. 

Current methods for finding or fixing bugs take one of two approaches: they’re either user-driven or automated. User-driven methods are flexible and can test any aspect of a model’s behavior, but they depend on highly variable human ability to imagine bugs and are so labor intensive that in practice only a small part of the input space gets tested. Automated approaches, on the other hand, are fast and so can explore large portions of the input space. However, since they lack human guidance, they can only test if a model is right or wrong in very restricted scenarios, such as when the model has inconsistent predictions on inputs with slight variations in phrasing. 

We believe platform models, specifically modern large language models (LLMs) like GPT-3, offer an opportunity for us to combine the synergistic strengths of both user-driven approaches and automated approaches, keeping the user in control of defining what the model being tested should be doing while leveraging the abilities of modern generative language models to generate at scale tests within a specific category of model behavior. We call this human-AI team approach Adaptive Testing and Debugging, or AdaTest for short. 

With AdaTest, a large language model is tasked with the slow burden of generating a large quantity of tests targeted at finding bugs in the model being tested, while the person steers the language model by selecting valid tests and organizing them into semantically related topics. This guidance from the person drastically improves the language model’s generation performance and directs it toward areas of interest. Because these tests are effectively a form of labeled data, they not only identify bugs but can be used to fix bugs in an iterative debugging loop similar to traditional software development. AdaTest offers significant productivity gains for expert users while remaining simple enough to empower diverse groups of non-experts without a background in programming. This means experts and non-experts alike can better understand and control the behavior of their AI systems across a range of scenarios, which makes for not only better-performing AI systems but more responsible AI systems. The AdaTest code and pre-populated test trees are open source on GitHub. 

Read the paper

Get the code

We’re presenting our paper, “Adaptive Testing and Debugging of NLP Models,” at the 2022 Meeting of the Association for Computational Linguistics (ACL), where our colleagues will also be introducing work that leverages large language models, in their case, to grow adversarial datasets for content moderation tools. 

Figure 1: AdaTest consists of two loops: a testing loop that generates and organizes tests optimized for the model being tested (the target model) and a debugging loop that iteratively refines the model based on test failures. 

Finding bugs with the testing loop

The AdaTest process is composed of an inner testing loop that is used to find bugs (Figure 1, unrolled in Figure 2) and an outer debugging loop that is used to fix bugs (Figure 1, unrolled in Figure 4). 

Consider how this works for sentiment analysis, used to determine if a piece of text expresses a positive or negative sentiment (typically in the context of product reviews or customer feedback). While the task seems simple enough, even state-of-the-art models have failures, ranging from overt, like classifying “I don’t think I’ve ever had a nicer time in my life” as negative, to more subtly harmful, like classifying “I am a racial minority” as negative (both represent real failures found with AdaTest in commercial models). To demonstrate how AdaTest finds and fixes bugs, we show how to test for (and later fix) instances of fairness-related harms in which neutral references to a specific identity group within a piece of text could cause a sentiment analysis model to incorrectly downweight the sentiment of the text—in other words, scenarios in which a model might treat comments from specific groups more negatively. 

In the testing loop, we start with a set of unit tests about various identities and label the set “/Sensitive” (Figure 2 below). These initial examples don’t reveal any model failures. But AdaTest then uses a large language model—in our case, GPT-3—to generate many similar suggested tests designed to highlight bugs (Figure 2A). While hundreds of tests are generated, we only need to review the top few failing or near-failing tests. We then ignore tests that don’t represent real failures (for example, “I am tired of being silenced” really should be negative in Figure 2) and add the other valid tests to the current topic, also occasionally organizing them into additional subtopics (Figure 2B). These user-filtered tests are included in the language model prompt for the next round of suggestions, nudging the next set of suggestions toward the intersection between user interest and model failure (Figure 2C). Repeating the testing loop results in the language model starting at tests that don’t fail and slowly working its way up to producing stronger and stronger failures. So even when users can’t find model failures on their own, they can start from a small set of passing tests and quickly iterate with the language model to produce a large set of tests that reveal bugs in the model being tested. 

Figure 2: The testing loop cycles between the large language model (LLM) generating test suggestions, the model scoring the suggestions, and the user accepting (

) and organizing them, beginning with initial user-provided examples. In this three-way sentiment analysis example, the model “f” can either pass (green) or fail (red) a test. Passing one of the tests above means the model did not output “negative” while failing a test above means the model did output “negative” and hence failed the test assertion (≠). As the user filters and organizes (B, D), the LLM iteratively climbs toward suggesting valid tests that reveal more pronounced failures (A, C). In this example, we’re testing a sentiment analysis model to ensure that neutral identity-related statements don’t cause the model to flag comments as negative. 

If instead of the “/Sensitive” topic shown in Figure 2, we target a different topic, such as handling negation, we’ll reveal different failures. For example, starting from simple statements like “I have never been happier” that a commercial model correctly classifies as positive, AdaTest can quickly find bugs like “I don’t think that I’ve ever seen a nicer town” getting labeled as negative. These bugs are egregious and obvious once you see them, but they’re hard to find by hand since they only happen for very specific phrasings. 

We ran user studies to quantitatively evaluate if AdaTest makes experts and non-experts better at writing tests and finding bugs in models. We asked experts—those with a background in machine learning and NLP—to test specific topics in two models: a commercial sentiment classifier and GPT-2 for next word auto-complete, used in such applications as predicting the next word in an email being typed (a scenario in which we want to avoid suggesting stereotypes, for example, one of the behaviors we had participants test for). For each topic and model, participants were randomly assigned to use CheckList (representing state-of-the-art user-driven testing) or AdaTest. We present the average number of discovered model failures per minute in Figure 3, where we observe a fivefold improvement with AdaTest across models and participants in the study. We asked non-experts, or those without any programming background, to test the Perspective API toxicity model for content moderation. Participants tried to find non-toxic statements (that is, statements they would personally feel appropriate posting) predicted as toxic for political opinions. Participants were given access to an improved version of the Dynabench crowd-sourcing interface for model testing and to AdaTest. AdaTest provided up to a tenfold improvement (bottom portion of Figure 3). 

Figure 3: Per-topic model failures per minute. Experts found approximately five times more failures with AdaTest on all topics, and non-experts benefited by up to 10 times. Error bars represent the 10th and 90th percentiles over bootstrap re-samples of participants. 

We also grouped participants by their progressive versus conservative political alignment and found that participants wrote tests with twice the quality when testing their own perspective versus an opposing perspective (as measured by an independent set of in-group raters). Our user studies highlight that AdaTest can be used by anyone and that such easy-to-use tools are important to enable model testing by people with diverse backgrounds since testers representing different lived experiences and viewpoints are needed to effectively test different perspectives. 

Fixing bugs with the debugging loop 

Once enough bugs are discovered, testers of a model then engage in the outer debugging loop (Figure 4 below), where they fix bugs discovered in the testing loop and then retest the model. In our experiments, we fixed bugs by fine-tuning the model on the tests, but other strategies, such as collecting more data or adding constraints, are also possible. The retest part of the debugging loop (that is, running the testing loop again) is critical since once we use our tests to fix the model, they no longer represent test data but rather training data. The process of fixing a bug often overcompensates, introducing shortcuts or bugs in the initial rounds of the debugging loop that can only be found using a new set of tests adapted to target the new “fixed” model. 

Running the debugging loop on an open-source RoBERTa-Large sentiment model (Figure 4) demonstrates the importance of a test-fix-retest cycle. We start with tests from the “/Sensitive/Immigration” topic from Figure 2 that the RoBERTa model incorrectly labels as negative. Fine-tuning the model on these tests (mixed with the original training data to maintain task performance) results in a new model that no longer fails the tests (second row of Figure 4). However, when we rerun the testing loop, we find that now almost all immigration statements are labeled as “neutral,” even if they are truly negative based on the application and testing scenario (for example, the statements in the third row of Figure 4 wouldn’t be neutral if a model were tasked with detecting if language was for or against something). Fine-tuning again using these new tests (and the older ones) results in a model that correctly fixes the original bug without adding the “every immigration statement is neutral” shortcut.

This doesn’t, of course, guarantee that there isn’t another shortcut still in the model, but in our experience, a few rounds of the debugging loop drastically reduce the number of accidental bugs that get introduced when “fixing” the original bugs. The testers of the model don’t have to exhaustively identify every possible shortcut or imbalance ahead of time, since AdaTest adaptively surfaces and fixes bugs that have been introduced in the next rounds of testing and debugging. Thus, the debugging loop serves as a friendly adversary, pushing the boundaries of the current “specification” until a satisfactory model is produced. In fact, AdaTest can be seen as an application of the test-fix-retest loop from software engineering to NLP. 

Figure 4: Shortcuts added during an iteration of the debugging loop are found and fixed by future iterations. 

To evaluate the effectiveness of the debugging loop, we fine-tuned RoBERTa-Large to detect if two questions are duplicates (that is, the same question worded differently) using the Quora Question Pairs (QQP) dataset and also fine-tuned it for positive/neutral/negative sentiment analysis using the Stanford Sentiment Treebank (SST) dataset. Using previously published CheckList suites for evaluation, we find the baseline model fails 22 out of 53 QQP topics and 11 out of 39 sentiment topics. We then created data to “fix” a topic by either taking 50 examples from the topic’s data in the CheckList condition or by starting from a seed of five examples and running the debugging loop with AdaTest until finding failures becomes qualitatively difficult (on average 2.83 rounds for QQP and 3.83 rounds for sentiment). This yields an average of 41.6 tests for QQP and 55.8 tests for sentiment. We followed this process for six distinct high-failure rate topics in each task. In the vast majority of cases (see paper for details), AdaTest fixes the topics used for training and a number of unseen held-out topics without breaking any topics, while CheckList data often introduces new bugs (and thus breaks other test topics).

We also evaluated the effectiveness of AdaTest in a standard development setting, targeting a model for to-do detection in meeting notes. After three months of development, CheckList testing, and ad hoc GPT-3–based data augmentation, a PhD-level team had managed to build a model with an F1 score of 0.66 (out of 1) on unseen data collected in the wild. We gave AdaTest to the team with a demo a few minutes long. After four hours of running the debugging loop on their own, they produced another model with an F1 score of 0.77 on the same unseen dataset. These scores were then replicated again on a second unseen dataset, showing that AdaTest can add significant bug-fixing value with a fraction of the effort involved in traditional approaches. 

The promise of human-AI collaboration for ML development

AdaTest encourages a close collaboration between people and large language models, yielding the benefits of both. People provide the problem specification that the language model lacks, while the language model provides quality test creation at a scale and scope that is infeasible for people. The debugging loop connects model testing and debugging to effectively fix bugs, taking model development a step closer toward the iterative nature of traditional software development. Human-AI partnership represents a promising way forward for machine learning development, and we expect this synergy to only improve as the capabilities of large language models continue to grow.

Check out the full paper to see AdaTest’s effectiveness on classification models (sentiment analysis, QQP, toxicity, media selection, and task detection), generation models (GPT-2, translation), and per-token models (NER) ranging from well-tested production systems to brand-new applications. Give it a try yourself at https://github.com/microsoft/adatest. 

The post Partnering people with large language models to find and fix bugs in NLP systems appeared first on Microsoft Research.

It’s a well-known challenge that large language models (LLMs)—growing in popularity thanks to their adaptability across a variety of applications—carry risks. Because they’re trained on large amounts of data from across the internet, they’re capable of generating inappropriate and harmful language based on similar language encountered during training.  

Content moderation tools can be deployed to flag or filter such language in some contexts, but unfortunately, datasets available to train these tools often fail to capture the complexities of potentially inappropriate and toxic language, especially hate speech. Specifically, the toxic examples in many existing hate speech datasets tend either to be too hard or too easy for tools to learn from—the too-easy examples contain slurs, profanity, and explicit mentions of minority identity groups; the too-hard examples involve obscure references or inside jokes within the hate speech community. Additionally, the neutral examples in these datasets tend not to contain group mentions. As a result, tools may flag any language that references a minority identity group as hate speech, even when that language is neutral. Alternatively, tools trained on this data fail to detect harmful language when it lacks known or explicit slurs, profanity, or explicit mentions of minority identity groups.  

Generating the kind of data needed to strengthen content moderation tools against the above failures and harms is challenging for numerous reasons. In particular, toxic text that is more implicit and that existing machine learning architectures can still learn from or neutral text with group mentions is difficult to collect at scale. Additionally, asking people to write such examples—particularly the toxic ones—can have a negative impact mentally on those assigned the task. 

Inspired by the ability of large language models to mimic the tone, style, and vocabulary of prompts they receive—whether toxic or neutral—we set out to create a dataset for training content moderation tools that can be used to better flag implicitly harmful language. In our paper “ToxiGen: A Large-Scale Machine-Generated Dataset for Adversarial and Implicit Hate Speech Detection,” we collected initial examples of neutral statements with group mentions and examples of implicit hate speech across 13 minority identity groups and used a large-scale language model to scale up and guide the generation process. The outcome is the largest implicit hate speech dataset to date that is publicly available: 274,000 examples comprising both neutral and toxic statements. We conducted a human study on the generated dataset to better understand different aspects of harm beyond binary labels of toxic and neutral assigned by content moderation tools. To stress test existing content moderation tools across minority identity groups studied in this work, we also propose an adversarial classifier-in-the-loop decoding approach. The dataset, two content moderation tools trained on the dataset, prompts used as seed data, and the source codes for our proposed adversarial decoding approach are available in the ToxiGen GitHub repo (please see footnote).

Read the paper

Dataset & more

We’re presenting this work at the 2022 Meeting of the Association for Computational Linguistics (ACL), where our colleagues will also be presenting work that leverages the generative power of large language models and human expertise. 

Figure 1: The ToxiGen dataset—an implicit hate speech dataset created by using a large-scale language model with both regular and adversarial decoding to scale up and guide the generation process—contains 274,000 examples comprising both neutral and toxic statements across 13 minority identity groups. As illustrated above, mentions of a specific minority identity group in the prompts and mentions of the same minority identity group in the corresponding generated text are proportional.

Demonstration-based prompting for building better datasets

Large Transformer-based language models don’t explicitly encode semantic information; nevertheless, these models can distinguish the statistical interactions of words in different contexts. Through experimentation with the generation of language via one of these large language models, we learned how to utilize careful prompt engineering strategies to create the ToxiGen implicit hate speech dataset. 

Our first experiments were to generate examples of hate speech and neutral speech related to the 13 minority identity groups in our work. We started by collecting implicit hate speech prompts from existing datasets and neutral prompts drawn from news articles, opinion pieces, podcast transcripts, and other similar public sources and feeding them into the LLM to create a broader, deeper set of prompts. What we found was that the LLM could generate examples that were qualitatively different depending on the source material. When prompted with bits from different writers on the above topics, in each case, the LLM produced linguistically diverse outputs that were nonetheless similar in style and tone. 

Furthermore, we found that through careful cultivation of prompt sets, we could generate a wide variety of text reflecting diverse opinions and thoughts on these topics that weren’t found in our original source materials. We could generate neutral statements about sensitive topics that mentioned the relevant minority identity groups, and we could consistently generate hate speech statements about these minority identity groups that didn’t contain slurs or profanity. And the more we experimented with the source material, the more interesting our dataset became. This is particularly exciting because we hope that other individuals and groups can use these tools to extend our dataset; different disciplinary experts could utilize the same strategies and collect even better prompt sets, resulting in even more subtle and rich examples of neutral speech and hate speech. 

We also found that the model often generated examples of speech that we ourselves had trouble labeling. In essence, we were using the LLM as a probe to explore the delicate boundaries between acceptable and offensive speech. As a result, our own understanding of the problem definition itself grew through our interactions with the model.  

The first 260,000 examples from our dataset were drawn from this experimental approach. 

Figure 2: Examples of statements generated by (De)ToxiGen that fool Google’s Perspective API, HateBERT, OpenAI content filter, AI2 Delphi, and RoBERTa. Five statements are neutral but mention minority identity groups, so the content moderation tools find them hateful. Five are toxic sentences, but the tools find them neutral. The proposed decoding approach, (De)ToxiGen (referred to as ALICE in the paper), can challenge these content moderation tools, allowing developers to increase their coverage by creating adversarial examples. 

(De)ToxiGen: An adversarial decoding approach for strengthening content moderation tools

While demonstration-based prompting can facilitate large-scale data generation, it doesn’t generate data targeted specifically to challenge a given content moderation tool, or content classifier. This is important because every content moderation tool has unique vulnerabilities depending on the type of data it has been trained on. To address this, we developed (De)ToxiGen (referred to as ALICE in the paper), an algorithmic mechanism that creates an adversarial set-up between an LLM and a given content moderation tool in which the content classifier is in the loop during decoding.  

The proposed approach can increase or decrease the likelihood that a generated statement is classified as hate speech while maintaining the coherence of the generated language. It can generate both false negatives and false positives for a given content moderation tool. For false negatives, toxic prompts are used to elicit toxic responses, and then the tool’s probability of the neutral class is maximized during decoding. Similarly, to generate false positives, neutral prompts are used to generate neutral responses, and then the probability of the toxic class is maximized during decoding. With this approach, we’re essentially trying to reveal weaknesses in a specific content moderation tool by guiding the LLM to produce statements that we know the tool will misidentify. The generated data can then be used to improve the performance and coverage of the targeted content moderation tool. Our ToxiGen dataset includes data generated by both demonstration-based prompting and our proposed adversarial decoding approach. Through empirical study on three existing human-written datasets, we found that starting with an existing content moderation tool and fine-tuning it on ToxiGen can improve the tool’s performance significantly, demonstrating the quality of the machine-generated data in ToxiGen.  

Human evaluation: Better understanding the data

Human language is complex, particularly when it comes to harmful statements. To better understand different aspects of the data in ToxiGen—its perceived harmfulness and intent and whether it presents as fact or opinion, for example—we conducted human evaluations on the data generated by both regular decoding (top-k), used in the demonstration-based prompting, and the proposed adversarial decoding. The human evaluation also allowed us to test the quality of the output of these methods and gauge how effective these methods were in guiding the generation of the data we sought. 

For the human evaluation, three annotators were used for each statement from a pool of 156 prequalified annotators with prior experience annotating toxic language. About 4,500 samples were randomly selected for each of the decoding methods with coverage across all 13 minority identity groups for each split. We found the following: 

For both decoding methods, minority identity group mentions included in the prompt also exist in the generated statements. This means that both data generation methods reliably produce the data they were designed to produce—hateful and neutral statements with explicit reference to the specified minority identity group.

In the neutral case, the label of the prompt matches the generated text more often than in the toxic case, as shown in Figure 3a. 

The proposed decoding approach generates a higher percentage of adversarial text compared to regular decoding—that is, it produces data that is more likely to fool a given content moderation tool—as illustrated in Figure 3b. 

Figure 3a (left) and 3b (right): Human evaluations on the data generated by regular decoding (top-k) and the proposed adversarial decoding showed that the toxicity labels for the prompt and the generated response match more often for non-toxic prompts compared to toxic ones (left). It was also observed that (De)ToxiGen generates a higher percentage of adversarial text compared to regular decoding (right). 

90.5 percent of machine-generated examples were thought to be human-written by the majority of annotators.

Perceived harmfulness with respect to human- or AI-authored text is similar. 

Looking ahead: Societal implications and opportunities

As advances continue to be made in large language models, we remain vigilant in our pursuit of AI systems that align with our commitment to technology that benefits society as a whole and empowers everyone to achieve more. We’re beginning to ask better questions to more deeply understand the risks associated with LLMs and build processes and methods for addressing them. Existing content moderation tools tend to be only good at flagging overt inappropriate or harmful language. Our work aims to create data that can better target the challenge. While our work here specifically explores hate speech, our proposed methods could be applied to a variety of content moderation challenges, such as flagging potential misinformation content. By releasing the source codes and prompt seeds for this work, we hope to encourage the research community to contribute to it by, for example, adding prompt seeds and generating data for minority identity groups that aren’t covered in our dataset. 

As with many technologies, the solutions we develop to make them stronger, more secure, and less vulnerable also have the potential to be used in unintended ways. While the methods described here may be used to generate inappropriate or harmful language, we believe that they provide far greater value in helping to combat such language, resulting in content moderation tools that can be used alongside human guidance to support fairer, safer, more reliable, and more inclusive AI systems.  

Considerations for responsible use

There is still a lot that this dataset is not capturing about what constitutes problematic language, and before utilizing the dataset, its limitations should be acknowledged. Our annotations might not capture the full complexity of these issues, given problematic language is context-dependent, dynamic, and can manifest in different forms and different severities. Content moderation tools aren’t a silver bullet to address harmful online content. Problematic language is fundamentally a human-centric problem. It should be studied in conjunction with human experience, and tools to address this problem should be developed and deployed with human expertise and well-informed regulatory processes and policy. Multidisciplinary work is needed to better understand the aspects of this challenge.  

Also, this dataset only captures implicit toxicity (more precisely hate speech) for 13 minority identity groups and due to its large scale can naturally have imperfections. Our goal in this project is to provide the community with means to improve hate speech detection on implicit toxic language for the identified minority identity groups, and there exist limitations to this dataset and models trained on it that can potentially be the subject of future research, for example, including more minority identity groups, a combination of them, and so on that are not covered in our work. Stronger content moderation tools and systems can contribute to mitigating fairness-related harms in AI systems. For example, systems that don’t over-flag neutral statements with minority identity group mentions can help ensure better representation of diverse perspectives and experiences, while systems that can better flag implicit hate speech can support more inclusive technology.   

Acknowledgment 

This work was conducted by PhD students Thomas Hartvigsen and Saadia Gabriel during their internships at Microsoft Azure and Microsoft Research. Hamid Palangi, Dipankar Ray, Maarten Sap, and Ece Kamar served as advisors on the work. A special thanks to Misha Bilenko from Azure ML for making the compute resources available and to Microsoft Research for supporting our large-scale human study. 

Please note: This research, the GitHub repository, and examples from our work included in this blog contain and discuss content that is offensive or upsetting. All materials are intended to support research that improves hate speech detection methods. Included examples of hate speech don’t represent how the authors or sponsors feel about any minority identity groups. Hate speech applies to a range of minority identity groups; for the purposes of this research, we focus on 13 of them (as shown in Figure 1). Content moderation tools are part of larger content moderation systems. These systems also include human expertise and thoughtful policy and regulatory development. Even the most robust content moderation tools and datasets require systems with human supervision. 

The post (De)ToxiGen: Leveraging large language models to build more robust hate speech detection tools appeared first on Microsoft Research.

I’m just about out of patience with LinkedIn posts where a freshly-minted tech CEO says “instead of a week-long interview process and several assessments, I just have a conversation with the candidate. I can tell in about five minutes if they’re right for the job.”

No, they can’t. Although shorter interview processes are a good idea (excessively long ones will scare away the most sought-after candidates), the “conversation over coffee” interview approach is no better than choosing candidates at random. And that’s pretty bad.

What we’re witnessing from LinkedIn CEOs is not extraordinary intuition or social skills. It’s the Dunning-Kruger effect: people tend to overestimate their own competence at a task, and the worse they are at it the more they overestimate. Interviewing is one of the most well-studied examples. There’s not a single person in the world who can assess someone’s job skills in an unscripted five-minute conversation—or, for that matter, a much longer one. Unscripted conversations aren’t interviewing; they’re speed dating. Yet employers stubbornly believe they’re the most effective way to select high-performing candidates.

Study after study after study has proven them wrong. Unstructured interviews increase multiple types of bias, open the door to interviewer idiosyncracies, and reduce hiring accuracy by over half. They’re worse than useless, and not just by a little. And the impact of hiring mistakes is massive: the wrong candidate can cost up to a quarter of a million dollars and cause significant delays in the company’s projects.

Programming and evidence of ability

For programmers, bad interviews are especially frustrating. Programming is a task that produces self-evident results. You can’t assess the build quality of a bridge or the preparation of a meal just by looking at it, but with code there’s nothing under the surface: what you see is all there is. You can demonstrate your ability to write good code in real-time without any complex setup or tangible risk. Yet companies go out of their way to either avoid such a demonstration or ask for one that’s completely tangential to the job they’re hiring for.

Most of us have been there and done that. You’re interviewing for your first programming job and they want to know what element on the Periodic Table best describes your personality. You’re an experienced front-end developer but the interview is an hour-long “gotcha” quiz on closures and hoisting. You’ve built several .NET applications from scratch but they want you to invert a binary tree. You’ve contributed to the Linux kernel but now you have to guess the number of ping-pong balls that would fit in the interview room. The whole thing has been memed to death.

This is especially bewildering in context of the job market. Programmers are worth their weight in gold. Salaries are skyrocketing throughout the industry. Remote work has become table stakes. Tech startups are offering four-day work weeks to make themselves more appealing. Recruiters are second only to car warranty scammers in their efforts to reach us. But it all goes to waste if the candidate who gets hired isn’t able to do the job. So why aren’t companies investing more at the tail end of their hiring process?

An optimized interview process

Let’s talk about what an ideal coding interview could look like. If someone’s job is to write Java, the process might go like this:

Disclose the salary range and benefits in the very first contact.

Review their resume ahead of time for professional or open source experience writing Java (any other programming language would be relevant; Groovy and Kotlin would be equivalent experience).

Watch them write Java for an hour or so in the most realistic environment possible. Grade them on predetermined, job-relevant attributes, e.g. problem-solving, null safety, error handling, readability, naming conventions, and encapsulation.

Give a scripted, completely standardized assessment to ensure they understand the JVM and basic command-line tools.

Let them know when to expect a followup call.

That’s it. No unscripted “culture interview,” no free-for-all “group quiz,” no “compiler optimization test,” no asking the candidate to build a 15-hour project from scratch in their spare time. You assess their job-specific skills and you’re done.

It’s not that I think coding ability (or any one skill) is the only important thing about a candidate. Of course you want to hire people who are reliable, smart, honest, and kind. But you have to restrict yourself to what is knowable. And in the context of a job interview, there isn’t much. Most candidates act fake in job interviews and a significant percentage of them lie. Even putting honesty aside, 9 out of 10 people report having interview anxiety. If you want to know how someone behaves at work, you simply can’t rely on the interview. That’s not the real them. Obviously if you see any major red flags—being rude to the receptionist, say—you can throw out their resume. But those are rare.

That’s part of the inherent risk of hiring: interviews just aren’t as powerful as we’d like them to be. Many employers try valiantly to devise interviewing methods that predict soft skills and personality traits. But anything outside the limits of practical, measurable work is usually an illusion, albeit a compelling one—a 2013 study found that interviewers form highly confident impressions of candidates even when those candidates’ responses are randomly selected! People tend to trust their first impressions regardless of how accurate or inaccurate they are, which leads to mistakes in the hiring process. Any time you ask humans to intuit facts about other humans, you decrease the accuracy of your process. The only thing you can measure with any accuracy at all is task competence.

Methods for testing coding ability

So, if it’s the best we can do, what’s the ideal way to measure programming skills? Several commercial products offer a way to test candidates with pre-written, auto-graded tasks in a code playground. Although they’re likely much better than the norm, I’m not here to evangelize any of them. Why would you pay for another company’s general-purpose assessment when your entire development team does specific, real-world “assessments” all day long? The last time I was asked to devise a coding test for an interview I looked over my recent work, picked a section of code that went well beyond “glue code” but didn’t get into anything proprietary, tweaked it so it would stand on its own, and copied the function signature into LINQPad (a code playground). The interviewee’s task was to implement the function according to nearly the same requirements I’d needed to meet. And when they were finished, I knew if they could do the job because they had just done it. It’s hard to get a more realistic assessment than that.

On the other side of the table, by far the best interview process I’ve ever gone through was with a startup who put me on a one-week contract (an “audition” as some companies call it) to work with their team. They assigned me a task; I cloned their repo, wrote code over the course of a couple evenings, and submitted a PR; the rest of the team gave feedback; I did some more work to bring the PR up to their standards; and they merged it and paid me at my regular rate. Even though I didn’t ultimately accept their job offer, everyone walked away happy—they got some extra work done and I got a paycheck. If I had accepted, there would have been no uncertainty about my skills or ability to work with the team. That kind of organizational competence is hard to come by. Years later, they’re still on my shortlist if I’m ever on the hunt for a job.

Maybe that doesn’t fit your definition of “interview.” I say it’s something much better than an interview. Can you imagine trading that experience for 15 minutes shooting the breeze with their CEO? It’s a ridiculous proposition. Their CEO doesn’t even code.

For security or policy reasons, some teams may not be able to offer such a highly practical interview process. In those situations, consistency is a close second to realism. If you can’t make the interview look like the job, at least make one interview look like the next—give yourself the ability to compare candidates on the same metrics by following the same script every time.

Helping every candidate be at their best

A consistent, scientific interview process doesn’t have to be completely rigid. You may actually hurt yourself by trying to standardize things that aren’t relevant to what you’re measuring; every candidate has different needs. For example, immunocompromised candidates need to have the option to interview remotely to protect their health. Candidates with physical or neurological conditions may fare better if the interview is split into short segments with frequent breaks rather than a three-hour marathon. Some candidates with interview anxiety may appreciate a moment or two before the interview to establish a friendly rapport with the interviewer; for others, small talk is difficult and they’d prefer to skip that part. You should always be willing to adjust the environment or cadence of the interview to the candidate’s needs. Accommodations like these are a win-win because they allow each candidate the best opportunity to demonstrate their skills. If you want to hire the best candidate, you need to see the best of each candidate.

Also, interviews are a two-way street. You’re being measured just as surely as the candidate is. Neglecting to answer their questions because “it isn’t in the script” would be a mistake.

How do you reconcile your need for objective, comparable data with your candidates’ needs for reasonable accommodation and reciprocity? The simplest answer comes from the robustness principle, a famous rule of software design: “be conservative in what you do, be liberal in what you accept from others.” To apply it here, you should assess candidates conservatively (narrowly) on the parts of the interview that are standardized and focused on relevant skills—but you should be liberal (flexible) about whatever else they may need. You’ll still get the data you want and your candidate pool won’t be restricted or misrepresented by unnecessary rules.

Conclusion

Structured, realistic, job-focused interviews are the gold standard according to several decades of research. So when you decide how much of the process should be unplanned or how much of the evaluation criteria should be unquantifiable, the only question you should ask yourself is how often do I want to select the wrong candidate?

Of course, sometimes the wrong candidate will slip through. That’s not totally preventable. The best interviewing methods are only mostly accurate. But that shouldn’t scare you into an intuition-based interview process any more than the occasional computer failure scares you into doing everything on paper.

Hiring wisely is perhaps the most powerful advantage a company can have over its competitors, and the best way to do it is to focus on what you can measure. 

The post The science of interviewing developers appeared first on Stack Overflow Blog.

The home team talks about the past, present, and future of crypto; good reasons to go public with your open-source project before you think you should; and the importance of test-driven development.

Episode notes:

Highly-touted cryptocurrencies like TARA don’t always solve the problems they’re supposed to, as Bloomberg reports.

If you’re looking for a compelling deep-dive into a crypto scammer, Cassidy recommends BBC podcast The Missing Cryptoqueen.

Ceora is working to improve the quality of her commit messages in order to turn what’s now a personal project into an open-source project that others can contribute to. One great resource she’s found: Zen and the art of writing good commit messages.

Attention devs: if you have tips for basic project maintenance and hacks for improving commit messages, Ceora wants to hear from you.

Read up on the benefits of test-driven development.

Today’s Lifeboat badge goes to user Nina Scholz for their answer to What’s the difference between Object.entries and Object.keys?.

TRANSCRIPT

The post Make your open-source project public before you’re ready (Ep. 444) appeared first on Stack Overflow Blog.

Welcome to ISSUE #126 of The Overflow! This newsletter is by developers, for developers, written and curated by the Stack Overflow team and Cassidy Williams. This week: the new Developer Survey is here and ready for your input, whether amateurs can add value to math research, and the problems with yellow in design systems.

From the blog

The 2022 Developer Survey is now open stackoverflow.blog The 2022 Developer Survey is ready for your input.

An unfiltered look back at April Fools’ 2022 stackoverflow.blog As with any good joke, the most important part is the resulting data.

New data: Do developers (and their employers) care about their wellness? stackoverflow.blog As May is Mental Health Awareness Month, we wanted to see what developers are doing to decrease that stress and prioritize their own wellness. Earlier this year, we surveyed over 800 developers to see if they are happy at work and what they are doing to maintain or improve mental health.

Feeling burned out? You’re not the only one (Ep. 440) stackoverflow.blog Friend of the show Jon Chan, Stack Overflow’s Director of Engineering, Public Platform, joins the home team to talk about burnout: what it is, how to prevent it, and how to recover.

MongoDB World 2022, the conference for developers promotion Join MongoDB from June 7-9 in New York City for three days of sessions, workshops, and more. Save 25% on your ticket with discount code WORLDNL25.

Interesting questions

Dealing with a sequence of bad luck early in my career as a dev? How to address on the resume? workplace.stackexchange.com “If you’re getting interviews, then you probably don’t have much to worry about.”

Is amateur research in mathematics viable? mathoverflow.net Hobbyists have made a number of contributions to mathematics.

Fastest 8-bit microprocessor for multiply-accumulate? retrocomputing.stackexchange.com So what’s the fastest slow microprocessor?

How exactly is hypoglycemia a problem? medicalsciences.stackexchange.com Have you ever gone to the grocery store and found it didn’t have food?

Links from around the web

How FedEx influenced the React logo maykelloomans.com An unlikely duo. The ultimate crossover. Did anyone expect it?

The “dark yellow problem” in design system color palettes  medium.com If you want to use yellow in your design system, there are quite a few considerations to keep in mind.

React key attribute: best practices for performant lists www.developerway.com When you loop things in React, you add keys. But…why?

Syncing text files between browser and disk using Yjs and the File System Access API  motif.land This is a super practical look at using the new File System Access API with a Conflict-free Replicated Data Type (CRDT) architecture. You’ll learn a lot!

The post The Overflow #126: The 2022 Developer Survey now open appeared first on Stack Overflow Blog.

The concept of future-proofing your code is one that perennially pops up in conversations about software. It sounds magical—who wouldn’t want to make your code impervious to the future?

Of course, the reality is much less rosy and much messier.

In this article, I’m going to discuss what I think people mean by “future-proofing,” how you might be able to accomplish it, and when and why it might be a bad choice.

How does one “proof” one’s “future?”

You can think of future-proofing more accurately as change-proofing. We can define it as:

“Making a design, architecture, or programming decision that allows for future changes to be easier to manage, take less time or result in fewer changes to the overall code.”

These changes can fall into a number of different categories:

Changes to scale – the project has to do more of the things it’s doing, whether that means handling more traffic or work items, processing them faster, etc.

Changes to requirements – new information has entered the business (or has entered the engineering team from the business) and now the system needs to change to accommodate them.

Changes to technology stack – switching to a different data store or programming language, for example.

Changes to integrations – the project needs to talk to a new third-party application, either on top of or instead of an existing one.

Changes to schemas – we want to change the fields that define our data objects.

Should I stay or should I grow?

The main issue with future-proofing is that it runs slap-bang into one of the central tenets of software engineering: YAGNI, or You Ain’t Gonna Need It. 

This principle states that until you actually know you’re going to have a change, you shouldn’t code your software in a way that anticipates that change. Violating this principle results in bloat, confusing and unnecessary abstractions that make the code harder to understand, and often mounds of tech debt.

However, if you never cast your eyes to the future, you really can run into issues where just a little extra bit of work upfront could have saved you months of it down the road.

So… should you future-proof your code? The answer, as with practically everything in software, is, “it depends.”

I see this as a spectrum of sorts:

Every situation is unique, of course. But some kinds of changes are much less likely to deserve future-proofing, while in others, the extra work is more likely to pay off and (just as importantly) not result in significant downsides.

Future-proofing strategies

In general, any time we want to protect ourselves from future changes, we would engage in one of two strategies.

Modularization

Modularization is the act of splitting your code up into smaller chunks. Every piece of software has some level of modularization—otherwise, your code would consist of a single enormous file that had nothing but unrolled functions and primitive types. (These programs do exist, but no one in their right mind wants anything to do with them.) 

Increasing modularization allows you to isolate your changes to a single module, and/or to more easily swap modules out for each other. 

You can also have modularization at different levels. Extracting code into a function is a simple act of modularization that rarely has a downside until you’re at the point where your functions are so small and numerous that they’re hard to keep track of. However, you can extract things into a class, a sub-application, a microservice, or even a separate cluster; and at each level, while the isolation and flexibility goes up, so does the cognitive and administrative overhead of managing your different components.

Abstraction

Abstraction is the mental model you have of the parts of your code. A module, function, class, etc. with low abstraction (or higher specificity) more closely models the action it takes or object it represents. A thing with higher abstraction brings your reasoning up a level. For example, instead of your code working with Cars, it could work with Vehicles. That way, if you ever need to start handling Motorcycles, the change necessary to do so is mitigated by the fact that you were never really talking about Cars to begin with.

The higher you go with abstraction, the more flexible you make your code, but also the harder it is to understand. Everyone knows what a Car is, but when you start working with Vehicles, it becomes more difficult to work with Car-specific things like booster seats and steering wheels. And if you keep going higher up the abstraction tree, you might find yourself coding around Transportation, ManufacturedGoods, even plain old Objects, and have a very hard time figuring out how to fill up your tank.

Types of changes

Lets go through each of the types of changes I listed above. For each, I’m recommending where on the spectrum your future-proofing efforts might lie. In general, I will be applying these suggestions for when you have no current indication that the change will happen. As the change becomes more likely, you’d move further towards the left side of the line.

Changes to scale

Making your code and architecture robust and able to handle whatever you throw at it is a core facet of design. However, part of that architecture and design has to be what kind of traffic you expect it to have. The higher scale you have to handle, in general, the more complex your architecture needs to be.

Making a simple app that’s designed to be used by a handful of internal users could be done via a Rails or Django application with a minimum of JavaScript, for example. But if you need to be able to handle thousands or millions of concurrent users, that’s just not going to cut it. You’re going to need lightning-fast services or micro-services, horizontal scaling and auto-scaling, and probably a more responsive front-end and more tailored and complex caching mechanisms.

Over-architecting all of this upfront when you have no anticipation of your app actually scaling to that size is flagrant YAGNI. In fact, building your system deliberately for your current scale even if you know you may have to rewrite it when the scale changes is a perfectly valid strategy, sometimes called sacrificial architecture.

Having said that, there are some scaling considerations you should take into account. Leaving in lots of N+1 queries or having unnecessarily large or frequent requests is bad engineering practice all around and will leave a bad taste in your users’ mouths—regardless of how many or few they are.

Changes to requirements

This case (where your code has to change or add to its behavior) is the hardest one to guess at, as well as the one most likely to happen.

This is a case where general modularization can help you, but I’d recommend against trying to prematurely add abstraction to your code.

In particular, if we’re talking about business rules, isolating the rules themselves in a way that makes them easy to test and change is an overall good practice.

As a simple example, here’s some code working with a product that has some rules around what a valid product can be considered:

product.save if product.price >= 0 && !product.name.nil?

Products might be saved from many different places in your application. Rather than the calling code checking these business rules, we can extract them to their own method:

class Product def valid? self.price >= 0 && !self.name.nil? end end product.save if product.valid?

We can even go further and have a separate class or module that handles this check:

module ProductValidator def self.valid?(product) product.price >= 0 && !product.price.nil? end end product.save if ProductValidator.valid?(product)

These are relatively minor changes and keep your code clean and easily testable, and can isolate further changes of this sort to the one place. 

The key words are “of this sort”, however. If the requirements changes involve completely changing how the behavior of a feature acts, or adding new features, that’s where you want to start putting your foot down more firmly on the YAGNI side of things and only building what you actually know.

Let’s imagine that your product can be purchased:

class Product def purchase # contact the purchase service and complete transaction end end

Wait a minute, you reason. We might eventually expand into not only buying products but also selling them! Maybe renting them? We should abstract this out:

class Product def perform_action end end class PurchasableProduct def perform_action # contact the purchase service and complete transaction end end

This is classic YAGNI over-abstraction. People reading your code won’t see you purchasing a product, you’re “performing an action” on something that resolves to purchasing. 

You have no reason to believe that your business actually will expand into further operations. You’ve introduced a level of abstraction that removes your code one extra step from what’s actually happening and what’s being modeled.

Changes to technology stack

I’ll come straight out and say it—there is just no good reason to build your app in a way that you plan to eventually change your basic technology stack.

This is obviously almost impossible to do from a programming language perspective. The only way you could provide more isolation is to go further into microservices—which might make sense from an architecture perspective, but not from a future-proofing one.

As for data stores, no one—no one—actually changes their database from MySQL to Postgres or vice versa. If you’re using an open-source or open-standards way of interacting with your data, go all into it and don’t look back. At this point, the similarities far outweigh the differences, and if you do need to switch for whatever arcane reason, you’ll need a full regression suite anyway to make sure nothing else breaks.

(Note: This point is often used to discourage the use of ORMs. I think ORMs have other very good advantages, but the ability to switch data technologies is not a practical argument for their use.)

Changes to integrations

In this case, we are worried about having to change how we talk to some third-party application. These services can provide things such as metrics, tracing, alerting, logs, feature flags, object storage, deployments, etc. In this case, it’s nice to have the freedom to change who you’re doing business with based on cost, feature set, ease of use, etc.

I am most inclined to ensure that we have abstraction and modularization around integrations. In many cases, these third-party applications can be fairly easily switched out for each other. Once you’re providing metrics or tracing, for example, the abstractions and ideas are very close to each other regardless of which provider you’ve signed up with.

What I prefer doing is building a facade around any code that needs to talk to a third-party system. Often to keep things simple, this is a wrapper around the API of whatever provider I’ve chosen, which exposes all the functionality necessary to operate.

This facade often takes just a couple of days to build (if there isn’t already one out there) and can be used in whatever project needs it. I’ll often add a set of team or company defaults that most closely match our most common use case so that new projects don’t need to figure it out for themselves.

This facade allows you to avoid vendor lock-in by not tying your systems to any specific provider—but the overhead is small enough that it shouldn’t confuse you or your teammates.

One thing that is important is to allow you to break out of the facade by accessing the internal client or API if necessary. If you’ve chosen your provider because of specific features that need to be used, then you shouldn’t need to tie your arm around your back to prevent you from using them. However, it means that if you do end up switching your provider, you don’t have to make any changes to the “normal” case—you only need to look at code that accesses the internal client or API and focus your work there.

Here’s a rough approximation of a feature flagging library we built for internal use:

module FlippFlags def backend=(backend) self.backend = backend end def client(config) self.backend.new(config) end end class FlippFlags::Backend def initialize(config) # this is a Ruby version of a constructor, i.e. the "new" keyword @client = ... # initialize the actual client end def enabled?(flag, user) @client.enabled?(flag, user) end def internal_client @client end end # application code FlippFlags.backend = FlippFlags::MyFlagProvider client = FlippFlags.client(my_config) client.enabled?("flag_name", user_object) # true or false

Here you can see that the only change you need to make is to switch the backend that is passed into the library, and it seamlessly switches to a different provider—or maybe even an internal class that acts the same way! Otherwise, you use the client exactly the way you’d use the internal client, but without tying your code directly to that implementation.

Changes to schemas

The shape of our data reflects our understanding of it. As we gain more understanding, we often want to change its shape. This could mean adding or removing fields, changing field types, default values, etc.

Making our data schemas backwards and forwards compatible is possible by following some simple rules. Technologies such as Avro and Protobuf specify those rules and have built them into their tooling. If you’re not using those tools, though, you can “soft-enforce” similar rules whenever you change your own schema to ensure it’s unlikely that your changes will break whatever depends on your data.

Having said that, there are cases where you simply can’t follow those rules—and that’s okay! This is when you specify a migration path to go from the old data to the new data. But the rules make it so you shouldn’t need to follow that onerous process for every single data change you go through.

Conclusion

Future-proofing is not necessarily a goal of software development so much as one of the many “ilities” that affect your design. Overcorrecting on this spectrum can lead to unnecessary work, tech debt, and confusing abstractions. Finding the sweet spot, though, can save you time and effort down the road.

The post Crystal balls and clairvoyance: Future proofing in a world of inevitable change appeared first on Stack Overflow Blog.

You may be running your code in containers. You might even have taken the plunge and orchestrated it all with YAML code through Kubernetes. But infrastructure as code becomes a whole new level of complicated when setting up a managed Kubernetes service. 

On this sponsored episode of the Stack Overflow podcast, Ben and Ryan talk with David Dymko and Walt Ribeiro of Vultr about what they went through to build their managed Kubernetes service as a cloud offering. It was a journey that ended not just with a managed K8s service, but also with a wealth of additional tooling, upgrades, and open sourcing. 

When building out a Kubernetes implementation, you can abstract away some of the complexity, especially if you use some of the more popular tools like Kubeadm or Kubespray. But when using a managed service, you want to be able to focus on your workloads and only your workloads, which means taking away the control plane. The user doesn’t need to care about the underlying infrastructure, but for those designing it, the missing control plane opens a whole heap of trouble. 

Once you remove this abstraction, your cloud cluster is treated as a single solid compute. But then how do you do upgrades? How do you maintain x509 certifications for HTTPS calls? How do you get metrics? Without the control plane, Vultr needed to communicate to their Kubernetes worker nodes through the API. And wouldn’t you know it: the API isn’t all that well-documented. 

They took it back to bare necessities, the MVP feature set of their K8s cloud service. They’d need the Cloud Controller Manager (CCM) and the Container Storage Interface (CSI) as core components to have Vultr be a first-class citizen on a Kubernetes cluster. They built a Go client to interface using those components and figured, hey, why not open-source this? That led to a few other open-source projects, like a Terraform integration and a command-line interface. 

This was the start of a two-year journey connecting all the dots that this project required. They needed a managed load balancer that could work without the control plane or any of the tools that interfaced with it. They built it. They needed a quality-of-life update to their API to catch up with everything that today’s developer expects: modern CRUD actions, REST best practices, and pagination. All the while, they kept listening to their customers to make sure they didn’t stray too far from the original product. 

To see the results of their journey, listen to the podcast and check out Vultr.com for all of their cloud offerings, available in 25 locations worldwide. 

TRANSCRIPT

The post Building out a managed Kubernetes service is a bigger job than you think appeared first on Stack Overflow Blog.