Truth is, even when he was home he didn't spend all his time with us. He'd go out to play cards during the day and go out drinking at night, so the time we spent together was mostly at mealtimes and before bed.

Even so, the feeling of him being there was completely different.

After I started working, I became the first one to leave home after Spring Festival.

Later, when I went home, I'd sometimes just sit on the sofa reading or playing on my phone. My sister would ask why I didn't get out more and talk to people, and I'd answer with what I've always thought: every year I come home and just sit here without saying a word, and that alone makes everything different.

Looking back now, much of the time it really is a comfort just to have the people who matter there. Everything else is secondary.

The day before the Qingming holiday (a Chinese spring festival for tending ancestors' graves), I went to bed half an hour early. On the first morning of the holiday I was bursting with energy and raring to go, and even dashed off a blog post, A Wrong Map Is Better Than No Map.

I thought, if half an hour early makes this much difference, imagine what happens if I go to bed even earlier. I'll take off.
So that day I tried sleeping at nine. I woke up at three in the morning and could not for the life of me fall back asleep, scrolled my phone for half an hour, and then managed to sleep through to seven.

I figured that with all that sleep I'd be full of energy yesterday. Instead, after breakfast and a coffee, I sat down at my desk and started nodding off, yawning nonstop. I flipped through a few ebooks and felt like I couldn't catch my breath, then picked up a paper book on alternative data and found it dull too.

So I stopped pretending and went back to sleep a little after ten in the morning, drifting in and out until noon. I got up, had lunch, and thought, surely I've slept enough now; I'll grab a coffee and grind straight through to dinner. But I was in the same state as in the morning, couldn't even hold a book, and before three I couldn't hold out any longer. I even had a bit of a physical headache. Damn it. Back to fucking sleep.

I drifted along until after five, then biked out for some Henan food. Riding past the bustling Big Roof, I finally felt a little bit alive again.
I was still sleepy in the evening, but to get back to a normal rhythm I forced myself to stay up until eleven, then slept straight through to seven-thirty.

When I woke up this morning, that muddled feeling from yesterday was finally gone. I was raring to go again and spent a very productive morning reading.

Comparing these three days, I really felt how important it is to look after your energy. Being driven and clear-headed versus being drowsy and worn out, with your head under a slow fog: those are two completely different people.
Yesterday I even started to suspect I'd gone lazy again, that Lu Shen's influence had worn off, and for a while I felt a real fear of backsliding. But comparing it with today, I'm sure it was only an energy problem, not a willingness problem.

Your body clock is hard to break casually. Right now the sleep window that suits me best is from eleven at night to seven-thirty in the morning. Minus the time it takes to fall asleep, that's just about eight hours. Once I drift outside that window, even drinking coffee like water doesn't do shit.

Lazy men and lazy women may just be products of low energy

Lazy men and lazy women are probably the most off-putting creatures in this world. Even when you want to help them, it feels like trying to prop up a pile of mud against the wall.

I'm not naturally a very energetic person. According to my BaZi (Chinese birth-chart astrology), I'm short on Fire among the five elements, so my spirit tends to turn inward.
In my own experience this actually matched up, and so it quietly became a self-fulfilling prophecy. In the past I often felt low on energy and assumed it was just my constitution. But comparing different ways of living over the past few years has convinced me that's really not the case. Without the comparison, you simply can't tell what a good state of living feels like.

Enough sleep at the right time (7 to 8 hours) + consistent exercise (for me, biking to and from work when it isn't raining, 25 km round trip): keep it up for half a year and your cardiovascular fitness and energy go up a level. As long as you don't break the routine on a whim, most days you'll have plenty of energy.

The old habits of only wanting to lie down on holidays, not being able to sit still at my desk, and not bothering with housework, all more or less learned helplessness, have completely disappeared, and I've come to experience the steady, always-improving way of living that Lu Shen has.
So much so that these days I really can't stand the lazy-man, lazy-woman state, including but not limited to:

1. Staying up late at night, numbly scrolling your phone and binge-watching shows
2. Lazing around in the morning and not being able to get out of bed
3. Being listless all day, never sitting when you could lie down
4. Doing housework sloppily, for example washing the dishes but not cleaning the sink, leaving paper towels everywhere, closing the window but not the screen, always doing only 80%. This matters especially for keeping a clean and tidy home: let that remaining 20% pile up for three days and the place becomes a nightmare.
5. Reading, studying and working in fits and starts: fishing for three days, drying the nets for two
6. ...

But all of these problems can be fixed bit by bit as long as you stay energetic.
And to stay energetic, a certain amount of self-discipline is indispensable. Especially for people who lack a sense of meaning and mission, it may even take some external constraint to complete the "enlightenment" process.

And once someone has experienced the contrast between the two states, I don't think anyone would fail to prefer a tidier, fuller life.

A quarter of 2024 is already gone. A night of rain has added a little more melancholy to the Qingming festival (the day Chinese families tend their ancestors' graves), but I'm in Hangzhou and can't go home to put fresh earth on the old graves of my forebears. After breakfast I made a cup of coffee and sat down by the window to sort through what I've been thinking and feeling lately.

My biggest cumulative gain of the whole first quarter was finishing Scholar's Advanced Technological System (a Chinese web novel), a hardcore science-fiction novel let down by its title. The author's academic range and prose are both solid. The main thread follows Lu Zhou, an academic prodigy, through his explorations in mathematics, as he pushes physics, biochemistry, materials science, energy, information science and human civilization as a whole forward.

Some people say a few of the book's details are quite wrong. Others say that even they, as outsiders, can tell the author is fooling outsiders.

To me, the first is a normal criticism. The second is just short-sightedness; that reader might as well not have read the book.

For me, the book pushed further the process of connecting the dots between some key experiences from my past, and it gave me a framework-level understanding of the central disciplines of today's world and their central questions. That is exactly why I called it a "cumulative gain" above: accumulation presupposes some consistent main thread. For me the big thread is how far I have come in experiencing and understanding the natural world and human society. The more focused sub-threads are investing and computing. But as I said in About me, understanding the world is the bigger question, the foundation of all foundations.

Which dots got connected?

The ACM campus contest, Ah Huang's transcendental number, USTC's admissions exam, a primary-school contest

When I took part in the campus round of the ACM programming contest in college, one problem involved very large numbers, that is, numbers beyond what 32-bit or 64-bit binary can represent. You need something like the built-in long long type some languages provide. Whether a language has built-in big-number handling isn't really the point. The point is that I had no background knowledge about big numbers at all, so I designed the algorithm with ordinary integer and floating-point types. Naturally the precision wasn't enough and the results were way off.

Only later did I learn that this is "the problem of limited precision in how computers and the C language represent data". My first reaction on learning it was: "What the hell, our college C course never taught this. Isn't this problem beyond the syllabus?"

Once in high school I ran into a hard math problem and felt I had cobbled together an answer. When I discussed it with Ah Huang, a classmate sitting next to me, he glanced at my result and said: your answer is wrong, because the problem is about a polynomial with integer coefficients, and your answer has a transcendental number in it. Me: ?????

Before that, I went from my county town to Zhengzhou to sit the independent admissions exam of USTC (the University of Science and Technology of China). Almost every question on the paper involved knowledge I had never seen. Beyond the syllabus, absolutely beyond the syllabus, damn it.

An even earlier memory: in first grade I entered a competition and did terribly. Later I heard that only second or third graders, or even third and fourth graders, could have had the knowledge it required.

A flash of insight: the valley of death on the learning curve, and the point of enlightenment

At first I assumed situations like these were much the same for everyone my age: some knew a bit more, some a bit less, but no qualitative difference.
So I never dug into it, and unfortunately no one was around to point it out to me. Only after several years of work, meeting more people and seeing more things, did my horizons widen enough for me to gradually understand what it meant.

1/ The point of enlightenment in learning.
On the learning path of every skill or discipline there is a critical point on the climb, which I call a true point of enlightenment, in exactly the sense Kant meant in his essay "What Is Enlightenment?".¹
Past this point, the student has walked out of the valley of death and can use their own reason to explore the much larger map still covered in fog.
Fail to get past it, and once the guidance stops, they keep circling and repeating at a low level, or even slide backwards.

2/ How to tell whether someone has reached it.
My own test is whether the student has built a small framework map out of what they already know, whether they have a vague picture of where things go from here, and whether, when they meet a suitable piece of knowledge, they can judge without anyone's help whether it fits into the map they already have and connects to it.

Only then has the master truly "led them through the door" (as the Chinese saying goes, the master leads you through the door, and the practice is up to you).

Only then do they have the foundation for taking in a high-volume flow of information.

3/ High-volume information input.
From what I later learned about the admissions results, plenty of people could indeed handle the questions that were beyond my syllabus. That was when I realized what "real teaching resources" mean, and what "material prepared for students with capacity to spare" means. The horizons of big-city teaching teams and the learning resources available there are simply not something a small county town can match, and they produced a cohort of talented students with very wide horizons. The questions that were beyond our syllabus had long been within their view. The shock I felt when Ah Huang mentioned transcendental numbers made me feel the gap deeply, and later I found many students like Ah Huang among my college classmates.

When I was at Liulou Primary School, my family didn't own a single book outside the textbooks. I coveted the school's little library room, but it was never opened to students before I graduated. In fourth grade I worked up the courage to ask the headmaster if I could borrow a book. He said all he had was Marx's Capital. I said Capital would do. He said even he couldn't understand that book, and didn't take me seriously. In the end I never borrowed a single book.
Going to middle school in the county town was painful but the right thing. With money I saved I bought Sherlock Holmes and Science Fiction World (a Chinese science-fiction magazine), and was lucky enough to pick up some popular-science magazines at a stall selling miscellaneous books. Sherlock Holmes and Science Fiction World in particular still have a very visible influence on me today.
Of course, back then nobody sold advanced reading for the main academic track. The only advanced book I read in high school was Chen Yuezeng's General Biology, borrowed from Ah Huang, and it influenced me a lot too.

Come to think of it, it makes sense. Having never been exposed to any of that, how could I possibly answer those questions by transferring knowledge?
When the difficulty in front of you lies too far beyond your knowledge and horizons, the gap is hard to cross. Besides, some knowledge simply cannot be derived from more basic knowledge. You can't work out integration on the spot in an exam hall from basic arithmetic.

4/ Drop the idea of "beyond the syllabus" entirely.
Perhaps this is what competitions are for. They tell contestants that there is no syllabus in any absolute sense in this world, and therefore no such thing as "beyond the syllabus". They demand that contestants broaden their horizons, recognize that a bigger picture exists, accumulate, lift the fog covering the areas humanity has already explored bit by bit, and then explore the edges of the picture.

For people with plenty of life experience this is easier to grasp. No one is going to draw a boundary around the problems you face in real life and promise you'll find the answer as long as you search inside it.

And to let go of this idea and look for solutions across a wider range, building a vague but global framework matters most of all.

Columbus and the maps of the Age of Discovery

Before the Age of Discovery, European maps mixed science, speculation and theology. They covered only the known world of Europe, parts of Asia and the northern coast of Africa, and their accuracy varied enormously. Take, for example, the world map drawn in the late 15th century by the cartographer Henricus Martellus Germanus ²:

Columbus, a navigator of the same late 15th century, relied early on precisely these error-ridden maps. The Age of Discovery he went on to open fundamentally changed mapmaking: as explorers returned from new lands, cartographers folded the new information into their maps, vastly extending the reach of world maps and improving their accuracy.

In hindsight, those early maps, full of holes as they were, gave people a macro framework of the world and a precious sense of finiteness. That finiteness and framework gave the courageous pioneers more confidence to explore the whole world and to look for new sea routes, and that exploration would in turn correct the errors and make both the framework and the details more precise.

A map full of errors is far better than no map at all

If you can meet or find a teacher, textbook, novel, article or any other means that suits you and gives you a top-down framework map, please don't miss it, and certainly don't give up such a precious opportunity just because someone says it has a few small mistakes.

The Web's original paradigm metaphor: it started as "a hypertext application"

1980: Tim Berners-Lee and ENQUIRE

From June to December 1980, Tim Berners-Lee worked as an independent contractor at the vast CERN ¹.

While there, he learned that CERN had roughly 10,000 staff, each with different hardware, software and personal requirements, and that much of the work got done through email and file exchange.

He clearly saw a Tech Space here, that is, room to use technology to help researchers share and update information with one another. So he built a new hypertext prototype system called ENQUIRE².

Looking at it from today's vantage point:

1. This prototype was really a simple, app-level hypertext program, something like today's Wikipedia, or note-taking systems such as Notion, Flomo and Obsidian.
2. Any software designed at the app level is similar: the system defines too much around its particular use case and requires users to organize their content according to its rules, which means its extensibility is bound to be limited.
3. What the real Web needed was something that connects and organizes all these app-level things, Wikipedia, Notion, Flomo, Feishu, Google Docs, Obsidian and so on, at a higher level.

Clearly, only a higher degree of abstraction could deliver that more general capability.

1989: A higher abstraction

After leaving CERN at the end of 1980, Tim Berners-Lee spent three years in business related to computer networking (at the time the bottom layer of networks was TCP/IP, with various application-layer protocols such as FTP on top). As someone with deep technical understanding and real insight into how a business runs, and working on a real-time remote procedure call project at that, he evidently absorbed a great deal of network design thinking from that experience. In 1984 he returned to CERN.

More experience, and his years of work on computer networks, made him realize that hoping to unify CERN's documentation through one app-level tool was unrealistic. He needed some way to link different documentation systems (many of them coexisted at CERN, including his own ENQUIRE):

Creating the web was really an act of desperation, because the situation without it was very difficult when I was working at CERN later.
Most of the technology involved in the web, like the hypertext, like the Internet, multifont text objects, had all been designed already.
I just had to put them together. It was a step of generalizing, going to a higher level of abstraction, thinking about all the documentation systems out there as being possibly part of a larger imaginary documentation system.
But then the engineering was fairly straightforward. It was designed in order to make it possible to get at documentation and in order to be able to get people — students working with me, contributing to the project, for example — to be able to come in and link in their ideas, so that we wouldn't lose it all if we didn't debrief them before they left.
Really, it was designed to be a collaborative workspace for people to design on a system together. That was the exciting thing about it.

—— Academy of Achievement interview with Tim Berners-Lee, 2007: Visionary of the Internet ³

(By "the situation without it" he means the pain of working across many documentation systems at once. "The Internet" here is the underlying internet, which is not the same thing as the World Wide Web.)

Based on these ideas, he designed and submitted the Web proposal ⁴, shown below:

He also built humanity's first web browser, the first web server, and a prototype website, http://info.cern.ch ⁵

The earliest prototype contained three core technologies, and you can tell from the names alone that they were born to "link different hypertext systems":

1. HTML: HyperText Markup Language.
   The markup (formatting) language for the web.

2. URI: Uniform Resource Identifier.
   A kind of "address" that is unique and used to identify to each resource on the web. It is also commonly called a URL.

3. HTTP: Hypertext Transfer Protocol.
   Allows for the retrieval of linked resources from across the web.

Again, looking at it from today:

1. These three elements are highly abstract. By following a few simple conventions, a user can run a documentation system entirely under their own control (rather than writing their documents inside a software structure someone else controls), or adapt an existing documentation system so that it can link to others. This forms the basic framework of the modern Web.
2. The first browser was already a container space that could hold many different websites.
3. The primary motivation at this point was linking different documentation systems. Unlike the operating systems of the day, it was not driven by heavier business-workflow needs, so naturally there was no CSS or JavaScript, which only came later.

All in all, the Web's basic architecture was settled from the very first version.
A highly abstract mechanism at the application layer (still far more concrete, of course, than lower-level abstractions like TCP/IP), plus a natural, distributed openness, formed an enormous space for collaboration.

Tim Berners-Lee's original top-down design had achieved its goal. What remained were larger forces of the era: the growth of personal computers, the businesses built on them, networks becoming a high-speed information medium for human activity, and so on. These began to drive the Web's development in the other direction, from the bottom up.

1991 – 2007: Three browsers, Mosaic, Netscape and IE

This was the early era of browsers. Its landmark events include:

1. 1993: CGI
   The National Center for Supercomputing Applications (NCSA) released the NCSA HTTPd web server, which included an early version of CGI. This allowed server-side web programs to run in response to page requests, marking the birth of dynamic content on the web. Dynamic content was quite limited back then, of course: submitting a form, that kind of thing.

2. 1994: CSS is invented
   CSS was first proposed by Håkon Wium Lie on October 10, 1994. In 1996 the W3C endorsed it as the CSS1 Recommendation. Its original goal was better typographic control over documents, mainly document layout; it didn't even have a well-defined "frame layout" mechanism.

3. 1995: JavaScript
   In 1995, Brendan Eich, working at Netscape, invented JavaScript, and it shipped in version 2.0 of the Netscape browser. There was clearly a business need here too, and as a new technical feature it helped Netscape's browser stay ahead.

Back to our present-day view again:

1. In this phase, the context of the time and concrete needs limited people's imagination. Many designs were driven bottom-up by business needs, with little forward-looking design.
2. CSS was originally aimed mostly at document typesetting (the way Word does it). It struggled when used for two-dimensional page layout, and many capabilities that are routine in graphic design had to be achieved by hacking a one-dimensional flow layout.
3. JavaScript was originally meant for small tasks inside web pages; it was not a carefully thought-out language. Fortunately, the requirements for a universal Turing machine are simple, and JavaScript is Turing complete too (it has sequential execution, conditional branching and loops). Given enough APIs, it can in theory do anything any other language can.
4. Because the Web is open, as the system grew, web developers and browser makers had to stay compatible with and accommodate one another. Every web developer wants their pages to render correctly in browsers, and every browser wants to render every page. Together this created strong path dependence on technologies already implemented: anything new had to be compatible with what already existed. That guaranteed openness and usability, but it also piled up a large historical debt for the Web.
5. Standards bodies with real authority began to emerge. The W3C, led by Tim Berners-Lee, has led the HTML and CSS specifications all along, and the venerable ECMA took over standardizing JavaScript⁶.

Personally, I think these three browsers and the new technologies they introduced all belong to one paradigm of thought: they kept moving forward along the path of Tim Berners-Lee's browser prototype.

In this period users had limited interaction with content; web pages were purely informational and static. But that was bound to be a short-lived phase.

Because the Web is public and easy to access, humanity had for the first time a shared information medium, a tool no earlier era ever had. Demand was simply enormous. Quantitative change, piling up, brings qualitative change: bottom-up demand kept pushing Web technology forward and placed ever higher demands on the engineering design of the Web as a whole.

Once people no longer wanted websites to be simple text pages but complete apps that could carry all kinds of business logic, what they wanted from the browser also shifted, from a single tool for displaying pages to an all-purpose, OS-like platform.

The Web's paradigm shift: from "a hypertext application" to "an operating system platform"

From here on, we're back to things most of us know well.

2000 – 2010: The rise of Amazon's cloud services

This too is background from the era. The main story of this period was remote computer clusters and operating-system virtualization. The rapid growth of server-side capability laid the foundation for thin clients.

2008: The V8 engine

Chrome's V8 JavaScript engine⁷ greatly improved JavaScript's runtime capability and efficiency.

2009: ChromeOS

Google announced the project in July 2009, initially describing it as a cloud-first operating system in which applications and user data would live in the cloud. ChromeOS was mainly for running web applications.

2005 onward: The CSS3 standard

Work on CSS3 began as early as 2005.
It aimed to meet the need for more dynamic and interactive web design and layouts that adapt to all kinds of devices (responsive design), screens and user interactions. Concretely, this includes media queries, the Flexbox model, Grid layout, and richer animation.
On the engineering side, it also brought modularity into the CSS specification, allowing features to be developed and adopted independently.

In short: stronger layout and interaction across different devices, and a more engineering-oriented design, to cope with increasingly complex needs.

2015: ECMAScript 6

Discussion of the next version began almost immediately after ES5 was released in 2009, and the ES6 standard was officially released in 2015.
ES6's biggest improvements were native support for modules, more convenient asynchronous calls, improved variable scoping, and a more object-oriented Class syntactic sugar.

2015: PWA

Alex Russell and Frances Berriman of Google coined the term "Progressive Web App" to describe applications that take advantage of new features supported by modern browsers, including service workers, manifests and other web platform features.

2018: Apple starts supporting PWAs

Of course, PWAs conflict with Apple's business ecosystem, and Apple's support for them is incomplete.

Likewise, looking at it from today:

1. The most fundamental paradigm became "treat the browser the way you treat an operating system." With that as the backdrop, the overall direction of engineering became very clear.
2. Standards actually lag behind real needs. Before new standards like CSS3 and ES6 were written, there were already countless similar engineering practices: all sorts of hacky layout techniques in CSS; module conventions like CommonJS and AMD in JavaScript; jQuery's wrapper around document selectors; and so on. CSS3 and ES6 borrowed heavily from and absorbed these practices. You could say these standards were the result of the whole Web community pushing.
3. Beyond the community push, a Web designed around the "operating system metaphor" could borrow more and more from OS app programming and absorb the more carefully considered language designs of Java, Python and others.

In short, as the underlying tech stack and engineering architecture matured, the Web became an operating system platform in the true sense. Web applications matured too, and came closer and closer to the experience of native apps.

Some discussion from a WeChat group

The Web is the most far-reaching revolutionary technology of the information age (probably no need to say "one of the most"), and its inventor, Tim Berners-Lee, is still alive. His views on the Web are well worth following.

This post was originally prompted by a Web-related discussion in a WeChat group. Newcomers to the field always seem to have similar confusions (and this friend is fairly sensitive to certain phenomena).

Here is a brief record of the discussion (unrelated content is blurred out to keep the logic flowing), to show some concrete scenarios.

The Past

In the first post on this blog, Origins, I quoted someone else's summary of the three stages of blogging:

Stage one: you've just discovered blogging, it feels fresh, and you try writing on some free hosting service.
Stage two: you find the free service too restrictive, so you buy your own domain and hosting and set up an independent blog.
Stage three: you find managing an independent blog too much hassle, and ideally you'd keep control while someone else handles the management, so you only have to write.

But after planting that flag, I only wrote on my independent blog for a year. When it came time to renew the domain, keeping my own domain going felt like a chore, so I gave the blog up.
After that I moved through Zhihu (a Chinese Q&A site), WeChat Official Accounts, and Xiaobaotong (a Chinese paid-newsletter platform) as places to publish my writing. I had thought that, just like the three stages above, handing my writing over to these platforms would let me focus more on the essence of writing. In practice, it was the exact opposite.

What Brought Me Back

Looking back, a few things led me to return to an independent blog.

1. These platforms are not convenient to use
Want to publish something publicly on them?
First you have to log in, then click around to find the publishing center, and only then can you write your first line (the editors on Zhihu and WeChat in particular are beyond words).

2. Playing away from home, without freedom
Almost every platform has its own explicit or implicit rules: what you can and can't write, what spreads easily, what sinks without a trace. These unspoken things shape my choice of topics and make me anxious.

All things considered, almost no writing platform is actually as convenient to write on as an independent blog. An independent blog does need some maintenance, but on the whole it's fairly infrequent. And more importantly, it gives you freedom.

3. I have engineering skills
After I left ByteDance in 2020, maybe because I came into contact with new things, maybe because I let go of some old obsessions, or maybe because I ran into a lot of problems in those years, with relationships and with my family's health, after a few years of struggle my view suddenly widened a great deal. I saw my own limits clearly, and society's limits, and I also saw what choices I should make within those constraints to keep going and keep living.

I also truly began to realize that technology is a very effective lever and a form of productivity, something that lets you make real changes by creating things.
Creating matters. Writing an article is creating, and using your own skills to build an independent product is creating too, and the latter compounds more strongly.

4. Some people's influence
Liu Shaonan, Bai Guang, and the indie developers on Jike (a Chinese social app) passionately searching for their own small business.
Huang Yan and Jiuzhang, whose independent blogs are still alive and still being updated.

Spending time around these people, I could feel the courage it takes to explore independence while being swept along by this huge, chaotic, noisy current of the world.

All in all, I've started to want to create things with a longer life span, and over these years I happen to have learned the skills to do it. Just as settling down in Hangzhou after turning thirty ended more than a decade of feeling adrift, these past two years I've increasingly wanted a home ground of my own, with all my writing gathered in one place.

What Changed

While trying to restore the old blog, I realized how carelessly I used to treat my writing. When I left ByteDance and switched computers, I backed up a lot of material, but not the source files of the old blog. Luckily, the GitHub repository still had the compiled files, so I restored the posts one by one, and one by one I reread those old articles from 2016. The writing is very immature, sometimes even deliberately mystifying, yet to me it somehow felt new every time I read it.

While rereading, I also found that many of the external links were dead. Some pointed to other people's independent blogs, some to general websites, and some even to ByteDance's old CDN image host. The GitHub and Google links, however, all still work.

Eight years really is a long time. Many things have changed or simply no longer exist, and some haven't changed at all. Putting them side by side makes you sigh.

Besides restoring the old stuff, there are a few changes:

• The blog's name changed from "Xiaohu Ruwei" (The Little Fox Wets Its Tail, an image from the I Ching) to the more concrete "Knots & Nets"
• I designed a new logo based on the imagery of my daughter's name (I've started dabbling in Mondrian, haha)
• I removed a lot of redundant visual decoration and simplified the UI to suit my current taste
• I care more about reader feedback, so I added Disqus for comments (Duoshuo, the old Chinese comment service, is long gone)

Also, the Jacman theme the old blog used is technically quite dated. I forked my own theme and started fixing the outdated or deprecated logic in it, for example gradually migrating the float layouts that were everywhere over to flex layouts.

Many choices can't be undone once made. Fortunately this one could. After going around the loop of third-party platform -> independent blog -> back to third-party platforms -> rediscovering how good an independent blog is, and back to an independent blog, I think I'll be staying on my independent blog for the long haul.

Bailujun North, Hangzhou, February 25, 2024

Introduction

The most accurate description is the official introduction¹:

We challenged ourselves to create a visual language for our users that synthesizes the classic principles of good design with the innovation and possibility of technology and science. This is material design.
Develop a single underlying system that allows for a unified experience across platforms and device sizes. Mobile precepts are fundamental, but touch, voice, mouse, and keyboard are all first-class input methods

Design principles

1. Material is a metaphor:
   This metaphor is the foundation for understanding everything else in material design. Whenever you think about the material design space and about how a whole series of elements should move and behave, don't forget the metaphor behind it.

2. Bold, graphic, intentional (that is, the design is arranged in advance; there is a deliberate motive behind it)
   It borrows the conventions of traditional print graphic design (typography, grids, space, scale, color, use of imagery) to guide visual treatment. These conventions don't just please the eye; they also serve to build hierarchy, convey meaning and direct focus (mostly in the sense of cognitive psychology, which I'll come to later).

3. Motion provides meaning
   Animation is a very important part of front-end interaction, but material design doesn't use it arbitrarily. Its motion is abstracted from reality and matches users' cognitive patterns.

Below I'll unpack this official summary.

The "science and technology" behind Material Design

Drawing on traditional graphic design

The official introduction says material design "synthesizes the classic principles of good design with the innovation and possibility of technology and science." The "classic principles" part is easy to understand. Back in the age of print, people had already explored a great deal about how to present information in forms that are more pleasing and easier to take in, and had distilled a number of graphic design principles that work:

• Consistency
• Reducing memory load
• Alignment, proximity
• …
  These guidelines, originally for traditional paper media, were carried over to the new electronic media with very good results. Keep in mind that early web pages, apart from a few hyperlinks, came nowhere near the richness of presentation in print. Ideas from traditional graphic design gave web design very useful guidance, the grid system for example:
  

As the web developed, electronic media acquired something of their own. They were no longer purely static like paper; they had real "interaction." The web's new features needed other theoretical support. If we can no longer borrow from traditional graphic design, how do we create a set of design guidelines suited to these new web features? This is where the material design docs' "innovation and possibility of technology and science" comes in.

Opening up new design territory

If traditional graphic design can't be borrowed directly, look at whether there is something behind graphic design worth digging into. When looking for theoretical support, you need to search deeper in the reductionist direction. Roughly, that direction goes like this:

Design rests on psychology
Psychology rests on biology
Biology rests on chemistry
Chemistry rests on physics

These sweeping propositions are highly contested, but within these disciplines you can at least say they are "partly true." Digging deeper for design basically takes you as far as psychology and biology. Finding deeper theory gives a foundation for "creative" design. It lets the expression of the design space change, so you're no longer confined to transplanting traditional graphic design onto the new web medium; you can improvise freely from these deeper theories and create design forms suited to the web.

All that said, what exactly are these innovations and possibilities of technology and science?

Cognitive psychology

Before talking about cognitive psychology, here's a small example from Toutiao Quan (a social feature in the Toutiao news app):

This small example shows, to some extent, that cognition matters in front-end interfaces far more than people imagine, though many people don't realize how important it is.

The way humans come to know things (perception, learning, reasoning, memory and so on) follows certain fixed patterns, and people's cognitive behavior in front-end interfaces doesn't escape them. These patterns are formed by human biology. Since the Industrial Revolution, humans and their environment have been rapidly transformed, but evolution works on a scale of millions of years. The evolution of human biology was essentially finished long ago, and for the foreseeable future that biology will stay stable. Psychology already has quite a few findings that can teach design something:

• How experience, environment and goals affect cognition
• Gestalt phenomena in perception
• Short-term and long-term memory
• Research on reading behavior
• The psychology of decision-making
• …

All of the research listed above has applications in design. As for why we should talk about cognitive psychology at all, Designing with the Mind in Mind has a passage on it (slightly edited and expanded):

Following user-interface design guidelines isn't as step-by-step as following a cooking recipe. Design guidelines often describe goals rather than actions. They are deliberately very general so that they apply broadly, but that also means people often interpret their precise meaning, and how they apply to a specific design situation, in different ways.

To complicate things further, for any one design situation, several rules often seem to apply at once. That requires trade-offs between designs, and those trade-offs can't rest on the designer's personal taste alone; they need data and theory behind them. Data can come from the operations team.

Theory, though, can only come from the deeper level of cognitive psychology.

If you understand users' cognitive patterns, you're less likely to end up with something that "follows the design principles but doesn't match how users think," which makes for a bad user experience. In the example above, decorating the "…" follows quite a few design rules: it's more consistent, fits better with the other icons, stands out more, and so on. But it doesn't match cognition, and users easily make mistakes there.

Humans live in the physical world, while their minds sometimes roam the internet. This creates a "context switch" between the physical environment and the internet environment: people understand things on the internet through certain features of the real world, the trash can standing for the recycle bin, for example. With the help of psychology, we can make the cognitive transition from reality to internet design fairly smooth. The same goes for motion. Motion in the real world is smooth; nothing moves in broken lines (collisions aside). That's why smooth transitions look natural, and because of that naturalness they feel familiar and are more readily accepted.

Another place where a "context switch" happens is moving between scenes within an app. Transition animations need careful design to guide the eye, so that the transition feels natural and smooth and the user's train of thought doesn't break. This is where Bézier curves come into their own.

Biology

This is mainly research into the characteristics of biological structure, where cognitive psychology can look for its theoretical foundation, for example:

• Neuroscience
• Eye-tracking research
• Research on how the eye forms images
• This research is more fundamental and touches on evolutionary theory to some extent, so I won't go into it here. Here's one piece of theory about the eye's photoreceptor cells:

The spatial resolution of human vision drops sharply from the center to the periphery, for three reasons:

Pixel density
Each eye has 6 to 7 million cone cells, and they are packed far more densely in the center of the visual field (a tiny region called the fovea) than at the edges (corresponding to central vision). The fovea has about 158,000 cones per square millimeter, while the rest of the retina has only 9,000 cones per square millimeter (corresponding to peripheral vision).

Data compression
The fovea's cones connect 1:1 to ganglion cells, the starting point for processing and transmitting visual information, while in the rest of the retina several photoreceptors (cones and rods) connect to a single ganglion cell. In technical terms, peripheral visual information is compressed (lossily) before it reaches the brain, while central vision is not.

Processing resources
The fovea makes up only about 1% of the retina, yet 50% of the visual cortex is devoted to input from the fovea, and the other half processes data from the remaining 99% of the retina.

As a result, visual resolution at the center of our visual field is far higher than anywhere else. At arm's length, the fovea's field covers roughly the area of a thumbnail, about the width of three characters on the screen you're reading now. This is the basis of visual focus.

—— Jeff Johnson, Designing with the Mind in Mind (quotation slightly adapted)

Biological research on the eye tells us that when people look at a screen, their visual focus is basically only the size of a thumbnail; everything outside it is "peripheral vision." Given this mode of vision, how to guide the eye and draw users' attention to the information you want them to see is a real challenge. The most direct example is where and how an error message appears when an input field has an error.

Other things

Abstraction from physics && sensible quantification

The official material introduction also says "material is metaphor". Material is a metaphor. A metaphor for what?

What material design metaphorically refers to is a design material. Let's call it the "material card." It is abstracted from reality, stripped of reality's impurities and randomness, keeping spatial relationships, change and transition, and light and shadow. This material has thickness, and pieces of it have spatial relationships with one another. This actually hints at a bigger metaphor: the whole of material design stands for a particular design space, a space with three-dimensional structure, and material cards are the basic material it is built from.

Why go to so much trouble to abstract a design material and a virtual space?
Because materials, together with the rules for building with them, determine consistency

To understand this, we can start from architecture in the real world:

All design can be vividly described as "dancing in shackles," and this is especially obvious in architecture. Real-world architecture requires using basic available materials (brick, concrete, steel, glass) to produce designs that obey the physical laws of the real world.

Keep in mind that most buildings in the world are much alike. Only a few architects can design buildings that both obey the rules and rise above the ordinary.

If the materials and rules are too restrictive, the result is rigid expression. Although material design is abstracted from the real world, it also adds magical properties that don't exist in reality: material design cards can be freely split apart, merged, transformed and so on.

MD has some quantitative standards, and they provide design with a stronger layer of regulation:

• Card thickness
• Card elevation differences
• Color choices
• Animation timing
  • The longest gap between successive images that still lets them fuse visually: 20ms
  • The lag between a visual event and our complete perception of it: 100ms
  • The editing "window" for an event to enter consciousness: 250ms
  • …

This idea of "quantified design with a basis" is ahead of its time. It means design no longer belongs exclusively to masters with rich experience, and is no longer just bursts of inspiration and the expression of personal taste. If you know some history of science, you'll see why this act of "bringing science into design" at the level of design language matters. To draw a comparison: it is like bringing quantitative, scientific method into alchemy, which had always been practiced in mysterious ways, and it became chemistry.

Traces of combining science and technology can be seen in earlier design too, but none of them reformed and regulated design as thoroughly as material design does, through the form of a metaphorical space.
These features can be said to be why "people without much experience can still make decent designs."

People without much experience can still make decent designs?

What is decent design

• Visually beautiful
• Helps convey information precisely
• Grounded in psychology and matched to users' cognitive habits (not just the designer's personal taste)
• Material design has already built all of the above into its basic vocabulary, so doing front-end design with material design to some extent sets its floor: it won't be terrible.

Material Design tries to find a design point between art and science. That is possible, although it demands a great deal of the designer.

In my view, Material Design has partly achieved this. Its thinking is very clear and highly inspiring. Even if Material Design as a whole dissolves one day, many of its most inspiring ideas will carry on.

A Material Design mind map

The mind map above was compiled from the Material Design guidelines. The branches on the right describe Material Design's most basic components and structures. The branches on the left introduce some empirical best practices. Using what I've described above, readers can see whether they can find traces of these principles and techniques behind them.

References

Links:

material design official site
How should we understand Google's Material Design language? (Zhihu, Chinese)
The Verge's interview with Matias Duarte

Books

Grid Systems in Graphic Design
Designing with the Mind in Mind: Simple Guide to Understanding User Interface Design Guidelines
Don't Make Me Think

Why I'm writing this

Front-end engineering is a peculiar profession. To borrow an elegant phrase, the front-end engineer is both painter and hacker. Along the product chain, front-end engineers sit between the server side and the designers, a bridge between engineering and design.

This means front-end engineers have to care not only about engineering but also about product design and user experience on the other end. Yet from what I see around me, many front-end engineers have built up far more on the engineering side than on the other.

This series introduces some of the more "humanistic" knowledge in front-end engineering, and it tries to cover only the parts where the humanistic can be explained with reason.

Explaining the humanistic with reason?

Take the relationship between design and psychology as an example:

Biology is grounded in chemistry, and chemistry is grounded in physics

In the natural sciences, one idea has had a far-reaching influence: reductionism. The sentence above roughly means that all biological phenomena can ultimately be traced back to the various chemical phenomena inside organisms, and all chemical phenomena can be traced back to the combinations and motions of various particles.
Reductionism is highly contested, but judging by the history of the natural sciences, it has been the most effective intelligible approach to research, producing a great many results through reductive analysis, and so it has been borrowed by other social disciplines (psychology, economics).
Modern psychology is barely a hundred years old. From the 1960s and 70s on, advances in medical technology and experimental methods, such as the arrival of EEG (electroencephalography) and MRI (magnetic resonance imaging), drove rapid progress in fields like cognitive psychology and neuropsychology, and also gave psychology a foundation that supports reductionism.

Psychology is grounded in biology

This sentence may be more contested, but following the reductionist line of thinking, at least some phenomena can be pushed one step further down to the level of biology.

The psychological foundations of design

Carrying the analogy all the way down, some parts of design can also find more universal psychological grounding, for example the alignment, proximity, contrast, and so on proposed in many design principles. This gives design a "universal explanatory foundation," so that design no longer looks like merely the expression of a designer's personal taste. When there are disagreements over a design, you can find a more relaxed way to settle them (relaxed compared with validating through data such as A/B tests): let's reduce it one level down and see what that says.

The plan

Roughly: design psychology, then information media, then philosophy of technology.
My knowledge is limited. Discussion and criticism is welcome.

The Neihan Duanzi operations team keeps getting user feedback asking "Does Neihan Duanzi have a favorites feature?", so much of it that you can't help wondering: why can't so many users see the "Favorite" and "Save" features? They sit right there on the panel that pops up when you tap Share, and by rights should be visible at a glance.

This is in fact a form of blindness known in psychology. Of course, once you know the two features exist and consciously look for them, the blindness disappears. But for many users that is a hard step to take. This post analyzes this baffling problem from the angle of cognitive psychology and proposes solutions.

The short version

A two-step analysis

Before the share panel opens, the share icon carries a strong goal cue (namely, sharing), which makes people subconsciously assume that every function in the panel that appears after tapping it is about sharing.

The Neihan Duanzi share icon:

Once the panel is open, attention, shaped by that goal, stays on the sharing functions, which leaves people blind to everything else. The function icons at the bottom also look very much like the iOS system style, which makes the blindness worse. (The image below is the panel after tapping; note the "Favorite" and "Save" functions at the bottom.)

What the user actually sees is this (the user only sees two spots: the top one when sharing, and the bottom one when cancelling. The middle part usually gets a quick sweep of the eye and is ignored):

How to fix it

It's actually simple. There are two directions:

1. Replace the share icon with "…" (meaning more functions): for this icon, users expect several functions to choose from once it opens, so they will look through it more carefully.
2. Keep the share icon for sharing only and split it out on its own (one icon, one function, as users expect), and put everything else under "…" (more functions).

Whichever direction you take, the style of the "Favorite", "Save" and other function icons needs to change so they don't look so close to the system's, ideally matching Neihan's own style.

For those who want to dig into the analysis, read on for the full version. It works mainly from a few theories of cognition and includes some quite interesting ideas from current cognitive psychology. Recommended.

The full analysis

Two questions (wish I could add a poll here)

1. Have you ever noticed "Neihan Duanzi's Favorite and Save features"?
2. If you have, how long after you started using the app did it take you to notice?

To describe the complete mental process of a user using Neihan Duanzi, I've included my own experience of using the app. If you're not interested, skip this section and go straight to the next major heading.

Background: my experience with the app

As of today I've been using the Neihan Duanzi app for about half a year. What puzzled me most was that sometimes I'd come across a picture or joke I really liked and wanted to save or favorite it, but couldn't find any feature for it.

As an engineer I consider myself very experienced with apps, yet I genuinely never found anything like that. In the end I could only guess the PM had left these two features out for some special reason. And to keep the jokes I liked, I had to share them straight to my own WeChat, as shown below (dumb, but I couldn't find a better way):

Today, at my first requirements meeting on Neihan, this question came back to me, so I asked the person in charge, and was told the "Favorite" and "Save" features do exist; I just hadn't found them. (Most people try to find something a few times without success, then basically give up and use some other workaround.)

So I went through every icon I could see, one by one. Sure enough, I finally found a few features I had overlooked for nearly half a year. For some reason they had been invisible to me, even though I had tapped the share icon countless times. As shown below, Favorite and Save are right there at the bottom of the panel:

Why were "Favorite" and "Save" invisible?

The reasons can be found in cognitive psychology. In short there are four main ones (I won't expand on them, just the conclusions):

The area where our gaze is focused (central vision) is very narrow. At normal screen distance (half an arm to an arm's length), our point of focus is roughly the size of a thumbnail.
Human attention is very limited. When looking at a phone, an ordinary person can basically focus on only one point (usually the point of gaze).
Attention is easily guided by goals. Things strongly related to the goal get noticed easily; the rest tends to be ignored.
How we perceive things is shaped by experience and habit.

Step-by-step analysis

Tapping the share icon

The share icon:

Anyone with experience of internet products knows this icon well, and usually taps it only when they want to share. Note the causality here: people generally don't tap the icon first and then decide whether to share. They come with the purpose from the start.

The share icon is so common that it sets up a strong representation (and a strong cue): the icon is the function. For experienced internet users, the share icon and the share function map one to one. (If functions other than sharing show up, that goes against what users expect.)

Taken together, these two points determine what users expect when tapping this icon: they're heading for sharing, and they expect to see sharing functions on the next screen (points 3 and 4 of the cognitive theory above).

Entering the share panel

The function panel:

Once in the share panel, because the user already has a goal and expectation and is shaped by them, their gaze and attention look like this (per the theory above, the user's point of focus is small, and it only lands on what most deserves attention), as below:

If the user decides not to share after all, habit sends their eyes straight to the bottom. Their gaze and attention at that moment look like this:

Another reason for the blindness
The Toutiao share panel && the Neihan share panel

Toutiao (ByteDance's news app):

Neihan:

Comparing the two, you can see that the buttons in the lower part of Neihan's share panel are styled very much like iOS, and this style uses light-gray line icons with very low visual salience (hard for peripheral vision to pick up). Drawing on their own experience of iOS, users subconsciously assume this part is just a bunch of miscellaneous built-in system functions, so their eyes sweep straight past it and the information is lost.

How to fix it

It's actually simple. There are two directions:

Replace the share icon with "…", meaning more functions, because for this icon users expect several functions inside once it opens, so they will look through it more carefully.

Keep the share icon for sharing only and split it out on its own (one icon, one function, as users expect), and put everything else under "…" (more functions).

Whichever direction you take, the style of the "Favorite", "Save" and other function icons needs to change so they don't look so close to the system's, ideally matching Neihan's own style.

I still remember how lost I felt when nearly twenty years of school came to an end, and my first sleepless night in the rented flat in Longze seems like only yesterday. Yet in the blink of an eye, a year has passed since graduation.

Work

It still gnaws at me that I missed the interview round of the graduate entrance exam in philosophy by a single point. In the end, right before graduation, I went back to the computer industry.

Looking back, I've never written a single word about that time preparing for the exam. Maybe my thoughts were buried under a year of busy work and I never found the time. From the summer after my junior year, I put all my energy into preparing for the philosophy of science exam. Those were probably the fullest days of my four years in college: every day in the library, going from ancient Egypt and Mesopotamia to ancient Greece, then Rome, the Islamic world and early modern Europe. I learned about the course of science, the development of philosophy, the rise of modern philosophy of science and philosophy of technology, and the differences between Eastern and Western civilizations. I still clearly remember the stomachache in the exam room and the slightly over-the-top post I put on WeChat Moments when the exam ended. But in the end I fell at the gates of Peking University. I made my compromise with reality and with my age, and that was the end of me and philosophy of science as a field. Still, the way of thinking that philosophy of technology gave me has become part of how I think, and it will keep benefiting me in daily life and at work.

Feeling uneasy, I started job hunting and went to a startup founded by an upperclassman from my school. I don't know whether it was my own ignorance or the company trusting me too much, but on my very first interview I was hired on the spot. Later this even led to some arguments with family and friends. Young and reckless indeed. I also had a doubt that, looking back now, sounds a bit like showing off: everyone says finding a job is hard, so what's going on here? But I didn't think much about it. Graduation was pressing, I joined the company and began life as a workplace rookie, and the graduation trip I had planned was canceled.

I think I was still lucky. Because I'd spent the summer after junior year and almost all of senior year preparing for the exam, I had nearly zero internship experience in computing, and when I ran into real projects I was badly stretched. But I got good guidance at the company and gradually came to be valued quite a bit, and things slowly improved. That was how I spent a hard but upbeat half year. During those six months I really had the fearlessness of a newborn calf: I dared to voice all kinds of opinions and ideas, and my efforts to help the company build its own wiki and proper engineering management did make a difference. Being recognized by a platform may be the rarest thing you get from a job.

After six months, the company's business focus gradually became clear. It dropped the original startup project and concentrated on crowdsourcing services. But that direction was too far from my own plans, work fell into a loop, it felt more and more boring, and I even had some very strange arguments with the tech lead. In the end I jumped to Toutiao and began my career at a second company, but that's another story.

In terms of skills, I went from half ops, half backend when I first started, to backend, and then to frontend. I've touched more or less the whole web stack, and in the end I chose frontend as my personal breakthrough point. I chose it because the frontend is close to users; it's the place in the computer industry closest to users, and perhaps also the most humanistic place. I've always believed that philosophy of technology can contribute something to the internet industry, and the easiest place to bring them together is the frontend.

Life

This year I moved from Zhichunlu to Longze, and then from Longze back to Zhichunlu, as if completing a full circle.

When you live in a city like Beijing, rent is always a topic, and it comes down to one word: expensive. But since I'd started living my own life, I didn't want to do what my parents' generation did when they worked away from home, renting any old run-down place and just making do. So I went to Longze to share a place with friends and tried to create the kind of atmosphere you see in Friends. That turned out to be not very realistic, but it felt pretty good.

When I first moved into the rented flat, to borrow a saying from my hometown, I "couldn't get a grip on anything." I had to call my mom for advice, and she said it was because I lacked the basic household things. Little by little I bought pots, pans and bowls, and thanks to a kind and capable girlfriend, the place slowly got sorted out and started to feel lived in.

Cooking for myself and having friends over for dinner happened now and then in the first half of the year after moving in, but in the second half I was swamped with all sorts of things and got much lazier in my spare time; everything was basically waiting to be picked up again. Moving back to Zhichunlu was a big turning point. No more squeezing onto the subway and buses every day, and I had a lot more time, so I found the feeling of living again.

Mind

Over a year, a lot changes inside you. Some of the shortcomings from my student days were gradually made up for, and at the same time many good qualities slowly faded away, such as spending time studying every day and the fearlessness of a newborn calf.

Afterword

I wrote this in June, during graduation season. Because I switched computers in the middle, my train of thought kept getting interrupted, and it's still incomplete even now. But as I've gained experience at the new company, I've slowly found my way back into a learning state, and that feeling of not yielding even to my teachers when it comes to what's right. I've also got a somewhat more mature plan for my career ahead. To be continued.

When I first heard the term "topological sort", I drew a complete blank. I ran through everything in my head and found nothing related to it. I'd studied graph algorithms as an undergraduate, but that mostly focused on graph connectivity, spanning trees, depth-first search, breadth-first search and so on. There didn't seem to be any "topological sort".

Only after learning what it basically is and where it's used did I realize that it had been there all along; I just hadn't clearly recognized the concept or its importance.

Module Dependencies in JavaScript

Anyone who writes JavaScript has probably run into the concept in the title. For example, there's a module A that depends on modules B, C and D; module B in turn depends on C, E and F; module C depends on D, E, F and G.

Put another way:

// module: {dependencies}
A: {B, C, D}
B: {C, E }
C: {D, E}
D: {E}
E: {}

If you're asked to write a program that computes the dependencies of every module in a system of interdependent modules, with the final output shown in the figure, how should you go about it?

Traversing the Graph Directly

Clearly, the module dependencies need to be converted into a graph data structure. Before I learned about topological sort, my instinct was that once the graph was built, I'd just pick a node, say A, start traversing from it, add every path A can reach, and merge them into A's dependencies. This could be optimized with DFS or BFS.

Topological Sort

See Topological Sorting directly.

The Heap Data Structure

What follows are only the parts I added myself.

A Different Analogy for the Call Stack

To explain the call stack, the original compares it to people getting in and out of an elevator. I didn't find that very vivid, so in my translation I replaced it with a human pyramid: the first person to climb on can only come down last, and the last person to climb on is the first to come down.

A Follow-Up Question

Question: is the delay you set in setTimeout exact?

The humble brick is a marvel: on every street, down every alley, in every odd corner, practically anywhere people live, you can find it.
Stones are not as ubiquitous as bricks, but they hold a respectable place too: you see them in all sorts of building materials, sculptures, stone tablets and so on.

If I asked what the difference is between these two utterly ordinary things, bricks and stones, some people might laugh. Bricks and stones have nothing to do with each other. Apart from some overlap in what they are made of and some uses in construction, they have almost nothing in common, and discussing their differences seems even more pointless.

That answer isn't really wrong, but it isn't a good line of thought, or a good angle from which to look at the question. Let's look at how bricks and stones differ from one particular angle.

What's the Difference?

First difference: bricks are man-made, stones are natural.

At first glance there's nothing really wrong with this, but it doesn't give you an "oh, that's it" feeling either. We keep asking: so what?

Second difference: bricks conform to a specific size, stones have basically no fixed shape.

Since bricks are man-made, they must serve some predetermined purpose. Bricks are used to build houses, while unprocessed natural stones come in all sorts of odd shapes and are of little use in construction. But this difference also seems to be a true but useless statement.

Now the third difference: bricks embody industrial standardization, stones embody the disorder of nature.

Now this gets interesting, because following this line of thought we can find a lot. A brick is no longer just a brick; it is also part of the process of industrialization. It is precisely this industrial standardization that lets bricks be produced more easily and cheaply, organized more easily, and serve industry itself better.

A small brick basically distills the most important qualities of industry. Compare stones: with all their irregular shapes, they're good for little more than piling up a makeshift wall. If you want stones to play a bigger role, you have to process them the way bricks are made, so they can interact more easily with their surroundings.

So from this angle, although bricks and stones differ in themselves, the bigger difference lies not in the things themselves but in what shows up when you put them in a group.

So What?

You may think all of the above is right, but what does it prove? It doesn't seem to affect our lives. We may not be able to say exactly how bricks differ from stones, yet we already use them the right way.

Of course, the difference between bricks and stones is quite interesting even when examined in depth, but what's more interesting is the difference it contains between industrial standardization and non-standardization. This idea of industrial standardization has been everywhere since the industrial age: memory, hard drives and cases that follow specific rules let us assemble a personal computer easily. If you drive abroad on vacation and lose a wheel or a screw, you can buy a dozen more on the spot instead of fetching them from home. By contrast, imagine a world without standardization: if you had a custom car built in your hometown and it broke down halfway through a trip, picture how awkward that would be.

In fact, all industrial standardization means much the same thing: some standardization lets things fit better into a group, and some lets things exist beyond space, time and particular people.

Fitting into a group is what the bricks above do. Existing beyond space and time means that people in different places and different eras can all, by convention, use the same specification without worrying about compatibility. Existing beyond particular people means that once something is standardized, its manufacture, maintenance and reproduction no longer depend on a specific person or group. McDonald's and Chinese cooking are good examples. Chinese cooking depends on personal experience and is hard to quantify and standardize; if certain master chefs kick the bucket, you may never again eat dishes as good as theirs. McDonald's is different: as long as there are quantified recipes and standardized production, it hardly matters who drops dead.

Standardization is so common that it has become transparent and vanished from our view, a kind of "vanished existence."

Web Standards

Recently I came across a classic front-end interview question again: "How do you understand web standards and the W3C?" The line of thought here is basically the same as above: whatever industrial standardization brings, web standardization brings too.

I won't go into the details of web standards here. Web standards are of course good. They provide sound, practical standards for content, presentation and behavior, and on that basis you can build web system architectures that are highly available, easy to understand and highly adaptable.

W3C

The value of a standard shows only after it has spread and is followed. The more people follow it, the greater the benefit, until what you finally have is one standard web system after another running on the internet. Because every web system is standard, they are friendly not only to humans but also to the higher-level systems that process them (search engines, for example).

There is a rule of thumb about the internet: the value of the internet is proportional to the square of the number of nodes it is made of. Web standardization is similar. Web standards let you build web systems with a clear structure, and these standard web systems join together into a larger standard system. This makes all kinds of work easier to carry out and lets more be built on top, creating more value. At that point it is like the bricks discussed earlier, or the shipping containers I mentioned in an earlier post.

The value of the web standards themselves is not what matters most; there may well be better standards than the current ones. A well-defined standard that spreads easily can create more value than a better standard that is hard to spread. History is full of similar examples.

As for the W3C (the World Wide Web Consortium), it leads the web standardization process and is responsible for drafting and promoting web standards. It has to make sure not only that the current standards are good in themselves, but also coordinate different vendors so that the standards are easier to accept and spread. It has a heavy responsibility and a long road ahead.

The clock, not the steam-engine, is the key-machine of the modern industrial age. — Lewis Mumford

The Clockmaker Who Beat Newton

I once happened to catch an episode of Luoji Siwei (a Chinese knowledge podcast) called "The Clockmaker Who Beat Newton." It tells the story of a clockmaker who built a highly accurate timepiece that made it possible to fix a ship's position at sea, giving a huge boost to navigation.

In fact, the story of clocks goes well beyond that. Determining longitude and advancing navigation was certainly a major event in human history, but compared with the role clocks played from the 10th to the 20th century, it is only the tip of the iceberg. It's just that the clock's other effects are almost "transparent" to us. Only after the philosophy of technology took shape did the veil slowly start to lift.

This post starts with the clock tower and briefly walks through some of the other stories of the clock, the spokesman for time.

Clock Towers Everywhere

Even today, we can still see clock towers in older big cities; they have long stood on university campuses and in churches. Even some newer cities put up a clock tower in a few key spots to remind people of the time.

The clock tower is a public timekeeping device. It appeared around the late 13th to early 14th century and quickly became a standard building in churches and universities. In monasteries and universities, people were required to arrange their daily activities strictly by the clock: prayers, services, meals and so on in a monastery all followed a strict rhythm and order.

In fact, if the clock tower was the product of people gathering into cities and needing to organize their lives and coordinate their activities, then monasteries already had timekeeping devices around the 10th century, well before clock towers appeared. Legend has it that a monk named Gerbert invented a mechanical water clock in the late 10th century. Regularity, order and punctuality were almost the natural requirements of monastic life, and they kept getting stronger over time. (See Lewis Mumford, Technics and Civilization.)

With the mechanical clock tower, timekeeping was no longer at the mercy of the weather. Water freezes, and the sun, moon and stars get covered up; a mechanical clock has none of these weaknesses and can run fairly reliably in all kinds of natural conditions. So it was quickly adopted by churches and monasteries, and later by the universities that emerged.

The meaning of the clock tower, and its disappearance
As mentioned above, the clock tower is a public timekeeping device, and it carries its own norms from birth: regularity, order, punctuality, and the quantification of time, something you can neither see nor touch.

A clock tower can coordinate and constrain human production and life over a wide area, so that all kinds of activities proceed in order. Inside churches and monasteries, its effect may have been strict prayers and services, a code of conduct in the religious sense. But once it left the church and entered the city, this clock-driven norm of punctuality and order coordinated human production and gradually brought the notion of time into the human world.

One effect of this notion of time is that it raises the efficiency of human production (before the "concept of efficiency" was born, efficiency itself already existed and was being strengthened, consciously or not). This in turn kept pushing the clockmaking trade forward.

A heavy timekeeping device like a clock tower usually served a whole large area, which was not very convenient. As clockmaking kept improving, especially after the invention of the mainspring in the 15th century, there was finally a technical basis for making clocks smaller. They shrank, and small table clocks began to enter people's homes. As clockmaking grew more precise and cheaper, this kind of timekeeping device began to spread.

The Revolution of Time

Some historians believe that in the centuries before the Industrial Revolution, another revolution had already been quietly under way: the so-called "revolution in the notion of time," or the "efficiency revolution." The changes it brought became the foundation of the Industrial Revolution.

The revolution of time can be traced, roughly, to the point when clocks became fully woven into human life. When people began to leave behind the primitive "relative notion of time," that is, sensing the passage of time entirely through the sun, moon, stars and the turning of the seasons, and adopted an artificial, mechanical means of keeping time, the revolution of time began. Time was no longer the unfolding and change of things; it became something independent, something that could be measured and quantified.

People began to mark time off according to the clock, ever more precisely. The introduction of hours, minutes and seconds made people clearly feel the passing of "time," and the important role time plays in daily life and in commerce. As commerce developed, producing more resources within a given time became an absolute pursuit. By then, the idea of "efficiency" had already begun to sprout.

People kept finding ways to raise efficiency: improving production processes, reforming production management, adopting new technologies, and so on. It was in this environment that the Industrial Revolution found soil in which to germinate.

Time Is Money

Today, "time is money" has become an everyday metaphor. In our culture, time has become a precious commodity, a limited resource we need to reach our goals. People are paid by the hour or the week, and phone calls are billed even more precisely, by the minute. We have borrowed a lot of money vocabulary to talk about time: time is something you can spend, waste, budget, invest, save or squander. In short, the notion of time has rebuilt the basic behavior of our daily lives and the way we understand the world.

All of this started with clock towers and other man-made timekeeping devices that run independently of nature.

The article talks about ellipses, but the same idea applies more or less to conic sections in general.

The optical property of the ellipse

Among the properties of conic sections, you often see a picture like the one below: a ray of light leaving one focus of an ellipse reflects off the wall of the ellipse and passes through the other focus.

Other conic sections, such as the parabola and the hyperbola, have similar properties — they are all about what happens to light coming from a focus.

But why?

Of course you can prove it mathematically: at the point of reflection (P in the figure), the incoming ray and the outgoing ray make equal angles with the tangent at that point. I'll skip the proof here. It takes a few turns, but it's not hard to follow.

The problem is that after working through a pile of math, I still wasn't satisfied with the answer. To me, a mathematical proof just takes one step back into mathematics and restates the original problem in mathematical terms. I could keep asking: why such a coincidence — why do the angle of incidence and the angle of reflection come out exactly equal after the calculation?

What I wanted to know was "why does the ellipse have this property", not "prove that the ellipse has this property" — just as Wang Yin (a Chinese programming-language researcher and blogger) puts it in his essay on causes and proofs. What I'm really getting at is the difference between the efficient cause and the final cause.

So could I have predicted this in advance, without going through such a complicated proof?

A more interesting explanation (from mechanics)

I puzzled over this, and over all the curious properties of the ellipse, for a long time in high school.

Then one day I noticed how my math textbook said to draw an ellipse: fix a string to two nails, pull the string taut with the tip of a pen, and go once around the nails — the result is an ellipse.

Thinking about it carefully, this description is interesting because it carries two hidden facts.

The first is obvious, and is essentially the definition of the ellipse: for any point on the ellipse, the sum of its distances to the two foci is constant.

The second is subtler, and you can see it by looking at the drawing process physically: as you draw, at any point of tangency the forces from the two ends of the string on the pen tip are equal. So the angle of incidence must equal the angle of reflection — otherwise the string wouldn't be in balance (it would slide). From this you can foresee that the calculation will come out this way.

At that point the "why" finally had an "intuitive" explanation.

Some fun generalizations

The same reasoning extends to the other conic sections, and even the ellipse itself has some extensions — for example, changing the distance between the two nails. What happens when the two nails coincide? The answer is obvious.

While learning front-end development, I ran into a puzzling term: 模态框 ("modal box"), the Chinese name for the modal dialog, a special kind of pop-up window. For a while it was hard to say what exactly was special about it, or what fundamentally set this kind of pop-up apart from other windows. Unlike plain words such as "dialog" or "panel," the Chinese word 模态 ("modal") feels mysterious through and through. It's hard to get an intuitive sense of what it is or what the word is trying to express, and you can't help being confused: what on earth does "modal" mean?

With computing vocabulary, I've always held this belief: if a term feels mysterious, there must be more hidden behind it, perhaps a metaphor, perhaps some deeper design principle. Either way, the meaning behind such terms is worth investigating properly; these small words may turn out to be doorways into one fascinating world after another.

What Does "Modal" Mean?

To understand the modal dialog, besides looking at what the term actually refers to (you can try Bootstrap's modal component), the first thing to do is figure out what "modal" means.

**The Chinese 模态 translates the English word modal.**

First, the dictionary definitions of "modal": relating to mode, to modality, to form.

Then look at some baffling technical terms built from the word:

• Modal analysis
• Modal matrix
• Modal logic
• Modal dialog

Let's guess how the word was introduced, or rather how the English modal was rendered in Chinese. You want to express content in a particular state; how do you translate that? We know the word 模型 ("model") carries the sense of "a fixed abstraction of something," so it can express "a fixed pattern," and the word 状态 ("state") describes how something behaves in a given situation. Put the two together, and 模态 practically suggests itself. Of course, the actual translation and coinage must have been far more complicated than this, but the core idea is the same: how to express a concept faithfully, clearly and elegantly.

After this analysis, we can say we understand "modal" a little. It refers to a particular state.

Now look again at the technical terms above, and they become easier to understand: they all carry the sense of "studying things in some particular state."

What Is a Modal Dialog?

With that understanding of the word, we can move on to the concept of the modal dialog. Literally, it should mean "a window in a particular state."

Seen only this way, it may still be a little hard to grasp. That's because we're missing something: the context in which modal dialogs are used. Only by adding that can our understanding be complete.

So what is that context? The term arose in software, so let's first look at how software is normally used. When we use software, we usually proceed step by step along our own line of thought. Say we're using a shopping system: we follow the process in our heads based on our existing understanding of the software, choosing products, adding them to the cart, placing the order and paying. These processes can be called the "normal state" of using software.

The modal dialog, a "window in a particular state," is defined relative to this normal state. A modal dialog is a window in a particular state that interrupts us out of the normal state and focuses our attention on handling that particular state. Look at how a modal dialog actually behaves: when it appears, it blocks all other operations, and what the user can attend to is limited to the inside of the current modal dialog, unless you deliberately close it, end the interruption, and return to the original normal flow.

That is the core idea of the modal dialog. Strictly speaking, the modal dialog is a concept from UI design. Wikipedia defines it as follows:

model window

In user interface design, a modal window is a graphical control element subordinate to an application’s main window which creates a mode where the main window can’t be used. The modal window is a child window that requires users to interact with it before it can return to operating the parent application, thus preventing the workflow on the application main window. Modal windows are often called heavy windows or modal dialogs because the window is often used to display a dialog box.

Modal windows are commonly used in GUI systems to command user awareness and to display emergency states, although they have been argued to be ineffective for that use. Modal windows are prone to produce mode errors.

Of course, the modal dialog as a design idea implies coercion. It forces the user's attention out of their normal train of thought and onto the contents of the dialog, which amounts to forced thinking. This design is generally used for warnings about fairly dangerous operations.

But criticism of modal dialogs comes in many forms, mainly of this coercive design approach and of whether it should be replaced by better solutions. Some even declare modal dialogs "evil." For these criticisms, see the following:

• Why are modal dialog boxes evil?
• Never Use a Warning When you Mean Undo

"Modeless"

Where there is a concept of modal, there is of course a concept of modeless. I'll set modeless aside for now.

A while ago our team kept arguing about whether an API should return UTC time strings or Unix timestamps. My position was "design for people first, then for machines", so I argued for the more readable UTC time.

The point of that example is that when we make a decision, we shouldn't just go with a passing thought and assume things should be a certain way. We should follow design principles that have clearly stood the test of time — especially when facing a hard trade-off.

Likewise, when we look at something new — HTML5, say (not that new anymore) — we shouldn't only look at its new elements and new APIs. We should think about the design philosophy and history behind it, and the pain points it solves. That's how we come to understand where these things come from.

The talk

The Chinese version of this post is a reposted translation of Jeremy Keith's keynote HTML5 Design Principles at Fronteers 2010 (October 21, 2010). Since the original is in English, read it in his own words: HTML5 Design Principles, by Jeremy Keith.

The talk walks through the principles behind HTML5:

• Avoid needless complexity
• Support existing content
• Pave the cowpaths
• Degrade gracefully
• Priority of constituencies: put end users first

In an earlier post, open questions, I raised a question that went roughly like this: in the field of computing, why should we ask about the motivation behind something? Learning front-end development lately has given me a deeper feel for this question, so I'm pulling it out to discuss on its own.

The "Four Causes" in the Philosophy of Science

As early as ancient Greece, philosophers began pondering the nature of things. One very famous philosopher put forward a set of arguments explaining motion and change in things, and the core idea is what the heading above describes: the "four causes". And, as you'd expect, the philosopher's name is Aristotle.

To save time explaining the concept of the four causes, I'll excerpt a few paragraphs from Professor Wu Guosheng's (a Chinese historian and philosopher of science) blog post Heidegger's Philosophical Reflections on Technology. Of course, you can also consult the English Wikipedia entry, or more specialized books on the philosophy of science:

What brings a thing into appearance is called the cause of that thing. Aristotle held that there are four kinds of cause: the "material cause", the "formal cause", the "final cause", and the "efficient cause". Take the making of a crown: gold is its material cause; the shape of the crown is its formal cause; the emperor's use of it to hold grand ceremonies and display his majesty and splendor is its final cause; and the craftsman, who gathers all the requirements and preparations and does his part to finally bring the crown forth, is its efficient cause.

Today we don't really understand the doctrine of the four causes. We don't know why four causes were proposed, much less why exactly these four. Because when we speak of a "cause", what we usually mean is only the thing that acts directly and produces and brings about the final result, which is only the efficient cause in Aristotle's sense (it can also be called the "proximate cause" —note by the quoter). For the crown as something made, from our point of view today, the craftsman is its only cause.

The efficient cause became the decisive one among the four, and ultimately determined how causality is defined. This happened after the 16th and 17th centuries (that is, the years when the mechanical view of nature was taking shape —note by the quoter), and it was extraordinary: it contained modern thought's exclusion of the final cause. The appearance of a thing came to be described as the result of another, earlier thing plus some acting force; in a sense, the thing became something ready-made, so-called "matter". Things no longer "come to" appear; they are simply present, permanently present. Causality no longer speaks of what makes a thing "appear", but of quantitative changes in matter that is fundamentally present.
This definition of causality makes it impossible for us to understand the Greek doctrine of the four causes. In particular, it stamps our understanding of the efficient cause in the Greek doctrine with the brand of our own era: acting on a thing so as to change it. But the inner connection among the four causes in the Greek sense is completely lost. Heidegger reminds us that understanding the essence of instruments depends on understanding the essence of causality, and we remain utterly ignorant of the essence of causality to this day, an ignorance shown especially in our misunderstanding of the Greek four causes.

Professor Wu's text above already explains quite clearly what the four causes are, and why today, when we mention the "cause" of something, the implied context is "one cause" rather than "four causes". This comes from the paradigm shift in the view of nature that took place in the 16th and 17th centuries. That shift treated the "efficient cause" among the four (which can also be called the proximate cause) as the primary factor in how things change, while the other causes faded into the background.

The Efficient (Proximate) Cause: An Effective Way of Inquiry in Traditional Natural Science

In traditional science, when we talk about the cause of something, we usually mean its proximate cause. Take a question I once answered on Zhihu (a Chinese Q&A site similar to Quora): Why is ethanol non-toxic, while methanol, propanol and butanol are all toxic?

Then look at the answers below it. The top-voted answer describes the molecular structure of alcohols and their corresponding chemical properties, as well as the different effects that different alcohols have inside the human body, and gives a quite professional answer. What it sets out from this angle is "the proximate cause of alcohol toxicity", that is, "how things currently are".

Since the Scientific Revolution, the thinking reflected in that answer has been the guiding idea of research in the natural sciences. This mode of inquiry, asking after the "proximate cause", has an enormous advantage: it leads humanity to set aside the most fundamental questions for the time being, concentrate on what can be studied at the current stage, and stay silent about what can't be studied or is temporarily out of reach. For example, to study the motion of objects, the most direct approach is to study the forces that push them and the different effects produced under different forces, while setting aside what Newton called the first mover and not considering it for now. It is precisely by advancing step by step, precisely, according to an internal logic, and accumulating continuously, that this form of research has achieved major results, as we can see.

But this doesn't give "our expectation" an explanation that feels more fundamental. Yes, the answer is very accurate and fits very well with the physiological and chemical knowledge humans currently have, but on that basis one can still ask: why is it like this? What caused this physiological and chemical state of affairs?

The Final Cause: Our Expectation

As the question at the end of the previous section suggests, we humans always psychologically want a "more fundamental" explanation, and somewhat stubbornly believe that "a more fundamental explanation ought to exist", even though for now many of these are just "illusions of causality". This shows up even more clearly in Darwin's theory of evolution, as in my answer to the question "Why is ethanol non-toxic, while methanol, propanol and butanol are all toxic?":

This is obviously a question of physiology plus evolution, and once evolution is involved, "causality" gets subtle.

When we say something is "toxic", there's an implied context: toxic to living things. Methanol, ethanol, propanol and butanol are structurally similar, so why is ethanol the least toxic? Answering from physiology and chemistry can only tell you "what" this toxicity looks like. In terms of causality, reasoning this way finds the "proximate cause" of "being toxic".

The "proximate cause" is the cause most directly related to the thing being studied. Looking for proximate causes is also the most common form of research in physics and chemistry. Its origin has a lot to do with the Scientific Revolution of the 16th and 17th centuries; the details are complicated, so I won't go into them here.

But in biology, studying only proximate causes explains "how it manifests, what it is like" well, but doesn't explain "why it is like this" well. In answering "what it is like", physiology and chemistry have done a fine job. As for "why it is like this", we can only turn to evolution.

My bold guess is this: ethanol occurs more readily under natural conditions (why it occurs more readily is another question), so living things have had more opportunities to come into contact with ethanol than with other alcohols. This has made life less sensitive to ethanol.

As for evidence, all I can say is that this guess is consistent with the logic of evolution. Producing concrete data would be hard.

Setting aside for now whether this answer is correct, and also not worrying about the headache-inducing causality in evolution, what I want to say is that even a "half-baked final-cause answer" like the one above gives us a feeling of "Ah, that's it!", as if it satisfied some longing deep in our souls.

This is in fact the charm of the "final cause": it satisfies a certain mysterious human craving for the truth, even when the final cause is sometimes illusory.

The "Final Cause" in Computer Science

The above describes the way of researching or learning in the traditional natural sciences. But computer science is different. In nature, all kinds of things exist independently of humans, and most of them existed countless years before humans did. It is hard, one could even say basically impossible, for us to trace back and see what things looked like when they first appeared, and where they sat within the complex system of nature.

In computing it's a completely different story. A great deal of it is "created by humans". These things were built out of certain motives, which matches human intuition very well: if it's done this way, there must be a "more fundamental reason" behind it, and that fundamental reason is the creator's purpose and intent.

This can basically be generalized to everything related to man-made things. Whenever we design or create something, there is some purpose in it rooted in the environment of the time. Take ordinary tables and chairs: every design carries the purpose its creator wanted to express. None of them appeared out of nowhere; they come accompanied by the intentions the creator planted in them. With a folding chair, the designer may have been thinking about saving space. With a chair built purely with mortise-and-tenon joints, the creator may have been after naturalness and environmental friendliness, or nostalgia for days gone by.

In computing, we should likewise ask about the "final cause" of something, that is, the original intention of the creator behind it. By asking about these original intentions, we can better understand the intent built into a thing and deepen our understanding of it. Take design patterns: why do these design patterns exist, and are there others? Once we understand this, we realize these things aren't set in stone, and that frees up our creativity.

Front-End Development

This is even more true in front-end development. The front end is purely a human construction, so there are many things we can ask about. For example, what is HTML5 for? Why was HTML5 proposed, and what were its proposers thinking?

Asking these questions lets us trace a problem back to the designers' intent and understand the original purpose, which in turn gives us a deeper understanding and a better grasp of what's to come.

Why Clear Concepts Matter So Much

In American TV shows we often hear characters mention or use the rhetorical device called "metaphor," and we vaguely know it is similar to "analogy," but we lack a clear understanding of it. In fact, we often use this device unconsciously in everyday conversation, and the reason it stays unconscious is that we have no clear concept of "metaphor."

It is not only metaphor. In daily life we often run into situations like this: before you have a clear concept of something, you don't pay it much attention even if it is around you all the time. Your awareness of it is in an "unconscious" state. Once you grasp the concept, you leave that muddled unconscious state, start deliberately noticing it around you, and are surprised to discover: oh, it was there all along.

To put it more philosophically, this is "being as non-being, the nothing that exists." From experience, humans build a concept of a thing in language, and that linguistic concept in turn makes us pay more attention to the corresponding thing, which finally deepens our understanding of both the concept and the thing.

In The Non-Designer's Design Book, Robin Williams opens with the story of the Joshua tree. She was given a book on identifying trees one Christmas, read that the Joshua tree could be recognized from just two clues, and decided from its odd-looking photo that she had certainly never seen one in her parents' neighborhood. When she walked outside with the book, she found Joshua trees in front of most of the houses on the street, where she had lived for years without ever noticing them. Her point: once you can name something, you notice it everywhere, and it becomes yours.

What this example describes is not a special case; behind it lie deeper questions about language and cognition. Concepts are the most basic units of thought, and we can hardly overstate their importance. Some work in cognitive science and linguistics now studies exactly these questions. This post briefly introduces the concept of "metaphor" and explains why metaphor matters in computer science.

I originally introduced metaphor inside another post, but I came to feel the concept was so important that it needed a post of its own. As described above, once the concept of "metaphor" is clear, you begin to leave your old unconscious use behind and consciously use this device to express concepts and things that were hard to put into words. You will also find that metaphor is everywhere and come to appreciate deeply the benefits it brings.

How Code Complete Treats Metaphor

In Code Complete, right after the welcome chapter, Steve McConnell introduces metaphors for software development and stresses their importance. He observes that computer science has the most colorful vocabulary of any field: walk into a carefully climate-controlled machine room and you will find viruses, Trojan horses, worms, bugs, logic bombs, crashes, flames, twisted sex changers and fatal errors. Where else would you run into all that?

These vivid metaphors describe particular phenomena and things in the software world, and lively metaphors like these can describe much broader phenomena too. With their help we can understand the process of software development more deeply.

So What Is a Metaphor?

A metaphor is a kind of figurative comparison. In Chinese the word literally means "hidden comparison," in contrast to the ordinary, explicit comparison, the "simile."

By comparison, the simile is the form of figurative comparison used most in everyday speech, as in Chinese stock phrases like "as beautiful as a flower" or "a father's love is like a mountain." We use similes mainly for aesthetic reasons: we want to add the aesthetic qualities of one thing to another. In the phrases above, we want to give a person the loveliness of a flower, or give a father the weight and towering steadiness of a mountain.

A metaphor is different. It is mostly used not for aesthetic reasons but to express some real connection between two things. Take Shakespeare's famous metaphor:

All the world's a stage,
And all the men and women merely players;
They have their exits and their entrances; and one man in his time plays many parts…
—William Shakespeare

Shakespeare does not use this metaphor to give the world and people the aesthetic qualities of a stage and actors. He genuinely believes that the way the world works resembles the way a stage play works in certain respects.

Many people (usually cognitive linguists) believe that metaphor has cognitive significance. In other words, it can help humans understand something better, and the means it uses is to connect a newer, unfamiliar concept to other concepts that are already known and easier to understand, using the similarity between the two to aid understanding.

Some Examples: Metaphors We Have and Haven't Noticed

In language, when something new appears and needs a word to name it, one approach is to extend an existing word. This extension is not arbitrary; there is usually some particular similarity between the new thing and the old one.

Some typical examples:

• socket: originally a plug or electrical socket. In computer communication it was extended to mean a mechanism for exchanging information, and like a wall socket it carries the sense of plugging two ends together.
• docker: originally a dock worker. Today there is a widely used virtualization technology called Docker. It also borrows the metaphors of dock workers and shipping containers to express standardized delivery.
• To be continued

Miscellaneous

Running scheduled commands with Linux crontab + Django commands

The problem

While building an archive management system, I had a task that needed to periodically pull data from a DB2 database, process it, and store it in MySQL. That meant running a standalone script inside Django that uses Django's environment.

The solution

Django provides user-defined custom commands for exactly this:

Writing custom django-admin commands¶

And Linux provides a handy tool for scheduling tasks:

linux crontab

Error 2006 'mysql server has gone away' when writing large amounts of data to MySQL

The problem

While building an archive management system with Django + MySQL, this error appeared when I wrote nearly 30,000 rows into MySQL in one go.

The solution

It turned out MySQL has a default limit on the size of data written. Just add the following to /etc/my.cnf (create the file yourself if it doesn't exist):

[mysqld]
max_allowed_packet = 16M

Of course, you can also solve this by limiting how much data you write to MySQL at a time.

Vim split screens and the Ctrl P plugin

I'd always heard how great the ctrl p plugin was. But when I first tried it, I really didn't see what was so powerful about it, until I later discovered Vim's split-screen feature.

Using Vim's split screens together with Ctrl P lets me open multiple files without leaving Vim, which saves a huge amount of time spent cd-ing around directories on the command line.

Reference: Vim's split-screen feature

Basic regular expressions

Once I needed to pad a bunch of short lines of text with trailing spaces so that every line had the same width.
At first I tried all kinds of column editing and block editing, and none of it worked. Did I really have to write a program just for this?

Later I found that a regular expression pretty much does the job:

%s/$/ n_spaces /g

In Vim, the $ symbol has a rich set of meanings; here it refers to the character just before the end of the line.

Front end

Mainly notes on things that are easy to confuse while learning front-end development. Personally, among HTML, CSS, and JavaScript, CSS is by far the hardest to pin down. Because of browser default styles and constraints, CSS inheritance, cascading rules, and so on all acting together, CSS often doesn't behave the way I expect. So there may be more CSS content here.
These are just scattered notes; for now I don't have the energy to organize them into standalone posts. Maybe later I'll expand some of the expandable bits into proper articles.

The difference between CSS pseudo-classes and pseudo-elements

Pseudo-classes
Pseudo-elements

Telling the concepts apart

An answer on Stack Overflow:
What is the difference between a pseudo-class and a pseudo-element in CSS?

The explanation on W3C:
pseudo-elements
pseudo-classes

Inheritance of relative font-size values in CSS

When font-size uses relative units like em, the calculations compound.

From Stylin' with CSS (quoted from the Chinese edition)

Generally, em uses the browser default as its base unit, that is, 1em = 16px. If you want to use em but also need to set specific pixel sizes, you can set the body's font-size to 62.5%. That changes the base size from 16 pixels to 10 pixels (16 × 62.5% = 10). Then the correspondence between em and pixels becomes very clear: 1em equals 10 pixels, 1.5em equals 15 pixels, 2em equals 20 pixels, and so on.

The p element's text is 12 pixels (the body's 16-pixel base × .75 = 12px). strong is a child of p, so its relative text size compounds level by level, and it should be 16px × 0.75 × 0.75 = 9px.

Note the difference from text-indent: font-size compounds, whereas with text-indent a later value overrides the earlier one.

text-indent is inherited by child elements. If you set text-indent on a div, all paragraphs in the div inherit that indent value. However, as with all inherited CSS values, the indent value is not the value set on the ancestor but the computed value. Here is an example.

Suppose a 400-pixel-wide div contains text indented 5%; the indent is then 20 pixels (5% of 400). Inside this div is a 200-pixel-wide paragraph. As a child element it inherits its parent's text-indent value, so its text is indented too. But what is the inherited indent value? Not 5%, but 20 pixels. That is, the child inherits the indent value computed from the parent's width. As a result, even though the paragraph is only half as wide as its parent, its text is still indented 20 pixels. This ensures that paragraphs are indented by the same distance no matter how wide they are. Of course, setting text-indent on the child element overrides the inherited value.

How inline-block behaves

This value causes an element to generate an inline-level block container. The inside of an inline-block is formatted as a block box, and the element itself is formatted as an atomic inline-level box.

Put more plainly: an inline-block element has both the block element's ability to have its width and height set and the inline element's default behavior of not breaking onto a new line. And that's not all; for example, an inline-block element can also take the vertical-align property. In short:
an inline-block element is a block container formatted as an inline element.

Open question:

Does an inline-block's margin push out the parent's height, while an ordinary block's margin collapses with the parent's margin?

References:
The past and present of inline-block

w3c: the ‘display’ property

mozilla: vertical-align

Using rem

Today I experimented with the rem relative unit in Chrome. My initial test code was:

<!DOCTYPE html>
<html>
<head>
    <meta charset="utf-8">
    <title>test rem</title>
    <style type="text/css">
        html{
            font-size: 62.5%;  // the behavior here is a bit odd
        }
        body{
            font-size: 1.2rem;
        }
        div{
            width: 300px;
            height: 300px;
            background-color: red;
            padding: 1rem;
        }
    </style>
</head>
<body>
    <p>测试字体大小</p>
    <div></div>
</body>
</html>

But at this point font-size was not 10px. I'd seen online that people set the root element's font-size to 10px to make later calculations easier, but here it didn't seem to take effect.

The reason is that Chrome's default minimum font size is 12px; any value set below that gets reset to 12px. In that case 1rem=12px. On Chrome's minimum font size limit,

see: What happened after Chrome dropped -webkit-text-size-adjust

Block-level and inline elements

• Block-level elements
  • Take up a whole line
  • width and height can be set
  • Default
• Inline elements
  • Setting width and height has no effect

Reference: The few most essential concepts in CSS

width and height

Defines the width of an element's "content area." The default value is auto, and it is not inherited. The browser computes the actual width.

When height and width use percentage relative units, they are calculated relative to the parent element's width (the parent's width works the same way). If not specified, the default is the image's own width and height; if the image fails to load, the alt attribute value is shown, or the browser's built-in broken-image display.

When to use each centering method

// Horizontal centering
// When using margin to center an element horizontally,
// you need to specify the width of the element being centered (it has nothing to do with the parent's width)
// Obviously this only applies to block-level elements
// It centers the element the property is set on, not its children or its parent
margin: 0, auto;   
text-align： center

How things behave when an inner box is bigger than its outer box can contain

Tools I've used

• reactjs react router
• underscore
• project-path (npm module)
• fixed-data-table (React component)

Over-engineering in code

What is over-engineering in code

Over-design, or over-engineering, means exactly what it says: when writing code, imagining too many scenarios that don't actually exist and designing for those imagined scenarios, such as constantly accounting for changes that might come in the future. In reality, the code at that point is still a long way from needing to consider any of that.

Over-engineering I've run into

When we were building a product before, we actually overemphasized possible future risks or changes in some places. For example, the product stored images on Qiniu cloud storage (a Chinese cloud storage provider), and at first everything was stored as URLs. But then an engineer imagined a scenario: "What if Qiniu's domain changes someday, or we stop using Qiniu?" So he set about changing the URLs to keys, only to find that the rewrite wasn't easy at all, and in the end it was abandoned.

The other case was over-design at the product level. While we were developing a product, the product manager's prototype changed countless times, and some of the last changes even altered the most basic data, rendering all the code we'd written completely useless. And this was precisely a product that needed fast iteration; to this day the first version still isn't done. Strictly speaking this example isn't over-design, but I'll put it here for now.

I don't understand how we, who are always going on about the MVP (minimum viable product), completely ignored it when building our own product.

Of course, for someone like me without much programming experience, it's still too early to be thinking about this. I'm just writing it down for now, to reflect on later.

Product experience

First impressions of iOS

My Nexus, after three years of service, finally got too old and decrepit and was honorably discharged. Torn by Android's endless variety, I ended up just getting an iPhone. Right away a few things felt really awkward. Some of that is naturally the unfamiliarity that comes with switching systems, but some of it is clearly bad design.

The useless pull-down menu

Compared with Android's powerful pull-down menu, iOS's pull-down menu is only used for notifications or recently used app features, and part of the settings is placed in a menu that slides up from the bottom. Even those settings are incomplete. Take Wi-Fi: you can only toggle it on and off, and can't use it as an entry point to configure Wi-Fi further. To configure Wi-Fi you have to go dig through the Settings app.

Now look at Android's pull-down menu: whether it's Wi-Fi or the flashlight, it's incredibly practical and convenient, and used very frequently.

The infuriating alarm settings

This one is even more obvious. On Android, setting an alarm takes one step. On the iOS alarm page, you first tap Edit in the top left to enter edit mode, then tap the time before you can choose it. A feature that could be done in one step gets stubbornly split into several. Using it is incredibly frustrating.

The baffling keyboard and system-wide copy and paste

The key proportions on the 9-key input layout are a real pain: the main area doesn't get enough space, while a big confirm button sits on the right, taking up a ton of room.

And the behavior of the keyboard and system-wide copy and paste is simply bewildering; there's no discernible pattern to any of it. It feels like one thing one moment and something else the next.

A messaging screen that doesn't make full use of the touchscreen

Editing on the iOS Messages screen still means an Edit button in the top left that you have to tap first. Tapping a message only opens its details, and a long press does nothing.

This "Edit button" pattern is everywhere in iOS, like the alarm editing mentioned above. It's very different from Android, where a long press generally means editing the item you pressed. iOS doesn't assign any action to a long press, which adds yet another step. It really doesn't make full use of the interaction advantages of the touchscreen itself.

The soy sauce bottle in the kitchen

There's a soy sauce brand called Haitian with an incredibly annoying seal design. You open the cap, pull the ring tab, pour out the rich soy sauce, and then you discover the cap won't close anymore! Soy sauce is something you use over a long time; I have no idea what a one-time design like a soda-can pull tab is for.

Compare that with another condiment bottle in the kitchen, the white vinegar, and the gap in design is obvious. The vinegar has a big cap that's easy to twist open, and to help control the flow when pouring, there's a narrow slit inside that regulates it. Convenient in the extreme.

Changelog

2016-01-18 Updated up to the try-except-finally statement
2016-04-07 Renamed the post. Split its structure into core parts and fuzzy points

What I consider the core of the book

1. The core of the language
2. Variable types
3. Strict mode
4. Lexical scope, functions, closures, this, call, apply, bind
5. The prototype system

Core parts

The language core

Variables and their types

Variables in JavaScript have no type of their own. When you assign a value to a variable, the variable is really just referencing that value. This is the same as in Python.

Both differ from fixed-type variables in languages like C. In C, when you declare a variable you have to specify its type, and the variable is actually a fixed-size chunk of space that holds the value directly. Arrays are the same: an array can only store values of a single type, and C decides how much memory to allocate based on the type, for example 4 bytes for an int.

In JavaScript and Python you don't declare a variable's type. A variable is just a reference to data, and the type is determined by the data itself: whatever type of value the variable references, that's the variable's type. Data types in JavaScript fall into two groups (apparently borrowed from Java):

Primitive types

• Primitive types (5 of them)
  • Numbers
  • Strings
  • Booleans
  • null
  • undefined

Note: primitives are not objects, but you may have seen calls of the form APrimitiveType.method. That's because numbers, strings and booleans have the concept of "wrapper objects".

Object types

• Object types (object) (every data type other than the 5 primitives is an object type)
  • Plain objects (name-value pairs)
  • Arrays
  • Functions - JavaScript can treat function objects like ordinary objects

Strict mode: use strict

If you know a bit of JavaScript history, you'll know Brendan Eich designed it in ten days, drawing mainly on Scheme, Self, C, Java and other languages. Even with existing languages to borrow from, a really rigorous design in that little time wasn't realistic, so JavaScript carried some hidden flaws from the very start. Add the rapid spread during the browser wars, and these flaws never had enough time to be fixed before becoming part of the standard, which is why they have stuck around.

For a quick look at these flaws, see Ruan Yifeng's (a well-known Chinese tech blogger) article Ten design flaws of JavaScript, and the book JavaScript: The Good Parts.

use strict is actually a directive added in ECMAScript 5. It's very close to a statement but differs from one: it contains no keyword, is just a string literal, and can only appear in particular positions.

The purpose of use strict is to make the code after it run in JS's "strict mode", a subset of the language that fixes some important flaws and provides robust error checking and stronger security. Strict and non-strict mode differ in some important ways:

• The with statement can't be used in strict mode
• In strict mode every variable must be declared with var first (in non-strict mode, using an undeclared variable adds a new property to the global object)
• In strict mode, the this value in a function call is undefined rather than the global object.

Fuzzy points

Wrapper objects

Primitives are not objects, but you may have seen calls of the form APrimitiveType.method. That's because numbers, strings and booleans have the concept of "wrapper objects". In other words, when you use the . operator on one of them, JavaScript automatically calls new String(s) | new Number(s) | new Boolean(s) to temporarily convert these three primitive types into objects that inherit the methods of the corresponding type.

// A classic example
var s = "test";
s.len = 4;  // Because of the dot operator, a temporary String object is created here, but it is destroyed right away
var t = s.len; // A String object is created here too, but it is not the one above, hence undefined

• null and undefined have no wrapper objects
• You can construct wrapper objects explicitly with the constructors.

Variable hoisting and variable configurability

• Hoisting
  • A variable declared with var is moved up to the top of the scope it lives in
• Configurability
  • Global variables are properties of the global object; local variables have no such rule
  • Global variables declared with var are non-configurable and can't be removed with the delete operator

The eval operator

eval() is actually a function, but it has long been treated as an operator. ECMAScript tries to constrain how eval() behaves so that it looks like an expression.

That's because treating eval as an ordinary function without any restrictions has many drawbacks. First, eval() executes code dynamically, which breaks the original code structure and is hard for the interpreter to optimize. Second, if eval were an ordinary function, it could be assigned to another name, say g = eval, so anywhere g is used would have the same problem, making JavaScript code hard to optimize and performance worrying.

In short, if eval behaves like an operator, these serious problems go away. So the ECMAScript grammar keeps constraining it to make it behave more like an operator.

Labeled statements

Mainly used together with continue and break to allow more flexible jumps.

throw/try/catch/finally statements

I wrote an earlier post, Python Pitfalls I Fell Into, which covered an easy trap with try/except/finally in Python.

JavaScript has the same trap. As that post explains, since finally is designed to run no matter what, once a jump statement appears in finally, whatever the other blocks return is ignored.

function finallyTest(){
    var x = 'catch', y = 'except', z = 'finally';
    try {
        k = x + y;
        a = b;
        console.log("here is try------------");
    } catch(e) {
        console.log("here is catch----------");
        return x;
    } finally{
        console.log("here is finally---------");
        return z;
    }
}
// Run
finallyTest()
// Return value
here is catch----------
here is finally---------
'finally'

Function names

// A function declared directly has a function name
function foo(){}
foo.name  // the function name is foo
var bar = foo
bar.name  // assigning a function to a variable just points the variable at the function structure; it does not change the structure itself
// A function expression produces an anonymous function
var foo = function(){}
foo.name // "" (empty string)
// The function name exists only inside the function's own structure, and passing it around does not change that structure
function foo(){}
var o ={
    a: 'a',
    bar: foo
};
o.bar.name // foo

Losing the this reference

// Can happen when a function is passed around several times
function foo(){
    console.log(this.a);
}
var obj = {
    a: 2,
    foo: foo
};
var bar = obj.foo;
var a="oops, global";
bar(); //"oops, global"
// Can happen in callbacks (async)
// same code as above
setTimeout(obj.foo, 100); // "oops, global"

Our GitLab was deployed with Docker, which was quick and convenient.

Then an application on the server developed a memory leak, and to my surprise, after rebooting the server the whole of Docker was dead. Starting a container kept hitting a series of fairly low-level "filesystem unknown" errors, and the error details came in more than one flavor. After some digging I found that the devicemapper storage driver Docker was using had a bug, and judging from Docker's official GitHub, plenty of people had run into it. The discussion is here: docker fails to mount the block device for the container on devicemapper. It felt like I had fallen into a bottomless pit.

The fix offered on Stack Overflow and GitHub was: stop Docker, delete everything in /var/lib/docker, then restart GitLab. That looked to me like possibly another pit, and I didn't dare jump in. So I chose to study the problem slowly, and in the meantime redeploy GitLab on another server and migrate the data over.

Migrating the GitLab data

I thought the migration would be simple. It turned out there were plenty of pits waiting there too.

First I backed up GitLab's volumes on the original server, copied the backed-up volume files straight to the new server, mounted them the same way as before, and started it up.

Problems I hit

First, Redis complained about not having permission for something or other, so I googled around and fixed the permissions.
Then came an "undefined method" error. It turned out to be caused by the database not having been migrated.

Another hidden pit is the GitLab version. For critical services like this, try not to use latest as the version tag. Otherwise, when something goes wrong you'll have to work out which version you were actually running, and if you can't find out... get ready to rely on pure luck.

The final solution

Jumping into the pits one at a time like this felt pretty dumb, and later I found that someone had already filled them all in: Migrate a Gitlab Docker container from version 8.0.4 to 8.2.0

Appendix: the original guide

The Chinese version of this post includes my full translation of the guide above. For English readers, please read the original directly: Migrate a Gitlab Docker container from version 8.0.4 to 8.2.0. It walks through twelve steps, from stopping the container and backing up the volumes, through fixing the known permission issues and re-running the database migrations, to clearing the Redis cache.

One note of my own on it: step 4, removing the GitLab container, is optional if you are moving the data to a Docker GitLab instance on a different server.

On "concept," from Wikipedia. To keep the concept complete, I've quoted a fairly long passage here:

A concept, also called an idea, is an abstract, general thought or notion, or an entity that serves as a category or class designating entities, events or relations. Concepts ignore the differences among the things in their extension and treat them as if they were identical, so concepts are abstract. They apply equally to everything in their extension, so they are universal. Concepts are also the basic elements of propositions, just as words are the basic semantic elements of sentences.

Concepts are bearers of meaning, not agents of meaning. A single concept can be expressed in any number of languages; terms are the forms in which concepts are expressed. The concept of a dog can be expressed as Hund in German, chien in French and perro in Spanish. The fact that concepts are, in a sense, independent of language is what makes translation possible: words in different languages have the same meaning because they express the same concept.

A concept is a person's understanding of a complex process or thing. Philosophically speaking, the concept is the basic unit of thought. In everyday usage, people often treat a concept as equivalent to a word or a noun (a term). Chinese culture is traditionally regarded as not emphasizing the definition of concepts, and therefore as not having developed logic and the related philosophy and science.

Conceptual analysis means tracing the meaning, origin and possible contexts of a word. It's extremely common in philosophy: before discussing any question, you first have to analyze every concept in it and get completely clear on what each word means, and only then can the discussion move forward.

Rhetoric and distortion in translation

There are many reasons conceptual analysis is needed. A concept's meaning may have shifted over time, or a word may be entirely foreign and have lost its original meaning in translation. Computer science is itself an imported science in China, and nearly all of its technical terms are translations. So conceptual analysis here leans more toward removing the distortions introduced by translation.

In computer science, none of the many books I've read so far, not even the formal computer science education I received, included anything on analyzing terms. And because Chinese tends to value rhetoric over analysis, images that were fairly plain in the original English may, once translated into Chinese, pick up implicit rhetorical devices for the sake of elegance or imagery, and start to become mysterious. At that point, going back to the original word becomes very important.

Some such translations:

• handle, translated as 句柄 (literally "phrase handle")
• socket, translated as 套接字 (literally "sleeve-joint word")
• meta-programming, translated as 元编程 (literally "primal programming")
  and so on. In English these meanings are fairly plain, but in Chinese, to keep the terms concise and elegant, the translators borrowed a lot of native imagery, and that imagery is itself rather hazy. The originally clear meaning becomes hazy and hard to grasp. These translations are products of their history and the technical background of their time. If they were made today, they probably wouldn't be done that way at all; the key words might even be kept in English.

When we run into such words, we need to look at when they were translated and what the source word was before it became Chinese. Once we find the source word, we should understand it within its own context. But even the source word deserves analysis, because it didn't come out of nowhere either. Computing is a very young science, and many of its words are newly coined. Some were made up outright; others also used some rhetoric, extending an original meaning into the computing domain.

A word isn't extended without reason. Behind it there's usually some inner connection between concepts, and this often forms a rhetorical device: metaphor.

The metaphors behind concepts

Another benefit of conceptual analysis is that it lets us see the metaphors behind concepts, which in turn helps us understand a concept's core idea better and more deeply.

For example, a few metaphors I personally like:

• docker: the dockworker metaphor
• handle: the door-handle metaphor
• callback: the call-you-back metaphor
• hook: the hook metaphor

What these metaphors have in common is that their plain original meaning is connected, in a somewhat implicit way, to the concept in computer science: the two share a very similar structure or meaning. If you want to grasp what a term means quickly, it's very important to combine its metaphor with the context at hand.

The philosophy behind conceptual analysis

Distinguishing the "efficient cause" from the "final cause"

To be continued

Iterator Types

First, the part on iteration in the official Python 2.7 documentation

The passages below are quoted from the official documentation, Iterator Types. Before reading, it helps to be clear on what a few words mean:

• "iterator"
• "iteration"
• "iterable"
• "container": an object that can hold other objects, such as a list or a dictionary

New in version 2.2.
Python supports a concept of iteration over containers. This is implemented using two distinct methods; these are used to allow user-defined classes to support iteration. Sequences, described below in more detail, always support the iteration methods.

One method needs to be defined for container objects to provide iteration support:
container.__iter__()
Return an iterator object. The object is required to support the iterator protocol described below. If a container supports different types of iteration, additional methods can be provided to specifically request iterators for those iteration types. (An example of an object supporting multiple forms of iteration would be a tree structure which supports both breadth-first and depth-first traversal.) This method corresponds to the tp_iter slot of the type structure for Python objects in the Python/C API.

The iterator objects themselves are required to support the following two methods, which together form the iterator protocol:

iterator.__iter__()
Return the iterator object itself. This is required to allow both containers and iterators to be used with the for and in statements. This method corresponds to the tp_iter slot of the type structure for Python objects in the Python/C API.

iterator.next()
Return the next item from the container. If there are no further items, raise the StopIteration exception. This method corresponds to the tp_iternext slot of the type structure for Python objects in the Python/C API.
Python defines several iterator objects to support iteration over general and specific sequence types, dictionaries, and other more specialized forms. The specific types are not important beyond their implementation of the iterator protocol.
The intention of the protocol is that once an iterator’s next() method raises StopIteration, it will continue to do so on subsequent calls. Implementations that do not obey this property are deemed broken. (This constraint was added in Python 2.3; in Python 2.2, various iterators are broken according to this rule.)

Sorting out the concepts

That's roughly the core of what Python 2.7 says about iteration. To sum it up:

For a container object to support iteration, it needs to implement an __iter__() method. This method returns an "iterator object" (iterator). If a container implements this method, we call the container "iterable".

An iterator also needs an __iter__() method, which returns the iterator itself, plus a next() method that returns the next item. An iterator is, of course, itself "iterable". Together, these two methods of an iterator are called the "iterator protocol".

In for and in statements, whether you use a container object (such as a list) or an iterator object, it has to support the __iter__() method. For a container, this method returns an iterator; for an iterator, it returns itself. That is then used for the iteration. In other words, even a container object is first turned into an iterator object before it's iterated over. Python's documentation says so too, in The for statement:

The for statement

The for statement is used to iterate over the elements of a sequence (such as a string, tuple or list) or other iterable object:

for_stmt ::=  "for" target_list "in" expression_list ":" suite ["else" ":" suite]

The expression list is evaluated once; it should yield an iterable object. An iterator is created for the result of the expression_list. The suite is then executed once for each item provided by the iterator, in the order of ascending indices. Each item in turn is assigned to the target list using the standard rules for assignments, and then the suite is executed. When the items are exhausted (which is immediately when the sequence is empty), the suite in the else clause, if present, is executed, and the loop terminates.

Generator Types

The official documentation's definition

From Generator Types:

Python’s generators provide a convenient way to implement the iterator protocol. If a container object’s __iter__() method is implemented as a generator, it will automatically return an iterator object (technically, a generator object) supplying the __iter__() and next() methods. More information about generators can be found in the documentation for the yield expression.

Sorting out the concepts

What this means is that a generator object is just a shortcut for implementing the iterator protocol. Normally, when we need a custom iterator, we have to implement the required __iter__() and next() methods by hand; with a generator object, you automatically get back an iterator that supports the iteration protocol. And to create a generator, all you need is to use a yield expression in a function. That creates a generator function, and when it's called, it returns an iterator (usually called a generator) that controls the execution of the generator function.

From Yield expressions:

The yield expression is only used when defining a generator function, and can only be used in the body of a function definition. Using a yield expression in a function definition is sufficient to cause that definition to create a generator function instead of a normal function.

When a generator function is called, it returns an iterator known as a generator. That generator then controls the execution of a generator function. The execution starts when one of the generator’s methods is called. At that time, the execution proceeds to the first yield expression, where it is suspended again, returning the value of expression_list to generator’s caller. By suspended we mean that all local state is retained, including the current bindings of local variables, the instruction pointer, and the internal evaluation stack. When the execution is resumed by calling one of the generator’s methods, the function can proceed exactly as if the yield expression was just another external call. The value of the yield expression after resuming depends on the method which resumed the execution.

All of this makes generator functions quite similar to coroutines; they yield multiple times, they have more than one entry point and their execution can be suspended. The only difference is that a generator function cannot control where should the execution continue after it yields; the control is always transferred to the generator’s caller

technological neutrality
Its core claim is that technology is pure and value-neutral, and has in itself no properties of right or wrong, good or evil; technology ends up in human use, and the results it finally produces should be borne by humans themselves.

Technological Neutrality in the Kuaibo Case

The debate in the recent Kuaibo case, as far as the value of technology goes, can be called a microcosm of the discussion technological neutrality has received in the philosophy of science: technological neutrality has persisted in the philosophy of science for a very long time, and it still has a large following today. Among the general public, technological neutrality is even the only value view of technology there is.

Within the value view of technological neutrality, there are some very common arguments:

• An example from daily life: if someone kills a person with a knife, the guilty party is the person, not the knife, and not the factory that made the knife.
• An example from science: the atomic bomb is innocent; the guilty are the people who misuse it.
• An example from the internet: trojans are innocent; the guilty are the people who use them for illegal activities.

These claims all argue for the value-neutral character of technology. I'll explain later where the holes in these arguments are.

Technological Neutrality in the Philosophy of Science

In fact, the view of technological neutrality appeared very early in the philosophy of science, and because of technology's own "self-concealing nature", the values attached to technology naturally tend toward neutrality. After the Industrial Revolution, the ills of technology gradually came to light, and reflection on the value of technology also made new progress in the philosophy of science.

The Premise for Discussing the Value of Technology: When We Discuss the Value of Technology, What Are We Discussing?

"Technology has no value bias. The consequences of any use of technology should, in the final analysis, be borne by humans, whether they are good or bad." Fundamentally, this statement is right, because without humans there would be no technology at all, and so no side effects brought by technology. In this sense, saying "all guilt ultimately falls on humans" is beyond reproach.

But in reality, we can't discuss the problem this way. Even in the philosophy of technology, discussion in the above sense is meaningless; it gives no substantive help to human understanding of technology.

The premise for discussing the value of technology is:

"Technology is created by humans and used by humans. Technology is not 'dead'; it has effects on humans."

Under this premise, when we discuss the value of technology, what we are discussing is: whether technology itself is really pure, and whether technology produces side effects on humans in certain respects. More precisely, technology is created by humans, so does this creation affect humans' construction of themselves?

The Intentional Structure of Technology

Imagine the following scenario:

There are two peoples. One gives its members knives and horses when they are very young. The other gives them poetry and books. What happens when these two peoples grow up? You can imagine that for the first people, the knife becomes their preferred way of speaking. For the second, poetry and books become their chosen way of speaking.

To take an example closer to daily life: once my boss asked me whether using a Mac felt great, and whether using a Mac, using these better tools, just makes you want to write things, makes you want to write code? Use a Mac and you want to write code; use Windows and you want to play games.

The examples above are meant to show that technology has its own intentional structure, and this intentional structure carries an attached value that unconsciously produces side effects on humans. The production of these side effects is not something humans can choose, especially in groups. Living inside their own paradigm, humans lack the ability to clearly discern the long-term effects of an intentional structure.

Note that this only says technology has its own intentional structure that in turn produces unconscious side effects on humans. It makes no value judgment about that intentional structure; that is, it does not judge whether it is good or bad. In fact, that is a very difficult task, and most of the time an impossible one.

On the other hand, saying technology has side effects on humans' construction of themselves is also unlike "determinism" in traditional philosophy, which leads to a situation where humans have no free will, and for any decision they make and any mistake they commit, they need not, and cannot, bear the consequences.

In the philosophy of technology, the thesis of technological non-neutrality holds that "technology and humans construct each other".

Back to the Technological Neutrality Argument in the Kuaibo Case

The intentional structure of technology means it will inevitably produce side effects on humans, and it also exposes a very large hole in the arguments listed earlier: they assume a premise, namely that humans can judge with complete clarity what potential side effects the use of a technology will have. In other words, that humans can clearly know, and control, the counter-effects brought by technology, including both the very obvious ones and the latent ones.

But this is unrealistic. The side effects of many technologies are hard to detect. It's hard for you to know clearly what effect long-term use of WeChat, as a technology, will have, or already has had, on you. Not to mention that humans live inside a great net woven from technologies.

Another aspect is that the side effects technology produces on groups are more complex than those it produces on individuals, because the behavior of a group is not a simple sum of individual behaviors, and the influence a technology has on a group runs deeper. Take programming languages: pure C programmers and Python programmers are bound to differ, and when these differences show up at the group level, they basically reflect the differences in the design philosophies of C and Python, that is, the differences in their own intentional structures.

In other words, the effect a technology produces within a group will end up closer to the technology's own intentional structure: if Kuaibo makes it easy to spread pornographic material, then under natural conditions at the current stage of civilization, it will inevitably spread pornographic material, not study material. Once you understand this, you can see clearly where the key lies: control, regulation, critique, reflection.

This is also what critical theory of technology holds:

"Even though technology has its own intentional structure, what this teaches us is to handle it more cautiously and more critically, rather than directly making a value judgment that affirms or negates it."

Philosophy of Science and Sociotechnical Decision-Making

To be continued

Appendix

I had never thought deeply about the philosophy of technology before. Only from the book Amusing Ourselves to Death did I sense that the relationship between technology and humans isn't as simple as splitting into two. Later I also thought about the problem of technological neutrality, and always felt something was wrong with it but couldn't put my finger on it, until I read Professor Wu Guosheng's Lectures on the Philosophy of Technology and got the answer I'm most satisfied with so far:

"Technology and humans construct each other."

Notes on the Key Points of Lectures on the Philosophy of Technology:

The Humanistic Essence of Technology

• What is technology?

  • Technology is a way of human existence; technology and humans are not split into two

• What is the essence of humans?

  • Humans are animals without an essence; the human essence is self-constructed

• Characteristics of technology

  • Technology is self-concealing, and the history of Western scholarship is a history that values only internal lines of reasoning
  • The problem with technological neutrality: technology has an intentional structure and contains particular value orientations
  • Technology is a link in constructing humans and the world: bodily techniques, humans are all born premature, shaping of the body, shaping of perception. The use of PPT.

Technology and Politics

• Does technology have politics?

  • Technology and politics are related. Politics intervenes in technology, and technology affects politics: gunpowder wiped out the knightly class and was the gravedigger of the feudal system. Hydraulic societies and centralized power.

• The political critique of technology

  • The hand mill gave rise to the feudal system; the steam mill gave rise to the capitalist system.

• Taylorism and Fordism

  • Technology leads to the retreat of human nature: seeing deaths in the news and gradually becoming indifferent

• Three views of technology

  • Technological optimism: technological neutralists; problems brought by technology can only be solved by technology
  • Technological pessimism: technological autonomy, technology has an internal logic of development; holds that modern society has entered a one-dimensional society, the sense of difference disappears, and the tentacles of technology reach into every field. But it views technology holistically, treating it as monolithic, without working out the various possibilities of technology. Only by going into the details can one get out of pessimism.
  • Critical theory of technology: go into the details of technology and examine every aspect: the theory of technological transformation. Assembly lines, computers, communications, transportation, propaganda technology. For every new technology, ask who benefits and who loses.

Technology and Philosophy

• The historical absence of the philosophy of technology

  • Beauty, justice, goodness, art and science are all focuses of philosophy; technology is not.
  • Technology itself: its self-concealing nature, and long-standing technological neutrality
  • Philosophy: philosophy emphasizes reasoning and internal logic: essentialism

• The rise of the philosophy of technology

  • Changes in technology itself: problems with technology have recently appeared; it is no longer transparent
  • Philosophy: the collapse of essentialism and the emergence of phenomenalism. Empiricism inherits nominalism.

Past experience, if not forgotten, is a guide for the future. (前事不忘，后事之师) — Strategies of the Warring States, "Strategies of Zhao I"

Module Imports in Python

Package Imports

This one is fairly simple, so a single example will do. Still, it did trip me up when I was starting out…

# File structure:
packages
├── __init__.py
└── my_module.py
# This is a package import
import packages
# What follows is obtained from attributes defined in __init__.py
packages.some_module.some_method
# This treats the package as one step on the search path
from packages import some_module
some_module.some_method

The Module Search Path

Consider the following scenario:

# Django has a common app, and it has been added to the installed apps list
# File structure of the common app
common
├── __init__.py
├── common.py
├── others.py
└── somethings.py
# somethings.py contains this import statement
from common.others import some_func   # raises an error

In the code above, because the names are the same and the search path by default starts from the directory of the code currently being executed, the common file is found before the common package, so it shadows the package on the search path. Once common is found, Python looks for some_func in the common file, can't find it, and raises an error.

Execution Order in try-except-finally

In Python exception handling, the finally clause is always executed, whether or not an exception is raised in the try clause. Because of this, finally is often used for cleanup, such as closing files or database connections.

If an exception occurs in try-except-finally and is not properly handled, Python first stashes the exception, and after the actions in finally have finished, it passes the stashed exception up to the caller. This execution order is what creates some potential traps.

Look at this example:

def finallyTest():
    print 'start------'
    while  True:
        try:
            print "running------"
            raise IndexError("r")   # raise an IndexError
        except NameError as e:
            print 'NameError happended {0}'.format(str(e))
            break
        finally:
            print 'finally executed'
            break # there is a break statement in finally
            
# run the test function
FinallyTest()
# output:
start------
running------
finally executed

In this example, the try clause raises an IndexError, and the except clause doesn't catch it. You would normally expect the error to propagate up to the caller (here, to the interpreter) and trigger an IndexError. But after the whole function runs, no exception appears.

The reason is that when an exception occurs in the try block and no matching handler is found in the except clauses, the exception is "stashed temporarily." When finally finishes, the stashed exception is "raised again." But if finally raises a new exception or executes a return or break statement, the stashed exception is lost, which leads to "exception masking."

Another, slightly more complicated example:

import logging
logger = logging.Logger(__name__)
def returnTest(a):
    try:
        if a == False:
            raise ValueError("data can not be False")
        else:
            return a  
    except ValueError as e:
        logger.exception("exception!, {detail}").fomat(detail)
        return "return exception in except statement" 
    finally:
        return "return in finally statement"
print returnTest(False)
print returnTest(True)

You would normally expect the first print to trigger the error, enter the except clause and log the exception, and since the statements in finally always run, it returns 'return in finally statement'. The second should return True. In fact, both return 'return in finally statement'.

The reason is the same as in the previous example. Whether in the try suite or the except suite, a return or break, any statement that makes the program jump, is saved first. Then the statements in finally are executed, and as soon as finally contains a jump statement, the program jumps right away, so the saved state above simply disappears.

The Limits of Dict Comprehensions

On one project I ran into an operation that had to query the database many times. Specifically, there were companies, each with attributes such as a name and a company code. Fetching company information from the database every time caused performance problems, so I decided to cache it in a dictionary and wanted to use a dict comprehension. I found that each iteration of a comprehension can only produce a single entry. I had never noticed this before, so I'm writing it down here.

# This works
>>>{model.key: model.value for Modle in models }
# The result is an ordinary dictionary
>>>{key1: value1, key2: value2, ...}
# But using it like this is not supported
>>>{model.key1: model.value1, moedel.key2: model.value2 for model in models}
# The workaround is to make the value above a dictionary, or to use a list comprehension and wrap it in one more list
>>>{model.key1: {models.key1: models.value1, model.key2: model.value2, ...} for model in models}

Here is the real example. The branch-related parts are branch company information, and the subbranch-related parts are sub-branch company information. The goal is to build a dictionary whose keys are branch company names and whose values are the branch's primary key, its number, and the sub-branches under it. The sub-branches are themselves a dictionary, keyed by sub-branch name with the sub-branch's other information as values. Together this forms a two-level dictionary cache of company data.

companys_by_name = {
        branch.name: {
            "pk": branch.pk,
            "number": branch.number,
            "subbranch": {
                subbranch.name: {"pk":subbranch.pk, "number":subbranch.number} for subbranch in SubbranchCompany.objects.filter(branch_company=branch)
            } for branch in BranchCompany.objects.all()
}
# In detail, the meaning is as follows; sub-branches are companies under a branch
company_info_dict = {
     branch_name:{
        "primary key": primary key of the current branch,
        "company number": company number of the current branch,
        "sub-branches":{
            sub_branch_name:{
                "primary key": primary key of the current sub-branch,
                "sub-branch number": number of the current sub-branch,
            } for sub_branch in sub-branches under the current branch
        }
     } for branch in all branches
}

Maybe because I've had a little education in the philosophy of science, I always try, while learning something, to understand its final cause. In natural science research, pursuing efficient causes is the best way to grow knowledge, while pursuing final causes often causes problems and isn't realistic. But in computer science, a great many things are defined directly by humans; you could say most of the computing world is something humans created themselves. In that situation, pursuing final causes is the best and deepest way to grow knowledge.

Anyway, I've written down the questions and my current thinking.
For me, the moment I can ask a question is the moment I really start to understand.

Engineering questions

• What is the relationship between RESTful and ROA (resource-oriented architecture)?
  • I roughly understand RESTful and have done some simple work with it.
  • Taken literally, resource-oriented architecture is an architecture centered on resources. On the backend, for example, you split things up by resource type and then combine fairly independent resources to produce the content you want.
  • As a metaphor, it's like constructing a building where all the raw materials (resources) are supplied in finely divided form, and in theory you can use them to build most kinds of building structures.

Meta questions

• How learning computer science differs from learning natural science.
  • This is mainly about methodology. In natural science, in physics for example, we might try to find the final cause of something, such as why the speed of light is finite, or, in biology, why ethanol isn't toxic to humans while methanol, with a similar structure, is highly toxic. In natural science research we go straight to finding out how something behaves; this is called the efficient cause, and this approach easily grows knowledge in natural science.
  • In computer science, most of the world is created by humans. Humans directly defined and built this world, so much of the time we can ask about final causes. Even if, in the end, some questions lead back to natural science or the formal sciences, pursuing final causes lets us get to the essence of a problem faster.

I'd occasionally hear metaphors about the three pillars of the web page: HTML is the skeleton, responsible for the page's structure; CSS is the skin, responsible for how the page looks; JavaScript is the muscle, powering the page's interactions. But that kind of description seems too broad to give you a concrete understanding.

So this post tries to lay out something between that metaphor and the low-level details. By describing the core role of HTML, CSS, and JavaScript in turn, I want to see whether I can get that bird's-eye view.

First, let's distinguish two front-end concepts: "typesetting" and "layout."

See two answers on Zhihu (a Chinese Q&A site): Typesetting and layout are two different things and Layout and typesetting are different

"Layout" refers to the macro-level division of the GUI, for example when you have several large blocks and want to arrange them in a specified way.

"Typesetting" refers to things like this: you have some paragraphs containing different fonts and font sizes, and you align them to different baselines, wrap lines, break words, and handle overflow sensibly.

Starting from the web page

A web page is the networked aggregate of all kinds of real-world information carriers

The web page is the most direct visible part of the front end. Its purpose is to "present information," and everything else serves that purpose. A web page isn't fundamentally different from any other carrier of information. Think of the information carriers we see in daily life: a Word document, a PDF, a questionnaire, a financial statement, and so on. A web page is a networked aggregate version of these carriers.

When humans look at a web page, what are they looking at?

Just as we treat the information carriers in our lives and the information they hold, we treat web pages in a similar way: first we pay attention to the content the document presents. If the content is formatted beautifully, we may enjoy reading it more and be more willing to read further. And if the carrier has a bit more interactivity, it's even more engaging.

In other words, when humans look at a web page, the order is still content -> style, interaction. Of course, as web pages grow richer and more colorful, style and interaction get more and more weight, to the point of being as important as the content.

Content corresponds to HTML; style and interaction are determined jointly by CSS and JavaScript.

HTML: a semantic document flow

The "two semantics" of a web page

Of course, as an information carrier a web page differs from physical objects in everyday life. Although the ultimate audience for the information on a web page is human, it reaches them through browsers, search engines, or other programs that can process web pages. This gives rise to two kinds of semantics on a web page:

The first is the meaning that is ultimately presented, understood by humans; this is semantics in the everyday sense.
The second is understandable by machine programs such as browsers and search engines; it is "machine-readable semantics."
The semantics ultimately presented are determined by human knowledge structures: humans interpret different content on a page as elements with specific meanings, such as "title," "body," "footnote," and "navigation." This is exactly why some HTML elements come with certain styles built in. Headings h1 through h6, for example, have a larger font size and are bold. That matches human common sense and fits the meaning to be presented, so this default is very convenient.

The second kind, machine-readable semantics, in the sense above, serves the first kind. Browsers, search engines, and other programs recognize the semantic elements in HTML code, then add styles and so on to present the final result. The more semantic the HTML, the easier it is for machines to understand. In software, semantics is usually built on sensible "structuring." Structuring means distinguishing content with different meanings by giving it different structures, and in HTML that structuring is done with the various HTML tags.

Presenting the first kind of semantics in a friendlier way is the overall goal of a web page. That takes close collaboration among designers, information architects, and programmers.

Making the second kind of semantics serve the first kind better is the essential work of the programmer.

Why the second kind of semantics matters

There's a very apt example of why the second kind of semantics matters. I don't know if you've run into this on your smartphone:

When you need to type something on a website, a keyboard pops up. But different input fields bring up different keyboard layouts: when you type an email address you get a keyboard with the @ symbol, when you type a password you get the full keyboard, and so on. Stranger still, this sometimes works and sometimes doesn't. When it works, typing feels very smooth. When it doesn't, you have to switch keyboard layouts by hand to find the symbol you need, which is really annoying.

Behind this is the second kind of semantics at work: in HTML there are many different types of input fields, some email, some password, some text. If a website distinguishes these meanings well and uses them in its HTML, the keyboard pop-up behaves very intelligently, because the machine can choose which layout to show based on the different semantics in the HTML. If it's email, it shows the layout with @; if it's a password, the full keyboard; and so on. Sites with poor semantics can't do this. They just show the generic layout, and the human has to figure it out and switch.

This example shows that as browsers and other software get smarter, semantic HTML becomes more and more important.

As mentioned above, browsers generally give certain HTML elements certain styles, but only because that matches human common sense and is convenient. Never use an element because you want a certain style; instead, choose the element whose meaning best fits the content you're marking up. Styling should be left to CSS. This is also why people sum it up as: "HTML cares only about semantics, not layout, and not style or interaction."

Understanding a web page through HTML semantics

Having covered why semantics matters, let's look at how to understand a web page through these elements of the second kind. Quite a few elements are involved here, so this is only a rough outline.

First, web pages have quite a few semantic elements for presenting information.

The most basic ones, like

<!--header-->
<header></header>
<!--main body-->
<main></main>
<!--footer-->
<footer></footer>
<!--paragraph-->
<p></p>
<!--section-->
<section></section>
<!--self-contained piece-->
<artical></artical>
...

and more specific ones:

<!--time-->
<time></time>
<!--address-->
<address></address>
...

These are all ordinary semantic elements for presenting information. But just as in real life, there are other ways to present information that help humans read and understand it better. For example:

• Lists
  They lay out the core information point by point and category by category, letting people quickly grasp the key aspects of something. In HTML

• Embedded elements
  Such as images; a picture is worth a thousand words.
  And music, video, and so on

Also, in real life we often hand out paper forms to collect information, and use reports to clearly present information sorted into categories in a fairly structured format. HTML has corresponding semantic elements for these:

• Forms (form)
• Tables (table)

They serve much the same purpose as their real-life counterparts. HTML forms come in many types: radio buttons, checkboxes, input fields (and input fields themselves have finer-grained semantic types). That mirrors the many kinds of surveys and forms in real life.

HTML also has some semantic elements of its own. For example:

<!--link, perhaps the most important unique element: it ties pages together over the network-->
<a></a>
<!--some other elements, which describe relationships between pages, page information, and so on-->
<link>
<meta>

Those are basically the kinds of HTML elements there are. Combined, they include parts for collecting information and parts for presenting it, and to present information better, HTML provides many semantic elements.

Put together, these elements form all the semantic information of a web page. They can implement and imitate almost every real-world information carrier, and in some respects even surpass them.

The document flow

Besides the elements mentioned earlier that carry some styles based on common sense (if those count as typesetting styles), HTML also has a simple default way of laying things out (let's call it layout style): elements are arranged in order from top to bottom and left to right. To put it more vividly, this is called the HTML document flow.

This default document flow gives HTML a common-sense, most basic layout, so that elements don't all end up heaped together in a mess. But I'll stress it again: although browsers give HTML some default styles and layout, these are all based on human intuition and common sense: "HTML cares only about semantics, not layout, and not style or interaction."

DOM

DOM stands for Document Object Module, that is, the Document Object Model. Here is the explanation from W3.org:

What is the Document Object Model?

The Document Object Model is a platform- and language-neutral interface that will allow programs and scripts to dynamically access and update the content, structure and style of documents. The document can be further processed and the results of that processing can be incorporated back into the presented page.

Why the Document Object Model?

“Dynamic HTML” is a term used by some vendors to describe the combination of HTML, style sheets and scripts that allows documents to be animated. The W3C has received several submissions from members companies on the way in which the object model of HTML documents should be exposed to scripts. These submissions do not propose any new HTML tags or style sheet technology. The W3C DOM Activity is working hard to make sure interoperable and scripting-language neutral solutions are agreed upon.

What are those two paragraphs saying?

In other words, the DOM is the Document Object Model. It isn't part of some specific language's standard library or anything like that; it's a model independent of language and platform. Any other language that wants to manipulate HTML documents (actually not just HTML, but also XML and some other structured languages) has to design its way of doing so according to this object model.

The reason the DOM came about is that a bunch of people used to be at each other's throats: you had your ideas, I had mine. To unify the martial world, the W3C proposed the DOM and has kept standardizing it, which is good for everyone.

The DOM itself is a fairly complex thing, but for the DOM we use day to day, it's enough to know roughly what it looks like and how to manipulate it to count as getting started. When we talk about the DOM, we really mean a tree structure of HTML:

With the tree structure in the figure above, other languages that implement DOM manipulation, such as JavaScript, can operate on these nodes according to certain rules. The DOM tree itself also defines some concepts of inheritance, which make it easier to control its nodes.

Summary

That's basically the whole core of HTML. To sum it up in a few words: semantics, common-sense default styles for elements, the document flow, and the DOM.

The basic setups of The Three-Body Problem and The Walking Dead

Both setups are heavily Darwinian and put great weight on the struggle for survival.

The basis of the theory of evolution

In a given environment, resources are limited and can support only a certain number of living things, while the number of living things keeps growing. To win the limited resources they need, they must struggle to survive: the winners live on and the losers are eliminated.

The setup of The Three-Body Problem

Big Liu extends this idea to the whole universe and sets up a similar sociology at the cosmic scale: the universe's total resources are also limited, all kinds of civilizations live in it, and the more advanced a civilization, the more resources it needs. At the same time, every civilization's highest principle is that "survival is the primary need of a civilization"; it wants to survive. In such a universe, Big Liu plays out civilizations' strategies for survival.

The setup of The Walking Dead

A sudden virus disrupts the existing world order and drops the surviving humans into a world of walkers, full of danger on every side and short of the resources needed to live, to show how humans survive at the end of the world.

Strategies in The Three-Body Problem and The Walking Dead

The Three-Body Problem

Whether it's the Trisolarans' interference by sophons or the Singer's erasure by dual-vector foil, the strategy is pure dark forest: no cooperation, no game playing. "I destroy you, and what does that have to do with you?" Blunt and direct. You could say it pushes the struggle for survival to its limit. The loser mentality mocked in "Trust in Losers and Never Be Redeemed" [1], the review that has long held the top spot in the Three-Body section of Douban (a Chinese book and film review site), is exactly this survival-competition thinking pushed to the extreme.

The Walking Dead

It adds more elements. The protagonist's job before the apocalypse was police officer: the best representative of the old world order and its rules. It also examines humanity and morality: police officers, soldiers, doctors, teachers, priests, ordinary citizens, children and others each stand for a different kind of person and are shown in different ways. Survival strategies of every kind, family-style, community-style, trap-style, are examined one by one.

Order in The Three-Body Problem and The Walking Dead

The Three-Body Problem

The whole story opens against the backdrop of the Cultural Revolution, and what it sets out to highlight is the tense atmosphere of suspicion of that period and the chaos of moral order at the time: a single political movement was enough to incite countless people to push the so-called morality and humanity of thousands of years of civilization into the abyss. One major theme of the Cultural Revolution was how to survive, and that was only within humanity itself. Extend these fragile things to different civilizations across the whole universe and magnify them, and the dark forest is not unimaginable. Losing reason can overturn morality and humanity, and so can complete reason. Big Liu shows one such possible cruelty: morality and the like are too unreliable, and what can be relied on are strict rules of the game. The Swordholder logic understands the rules; Cheng Xin (in my view the "saintly" Cheng Xin is the spokesperson for morality and humanity) doesn't understand them, or rather doesn't want to. "Lose humanity, and you lose much; lose bestiality, and you lose everything." Big Liu stresses this point in different ways [2].

The Walking Dead

It keeps the story on Earth and clearly examines different kinds of people in much more detail and in many different ways. As the plot develops, the conflict between humans and walkers, though always an important backdrop, gradually fades, while conflict between humans and humans comes to the fore. Tens of thousands of years ago humans gradually began forming societies and used language, technology and so on to discipline themselves, eventually more or less completing their own domestication and forming a world of meaning that represses themselves [3]. I think this could be called the technological origin of morality and world order. But this order arose from a particular stage of humanity combined with a particular environment, and the apocalypse ended the old order along with the social and natural conditions that went with it. In the new environment, how should highly evolved modern humans survive? Can they rebuild a different world order? How will moral standards change? After going through many situations, Rick and his group gradually shake off the influence of the old world order. In this group led by a policeman you can still see things beyond survival, and their minds still waver, but the struggle for survival has sunk deeper and deeper into the group's consciousness. I don't know what strategy would serve their survival best, but it will certainly differ from the dark forest of The Three-Body Problem.

Darwinian thought and sociology

Darwin's idea of the struggle for survival holds a significant place in sociology, and a social Darwinism grew out of it. Applying Darwin's ideas to sociology also produces analyses that are logically clear and especially seductive, much like analysis in the sciences (The Three-Body Problem is one such case). But Darwin himself overemphasized the role of the struggle for survival and neglected phenomena beyond it, and when applied to society it struggles to handle complex social phenomena. Besides, Darwin's idea is only "evolution" in the sense of change over time, not "evolution" in the sense of progress.

January 2016, Huilongguan, Beijing

Notes

[1]: For details see "Trust in Losers and Never Be Redeemed" (信卢瑟，永世不得超生). I personally don't agree with this review. What I disagree with is not whether its criticism hits The Three-Body Problem where it hurts. Quite the opposite: the review is right that the book does adopt an extreme, even radical, principle of survival competition. What I disagree with is its misunderstanding of science fiction as a genre. A common technique in science fiction is to assume a premise and then build out a self-consistent world from it. Since Big Liu assumed that survival is the primary need of a civilization, a novel written from that premise is bound to look like this. Of course you could assume that survival is not the primary need of a civilization and describe a relatively harmonious universe, but that would be a different story. A similar example is the classic science-fiction stories built on the premise "what if ice were denser than water". Under such a counterfactual premise, you can't fault the author for the premise itself. What you should look at is whether the world the author shows under that premise is self-consistent, and how deep the portrayal goes.

[2]: In his new book Liu Cixin on Science Fiction, Liu Cixin reveals that he is a scientism believer and a worshipper of technology. The worlds he builds are often as hard and cold as steel, short on humane concern. I think this is a weakness, but also a strength.

[3]: Here I draw on Wu Guosheng's Lectures on the Philosophy of Technology, which gives a lecture-style overview of the philosophy of technology. Human self-discipline is one of its important themes.

The scope of science studied in the history of science goes beyond what we mean by science in everyday life. The history of science deals with science in a broader sense, which generally includes the following:

• The rational science of ancient Greece
• The mathematical-experimental science born of the Scientific Revolution
• Natural history in a broader sense
• Science in the widest sense, including technological systems

The main line of the "history of Western science" that the question refers to runs:

• Ancient Greece
• Rome
• The Middle Ages
• The Renaissance
• The Scientific Revolution
• The present

"Science" in what follows is also used in this broad sense, but the main line and the side branches of Western scientific development will be obvious at a glance. One very important thread is the relationship between the type of civilization and the type of science.

II: East and West in the History of Science

If we trace science back to its origins, there are two fairly typical types of science in its history: the "practical science" of the East and the "rational science" of ancient Greece. The civilizations that carried these two types of science also fall into two types: hydraulic civilizations and non-hydraulic civilizations.

Hydraulic Civilizations and Practical Science:

The representatives are the four great ancient civilizations:

• The Mesopotamian civilization between the Tigris and the Euphrates
• The Egyptian civilization of the Nile valley
• The Indian civilization of the Indus and Ganges valleys
• The Chinese civilization of the Yangtze and Yellow River valleys

All of these civilizations grew up beside water, with people gathering along well-developed river systems to farm. In this kind of civilization, keeping the state running normally and keeping the people fed were the ruling class's first concerns. Accordingly, science, which was held by the ruling class, leaned toward the practical, concentrating on water management (indispensable for irrigation), astronomy (needed for farming, for daily life, and for predicting the fate of the dynasty), and medicine.

Non-Hydraulic Civilization and Greek Rational Science:

Geographically, Greece lies on the Mediterranean coast. The land is mountainous, with no large rivers and no broad, fertile alluvial plains, so irrigation depended more on rain than on river water management. Unable to sustain themselves on local conditions alone, the Greeks gradually developed strong seafaring and trade. At the same time, Greece was neither too close to nor too far from Mesopotamia and Egypt: close enough for exchange, far enough not to be fully assimilated. In addition, the mountains split Greek political life into many small city-states, which both cooperated and competed with one another. In this environment the Greeks developed the democratic city-state and slavery, which guaranteed the emergence of a leisured class, while a free, democratic environment made political debate the norm. All of this cultivated the Greek spirit of reason. Unlike the administrators of the East, who had to strain to keep a centralized empire running, the Greek leisured class paid more attention to thinking about questions such as political legitimacy. These conditions made Greece a remarkable anomaly in the history of science. The Greeks were good at argument, had a passion for getting to the bottom of things, and had a scholarly spirit that rose above worldly concerns. Greek astronomy, for example, differed from the practical astronomy of the East: it was a mathematical planetary astronomy whose aim was to explain nature, not to serve farming, astrology or other practical ends. Explaining nature by nature itself, without appealing to gods or other external agents, and moving from Thales's "water is the origin of all things" to Plato's theory of Forms and on to Aristotle's comprehensive natural philosophy, clearly shows how rational science emerged. This is also the oldest source of modern science.

III: How the Two Types of Science Developed

The Inheritance and Development of Greek Rational Science:

Greece developed a rational scientific tradition that was disinterested, fairly systematic and driven by its own internal logic: Ptolemy's Almagest and Euclid's Elements, for example, and, more importantly, the rational paradigms founded by Plato and Aristotle. Plato's theory of Forms and Aristotle's natural philosophy differ, but both are rational systems with an internal logic. After the fall of Greco-Roman civilization, this rationality was absorbed into Christian doctrine, forming patristic philosophy in the earlier period and scholasticism in the later. Greek culture was at the same time transplanted into the Islamic world, and after the Crusades it began to revive in Europe. After the Age of Discovery, the Reformation and the Renaissance, the main line of Greek rational science was joined up again in Copernicus. Close behind came the Scientific Revolution of the early modern period. Note that this was the sixteenth and seventeenth centuries: capitalism played a role in the development of science, but its influence was nowhere near what it would be during the Industrial Revolution. The main driving force in this period was still the internal logic of rational science, and the religious spirit also gave it a considerable push. (The role of religion in the birth of modern science is also very important, but discussing it would take a great deal of space. The "strict constraints of religion" mentioned in the question are not quite fair.)

The Development of Practical Science in the Hydraulic Civilizations of the East:

Practical science has no internal logic of development. It develops according to the needs of its external environment, and once that environment no longer demands anything, it loses its momentum. This shows very clearly in Chinese history. China is a very typical hydraulic civilization, characterized by centralized politics and officially controlled science. Science driven by external demand of this kind is tied very closely to those demands, and over the long run this shows up as a discontinuous development of science and technology that never forms a system. Moreover, in a highly centralized state like China, once a corresponding culture of statecraft has formed, that culture strongly restricts scholarship that is transcendent and useless. (Whether this is religion or culture is a question in itself.) Over two thousand years of history, China's science and technology (in the sense of technological systems and natural history) was in fact always very strong, reaching its peak in the Ming dynasty. By that point external needs had largely been met. Unless there was a change in social structure (like capitalism and the rise of cities), or science itself developed a structure like Greek rational science, it was hard for science and technology to move further forward. Both were difficult: in the first case, feudal forces kept stifling the sprouts of capitalism; in the second, China's political culture had already set and become a strong constraint. This happened to be exactly when the West picked up the thread of Greek rationality and began its revival, and from then on China began to fall behind in technology as well.

IV: Conclusion

In short, from a Whig-history perspective, the logic of Western scientific development is fairly clear at the macro level. The main line is the Greek thread of rationality; the side branch is the transplanting and re-transplanting of Greek learning by the Islamic world during the Middle Ages. The Islamic world, however, always remained a typical Eastern civilization. Even though it transplanted Greek learning and had an extremely important influence on the birth of modern science, it never managed to become the birthplace of modern science.

Aristotle held that free scholarship requires three conditions:

• A leisured class
• A critical tradition
• Wonder at the world

Looking at the history of science, both ancient Greece and early modern Europe fit these conditions fairly well, while centralized states find them hard to meet. Note that "democracy" is not emphasized here. The point is only that scholarship needs a relatively free environment to develop; once a centralized state has shed its ignorance, it can also create similar conditions.

Huilongguan, Beijing, January 7, 2016

In terms of features, Django's ORM is only a subset of SQL. That means there are many things SQL can do that the Django ORM can't, not to mention that SQL is even Turing-complete. For example, it wasn't until version 1.8 that Django gradually added control-flow constructs like CASE, WHEN and IF. Yet these are very common in certain kinds of table operations, such as reporting.

Fortunately, Django provides an interface for raw SQL, so you can use raw SQL to implement more complex features.

SQL Control Flow: A CASE WHEN Example

Say we have an archive table, archives:

Fields:

• number archive number
• type archive type
• status archive status
• company company
• branch_company branch company

The requirement is to compute, for each type and each branch company, the total number of archives, the number of archives with status=01, and their share of the total.

Of course you could do this in a programming language, but it would be fairly involved. Here you can use a CASE WHEN statement to precisely control which values of the same field get selected.

SELECT 
      COUNT(CASE WHEN status='01' THEN status END) AS status_01,
      COUNT(*) AS total,
      CONCAT(FORMAT(COUNT(CASE WHEN status='01' THEN status END)/
      COUNT(*)*100, 2), '%') AS percentage
FROM
      archives
GROUP BY
      status, branch_company

That solves the problem above. Since this table was made up for the example, I won't show the results here.

Chained Queries on Top of That

One very handy feature of Django's ORM is chained queries: you can keep chaining filter and similar methods to narrow down to exactly what you want.

This is especially useful in certain scenarios. With the table above, for example, sometimes you want data for a particular branch or sub-branch company, and sometimes data for a single type. Writing a separate SQL query for every combination of conditions gets very complicated, and sometimes there are a huge number of combinations. Chained queries solve this problem quite neatly.

To give raw SQL a simple form of chained querying, we need to keep appending condition clauses to the WHERE. A small class does the job:

class GenQuerySQL(object):
    def __init__(self, table):
        self.table = table
        self.group_by_fields = " "
        self.where_conditions = " 1=1 "
        self.fields = " "
        self.order_by_fields = ""
    def where(self, where_condition):
        if where_condition:
            self.where_conditions += " and " + where_condition
        return self
    def add_field(self, fields):
        self.fields += " " + fields
        return self
    def group_by(self, group_by_field):
        self.group_by_fields = group_by_field
        return self
    def order_by(self, order_by_field):
        self.order_by_fields = order_by_field
        return self
    def sql(self):
        SQL = 'SELECT ' + self.fields + ' FROM ' + self.table + ' WHERE ' + self.where_conditions + ' GROUP BY ' + self.group_by_fields + ' ORDER BY ' + self.order_by_fields + ";"
        return SQL

This class gives a simple imitation of chained queries. For example:

archive_statistics = GenQuerySQL('SOME_TABLE')
raw_sql = archive_statistics.add_field('fields').where('where_condition').group_by('group_by_fields').order_by('order_by_fields').sql()

Here .where can be chained multiple times. You could also do it another way: first build the WHERE clause from your conditions, then assemble the final SQL statement. The idea is the same: filter the conditions first, then query the database.

Huilongguan, Beijing, January 2016

Reference: gitlab documentation

First, pull the Docker image with the dao accelerator:

dao pull gitlab/gitlab-ce

Creating volumes

Simply put, volumes are storage outside the container that can be mapped into it. You can keep fairly generic configuration files, data or code as volumes on the host machine where the container runs. This not only keeps the data permanently and safely, it also makes it easy to change what's in the volumes and map them back into the container, which is very useful for containers whose files need frequent changes.
For this GitLab deployment, I created three volumes: the config, logs and data directories under /mnt/volumes/gitlab.
Start a GitLab container:

sudo docker run --detach \
    --hostname git.xiaohuruwei.com \
    --publish 8443:443 --publish 8080:80 --publish 2222:22 \
    --name gitlab \
    --restart always \
    --volume /mnt/volumes/gitlab/config:/etc/gitlab \
    --volume /mnt/volumes/gitlab/logs:/var/log/gitlab \
    --volume /mnt/volumes/gitlab/data:/var/opt/gitlab \
    gitlab/gitlab-ce:latest

What the options mean:

–hostname
The domain bound inside the container, used when repositories are created. Here it is bound to git.xiaohuruwei.com.

–publish
Port mapping: the part before the colon is the host port, the part after is the port the container exposes.

–volume
Volume mapping: the part before the colon is a path on the host, the part after is the path inside the container.

Configuring Nginx for HTTPS

Reference: gitlab set nginx

Nginx configuration file

server {
    listen 80;
    server_name git.xiaohuruwei.com;
    access_log /var/log/nginx/gitlab.xiaohuruwei.access.log;
    error_log /var/log/nginx/gitlab.xiaohuruwei.error.log;
    rewrite ^ https://git.xiaohuruwei.com;
}

HTTPS proxy

server {
    listen       443 ssl;
    server_name  git.xiaohuruwei.com;
    access_log /var/log/nginx/https-gitlab.xiaohuruwei.access.log;
    error_log /var/log/nginx/https-gitlab.xiaohuruwei.error.log;
    # SSL certificate config. This uses a self-generated certificate, so browsers will warn about it; just choose to trust it.
    # For a publicly trusted certificate, you need to get one from a certificate authority
    ssl on;
    ssl_certificate /etc/nginx/ssl/getbase.crt;
    ssl_certificate_key /etc/nginx/ssl/getbase_nopass.key;
    location / {
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_pass https://localhost:8443;
    }
}

Enabling HTTPS in GitLab

Having Nginx reverse-proxy HTTPS isn't enough on its own; you also need to turn on HTTPS support in GitLab.

Edit the configuration: add the following to gitlab.rb in /mnt/volumes/gitlab/config/:

# note the 'https' below
external_url "https://gitlab.example.com"

Create an ssl directory and put the SSL certificate files in it, named to match the domain above:

git.xiaohuruwei.com.crt
git.xiaohuruwei.com.key

Restart the container:

docker restart gitlab

Testing Access to GitLab

Open the web interface. The default login is root and the password is 5iveL!fe (since changed to the default password of Limi Jiaoyin, the company I was working for). Create a project repository: test.
Because the SSL certificate is self-generated and not trusted across the web, you have to choose to trust it for now. Set an environment variable locally:

export GIT_SSL_NO_VERIFY=1

Then clone the repository: git clone https://git.xiaohuruwei.com/root/test.git

Enabling Email

The default mail service doesn't work very well, so I configured my own. See the official GitLab SMTP documentation.

Using a 163.com mailbox and following the official docs, you'll find that emails don't get sent.

gitlab_rails['smtp_enable'] = true
gitlab_rails['smtp_address'] = "smtp.163.com"
gitlab_rails['smtp_port'] = 25
gitlab_rails['smtp_user_name'] = "xiaohuruwei@163.com"
gitlab_rails['smtp_password'] = "xxxx"
gitlab_rails['smtp_domain'] = "163.com"
gitlab_rails['smtp_authentication'] = "login"
gitlab_rails['smtp_enable_starttls_auto'] = true
gitlab_rails['smtp_tls'] = false
gitlab_rails['smtp_openssl_verify_mode'] = 'none'

The logs were messy and I didn't spot the problem in them. Later I found that someone in the Ruby community had solved the same problem:
GitLab 配置通过 smtp.163.com 发送邮件 (Sending GitLab email through smtp.163.com)
along with NetEase's official explanation:
NetEase's SMTP servers require the authentication account and the sender account to be the same; if they differ when a user sends an email, the email is rejected.

So I added two more lines, and after testing it worked:

gitlab_rails['gitlab_email_from'] = "xiaohuruwei@163.com"
user['git_user_email'] = "xiaohuruwei@163.com"

Access over SSH

Because GitLab is deployed with Docker, access over SSH (for example, git clone git@git.xiaohuruwei.com) goes through two layers of authentication:

One is authentication by the freelancer server.
The other is GitLab's own authentication.
The latter needs an SSH key.
The former might need a reverse proxy for SSH itself (the Nginx we use doesn't reverse-proxy anything other than HTTP and HTTPS).

For now, port forwarding turned out to be rather difficult, but you can change GitLab's default SSH port to a non-standard one by editing the variable directly in the GitLab configuration file:

gitlab_shell_ssh_port = 2222

Then restart the Docker container, and you'll see the SSH address change in the web interface:
ssh://git@git.xiaohuruwei.com:2222/root/test.git, after which you can carry on with git clone as usual.

Huilongguan, Beijing, January 2016

From Wikipedia: Cargo cult programming

For readers who can't reach Wikipedia because the GFW (China's Great Firewall) blocks it:

Cargo cult programming is an anti-pattern in computer programming, characterized by the ritual use of code or program structures without understanding them. It is typically a sign that the programmer understands neither the bug they are trying to fix nor the apparent solution.

The term sometimes also refers to unskilled or inexperienced programmers copying code from one place to another without really knowing how it works, or whether it is needed in its new place. It can also mean applying design patterns, coding styles or programming methods incorrectly or excessively, without understanding the principles behind them.

Cargo cult programming takes its name from the term "cargo cult". A related coinage is "cargo cult software engineering".

When reading any tutorial, don't practice cargo cult programming. Some things need to be changed to fit your own situation, and you need to understand them. If you don't understand something, get familiar with the relevant background first.

About DaoCloud

DaoCloud offers a Docker-based continuous integration service. With it you can easily set up automated builds and continuous delivery for a project.

Of course, to use DaoCloud for continuous integration you first need to register a DaoCloud account. If you work for a company, you also need to ask the company to add your account to its user group, so that you can use the company's resources.

Continuous integration for a simple HTML5 app

Scenario for this section

• You have your own host
• You have added the host to DaoCloud
• You only use DaoCloud's automated build and deployment features
• The image is also deployed to your own host.

What is a "simple HTML5 app"

Front-end pages only, served by nginx.

The general approach to deployment

This post assumes one scenario: you have your own host, it is bound to DaoCloud, and you want to deploy continuously to that host through DaoCloud. To deploy a pure front-end project, you bundle nginx in and let nginx serve it.

Preparing before building the app

App directory structure

Because DaoCloud is based on Docker, before using DaoCloud we need to follow Docker's conventions and turn our project into an app that can be packaged into an image. For an HTML5 project, that means adding a Dockerfile and setting up the nginx config file it needs. For example, I have an app called jinli, with this directory structure (yours may well differ):

jinli/  # project root
├── jinli/  # the actual app directory
│   ├── images/ # images
│   ├── index.html  # default index html
│   ├── scripts/  #  js folder
│   └── styles/    # folder for css
├── Dockerfile   # Dockerfile
├── jinli.conf   # nginx config for this app
└── log/  # log files

In this structure, the file related to building the Docker image is the Dockerfile. jinli.conf is a file that will be packaged into the Docker image along with it. Everything else belongs to the app itself.

Creating the Dockerfile

A Dockerfile is Docker's rules file, much like a Makefile for make. It describes the process of building your app into a Docker image.
In this example, the Dockerfile looks like this:

# use the nginx service
FROM nginx:1.9.5
# copy the code
COPY . /src
# add the nginx config file
COPY jinli.conf /etc/nginx/conf.d/
# remove the default nginx config file
RUN rm /etc/nginx/conf.d/default.conf

Explaining the Dockerfile:

• First, picture the Docker image you're about to build as containing a Linux operating system.
• FROM nginx:1.9.5 means: bring nginx version 1.9.5 into your image. (This is actually the step where the operating system comes in. nginx is itself an image built on top of a Linux operating system, so when you pull nginx, you pull it together with the whole Linux system it sits on.)
• COPY . /src means: copy the code in the current directory to the /src directory in the image (in the image's operating system).
• COPY jinli.conf /etc/nginx/conf.d/ means: copy the nginx config file you wrote in the current directory to the corresponding directory in the image (in its operating system).
• Delete the original default nginx config (because the default config file takes up localhost; I don't know whether there is a better way around this).

Configuring the nginx config file used inside Docker

In this example, the nginx config file is named jinli.conf. It is packaged into the image to serve the HTML5 app. This is not very different from deploying directly on a server as usual. The things to watch are:

• server_name is localhost, because this server_name is used inside the Docker container.
• The root folder in location is a path inside the Docker container, not on the host machine.

server {
    listen 80;
    server_name localhost;
    access_log /src/log/jinli.access.log;
    error_log /src/log/jinli.error.log;
    # you can also add options such as gzip here as needed
    location / {
    root /src/jinli/;
    }
}

Then push the app to GitHub, for example to: xiaohuruwei/jinli.git

That's all the material you need. Now on to DaoCloud.

Using DaoCloud to build images and release continuously

The previous steps produced a Docker-ready version of the code, and it doesn't need to connect to external services like MySQL, so it can be built directly as a single-app image.

DaoCloud already has thorough documentation on this; you can follow its chapters on creating a new project and on continuous integration.

Section 4, Creating a new project: choose the git source to bind according to your situation
Section 5, Continuous integration and image building: for a simple HTML deployment, you don't need to know how to write daocloud.yml
Section 9: Releasing an app to your own host cluster

The core parts of the process above are:

• Bind a git source
• Choose the git repository to deploy and build it into an image
• Deploy to your own host
• Turn on the settings for subsequent automatic continuous deployment
• In the "Settings" of the specific project under "Code Build", turn on continuous integration and set up the image and the CI trigger rules, for example: trigger automatic CI whenever code is pushed to the docker-support branch.
• In the "Release" settings under "App List", turn on automatic release.

What to do after release

After release, you can see the details of the created container on DaoCloud, including the port mapped to the host. Click the port number to see the app's access address, and open it to test whether the deployment worked.

For convenience in what follows, assume the mapped port is 8888. Since we released to our own host, we also need to set up a domain and an nginx reverse proxy ourselves to serve external traffic. Assume a domain is already bound to the host; here, say, xiaohuruwei.com.

Log in to your host and configure nginx. If nginx was installed with a package manager like apt-get, add jinli.conf under /etc/nginx/conf.d/:

server {
    listen 80;
    server_name xiaohuruwei.com;
    access_log /var/log/nginx/jinli.access.log;
    error_log /var/log/nginx/jinli.error.log;
    client_max_body_size 200m;
    # you can set other options as needed
    location / {
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_pass http://127.0.0.1:8888;
    }
}

Then visit xiaohuruwei.com and you'll see the content.

Also, when using DaoCloud, turn on both automatic build and continuous release. That way, next time you only need to push code to GitHub; once your update rule is triggered, the image is rebuilt and released automatically.

Continuous integration for an app that depends on external services

Using a Django app as the example

Scenario this section applies to

• You have your own host
• You have added the host to DaoCloud
• You only use DaoCloud's automated build and deployment features
  The external services it depends on already exist on your host (as Docker containers), and the image is also deployed to your own host. At deploy time, it connects to the external services already on the host (such as MySQL, Redis and so on).

Apps that depend on external services

In the Docker world, if a Docker image needs other images to provide services to it, we say the image has external services.

Take a simple Django app. If it can't do its job on its own and needs MySQL or another database for data persistence, then this Django app depends on external services.

Preparing before building

Preparation is close to the HTML5 case. Because it needs to connect to external services, and because uwsgi doesn't handle static files by default, there are a few more changes to the original code.

Project structure

First, the Django app's directory. Say there's an app called taikang, with this structure:

taikang    # project root    
├── Dockerfile    # Dockerfile
├── logs    
├── manage.py
├── Makefile    # makefile
├── requirements.txt    # python libraries the app depends on
├── templates
└── taikang        # app directory
    ├── __init__.py
    ├── models.py
    ├── settings.py   # django settings file
    ├── urls.py
    ├── views.py
    └── wsgi.py

Writing the Dockerfile
The Dockerfile means exactly the same thing as in the HTML5 deployment. Here it is:

# use python 2.7.10 as the base service
from python:2.7.10
copy everything in the current project into /src in the image
COPY . /src
# install the libraries the django app depends on, using the douban pypi mirror
RUN cd /src; pip install -r  requirements.txt -i http://pypi.douban.com/simple --trusted-host pypi.douban.com
# expose port 8000 from the image
EXPOSE 8000
# set the working directory to where the code is
WORKDIR  /src
# set the app's start command
CMD ["make", "start-uwsgi"]

Writing the Makefile

You may notice that the start command at the end uses make. That's what the Makefile in the directory structure above is for. I won't go into makefiles in detail here.
In short, it makes the app easier to manage. Here is the makefile:

# host and port uwsgi starts on
host:=0.0.0.0
port:=8000
# when debugging, start with django's built-in server
debug:
    ./manage.py runserver $(host):$(port)
# start the uwsgi service
start-uwsgi:
    uwsgi --socket $(host):$(port) \
        --chdir $(shell pwd) \
        --wsgi-file taikang/wsgi.py \
        --master \
        --process 4 \
        --pidfile $(shell pwd)/uwsgi.pid
# stop the uwsgi service
stop-uwsgi:
    uwsgi --stop uwsgi.pid
# reload the uwsgi service
reload-uwsgi:
    uwsgi --reload uwsgi.pid
# collect static files
collectstatic:
    ./manage.py collectstatic --noinput
.PHONY: debug \
    collectstatic \
    reload-uwsgi \
    start-uwsgi \
    stop-uwsgi

Static file support

When uwsgi serves the app, static files need to be handled separately. The currently recommended option is the whitenoise library for Django. It makes serving static files easy and needs only a few lines of configuration.

Connecting to services through environment variables

When deploying with Docker and you need to connect to external services, you usually do it through links plus environment variables. To support environment variables, our Django app needs some changes too: specifically, changing the configuration in settings.py.

Modify settings.py, using MySQL as the example:

# the original mysql config
#DATABASES = {
#    'default': {
#        'ENGINE': 'django.db.backends.mysql',
#        'NAME': 'starding_taikang',
#        'USER': 'root',
#        'PASSWORD': 'your_password_here',
#        'HOST': 'localhost',
#        'PORT': '3306'
#    }
#}
# get the service from environment variables
import os  # import the os library
mysql_name = os.environ.get('MYSQL_INSTANCE_NAME') or "taikang"  # get the database name
mysql_user = os.environ.get('MYSQL_USERNAME') or "root"  # get the mysql username
mysql_password = os.environ.get('MYSQL_PASSWORD') or "your_password"  # get the password
mysql_host = os.environ.get('MYSQL_ADDR') or "127.0.0.1"  # get the service address
mysql_port = "3306"
# use the variables above instead of strings
DATABASES = {
   'default': {
       'ENGINE': 'django.db.backends.mysql',
       'NAME': mysql_name,
       'USER': mysql_user,
       'PASSWORD': mysql_password,
       'HOST': mysql_host,
       'PORT': mysql_port
   }
}

Deploying the built taikang project with DaoCloud

The steps are basically the same as the HTML5 deployment. Suppose that after the image is built, its address is:

daocloud.io/xiaohuruwei/taking:latest

Next, when deploying from the image, you need to connect the external services, which means editing the yaml file by hand.

takang_test:
  image: daocloud.io/xiaohuruwei/taking:latest
  restart: always
  external_links:
  - mysql1:mysql      # mysql1 is the existing mysql container instance on your host; after the colon is the alias
  ports:
  - '8000'
  environment:    # environment variables    
  - MYSQL_ADDR=mysql

But this approach seems to have a bug. If it doesn't work, deploy directly first, and once that succeeds, edit the yaml file into the form above. When you edit the yaml file, DaoCloud will prompt you to redeploy; just confirm.

At this point the Django project is deployed too. Then, just like the HTML5 project, turn on automatic build and automatic release.

Note: because the Django project is deployed with uwsgi, visiting the address DaoCloud gives you directly won't work. You must configure an nginx reverse proxy.

Here is the nginx proxy config for a Django app deployed with uwsgi:

server {
    listen 80;
    server_name xiaohuruwei.com;
    access_log /var/log/nginx/taikang.access.log;
    error_log /var/log/nginx/taikang.error.log;
    client_max_body_size 200m;
    gzip on;
    gzip_min_length 1k;
    gzip_buffers 16 64k;
    gzip_http_version 1.1;
    gzip_comp_level 6;
    gzip_types text/plain application/x-javascript text/javascript text/css application/xml;
    gzip_vary on;
    location / {
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        include uwsgi_params;
        uwsgi_pass 127.0.0.1:32801;
    }
}

Huilongguan, Beijing, January 2016

Let's start with Wikipedia's definition:

Docker is an open-source project that automates the deployment of applications inside software containers, by providing an additional layer of abstraction and automation of operating-system-level virtualization on Linux. Docker uses the resource isolation features of the Linux kernel such as cgroups and kernel namespaces, and a union-capable filesystem such as aufs and others to allow independent “containers” to run within a single Linux instance, avoiding the overhead of starting and maintaining virtual machines.

From this fairly academic definition you can vaguely gather that Docker is a tool that automates things like deployment, but you can't really see what makes it so damn great, or where exactly its strengths lie. To figure that out, reading the definition is nowhere near enough. One way is to use Docker for a long time and gradually come to appreciate its advantages. But for a newcomer who has never touched Docker, and who may not even know much about deployment, how do you get there quickly? A good approach is to lean on a metaphor.

The Docker metaphor

The concept of "metaphor" gets its own post: "Metaphor"

Likewise, if you want to understand Docker quickly and grasp its core idea, a metaphor matters a lot.

The literal meaning: Docker, the dock worker

The name Docker is itself a metaphor. In the global logistics system, one of the most important inventions is the shipping container.

Why is the container so important? To see why, look at how logistics worked before containers existed: goods are produced and packed into crates, then loaded crate by crate onto a truck, then unloaded crate by crate and put on a train, carried to the station near the port, unloaded crate by crate again, loaded onto trucks, hauled to the freighter, loaded on crate by crate...

You can see that a huge amount of time, labor, and material goes to waste on all the loading and unloading in between. In logistics, the time goods actually spend on the road is fixed, and until transportation technology improves it's hard to shorten. So the time spent loading and unloading became the bottleneck of logistics, and that situation improved enormously once the container appeared.

What makes the container important is that it provides a universal, standard specification for packaging goods (put plainly, its size and shape conform to a standard). This produces a huge advantage: in transport, the goods only need to be packed once before shipping. The container can then go onto a train or a truck, be hauled to the port, be placed directly on the ship, and after unloading go straight onto a train or truck again to its destination. On top of that, the container's standard makes it very easy to handle mechanically, which kicked off a standardization of the entire global logistics system around the container. That saved enormous amounts of time and labor and accelerated the flow and reallocation of resources around the world.

Docker (the dock worker) borrows exactly the container metaphor

Docker is like the dock worker loading goods into a container: it packages your application into a container of a standard specification. In computing terms, this container packed to a given specification is called an "image." Really it's your original code plus some extra content, format, and so on, turned into a static application. And just as a disc image can only be parsed by software that understands that disc image format, this image can only be parsed by Docker.

The container reduces the work of transporting goods. What similar advantage does a Docker image offer? First, let's look at what deploying an application was like before Docker.

Before Docker, say I wanted to deploy a Django application. What would I have to do?
First I need a Python environment. Say I use Python 3 and your machine has Python 2. OK, install Python 3 first. Turns out that's a pain: you have to install all kinds of dependencies first and sort out possible conflicts. No way around it, so you grit your teeth and do it.

Once Python is installed, thanks to magical tools like pip, Django and the Python libraries you need go in quickly. Huh, turns out you also need MySQL, and Redis too. No way around it, keep going: download, install, configure. After a herculean effort it's finally done. And just like that, the day is gone.

What? You're telling me the old server is being retired and we're switching to a new one? Dammit, fine, let's do it all over again. With yesterday's experience, it only takes most of a day.

What? You're saying our app is so good we're going to promote it, and we need to walk other people through deploying it? I choose death. Delete the code and quit.

That description may be a bit exaggerated, but it's by no means rare. Before Docker, this is exactly what ops people spent much of their time doing, wasting huge amounts of manpower and resources on endlessly repeated work.

After Docker

A standard deliverable

Saying Docker is like a shipping container, the key point is "standardization." What is its concrete product? It's the "image." Honestly, the Chinese word for it, jingxiang (镜像, "mirror image"), sounds pretty mystical, and the original English word "image" is pretty mystical too. It's mystical because its original meaning has been extended for special contexts. It originally means "picture, likeness," and a picture also carries the sense of "an abstract depiction of a real object." A picture can also be copied easily, which later gave it the sense of "a copy of the original picture," and later still it was used directly for disc images (something that stores an image and is easy to copy, except that a picture is drawn on paper while this kind of image exists digitally on a disc). Of course, jingxiang already means "copy" in Chinese, but it picks up the particular mood of Chinese characters and comes across as mystical and hazy.

"Image" was later extended further. In Docker, it means taking your application itself and packaging it in a certain format (really just a series of command lines executed according to rules) into something with a standard specification (much like a container packaging your goods). It's just like how a disc image packages the original content into a format that can be burned directly onto a disc. This format can only be parsed by Docker itself, just as your disc image can only be parsed by specific tools.

In Docker, an image can't be run directly. My guess is that this isn't for technical reasons but for engineering-design reasons. Generally, a specific version of a piece of software gets packaged into exactly one image. If an image could be configured and run, it could easily get damaged in use.

So how do you avoid that? Add another layer, which amounts to a cloning technique: as long as the original body is fine, the clones can crash and burn all they like without anything really dying. This extra clone layer is called a container. The name is pretty apt: it's like a bottle or some other vessel, your application runs inside it, and you get an extra layer of safety. If you want to use a service or get your application running, you just create a new container from the image (also a single command), while the image itself stays put, untouched. Can't help it, it's precious.

Roughly speaking, there are two kinds of images:

One kind, let's simply call base standalone images. Each is usually a standalone service, and the most standalone service images of all are the various stripped-down operating systems. They're packaged into images as the most basic service, and practically every image depends on one. But for ordinary application developers like us, we generally don't use these images directly.
If we do use one, our Dockerfile usually looks something like this:

# Base operating system
From ubuntu:latest
# Build the other services we need, plus our own code
RUN apt-get install mysql
COPY .  /src/

The other kind, let's call composite images. They're built on top of a standalone image, combining other services on top of it, such as a Python service or something like MySQL. As fairly top-level application developers, we generally use these pre-combined service images directly.
The Dockerfile then looks something like this:

# Base python service
From python:2.7.10 
# Our own code
RUN apt-get install mysql
COPY .  /src/

In fact, a Python service like the one above is itself built on top of a base operating system. If you look at the Python Dockerfile on Docker Hub, this becomes clear:

FROM buildpack-deps:jessie    # jessie is a stripped-down debian operating system
# remove several traces of debian python
RUN apt-get purge -y python.*

The images we build ourselves can also be called a kind of composite image. We usually use a Dockerfile to combine our own application code with some of the specific service images above (such as python), and that's how we build our own application.

What exactly does Docker simplify?

Docker takes exactly those repetitive parts of the deployment process and does them automatically. You only need to build a working Docker image during the first deployment. After that, a few short commands are enough to pull the image, create a container from it, and get the service running. All you need is a server with Docker installed, a Dockerfile, and a reasonably smooth network connection. Truly "build once, deploy anywhere."

Need a Python 3 environment? Just from python:3.x and you're done.
Need to migrate servers? Copy the application along with its Dockerfile and volumes data to the new server, a few commands, done again.
Need to offer it as a service to others? The Dockerfile is the clearest deployment document there is. Maintain an official image, and whoever needs it just pulls it down and deploys it with a few commands.
By now you may have noticed that the Docker image has become a standard shipping unit, just like a container. Unlike traditional software delivery, which just hands you the code and some documentation and calls it a day, it hands you a standard Docker shipment directly. That might be a Dockerfile, or the image itself, and this standard unit includes not only the code but the whole environment the code runs in, including the OS.

So whoever wants to use my service just pulls the image and instantiates a container, and it directly provides the service you need. No more tedious installation process like before: someone has already done all of that for you.

And the advantages don't stop there

Just as the "standardization" the container brought to logistics didn't only affect the container itself, but standardized and improved the efficiency of the whole field, from transport machinery to logistics management.

In other words, once you have "standardization" of one core thing, you can do much more on top of it, like automated mechanical handling of containers. Docker is similar: once this kind of standard software unit is established, the standard unit and the corresponding ways of managing it can bring about many more changes. For example:

Unified service management

Applications deployed with Docker all fall within Docker's management. This is another huge advantage of Docker: it provides an isolated space and brings the scattered applications deployed on a server together under centralized management.

For example, say I have a ton of services deployed on one server: MySQL, Redis, RabbitMQ, plus a pile of other applications. One day the server suddenly loses power and reboots. Brace yourself for a tragedy. The apps that weren't set to restart automatically, the apps that broke on restart, the important apps hiding in some corner you didn't even know about that failed to start...

With Docker, though, you can see at a glance which applications started normally and which ones have problems. Then you just deal with them methodically, and you're fine.

Use in continuous delivery

Continuous delivery is somewhat beyond the scope here. Go find the answers yourself.

A misconception about Docker

Someone might ask: I only have one server, and I'm unlikely to migrate. My Python service and MySQL service only need to be deployed once and can be reused from then on. Does Docker still have any advantage for me? After all, Docker has a learning cost too.

If you're really sure your applications are one-off and only for your own use, then Docker's advantage in this scenario isn't especially obvious: even with Docker, someone has to do the initial build, and that's work. It's about the same amount of work as a one-time deployment. Even so, it still has an upside: it gives you a clean, isolated environment.

Also, when many people say Docker changed the world of ops, they're speaking at the level of the whole community. For applications with huge user bases like MySQL and Python, and for applications that are frequently migrated, changed, and redeployed, the time saved across the whole community after Dockerizing them is enormous.

That is, Docker can't reduce deployment work "to zero." In the good case it's "reduced to basically one," because no matter what, you have to build the application and its environment once. Just as the container eliminates the handling along the way but can't avoid the one operation of loading the goods into the container. But that one build can become the foundation of "build once, deliver anywhere." The first time you build your own image is similar to loading a container: it takes about as much work as deploying directly on the machine. At that point, Docker's efficiency gain is very small.

But are you really sure the work you're doing is only a one-off?

Huilongguan, Beijing, January 2016

Closures

Let's start with the definition from Wikipedia:

In programming languages, closures (also lexical closures or function closures) are a technique for implementing lexically scoped name binding in languages with first-class functions. Operationally, a closure is a record storing a function together with an environment: a mapping associating each free variable of the function (variables that are used locally, but defined in an enclosing scope) with the value or storage location to which the name was bound when the closure was created. A closure—unlike a plain function—allows the function to access those captured variables through the closure’s reference to them, even when the function is invoked outside their scope

Then let's examine the definition:

There are a few key points in the passage above:

• A closure is a technique.
• A closure is a record made up of a function and the environment it lives in.
• When a closure, i.e. the record above, is created, it binds the free variables used in the function (variables defined in an enclosing lexical scope but used locally in the function) to their values.
• Unlike a plain function, a closure lets the function use the variables it has captured (bound), even when those variables are defined outside its scope.
  Looking only at the definition, plus whatever intuition the word "closure" gives you, you end up with a fuzzy idea at best: this thing is roughly some enclosed structure that contains a function and variables defined outside the function. To understand it more deeply, you have to look at real code.

An example of a closure in Python:

# function that creates the closure
def counter(start_at=0):
    count = [start_at]
    def incr():
        count[0] += 1
        return count[0]
    return incr
# assign the closure to another variable
counter1 = counter()
# assign it to yet another variable
counter2 = counter(5)

In the code above, counter is a function that creates closures, and the next two lines assign its results to two different variables. This actually produces two closures, counter1 and counter2, each consisting of the function itself plus a scope attached to it from outside the function.

These two closures are a lot like two instantiated objects: the scopes attached to them outside the function are independent and don't affect each other.

Decorators

First, the literal meaning.
The word "decorator" literally means "a tool for decorating some particular thing." Implicit in the word is the idea that the thing being decorated is the core, while the decorating tool is just an embellishment that adds a little something extra.

Take a metaphor: at Christmas we hang little ornaments on the Christmas tree. Each ornament is a small decorator, and the core being decorated is the tree. This also implies that the tree is what matters; the ornaments merely give it some extra fun features.

It's similar in Python. Python doesn't actually have an object type called "decorator" the way it has functions, dicts and lists. Rather, Python lets you build a certain kind of callable (usually a function or a class) that can be used to decorate other objects, and we call any object with this decorating ability a decorator. At bottom, it's just a callable that plays a decorating role.

That raises two questions:

What does it decorate?
And how does it do the decorating?
Again, let's look at code. An example of a Python decorator:

import time
from functools import wraps
# the main function being decorated
def foo():
    # do whatever we actually want to do
    # ...
    # ...
    # and finally print something
    print 'in foo()'
# the decorator    
def timeit(func):
    '''
    Decorator that reports the execution time.
   '''
   @wraps(func)
    def wrapper(*args, **kwargs):
        start  = time.clock()
        result = func(*args, **kwargs)
        end = time.clock()
        print 'used: {0}'.format(end-start)
        return result
    return wrapper

First we need to be clear about what the core of our program is. We want the foo function to carry out some specific task and print something at the end, so our core function is foo.

At the same time, we'd also like to know how long the task takes, which is exactly something a decorator can do. So we build a function, timeit, as a decorator to decorate foo.

One point worth stressing again: the decorator timeit exists to decorate foo. It only adds the ability to report execution time; it doesn't change what foo was meant to do. It has no effect whatsoever on foo's original job.

If you look closely, though, you'll notice that the decorator above uses closures (just not very visibly, since it doesn't carry an extra scope along, though adding one would be fine). More precisely, both closures and decorators rest on the fact that callables in Python can be passed around.

The Decorator Pattern

So where do decorators come from? The idea comes from the decorator pattern in design patterns: without changing the core object being called, add some interesting and possibly useful features to it. That is exactly what "decorating" means.

You don't have to touch the original core object at all, and you still get what you want. Pretty clever.

Huilongguan, Beijing, January 2016

Looking back over these years, I started trying to blog in my first year of college. Back then I didn't know much about technology, so most of my posts were miscellaneous essays, mostly my thoughts on current events.

At first I didn't even know independent blogs existed, and I posted everything on QQ Zone (a social blogging space attached to the QQ messenger). That was because QQ Zone was the only place I knew of where you could publish and keep journal entries online, so I used it as a blog. QQ Zone did have some advantages: since it was built on the QQ social platform, whatever I wrote could always be seen by friends and others, and I'd get some feedback now and then, which was one of the things that kept me writing.

But later, as what I wrote leaned more and more toward my major, and I rarely wrote those essays that oozed edgy teenage thinking anymore, QQ Zone's advantages faded. Its writing experience was also terrible (laggy, rich text only): fine for a short plain-text post, hopeless once a post had a few more elements.

I've blogged in a lot of places

So I went looking for other places to blog: Douban, NetEase and so on. But they were either awful to write in or full of ads that really spoiled the mood. Around then I happened to learn about independent blogs. So I registered a domain, bought a server, set up my own WordPress, hunted for a good theme, and after a lot of tinkering finally had my own independent blog.

After having it for a little over a year, though, I found that I spent less time on the writing itself than on maintaining the blog. One headache after another, until I was too worn out to care anymore.

That's when I realized writing was the real goal all along, and I no longer wanted to spend effort maintaining an independent blog. So began my second search for a suitable writing platform. I tried a wave of them, NetEase's Lofter among others, and only Jianshu was more or less acceptable, but Jianshu wasn't very customizable.

Then it also hit me that I'd used far too many platforms, and what I'd written was scattered all over the place, which was bad for building up and organizing my own knowledge. So I decided to keep looking.

In the end I found GitHub's twin: GitHub Pages.

Bringing it all together at last

GitHub is where programmers hang out most anyway, and GitHub Pages had great reviews. It took me just half a day to set up a semi-independent blog, and one look at the result told me this was exactly what I wanted.

And so began the road to bringing it all together...

Later I happened to see someone online summarize "the three stages of blogging," and it rang very true. Here it is:

Stage one: you've just discovered blogging, it feels fresh, and you try writing on some free hosting service.

Stage two: you find the free service too restrictive, so you buy your own domain and hosting and set up an independent blog.

Stage three: you find managing an independent blog too much hassle, and ideally you'd keep control while someone else handles the management, so you only have to write.

.
.
.
.
.

An addendum from February 2024

Reopening My Independent Blog