Mahmoud

How not to structure your database-backed web applications: a study of performance bugs in the wild Yang et al., ICSE’18

This is a fascinating study of the problems people get into when using ORMs to handle persistence concerns in their web applications. The authors study real-world applications and distil a catalogue of common performance anti-patterns. There are a bunch of familiar things in the list, and a few that surprised me with the amount of difference they can make. By fixing many of the issues that they find, Yang et al., are able to quantify how many lines of code it takes to address the issue, and what performance improvement the fix delivers.

To prove our point, we manually fix 64 performance issues in [the latest versions of the applications under study] and obtain a median speed-up of 2x (and up to 39x max) with fewer than 5 lines of code change in most cases.

The Hyperloop website provides access to a tool you can use to identify and solve some of the common performance issues in your own (Rails) apps.

I’m going to skip the intro parts about what ORMs do and how a typical web app is structured, on the assumption that you probably have a good handle on that already. Note that fundamentally a lot of the issues stem from the fact that the ‘O’ in ORM could just as easily stand for ‘Opaque.’

On one hand, it is difficult for application compilers or developers to optimize the interaction between the application and the underlying DBMS, as they are unaware of how their code would translate to queries by the ORM. On the other hand, ORM framework and the underlying DBMS are unaware of the high-level application semantics and hence cannot generate efficient plans to execute queries.

I remember in the early days of Spring (when Hibernate was young too, and JPA didn’t exist) a framework called iBATIS was quite popular amongst people I considered to be good developers. It turns out iBATIS is still going strong, though now it’s called MyBatis. The key selling point is that you retain control over the SQL, without the overheads of directly using an API like JDBC.

Finding and profiling real-world applications

The study focuses on Ruby on Rails applications, for which many large open source applications: “compared to other popular ORM frameworks such as Django and Hibernate, Rails has 2x more applications on GitHub with 400 or more stars than Django and Hibernate combined.” Six popular application categories (covering 90% of all Rails apps with more than 100 stars on GitHub) are further selected, and then the two most popular applications in each category. Resulting in the following twelve apps:

They have been developed for 5-12 years, are all in active use, and range from 7Kloc to 145Kloc (Gitlab). To generate realistic data for these apps, the team collected real-world statistics based on either the app in question where available, or similar apps otherwise, and implemented a crawler that fills out forms on the application’s websites following the real-world stats. For each application, three sets of data were created: with 200, 2000, and 20,000 records in the main database table.

When we discuss an application’s scalability we compare its performance among the three above settings. When we discuss an application’s performance we focus on the 20,000-record setting, which is a realistic setting for all the applications under study. In fact, based on the statistics we collect, the number of main table records of every application under study is usually larger than 20,000 in public deployments…

(E.g., more than 1 million records).

With databases in hand, the applications are then profiled by visiting links randomly for two hours and the resulting logs produced by Chrome and the Rails Active Support Instrumentation API are processed to obtain average end-to-end loading time for every page, the detailed performance breakdown, and the issued database queries. For each app, the 10 pages with the worst loading time are plotted in the figure below.

11 out of 12 applications have pages whose average end-to-end loading time (i.e. from browser sending the URL request to page finishing loading) exceeds 2 seconds; 6 out of 12 applications have pages that take more than 3 seconds to load…. Note that our workload is smaller, or for some applications, much smaller, than today’s real-world workload.

Looking at where the time goes for these queries, server time (app server + DBMS) contributes at least 40% of this latency in at least 5 out of the top 10 pages for 11 of the 12 apps.

In total, 40 problematic actions are identified from among the top 10 most time-consuming actions of every application. 34 of these have scalability problems, and 28 take more than 1 second of server time. 64 performance issues were found in these 40 actions, as summarised in the table below.

In addition to studying these actions, the authors also look at performance issues reported in the application’s bug trackers.

The causes of inefficiency are categorised into three main areas: ORM API misuse, database design, and application design.

ORM API Misuse

Inefficient computation (IC) occurs when the same operation on persistent data can be implemented via different ORM calls. These often look similar on the surface, but can have very different performance implications. For example the difference between any? (scans all records in the absence of an index) and exists? as shown here. Another example is use of Object.where(c).first instead of Object.find_by(c).

There all also opportunities to move computation to the DBMS, for example, replacing pluck(:total).sum with sum(:total). Sometimes we can also take advantage of data already in memory (e.g. replace Object.count— a db query— with Object.size which will count in-memory objects if they have already been loaded).

Unnecessary Computation (UC) occurs when redundant or useless queries are issued. A classic example is a query inside a loop body, that could be computed once outside the loop body.

Inefficient data accessing (ID) results in slow data transfers by either not getting enough data (resulting in the familiar N+1 selects problem), or getting too much data. This can be due to inefficient lazy loading (still prevalent in this study), or too-eager eager loading. Here’s an example of a patch in Lobsters that replaces 51 database queries with one:

A similar inefficient updating issue occurs when N queries are issued to update N records separately, instead of using update_all.

Unnecessary data retrieval (UD) occurs when an application retrieves persistent data that it doesn’t then use.

Inefficient Rendering (IR). This one was a surprise to me in the measured impact. It’s very common in Rails to do something like this, which calls link_to within a loop to generate an anchor:

Replacing it with one call to link_to outside of the loop, and the use of gsub within the loop is faster.

There’s a readability trade-off here, but with complex rendering functions it is clearly worth it. 5 problematic actions in the study dropped their time by over half as a result.

Database design issues

Missing Fields (MF) occurs when an application repeatedly computes a value that it could just store.

Missing Database Indexes (MI). Need I say more?

… missing index is the most common performance problem reported in ORM application’s bug tracking systems. However, it only appears in three out of the 40 problematic actions in latest versions.

Application design issues

Content Display Trade-offs (DT). Typically returning all records instead of using pagination. This becomes a problem as the number of records in the database grows.

Application Functionality Trade-offs (FT). Sometimes an application has a side information on a page that is actually quite expensive to compute. Removing this when it is not essential to the user’s task in-hand can make things go faster.

Fixing inefficiencies

The authors study all 40 of the problematic actions and manually fix 39 of the (the other one spends its time on file-system actions). This leads to 64 fixes being applied.

Many fixes are very effective. About a quarter of them achieve more than 5x speedup, and more than 60% of them achieve more than 2x speedup. Every type of fix has at least one case where it achieves more than 2x speedup.

40 of the 64 fixes alter neither the display nor the functionality of the original application, achieving an average speed-up of 2.2x (maximum 9.2x).

Per a recent comment from Steve Powell, it’s actually quite confusing to talk about a ‘2x speed-up’ when you’re talking about the time something takes! I guess we could interpret this as either 2x requests-per-second (that’s a speed metric) , or that an action takes on average half the time it used to (a ‘/2’ latency reduction?). With a single thread of execution, they both amount to the same thing.

The average server time is reduced form 3.57 seconds to 0.49 seconds, and the end-to-end page load times from 4.17 seconds to 0.69 seconds. More than 78% of fixes require fewer than 5 lines of code.

In other words, by writing code that contains the anti-patterns discussed earlier, developers degrade the performance of their applications by about 6x.

The authors wrote a static analyser (you can find it here) that looks for some of the simple API misuse patterns and ran it on the latest versions of the 12 ORM applications. There are still plenty of incidences! (Some of these in actions that weren’t identified as worse performing actions during profiling of course).

By profiling the latest versions of 12 representative ORM applications and studying their bug-tracking systems, we find 9 types of ORM performance anti-patterns and many performance problems in the latest versions of these applications. Our findings open up new research opportunities to develop techniques that can help developers solve performance issues in ORM applications.

Now if you’ll excuse me, I’ve got a Ruby codebase I just need to go and look at…

• St. James Park
• Guadalupe River Park - Discovery Meadow
• Guadalupe River Park - Arena Green
• Diridon Station

hugo-cumboss:

EXTREME POWER

antoine-roquentin:

you know this sort of round-table format (which wasn’t invented by bill maher, there were a number of shows based on it like meet the press and mcloughlin group) has so much potential as long as the guests aren’t in complete agreement with each other. and bill maher’s show in the 90s was one of the few cases i’ve seen where that happened from time to time, because something like meet the press has only the insider washington set on, making it so staid that there’s rarely been an episode worth watching over 50 years on air, while there’s episodes of politically incorrect that are still worth watching today. but then maher got 9/11 derangement syndrome like every other new atheist and decided that supporting wars in the middle east was a great thing, in part because of the access it provided them to rich and powerful washingtonians. and from then on, his stupid show has a lineup like it did on friday. dan savage, bari weiss, evan mcmullin. what a great lineup of islamophobes. please, tell me more about the barbaric brown-skinned palestinian hordes coming for the peace-loving israeli people. completely fucking worthless to watch now, despite its incredible potential, because he refuses to have a real debate on his debate show. they need to revive after dark but go harder. is there no network willing to pay a small fee for a show to air at the wee hours of the morning and then weather the potential controversy, in this era of massive programming spending?

he’s a very strong and persuasive alpha male

Blockchain's Two-Flavored Appeal

A recent story in Medium describes yet again quite well why blockchains don't solve any real problems: Blockchain is not only crappy technology but a bad vision for the future.

So what is their irresistible appeal?

Bitcoins remind me of a story from the late chair of the Princeton University astronomy department. In 1950 Immanuel Velikovsky published Worlds in Collision, a controversial best-selling book that claimed that 3500 years ago Venus and Mars swooped near the earth, causing catastrophes that were passed down in religions and mythologies.

The astronomer was talking to an anthropologist at a party, and the book came up.

"The astronomy is nonsense," said the astronomer, "but the anthropology is really interesting."

"Funny," replied the anthropologist, "I was going to say almost the same thing."

Bitcoin and blockchains lash together an unusual distributed database with a libertarian economic model.

People who understand databases realize that blockchains only work as long as there are incentives to keep a sufficient number of non-colluding miners active, preventing collusion is probably impossible, and that scaling blockchains up to handle an interesting transaction rate is very hard, but that no-government money is really interesting.

People who understand economics and particularly economic history understand why central banks manage their currencies, thin markets like the ones for cryptocurrencies are easy to corrupt, and a payment system needs a way to undo bogus payments, but that free permanent database ledger is really interesting.

Not surprisingly, the most enthusiastic bitcoin and blockchain proponents are the ones who understand neither databases nor economics.

Previously, previously, previously, previously, previously, previously.

This post introduces glom, Python's missing operator for nested objects and data.

If you're an easy sell, full API docs and tutorial are already available at glom.readthedocs.io.
Harder sells, this 5-minute post is for you.
Really hard sells met me at PyCon,
where I gave this 5-minute talk.

The Spectre of Structure

In the Python world, there's a saying: "Flat is better than nested."

Maybe times have changed or maybe that adage just applies more to code than data. In spite of the warning, nested data continues to grow, from document stores to RPC systems to structured logs to plain ol' JSON web services.

After all, if "flat" was the be-all-end-all, why would namespaces be one honking great idea? Nobody likes artificial flatness, nobody wants to call a function with 40 arguments.

Nested data is tricky though. Reaching into deeply structured data can get you some ugly errors. Consider this simple line:

value = target.a['b']['c']

That single line can result in at least four different exceptions, each less helpful than the last:

AttributeError: 'TargetType' object has no attribute 'a'
KeyError: 'b'
TypeError: 'NoneType' object has no attribute '__getitem__'
TypeError: list indices must be integers, not str

Clearly, we need our tools to catch up to our nested data.

Enter glom.

Restructuring Data

glom is a new approach to working with data in Python, featuring:

• Path-based access for nested structures
• Declarative data transformation using lightweight, Pythonic specifications
• Readable, meaningful error messages
• Built-in data exploration and debugging features

A tool as simple and powerful as glom attracts many comparisons.

While similarities exist, and are often intentional, glom differs from other offerings in a few ways:

Going Beyond Access

Many nested data tools simply perform deep gets and searches, stopping short after solving the problem posed above. Realizing that access almost always precedes assignment, glom takes the paradigm further, enabling total declarative transformation of the data.

By way of introduction, let's start off with space-age access, the classic "deep-get":

from glom import glom

target = {'galaxy': {'system': {'planet': 'jupiter'}}}
spec = 'galaxy.system.planet'

output = glom(target, spec)
# output = 'jupiter'

Some quick terminology:

• target is our data, be it dict, list, or any other object
• spec is what we want output to be

With output = glom(target, spec) committed to memory, we're ready for some new requirements.

Our astronomers want to focus in on the Solar system, and represent planets as a list. Let's restructure the data to make a list of names:

target = {'system': {'planets': [{'name': 'earth'}, {'name': 'jupiter'}]}}

glom(target, ('system.planets', ['name']))
# ['earth', 'jupiter']

And let's say we want to capture a parallel list of moon counts with the names as well:

target = {'system': {'planets': [{'name': 'earth', 'moons': 1},
                                 {'name': 'jupiter', 'moons': 69}]}}

spec = {'names': ('system.planets', ['name']),
        'moons': ('system.planets', ['moons'])}

glom(target, spec)
# {'names': ['earth', 'jupiter'], 'moons': [1, 69]}

We can react to changing data requirements as fast as the data itself can change, naturally restructuring our results, despite the input's nested nature. Like a list comprehension, but for nested data, our code mirrors our output.

And we're just getting started.

True Python-Native

Most other implementations are limited to a particular data format or pure model, be it jmespath or XPath/XSLT. glom makes no such sacrifices of practicality, harnessing the full power of Python itself.

Going back to our example, let's say we wanted to get an aggregate moon count:

target = {'system': {'planets': [{'name': 'earth', 'moons': 1},
                                 {'name': 'jupiter', 'moons': 69}]}}


glom(target, {'moon_count': ('system.planets', ['moons'], sum)})
# {'moon_count': 70}

With glom, you have full access to Python at any given moment. Pass values to functions, whether built-in, imported, or defined inline with lambda. But glom doesn't stop there.

Now we get to one of my favorite features by far. Leaning into Python's power, we unlock the following syntax:

from glom import T

spec = T['system']['planets'][-1].values()

glom(target, spec)
# ['jupiter', 69]

What just happened?

T stands for target, and it acts as your data's stunt double. T records every key you get, every attribute you access, every index you index, and every method you call. And out comes a spec that's usable like any other.

No more worrying if an attribute is None or a key isn't set. Take that leap with T. T never raises an exception, so worst case you get a meaningful error message when you run glom() on it.

And if you're ok with the data not being there, just set a default:

glom(target, T['system']['comets'][-1], default=None)
# None

Finally, null-coalescing operators for Python!

But so much more. This kind of dynamism is what made me fall in love with Python. No other language could do it quite like this.

That's why glom will always be a Python library first and a CLI second. Oh, didn't I mention there was a CLI?

Library first, then CLI

Tools like jq provide a lot of value on the console, but leave a dubious path forward for further integration. glom's full-featured command-line interface is only a stepping stone to using it more extensively inside application logic.

$ pip install glom
$ curl -s https://api.github.com/repos/mahmoud/glom/events \
  | glom '[{"type": "type", "date": "created_at", "user": "actor.login"}]'

Which gets us:

[
  {
    "date": "2018-05-09T03:39:44Z",
    "type": "WatchEvent",
    "user": "asapzacy"
  },
  {
    "date": "2018-05-08T22:51:46Z",
    "type": "WatchEvent",
    "user": "CameronCairns"
  },
  {
    "date": "2018-05-08T03:27:27Z",
    "type": "PushEvent",
    "user": "mahmoud"
  },
  {
    "date": "2018-05-08T03:27:27Z",
    "type": "PullRequestEvent",
    "user": "mahmoud"
  }
...
]

Piping hot JSON into glom with a cool Python literal spec, with pretty-printed JSON out. A great way to process and filter API calls, and explore some data. Something genuinely enjoyable, because you know you won't be stuck in a pipe dream.

Everything on the command line ports directly into production-grade Python, complete with better error handling and limitless integration possibilities.

Next steps

Never before glom have I put a piece of code into production so quickly.

Within two weeks of the first commit, glom has paid its weight in gold, with glom specs replacing Django Rest Framework code 2x to 5x their size, making the codebase faster and more readable. Meanwhile, glom's core is so tight that we're on pace to have more docs and tests than code very soon.

The glom() function is stable, along with the rest of the API, unless otherwise specified.

A lot of other features are baking or in the works. For now, we'll be focusing on the following growth areas:

• Validation functionality, in the vein of schema and cerberus
• CLI robustness, better error messages, etc.
• Extension API, clean up some internal code, open up extensions
• Automatic default registration of default behaviors for co-installed packages (e.g., Django)

We'll be talking about all of this and more at PyCon, so swing by if you can. In either case, I hope you'll try glom out and let us know how it goes!

Reminder: the term-of-art for AirBnB's business model is "Tax Fraud".

San Francisco City Attorney Dennis Herrera today threw the book at pair of city landlords who, having already settled a suit charging they were operating more than a dozen illegal Airbnb hotels across the city, allegedly spent the last two years fabricating an elaborate scheme to continue cashing in hundreds of thousands of dollars running unlawful short-term rentals. [...]

"Each of the 8 properties inspected had been staged to appear as if a tenant lived there, but it was obvious that it was a ruse. Every apartment had the same staging: the same Costco food items scattered about, the same arrangement of dirty breakfast dishes in every kitchen sink, same personal products in each bathroom, same damp towels artfully draped over doors as though someone had recently showered, the same collection of shoes and clothes in closets, and same houseplants in each apartment."

"Defendants also submitted six faux leases to create the appearance that certain apartments were rented to genuine tenants, then positing that the tenants were conducting unsanctioned illegal short-term rentals. However, the purported tenants were in fact Defendants' friends, family, employees and/or associates merely posing as tenants." [..] Tellingly, all but one of their Airbnb host accounts was created using the same IP address. [...]

"While teachers, families and long-time residents are struggling to stay in their homes, the Lees were taking precious housing and turning it into an illegal hotel chain," summed up Herrera. [...]

At a max of $6,000 per violation and more than 5,000 violations, the ceiling for this litigation exceeds $30 million. But the city instead requested the "more modest civil penalty" of $5.5 million, plus fees for "well over 400 attorney hours, 250 paralegal hours, and 80 investigator hours unraveling Defendants' scheme and bringing this Motion to Enforce."

Previously, previously, previously, previously, previously, previously.

The post Shocked appeared first on The Perry Bible Fellowship.

Off the coast of Yemen, in the Arabian Sea, lies isolated Socotra Island, where hundreds of plants and animals have developed into species unique to the island. Socotra is the largest island in an archipelago that includes three other islands. The Socotra Archipelago has been isolated from any large landmass for millions of years, and is now home to a surprising display of biodiversity. Probably the best-known of its endemic flora is the dragon blood tree, with red-colored sap and tightly-clustered branches that look like roots turned upside down. Below, a collection of images of the landscape of Socotra, and the plants and animals that call it home.

E-waste recycler Eric Lundgren loses appeal on computer restore disks, must serve 15-month prison term:

A California man who built a sizable business out of recycling electronic waste is headed to federal prison for 15 months after a federal appeals court in Miami rejected his claim that the "restore disks" he made to extend the lives of computers had no financial value, instead ruling that he had infringed Microsoft's products to the tune of $700,000.

The appeals court upheld a federal district judge's ruling that the disks made by Eric Lundgren to restore Microsoft operating systems had a value of $25 apiece, even though they could be downloaded free and could be used only on computers with a valid Microsoft license. [...]

But he said the court had set a precedent for Microsoft and other software-makers to pursue criminal cases against those seeking to extend the life span of computers. "I got in the way of their agenda," Lundgren said, "this profit model that's way more profitable than I could ever be."

Lundgren said he wasn't sure when he would be surrendering. He said prosecutors in Miami told him he could have a couple of weeks to put his financial affairs in order, including plans for his company of more than 100 employees. "But I was told if I got loud in the media, they'd come pick me up," Lundgren said. "If you want to take my liberty, I'm going to get loud."

"I am going to prison, and I've accepted it," Lundgren said Monday. "What I'm not okay with is people not understanding why I'm going to prison. Hopefully my story can shine some light on the e-waste epidemic we have in the United States, how wasteful we are. At what point do people stand up and say something? I didn't say something, I just did it."

Previously, previously, previously, previously.

• September 15 (Saturday) - Zlata Chochieva –Gramophone includes her in their Top Ten Chopin CDs; her Rachmaninoff is so masterful that at age 12 she gave a recital shown on Russian TV.
• October 13 (Saturday) - Vladimir Ovhinnikov – The only pianist to win the top prize at both the Tchaikovsky and the Leeds Competitions. He is now a professor at the Moscow Conservatory.
• November 11 (Sunday) - Henry Kramer - Laureate of the 2017 American Pianist Association Awards and lauded as “triumphant” and “thrilling” by The New York Times.  
• December 8 (Saturday) – Sandra Wright Shen - A Bay Area treasure, international star, and Steinway Artist who is among Silicon Valley’s most treasured classical musicians.  
• January 12, 2019 (Saturday) - Kate Liu -  First Prize winner at both at the New York and the Asia-Pacific International Chopin Competitions, she has appeared on PBS’s “From the Top at Carnegie Hall.”
• February 10 (Sunday) – Vyacheslav Gryaznov – He thrilled our audience receiving three encores in 2016, has played Berlin Philharmonie and Carnegie Hall, and composes piano transcriptions that are popular world-wide. 
• March 12 (Tuesday) – Benjamin Grosvenor – One of the world’s most sought-after pianists, he has been featured in two BBC documentaries and is the youngest British musician ever signed by Decca Classics.
• April 6 (Saturday) –  Nikolay Khozyainov - The New York Times praised his “stunning virtuosity and prodigious technique.” His sold-out concerts bear witness to that.
• May 11 (Saturday) – Charlie Albright – Praised by The New York Times for his “jaw-dropping technique and virtuosity meshed with a distinctive musicality” the super-star American pianist has performed with Bobby McFerrin, Joshua Bell, and Yo-Yo Ma.

Economists are waking up to the biases in neoclassical macroeconomics and how that provides an excuse for plutocratist cruelty.

There is no such thing as "the free market" — every market is regulated somehow, and a market without proper regulation is likely to have instabilities (such as crashes).

The EU's Latest Copyright Proposal Is So Bad, It Even Outlaws Creative Commons Licenses.

This is a post that was drafted months ago but never published.   Seems appropriate now more than ever so we’re going to release it into the world.

Something about this post title makes me think of some discount warehouse liquidator.  “We’ve eliminated the middle man and passed on the savings to you!”  But the truth is, that there are times when a middle man is nothing but destructive to a relationship and everybody gains by their removal.

Facebook is the ultimate middle man.

• They tax the creators to reach their followers.  Boost!
  
• They only show content that leads to stronger performance with their advertisers.
  

Instagram is the same now.  I know we all wish it wasn’t, but it is.

There are really only two parties that matter.  The writer and the reader.  A middle man always has dubious plans because they provide limited value.  Particularly on the open web.

RSS had this right years ago.  I realize, technically speaking, RSS platforms like ours are in the middle.  The difference is that we’re replaceable.  It’s not a private network and people come to us from other platforms all the time and vice versa.  It’s healthy.

So why did we all leave to the private network and entrust the middle man?  It’s because services like Facebook always start out as free and easy.  They are just here to help connect you to your friends.  For free.  No ads even… isn’t this great?

Skip ahead about half way to see it with satellite imagery instead of flat coloring. That version is a little dark, so you'll want to full-screen it.

Anyway, Planet Flatland has a very strange sun, is what I'm saying.

This update will be in the next release of XScreenSaver but I figured I'd post the video now anyway, because it's neat.

Oh yeah, also,

"We must do away with the absolutely specious notion that everybody has to earn a living. It is a fact today that one in ten thousand of us can make a technological breakthrough capable of supporting all the rest. The youth of today are absolutely right in recognizing this nonsense of earning a living. We keep inventing jobs because of this false idea that everybody has to be employed at some kind of drudgery because, according to Malthusian-Darwinian theory, he must justify his right to exist. So we have inspectors of inspectors and people making instruments for inspectors to inspect inspectors. The true business of people should be to go back to school and think about whatever it was they were thinking about before somebody came along and told them they had to earn a living." -- R. Buckminster Fuller, 1970

Previously, previously, previously, previously, previously, previously, previously, previously, previously, previously, previously, previously, previously, previously.