On March 25 is this years Document Freedom Day and, depending on your time zone, it has already started. Document Freedom Day is the global campaign for document liberation by local groups throughout the world. So far more than 50 groups registered their events in over 25 countries ranging from Asia, Europa, Africa, to South and North America.
Open Standards are crucial to ensure that different computer systems can work together, and that users can access documents regardless of the computing platform or device they use. They are the foundation of the Internet and the World Wide Web as well as all kinds of technical communication.
Every year, on the last Wednesday of March and the days around, people highlight these important aspects of our digital sovereignity by celebrating Document Freedom Day. In 2015, Document Freedom Day is happening around the world beginning on Tuesday 24, 12 UTC until Thursday, 12UTC. Following the path of the sun, the first events are happening in Japan, Taiwan and India, followed by more than 20 events in Europe, 1 in Africa, and more than 20 in North- and South America. Taken together, all these events make DFD the biggest campaign to promote Open Standards on a local level.
And Document Freedom Day is growing even beyond: In the past years, Document Freedom Day achieved to establish a view on technological boundaries and freedoms under social aspects. More and more, Document Freedom Day is becoming a day to raise political attention also on a national level or global. In 2015, different organisations will use Document Freedom Day to publish and spread political statements. If you are not close to any event or you miss the time to pass by, watch out #DFD2015 in the news, blogs and media, participate and help to get the world interconnected in freedom.
You can still participate: use this day to invite your friends or work mates for a drink or a coffee break and tell them about the international day of Open Standards and why Open Standards are important to you - and for everyone.
Before PostgreSQL got streaming replication, back in version 9.0, people kept asking when we’re going to get replication. That was a common conversation-starter when standing at a conference booth. I don’t hear that anymore, but this dialogue still happens every now and then:
- I have streaming replication set up, with a master and standby. How do I perform failover?
- That’s easy, just kill the old master node, and run “pg_ctl promote” on the standby.
- Cool. And how do I fail back to the old master?
- Umm, well, you have to take a new base backup from the new master, and re-build the node from scratch..
- Huh, what?!?
pg_rewind is a better answer to that. One way to think of it is that it’s like rsync on steroids. Like rsync, it copies files that differ between the source and target. The trick is in how it determines which files have changed. Rsync compares timestamps, file sizes and checksums, but pg_rewind understands the PostgreSQL file formats, and reads the WAL to get that information instead.
I started hacking on pg_rewind about a year ago, while working for VMware. I got it working, but it was a bit of a pain to maintain. Michael Paquier helped to keep it up-to-date, whenever upstream changes in PostgreSQL broke it. A big pain was that it has to scan the WAL, and understand all different WAL record types – miss even one and you might end up with a corrupt database. I made big changes to the way WAL-logging works in 9.5, to make that easier. All WAL record types now contain enough information to know what block it applies to, in a common format. That slashed the amount of code required in pg_rewind, and made it a lot easier to maintain.
I have just committed pg_rewind into the PostgreSQL git repository, and it will be included in the upcoming 9.5 version. I always intended pg_rewind to be included in PostgreSQL itself; I started it as a standalone project to be able to develop it faster, outside the PostgreSQL release cycle, so I’m glad it finally made it into the main distribution now. Please give it a lot of testing!
PS. I gave a presentation on pg_rewind in Nordic PGDay 2015. It was a great conference, and I think people enjoyed the presentation. Have a look at the slides for an overview on how pg_rewind works. Also take a look at the page in the user manual.
Crossposted from Asia Times
The “Wizard of Oz” is the best single-source explanation of American politics. Specialists, to be sure, will want to read the Federalist papers, de Tocqueville and the speeches of Lincoln, but the 1939 MGM movie tells most of the story. We are a nation of scarecrows without a brain, tin men without a heart, and lions without courage. Nothing is going to fix us, but the next best thing is to feel better about ourselves. A broken-down carnival huckster impersonating the Great and Powerful Wizard of Oz gives our three national archetypes not what they need, but the next best thing: A phony diploma, a testimonial, and a medal. The party favors don’t help the feckless trio (the Scarecrow proceeds to recite a comically mistaken formula for the length of the sides of an isosceles triangle) but they did wonders for their self-esteem.
America is a winner’s game. America succeeds because it breeds success and ruthlessly crushes failure. A main purpose of American politics, in turn, is to make losers feel better, without, of course, preventing them from losing. No president in American history more closely resembles the Wizard than Ob the Great and Powerful, the present occupant of the Oval Office. Self-esteem is America’s consolation prize and Obama spreads it with a shovel.
Six years after America inaugurated its first African-American president, the social condition of black Americans remains dismal and appears to be deteriorating. 49% of black males have been arrested by the age of 23; a third of black males will probably spend time behind bars. 54% of black men graduate from high school vs. more than 75% of whites. Only 14% of black eighth graders score at or above the threshold of proficiency. And nearly three-quarters of black births are to unmarried mothers. On the 50th anniversary of the Civil Rights Act of 1965, these are catastrophic results. Magical thinking now infects what was in the past a civil rights movement, for example the “Hands Up, Don’t Shoot” slogan adopted from an incident that Obama’s Justice Department proved never to have happened.
Hypocrisy of this sort plays well in America. It began with the beaten South, which lost nearly 30% of its military age in men in the Civil War, but earned a romantic image of gallantry in America’s popular culture. Claptrap like “Gone With the Wind” was the Confederacy’s consolation prize, with a revoltingly false portrait of a halcyon era of beautiful belles and elegant balls on antebellum plantations. The slaves’ descendants in their days of woe have been accorded the same, slim consolation. Rather than Scarlett and Rhett, African-Americans have rage-spewing rappers like Obama’s regular White House guest Jay-Z. Meanwhile young black men are fed into the maw of the criminal justice system. And Ob the Great and Powerful is there to hand out diplomas, testimonials and medals.
Hypocrisy is a luxury that America can afford. The number of Americans under judicial supervision (prison, probation or parole) rose from 2 million in 1980 to 7.5 million in 2008, and the violent crime rate fell correspondingly, because so many criminals were locked up. But hypocrisy is a far less effective palliative in the Middle East, now that times have gotten tougher. President Obama evinces the same concern for the supposed victims of imperialism around the world as he does for African-Americans–for example the Persians, who are less imperial victims than the rancorous remnant of a failed empire. His concern for the welfare of the Persians is almost child-like; in a sense it is child-like, for it is the concern of the child of the Bohemian anthropologist Stanley Ann Dunham, who married and divorced two Muslim men, and left young Barry with her parents while she set out to save the ironworkers of Indonesia from globalization. Here is what he broadcast to Iranians on Nowruz, the Persian New Year:
Hello! To everyone celebrating Nowruz—across the United States and in countries around the world—Nowruz Mubarak. For thousands of years, this has been a time to gather with family and friends and welcome a new spring and a new year. Last week, my wife Michelle helped mark Nowruz here at the White House. It was a celebration of the vibrant cultures, food, music and friendship of our many diaspora communities who make extraordinary contributions every day here in the United States. We even created our own Haft Seen (symbolic fruit plate), representing our hopes for the new year…As you gather around the Nowruz table—from Tehran to Shiraz to Tabriz, from the coasts of the Caspian Sea to the Persian Gulf—you’re giving thanks for your blessings and looking ahead to the future…As the poet Hafez wrote, “It is early spring. Try to be joyful in your heart. For many a flower will bloom while you will be in clay.”
There is no consolation for the Persians, however; their fertility rate has fallen from nearly 7 children per female in 1979 to just 1.6 in 2012, which means that they will have a higher proportion of elderly dependents than the United States a generation from now. After 3,000 years Persian culture has reached its best-used-by-date. The end of traditional society and the education of women has ruined it. Iran knows that it is going to die, and all the schmoozing in the world from an American president will do it no good.
The same sort of hypocrisy applies to Obama’s concern for Palestinian self-esteem. When Obama told the Huffington Post March 21 that Netanyahu had placed a strain on Israel’s democratic fabric, he did not refer merely to the prime minister’s factual campaign statement to the effect that foreign-financed campaign organizations were bringing large numbers of Arab voters to the polls. He meant (as Dana Milbank made clear in a March 22 op-ed in the Washington Post):
Without a Palestinian state, Israel can be either a Jewish state or a democracy but not both. If it annexes the Palestinian territories and remains democratic, it will be split roughly evenly between Jews and Arabs; if it annexes the territories and suppresses the rights of Arabs, it ceases to be democratic. There are roughly 4.4 million Palestinians in the West Bank, Gaza and East Jerusalem and another 1.4 million living inside Israel . That puts them in rough parity with Jews, who number just over 6 million. Higher Palestinian population growth and fertility rates indicate that Jews will be a minority between the Jordan River and the Mediterranean in a few years.
This is simply false: close examination of Palestinian statistics (e.g., comparison of actual birth and school registration records against the computer model output of the Palestinian authority) shows that the Palestinian population data are inflated by about 1.4 million. With the Jewish birth rate and the Arab birth rate between the Jordan River and the Mediterranean Sea converging on 3 children per female, there will be no significant change in relative population in the foreseeable future.
Nonetheless, what Obama means, quite simply, is that any outcome other than the establishment of a Palestinian state will be “undemocratic.” That is an act of hypocrisy Israel cannot afford. Then Prime Minister Ariel Sharon undertook perhaps the most hypocritical act in Israeli history, withdrawing unilaterally from Gaza in 2005. Sharon expected the result to be catastrophic, but hoped that the ensuing mess would persuade the world that Israel could not afford to give up more territory. Hypocrisy breeds hypocrisy, though, and Sharon vastly underestimated the world’s capacity to ignore the obvious.
There is an old country joke about the shtetl family that invites a poor man to Shabbat dinner. The hostess brings out a dish of whitefish, and the poor man proceeds to gobble it up. The hostess, somewhat chagrined, says gently, “Whitefish is very expensive,” to which the poor man replies, “Believe me, it’s worth it!” From the vantage point of the utopians in Obama’s camarilla—Susan Rice, Samantha Power, Valerie Jarrett and Ben Rhodes—allowing Hamas to turn Judea and Samaria into an artillery platform is worth it. They identify with failed or failing peoples, and cannot live with the knowledge that civilizations die out for the most part because they want to.
Israel’s election was in effect a referendum on the two-state solution, I argued earlier in this publication. Netanyahu had to tell the truth to his own base to persuade them to vote for him, and he told them that under foreseeable circumstances a Palestinian state simply would not happen—not while Mahmoud Abbas survives on the support of the Israeli Army, and Iran is sending weapons to Hamas, and the surrounding countries (especially Syria) have disintegrated into chaos and polarized radicalism. Israel’s national fund of hypocrisy, already depleted by Ariel Sharon’s profligacy in 2005, was exhausted.
Nothing that Mr. Netanyahu can do will placate the Great and Powerful Ob. It may be painful and in some ways damaging, but he has no choice but to ignore the man behind the curtain.
Update: It’s disappointing to see qualified, sheepish support for Netanyahu from the likes of the Wall Street Journal, which today wrote: “But in the closing days Mr. Netanyahu played up that foreigners (read: President Obama) wanted him defeated, and he rejected statehood for Palestinians, reversing a position he had taken in 2009. The reversal gave the impression of opportunism, even desperation, but it also rallied conservative voters who had hinted at growing “Bibi fatigue” after his long tenure as premier.” Opportunism? On the contrary, Netanyahu finally told the truth. Concludes the Journal: “Israel’s raucous democracy is imperfect, like America’s, but it is the only reliable one in the bloody cauldron of the Middle East.”
Ha’aretz explains the discrepancy between the Israeli polls and the outcome as follows: “[Netanyahu] won this election by convincing over 200,000 voters who were planning to vote for Habayit Hayehudi, Shas, Kulanu and Yahad to change their minds in the last six days of the campaign.” Why did they switch? The answer is simple and obvious: Most of those small-party voters oppose a Palestinian state, and Netanyahu ruled out a Palestinian state on his watch a day before the election. Outside of the Eurocentric secular elite in Tel Aviv, most Israeli voters look at the chaos in the Muslim world and draw the obvious conclusion that a Palestinian State would be absorbed into the maelstrom of extremism surrounding it, and turn into an artillery platform for terrorists. Last July I argued in Tablet magazine that it’s not the settlers, but the “unsettlers” (ISIS and Iran) who have forced a one-state solution on Israel. Netanyahu went through the motions of diplomacy until Monday in order to maintain correct relations with Washington. That pushed the one-state vote out to minor parties. Once Netanyahu acknowledged the obvious, 200,000 minor party voters came back to Likud.
Netanyahu has decided, with some justification, that he might as well be hanged for a sheep as for a lamb. His disagreement with the Obama administration is not about the best way to contain Iran’s nuclear ambitions. The Obama administration views Iran as a positive force with benign motives: the Director of National Intelligence’s Threat Assessment speaks of “Iran’s intentions to dampen sectarianism, build responsive partners, and deescalate tensions with Saudi Arabia,” and dismisses Iran’s threat to regional stability as mere “secondary consequences” of its support for Shia communities. Iranian official propaganda quotes it with enthusiasm. And after the appointment of Robert Malley as the Middle East Coordinator for the White House–the man Obama kicked out of its 2008 campaign because of his close ties to Hamas–Netanyahu has little to lose in terms of good will.
We now read that the White House will abandon Israel to the wolves at the United Nations. The political damage to the Democrats will be extensive, but less than one might guess at first glance: the majority of American Jews will continue to choose liberal utopianism over Israeli security. Leftwing Jewish outlets like The Forward echo the cri de coeur of the Israeli left: the “Zionist dream” of socialist equality and religious harmony is dead, and Bibi is its gravedigger. That corpse has been high for some some time, but no matter. The West is choking on its own illusions and hypocrisy; Israel once again may become an Asian nation.
The one chart you should remember from this post is this one, GIN speedup between 9.3 and 9.4:
Interpreting this chart is a bit tricky - x-axis tracks duration on PostgreSQL 9.3 (log scale), while y-axis (linear scale) tracks relative speedup 9.4 vs. 9.3, so 1.0 means 'equal performance', and 0.5 means that 9.4 is 2x faster than 9.3.
The chart pretty much shows exponential speedup for vast majority of queries - the longer the duration on 9.3, the higher the speedup on 9.4. That's pretty awesome, IMNSHO. What exactly caused that will be discussed later (spoiler: it's thanks to GIN fastscan). Also notice that almost no queries are slower on 9.4, and those few examples are not significantly slower.
While both pgbench and TPC-DS are well established benchmarks, there's no such benchmark for testing fulltext performance (as far as I know). Luckily, I've had played with the fulltext features a while ago, implementing archie - an in-database mailing list archive.
It's still quite experimental and I use it for testing GIN/GiST related patches, but it's suitable for this benchmark too.
So I've taken the current archives of PostgreSQL mailing lists, containing about 1 million messages, loaded them into the database and then executed 33k real-world queries collected from postgresql.org. I can't publish those queries because of privacy concerns (there's no info on users, but still ...), but the queries look like this:
SELECT id FROM messages WHERE body_tsvector @@ ('optimizing & bulk & update')::tsquery ORDER BY ts_rank(body_tsvector, ('optimizing & bulk & update')::tsquery) DESC LIMIT 100;
The number of search terms varies quite a bit - the simplest queries have a single letter, the most complex ones often tens of words.
The PostgreSQL configuration was mostly default, with only minor changes:
shared_buffers = 512MB work_mem = 64MB maintenance_work_mem = 128MB checkpoint_segments = 32 effective_cache_size = 4GB
We have to load the data first, of course. In this case that involves a fair amount of additional logic implemented either in Python (parsing the mbox files into messages, loading them into the database), or PL/pgSQL triggers (thread detection, ...). The time needed to load all the 1M messages, producing ~6GB database, looks like this:
Note: The chart only shows releases where the performance changed, so if only data for 8.2 and 9.4 are shown, it means that the releases up until 9.3 behave like 8.2 (more or less).
If you look at PostgreSQL 8.2, the oldest release supporting GIN indexes, that certainly was true - the load took ~1300 seconds with GIN indexes and only ~800 seconds with GiST indexes. But 8.4 significantly impoved this, making the GIN indexes only slightly more expensive than GiST.
Of course, this is incremental load - it might look very differently if the indexes were created after all the data are loaded, for example. But I argue that the incremental performance is more important here, because that's what usually matters in actual applications.
The other argument might be that the overhead of the Python parser and PL/pgSQL triggers is overshadowing the GIN / GiST difference. That may be true, but that overhead should be about the same for both index types, so read-world applications are likely to have similar overhead.
So I believe that GIN maintenance is not significantly more expensive than GiST - at least in this particular benchmark, but probably in other applications too. I have no doubt it's possible to construct examples where GIN maintenance is much more expensive than GiST maintenance.
The one thing that's missing in the previous section is query performance. Let's assume your workload is 90% reads, and GIN is 10x faster than GiST for the queries you do - how much you care if GIN maintenance is 10x more expensive than GiST, in that case? In most cases, you'll choose GIN indexes because that'll probably give you better performance overall. (It's more complicated, of course, but I'll ignore that here.)
So, how did the GIN and GiST performance evolved over time? GiST indexes were introduced first - in PostgreSQL 8.0 as a contrib module (aka extension in new releases), and then in core PostgreSQL 8.3. Using the 33k queries, the time to run all of them on each release is this (i.e. lower values are better):
Interesting. It took only ~3200 seconds on PostgreSQL 8.0 - 8.2, and then it slowed down to ~5200 seconds. That may be seen as a regression, but my understanding is that this is the cost of move into core - the contrib module was probably limited in various ways, and proper integration with the rest of the core required fixing these shortcomings.
What about GIN? This feature was introduced in PostgreSQL 8.2, directly as in-core feature (so not as contrib module first).
Interestingly it was gradually slowing down a bit (by about ~15% between 8.2 and 9.3) - I take it as a sign that we really need regular benchmarking as part of development. Then, on 9.4 the performance significantly improved, thanks to this change:
also known as "GIN fastscan".
I was discussing GIN vs. GiST maincenance cost vs. query performance a few paragraphs back, so what is the performance difference between GIN and GiST?
Well, in this particular benchmark, GIN indexes are about 10x faster than GiST (would be sad otherwise, because fulltext is the primary use of GIN), and as we've seen before it was not much slower than GiST maintenance-wise.
So what is the GIN fastscan about? I'll try to explain this, although it's of course a significantly simplified explanation.
GIN indexes are used for indexing non-scalar data - for example when it comes to fulltext, each document (stored in a
TEXT column as a single value) is transformed into
tsvector, a list of words in the document (along with some other data, but that's irrelevant here). For example let's assume document with ID=10 contains the popular sentence
10 => "The quick brown fox jumps over the lazy dog"
This will get split into an array of words (this transformation may even remove some words, perform lemmatization):
10 => ["The", "quick", "brown", "fox", "jumps", "over", "the", "lazy", "dog"]
If you build GIN index on this "vector" representation, the index will effectively invert the direction of the mapping by mapping words to IDs of all the rows containing that word (each row ID is a pair of block number and offset on that block):
"The" => [(0,1), (0,10), (0,15), (2,4), ...] "quick" => [(0,1), (0,2), (2,10), (2,15), (3,18), ...] "brown" => [(1,10), (1,11), (1,12), ...] ...
Then, if you do a fulltext query on the document, say
SELECT * FROM documents WHERE to_tsvector(body) @@ to_tsquery('quick & fox');
it can simply fetch the lists for
fox and combine them, to get only IDs of the documents containing both words.
And this is exactly where GIN fastscan was applied. Until PostgreSQL 9.4, the performance of this combination step was determined by the longest list of document IDs, because it had to be walked. So if you had a query combining rare and common words (included in many documents, thus having a long lists of IDs), it was often slow.
GIN fastscan changes this, starting with the short posting lists, and combining the lists in a smart way (by using the fact that the lists of IDs are sorted), so that the duration is determined by the shortest list of IDs.
How much impact can this have? Let's see!
The fastscan is not the only improvement in 9.4 - the other significant improvement is compression of the posting lists (lists of row IDs). If you look at the previous example, you might notice that the posting list can be made quite compressible - you may sort the row IDs (first by block number, then by row offset). The block numbers will then repeat a lot, and the row offsets will be an increasing sequence.
This redundancy may be exploited by various encoding schemes - RLE, delta, ... and that's what was done in PostgreSQL 9.4. The result is that GIN indexes are often much smaller. How much smaller really depends on the dataset, but for the dataset used in this benchmark the size dropped to 50% - from ~630MB to ~330MB. Other developers reported up to 80% savings in some cases.
The following chart (already presented at the beginning of this blog post) presents speedup of a random sample from the 33k queries (plotting all the queries would only make it less readable). It shows relative speedup depending on the duration on PostgreSQL 9.3, i.e. each points plots
So if the query took 100 ms on PostgreSQL 9.3, and only takes 10 ms on PostgreSQL 9.4, this is represented by a point [100, 0.1].
There are a few interesting observations:
This is perfectly consistent with the GIN fastscan - the queries that combine frequent and rare words took time proportional to the frequent word on PostgreSQL 9.3, but thanks to fastscan the performance is determined by the rare words. Hence the exponential speedup.
While I'm quite excited about the speedup, the actual performance depends on other things too - for example what dictionary you use. In this benchmark I've been using the
english dictionary, based on a simple
snowball stemmer - a simple algorithmic stemmer, not using any kind of dictionary.
If you're using a more complicated configuration - for example a dictionary-based stemmer, because that's necessary for your language, this may take quite a significant amount of time (especially if you're not using connection pooling and so the dictionaries need to be parsed over and over again - my shared_ispell project might be interesting in this case).
PostgreSQL does not have traditional bitmap indexes, i.e. indexes serialized into simple on-disk bitmaps. There were attempts to do that feature in the past, but the gains never really outweighter the performance issues (locking and such), especially since 8.2 when bitmap index scans were implemented (i.e. construction of bitmaps from btree indexes at runtime).
But if you think about that, GIN indexes are really bitmap indexes, with different bitmap serialiation format. If you're craving for bitmap indexes (not uncommon in analytical workloads), you might try btree_gin extension which makes it possible to create GIN indexes on scalar types (by default GIN can be built only on vector-like types -
tsvector and such).
I also submitted a talk entitled: "Suck it! Webscale is Dead; PostgreSQL is King!". This talk was submitted as a joke. I never expected it to be accepted, it hadn't been written, the abstract was submitted on the fly, improvised and in one take. Guess which talk was accepted? "Webscale is Dead; PostgreSQL is King!". They changed the first sentence of the title which is absolutely acceptable. The conference organizers know their audience best and what should be presented.
What I have since learned is that the talk submission committee was looking for dynamic talks, dynamic content, and new, inspired ideas. A lot of talks that would have been accepted in years past weren't and my attempt at humor fits the desired outcome. At first I thought they were nuts but then I primed the talk at SDPUG/PgUS PgDay @ Southern California Linux Expo.
I was the second to last presenter on Thursday. I was one hour off the plane. I was only staying the night and flying home the next morning, early. The talk was easily the best received talk I have given. The talk went long, the audience was engaged, laughter, knowledge and opinions were abound. When the talk was over, the talk was given enthusiastic applause and with a definite need for water, I left the room.
I was followed by at least 20 people, if not more. I don't know how many there were but it was more than I have ever had follow me after a talk before. I was deeply honored by the reception. One set of guys that approached me said something to the effect of: "You seem like you don't mind expressing your opinions". At this point, some of you reading may need to get a paper towel for your coffee because those that know me, know I will readily express an opinion. I don't care about activist morality or political correctness. If you don't agree with me, cool. Just don't expect me to agree with you. My soapbox is my own, rent is 2500.00 a minute, get in line. I digress, what did those guys ask me about? Systemd, I don't think they were expecting my answer, because I don't really have a problem with Systemd.
Where am I going with this post? I am stomping my way to PgConf.US with an updated version of this talk (You always learn a few things after giving a performance). I am speaking in the first slot on Friday and I am going to do everything I can to bring it. I can't promise to be the best, I can promise to do everything in my power to be my best. I am being recorded this time. My performance will be on the inner tubes forever. I have no choice.
A challenge, do you accept?
I challenge all speakers at this voyage of PgConf.US to take it up a notch. If you were accepted, you have a responsibility to do so. Now, now, don't get me wrong. I am not suggesting that you put on a chicken suit and Fox News t-shirt to present. I am however suggesting that if you are a monotone speaker, try not to be. If you are boring, your audience will be bored and that is the last thing the conference, you or the audience wants. So speak from your diaphragm, engage the audience and make their time worth it!
Each for its own reasons, the world’s major powers have decided to accept Iran as a regional hegemon, I wrote March 4 in Asia Times, leaving Israel and the Sunni Arabs in isolated opposition. The global consensus on behalf of Iranian hegemony is now coming clearly into focus. Although the motivations of different players are highly diverse, there is a unifying factor driving the consensus: the Obama administration’s determination to achieve a strategic rapprochement with Tehran at any cost. America’s competitors are constrained to upgrade their relations with Iran in order to compete with Washington.
The Obama administration’s assessment of Iran’s intentions is so positive that Iranian official sources quote it in their own propaganda. As Jeryl Bier observed at the Weekly Standard, the just-released Threat Assessment report of the director of National Intelligence makes no mention of Iran’s support for terrorism, in stark contrast to the explicit citation of Iranian terrorism in the three prior annual reports. The omission of Iran’s terrorist activities is noteworthy. What the report actually says is even more disturbing. It praises Iran with faint damn:
Despite Iran’s intentions to dampen sectarianism, build responsive partners, and deescalate tensions with Saudi Arabia, Iranian leaders—particularly within the security services—are pursuing policies with negative secondary consequences for regional stability and potentially for Iran. Iran’s actions to protect and empower Shia communities are fueling growing fears and sectarian responses.
Iran supposedly is doing its best to “dampen sectarianism, build responsive partners, and deescalate tensions with Saudi Arabia” — complete and utter falsehood. Iran is infiltrating Saudi Arabia’s Shi’te-majority Eastern Province (also its most oil rich) to agitate against Saudi control, and sponsored a coup against a Saudi-allied regime in Yemen. The report attributes nothing but good intentions to the Tehran regime, and worries only that its policies will have “negative secondary consequences” due to its (understandable, of course) efforts to “protect and power Shia communities.” Iran’s primary motivation, in the administration’s view, is to be a good neighbor and a fountain of good will. Neville Chamberlain never said such nice things about Hitler.
A sign of Saudi Arabia’s waning influence was Pakistan’s decision March 15 to refuse a Saudi request for Pakistani troops to deploy on its border with Yemen, now controlled by pro-Iranian Houthi rebels. A senior Pakistani official told the local press, “Pakistan would not rush to join the anti-Iran alliance that is being forged,” in the wake of Pakistani Prime Minister Nawaz Sharif’s visit to Saudi Arabia last week. “We cannot afford to involve ourselves in the disputes among the Muslim countries,” the official said, adding that Pakistan could spare no additional troops for Saudi Arabia.
That is a serious rebuff for Riyadh, which reportedly financed Pakistan’s nuclear weapons program as a last-ditch guarantee of its own security. As Akhilesh Pillalamarri wrote March 12 in The Diplomat, “Pakistan may be Saudi Arabia’s best bet for a strong long-term security guarantee”:
Pakistan has long had a close relationship with Saudi Arabia and has been involved in protecting that country and the House of Saud. Pakistan has much friendlier relations with Iran than Saudi Arabia does, but ultimately it is more dependent on Saudi Arabia. Saudi Arabia, for example, gave oil to Pakistan in 1998 to help Pakistan weather international sanctions against it for conducting a nuclear test. The Saudis also saved Nawaz Sharif after he was overthrown in a coup in 1999, and he is thus beholden to them.
Pakistan may have been Saudi Arabia’s best bet, but it is a bet that has not paid off. Pakistan is not beholden enough, it appears: Pakistan also is beholden to both the United States and China. The right question to ask is whether Washington intervened with Pakistan to block the Saudi proposal. And China, as I reported in my March 2 analysis, has decided that Iranian regional hegemony is the least bad alternative for the time being. China’s overriding concern is the security of its energy supplies, and it wants to avoid a full-dress Sunni-Shi’ite war in the region. Until early 2014 China thought it could rely on the United States to guarantee energy security in the Persian Gulf. With America’s strategic withdrawal from the region and the rise of ISIS, China has found itself without an American guarantee and without the resources to assert its own security interests. China’s shift towards Iran reflects these considerations.
Another issue for China, Paul Nash and Reza Akhlaghi wrote in the Diplomatic Courier March 16, is that “the rise of militant Sunni Islam is aligning China’s interests with Iran’s.” Nash and Akhlaghi argue:
The rise of the Islamic State in Iraq and Syria has thus emerged as a new component of the Chinese security calculus. Beijing is worried that the rise and spread of Sunni militant Islam so close to its borders, including neighboring former Soviet “Stan” countries of Central Asia, will kindle radical elements in Xinjiang. Sunni militant Islam also threatens to become a strategic and an ideological nightmare for China’s massive and unprecedented multi-billion dollar investments from Xinjiang westward across Central Asia, the linchpin of Beijing’s future vision of energy security and economic development. Sunni radicalism could hinder, if not derail, the realization of Beijing’s Silk Road Belt initiative, presenting a major obstacle to building out a vast overland transcontinental transportation and energy infrastructure.
In an effort to maintain stability in Xinjiang, China has set about strengthening ties with Turkey. But this is no easy task. According to a Pew Research Center poll published last July, Turkey has the most unfavorable view of China amongst the Middle Eastern countries surveyed, with 69 percent of Turks expressing a negative opinion of China, and 57 percent saying that China’s growing economy is not good for Turkey.
And so China gravitates increasingly towards Iran, which it believes can act as a buffer zone against the eastward advance of Sunni radical Islam.
Reality is a bit more complex: China envisions Turkey as a terminus for the “One Belt, One Road” initiative, but it also rankles at covert Turkish support for Chinese Uyghurs. Contrary to Nash and Akhlaghi, China will continue to balance relations between Iran on the one hand and the Saudis and Turks on the other, but it does not want to confront Iran at a moment when Iran provides an important counterweight to ISIS in Iraq, a growing source of Chinese oil imports.
If militant Sunni Islam is an important (if not dominant) concern for China, it is a primary concern for both India and Russia. Russia’s problems in the Caucasus lie with Sunni rather than Shi’ite Muslims. ISIS’ success has inspired copycat terrorists in Russia such as the Caucasus Emirate. An estimated 2,500 Muslims from Chechnya and elsewhere in the Caucasus have joined ISIS, and ISIS has declared its intention to “liberate” the Caucasus from Russian control. Russia warned the West a year ago that it would align with Iran to punish the West over the Ukraine conflict.
For India, an increase in Iran’s influence represents a distraction for its main opponent Pakistan, which is 80% Sunni and shares a border with Iran in fractious Baluchistan. India may not relish the prospect of Iran as a nuclear power, but it has no more sense of urgency about this than does Israel about North Korean nuclear weapons. China does not want a nuclear arms race in the Persian Gulf, but it needs time to develop a policy response independent of the United States. Washington’s embrace of Tehran has made Iranian regional hegemony the path of least resistance. For the time being, it’s Iran’s show.
A cobrança não virá daqueles que sempre tiveram acesso a bens de consumo, serviços e estão pagando (ainda) mais caro agora. A cobrança virá daqueles que experimentaram o acesso ao mercado e o perderão se a economia continuar apontando para baixo. Essa cobrança, a verdadeira, está apenas no começo.
Now that Britain will join China’s Asia Infrastructure Bank — bringing “One Belt/One Road” to the English Channel–Americans should spent some time learning why China’s global economic outreach is so compelling to our closest ally. Below is a Nov. 14, 2014 piece cross-posted from Asia Times.
The Sino-American comedy of errors
BEIJING – Everything in tragedy happens for a reason, and the result always is sad; most things in comedy happen by accident and the outcome typically is happy. Sino-American relations are not destined for conflict, although that is possible. The misunderstandings that bedevil relations between the world’s two most powerful countries remain comedic rather than tragic. That probably is as good as it gets, for no amount of explanation will enable Chinese and Americans to make sense of each other.
Where the Chinese are defensive and cautious, the Americans tend to perceive them as aggressive; where the Chinese are expansive ambitious, the Americans ignore them altogether. The United States is a Pacific power accustomed to maritime dominance. To the extent that Americans focus on China’s foreign policy, it is to express alarm at China’s territorial claims on small uninhabited islands also claimed by Japan, Vietnam and the Philippines. Apart from some overheated and self-serving rhetoric from a few Chinese military leaders, though, the contested islands are of negligible importance in China’s scale of priorities.
The issue may be moot by this writing: last week, China and Japan released a “Principled Agreement on Handling and Improving Bilateral Relations”, following meetings between Japan’s national security adviser, Shotaro Yachi, and Chinese State Councilor Yang Jiechi. The document promises to “establish crisis management mechanisms to avoid contingencies” and to employ “dialogue and consultation”.
Neither Japan nor China had any interest in a military confrontation in the Pacific, although both sides employed the island disputes to play to their own nationalist constituencies. The Principled Agreement sends a signal that the Kabuki show had gone far enough.
A common American meme in response to supposed Chinese expansionism in the Pacific projected an Indian-Japanese military alliance to contain Chinese ambitions under US sponsorship. Although a few Indian nationalists enthused over the idea, it was an empty gesture from the outside. If India got into a scrap with China over disputed borders, for example, just what would Japan do to help?
The newly-elected Indian government under Narendra Modi never took the idea seriously. On the contrary, after President Xi Jinping’s recent state visit to India, Modi envisions Chinese investment in urgently needed infrastructure. Economics trumps petty concerns over borders in the mountainous wasteland that separates the world’s two most populous nations.
There also is a strategic dimension to the growing sense of agreement between China and India. From India’s vantage point, China’s support for Pakistan’s army is a concern, but it cuts both ways. Pakistan remains at perpetual risk of tipping over towards militant Islam, and the main guarantor of its stability is the army. China wants to strengthen the army as a bulwark against the Islamic radicals, who threaten China’s Xinjiang province as much as they do India, and that probably serves India’s interests as well as any Chinese policy might.
Chinese analysts are dumbfounded about the US response to what they view as a sideshow in the South China Sea and only tangentially concerned about India. They struggle to understand why a vastly improved relationship with Russia has emerged in response to US blundering in Ukraine.
As a matter of diplomatic principle, China does not like separatists because it has its own separatists to contend with, starting with the Muslim Uyghurs in Xinjiang province. Washington thought that the Maidan Revolution in Kiev last year would take Crimea out of Russian control, and Russia responded by annexing the peninsula containing its main warm-water naval base.
When the West imposed sanctions on Russia in retaliation, Moscow moved eastwards – an obvious response, and one that strongly impacts Western power. Not only has Russia opened its gas reserve to China, but it has agreed to supply China with its most sophisticated military technology, including the formidable S-400 air defense system. Russia was reluctant to do so in the past given Chinese efforts to reverse-engineer Russian systems, but the Ukraine crisis changed that.
Western analysts, to be sure, now observe that the new Russian-Chinese rapprochement might be a challenge for the West. The New York Times devoted a front-page feature to the opinions of the usual suspects among Soviet watchers in its November 9 edition.
This was obvious months ago, and should have been obvious before the fact: the West merely threw B’rer Putin into the briar patch to his east. Of all the miscalculations in Western policy since World War II, this was perhaps the stupidest. The Chinese are left to scratch their heads about their unanticipated good luck.
It is wrong to speak of a Russian-Chinese alliance, to be sure, but there is a developing Sino-Russian condominium in Asia. The energy and defense deals between Moscow and Beijing are important in their own right, but they take on all the more importance in the context of what might be the most ambitious economic project in history: the New Silk Road. The Pacific holds little promise for China. Japan and South Korea are mature economies, customers as well as competitors of China.
Expansion in the Pacific simply has nothing to offer China’s economy. What China wants is to be impregnable within its own borders: it will spend generously to develop surface-to-ship missiles that can take out US aircraft carriers, hunter-killer submarines, and air defense systems.
China’s prospects are to the west and south: energy and minerals in Central Asia, food in Southeast Asia, warm-water ports on the Indian Ocean, a vast market, and access to world markets beyond. The network of rail, pipelines and telecommunications that China is building through the former Soviet republics and through Russia itself will terminate at the Mediterranean and provide a springboard for Chinese trade with Europe.
The whole Eurasian landmass is likely to become a Chinese economic zone, especially now that Russia is more amenable to Chinese terms. That the Americans would have helped bring this to fruition by tilting at windmills in Ukraine baffles the Chinese, but they are enjoying the result.
The economic impact of this is hard to fathom, but it is likely to extend Chinese influence westwards on a scale that the West simply hasn’t begun to imagine. It is not at all clear whether China has a clear idea of what the implications of the New Silk Road might be. The implosion of America’s geopolitical position has placed risks and opportunities at Beijing’s doorstep, to Beijing’s great surprise.
A year ago, Chinese officials privately reassured visitors that their country would “follow the lead of the dominant superpower” in matters relating to Middle East security, including Iran’s attempts to acquire nuclear weapons. For the past several decades, China has allowed the US to look out for the Persian Gulf while it increased its dependency on Persian Gulf oil. By 2020, China expects to import 70% of its oil, and most of that will come from the Gulf.
The Chinese view has changed radically during the past few months, in part due to the collapse of the Syrian and Iraqi states and the rise of Islamic State. It is hard to find a Chinese specialist who still thinks that the US can stand surely for Persian Gulf security. Opinion is divided between those who think that America is merely incompetent and those who think that America deliberately wants to destabilize the Persian Gulf.
Now that the US is approaching self-sufficiency in energy resources, some senior Chinese analysts believe it wants to push the region into chaos in order to hurt China. One prominent Chinese analyst pointed out that Islamic State is led by Sunni officers trained by the United States during the 2007-2008 “surge” as well as elements of Saddam Hussein’s old army, and that this explains why IS has displayed such military and organizational competence.
The complaint is justified, to be sure: General David Petraeus helped train the 100,000-strong “Sunni Awakening” to create a balance of power against the Shi’ite majority regime that the US helped bring to power in 2006. How, the Chinese ask, could the Bush administration and Petraeus have been so stupid? To persuade the Chinese that they were indeed that stupid is a daunting task.
China’s attitude towards Washington has turned towards open contempt. Writing of the mid-term elections, the official daily newspaper Global Times intoned: “The lame-duck president will be further crippled ? he has done an insipid job, offering nearly nothing to his supporters. US society has grown tired of his banality.”
But the decline of American influence in the region from which China obtains most of its oil is not a happy event for Beijing.
China did not anticipate the end of the free ride from the Americans, and it isn’t sure what to do next. It has tried to maintain a balance among countries with whom it trades and who are hostile to each other. It has sold a great deal of conventional weapons to Iran, for example, and some older, less-sophisticated ballistic missiles.
But China has sold Saudi Arabia its top-of-the-line intermediate range missiles, giving the Saudis a “formidable deterrent capability” against Iran and other prospective adversaries. China obtains more oil from Saudi Arabia than any other country, although its imports from Iraq and Oman are growing faster. Because the latter two countries are closer to Iran, China wants to strike a balance.
Chinese opinion is divided about the implications of Iran acquiring nuclear weapons: some strategists believe that the balance of nuclear power in the region will suffice to prevent the use of such weapons, while others fear that a nuclear exchange in the Gulf might stop the flow of oil and bring down China’s economy. China has joined the P-5 plus 1 negotiations (involving the UN Security Council permanent five members plus Germany) on Iran’s nuclear status, but has not offered a policy independent of President Barack Obama’s.
Meanwhile the rise of Islamist extremism worries Beijing, as well it should. At least a hundred Uyghurs reportedly are fighting with Islamic State, presumably in order to acquire terrorist skills to bring back home to China. Chinese analysts have a very low opinion of the Obama administration’s approach to dealing with IS, but do not have an alternative policy. This is an issue of growing importance. Instability threatens the Silk Road project at several key notes.
China has no sympathy whatever for what analysts there like to call “political Islam”. America’s flirtation with the Muslim Brotherhood – both from the Obama administration and from mainstream Republicans such as Senator John McCain – strikes the Chinese as incompetence, or worse. But China has no capability to go after the Islamists, except for a very limited deployment of marines off the coast of Somalia.
China’s policy-making is careful, conservative and consensus-driven. Its overriding concern is its own economy. The pace of transformation of the Middle East has surprised it, and it is trying to decide what to do next.
Its pro forma policy is to join the Iran talks, and offer to join the Quartet (the UN, the US, the European Union, and Russia) talks on the Israel-Palestine issue, but neither of these initiatives has much to do with its actual concerns.
What China will do in the future cannot be predicted. But it seems inevitable that China’s basic interests will lead it to far greater involvement in the region, all the more so as the US withdraws.
The Four Thirds Group Mathcing Simulator page (Click here) just added the new and yet still not released Olympus 7-14mm f/2.8 PRO MFT lens. The lens is scheduled to be released this summer and price is yet unclear.
The lens is already listed at BHphoto (Click here).
No Brasil, quando se trata de escrever, de produzir literatura, a repugnância ao profissionalismo chega a ser arrebatadora.
Para muitos, o escritor ainda é um romântico que se embebeda noite após noite, um irresponsável que não cumpre horários, veste-se com desleixo, despreza as instituições, considera-se superior a todos, vive de maneira miserável e, o principal, escreve quando sua fada-madrinha bate à janela e lhe sussurra o tema de um continho rápido.
Para os que pensam assim, a literatura ainda é um presente caído dos Céus — ou o fado terrível, cujo peso condena o escritor a ser maldito até o fim de seus dias.
Para os que pensam assim, o escritor disciplinado — que estuda, trabalha, reserva algumas horas diárias para escrever e ainda publica seus livros — é um verme burguês que deve ser pisoteado.
Para os que pensam assim, o jovem que se inscreve numa Oficina de Escrita Criativa e se dedica seriamente a entender a técnica da escrita e como grandes escritores a utilizaram não passa de um tolo ingênuo.
Para os que pensam assim, o escritor não deve nem mesmo planejar seus livros, mas deixar que a escrita flua como uma energia cósmica. Para eles, o escritor é um xamã, a voz que revela segredos milenares, o vidente que anuncia uma nova era. E outras bobagens.
Para os que pensam assim, o escritor que enfrenta a luta diária com as palavras seguindo um método severo de trabalho, sem obedecer a panelinhas, partidos ou teorias estéticas, é um pária digno do desprezo de todos os departamentos de Teoria Literária.
E se esse escritor — depois de anos debruçado sobre a escrita, trabalhando durante o dia como funcionário público, professor, jornalista, bibliotecário ou pedreiro, e dedicando suas horas de descanso ao que dá sentido à sua vida —, se esse escritor ainda consegue publicar seu livro e obter relativo sucesso, então está comprovado: ele não vale nada mesmo, certamente se vendeu para os porcos capitalistas e só é lido porque produz uma literatura rasteira, pequena, sem originalidade.
Bem, se você chegou até aqui, tenho algo importante a lhe dizer: não acredite nessas conversas de botequim. Não acredite no discursinho afetado de quem se pavoneia apenas para esconder a própria mediocridade.
Não há grandes escritores sem disciplina, sem seriedade, sem estudo. Sempre que você escutar esses discursinhos fantasiosos, não perca tempo retrucando. Muitas vezes, o tal discursador faz exatamente o contrário do que apregoa — primeiro, para estar bem com os medíocres que defendem essas teses; segundo, para não alimentar possíveis rivais literários.
Imagine o cotidiano desses escritores, pense nas horas cheias de solidão, nas leituras diárias, nas raríssimas amizades, na eterna dúvida em relação à qualidade do próprio trabalho, no peso das palavras, no volume de rascunhos escritos, reescritos e jogados fora, na vida aparentemente banal, apagada, sem um gesto de heroísmo visível, sem blog, sem frequentar rodinhas de literatos para encher a cara de cerveja ou fumar baseados.
Pense numa vida inteira dedicada à escrita, esse trabalho muitas vezes enlouquecedor, e no resultado final: três, quatro, cinco livros bons, talvez um genial.
E pense em você mesmo, escrevendo todos os dias, lutando para encontrar sua própria voz, explicando a alguns o que você almeja — e recebendo em troca o olhar perdido de quem não entendeu nenhuma palavra.
Então recorde os discursinhos. E você verá como as bobagens que tantos repetem — por irresponsabilidade, perfídia, desconhecimento ou cinismo — não se sustentam. Elas se dissolvem quando você as coloca diante do cotidiano, diante dos escritores que produziram obra consistente, capaz de vencer o tempo.
Contra as mentiras românticas e esquerdistas, o melhor remédio é simples — e está sempre à mão: chama-se realidade.
About a week ago, I posted comparison of pgbench results since PostgreSQL 7.4, which was one of the benchmarks done for my pgconf.eu 2014 talk. For an explanation of the whole quest, please see the first post.
Now it's time to discuss results of the second benchmark - TPC-DS, which models an analytical workload, i.e. queries processing large amounts of data, with aggregations, large sorts, joins of large tables, TOP-N queries etc. That is very different from pgbench, executing queries manipulating individual rows mostly through primary keys, etc.
The one chart you should remember from this post is this one, illustrating how long it takes to execute 41 queries (subset of TPC-DS queries compatible with PostgreSQL since 7.4) on a 16GB dataset (raw CSV size, after loading into database it occupies about 5x the size because of overhead, indexes etc.).
The numbers are runtime in seconds (on the i5-2500k machine), and apparently while on PostgreSQL 8.0 it took ~5100 seconds (85 minutes), on 9.4 it takes only ~1200 seconds (20 minutes). That's a huge improvement.
Notice the 8.0 results are marked with a star - that's because on 8.0 one of the queries did not complete within an arbitrary limit of 30 minutes, so it was cancelled and was counted as taking 1h. Based on several experiments, I believe the actual runtime would be even longer than that - in any case it was much longer than on PostgreSQL 8.1, where this particular query got significantly improved by bitmap index scans.
Considering that this is quite I/O intensive (the database size is ~5x the RAM), that's a huge improvement. As we'll see later, with smaller datasets (that completely fit into RAM), the speedup is even larger.
BTW I've had trouble making this work on PostgreSQL 7.4 (without making the results difficult to compare), so I'll only present results for PostgreSQL 8.0 and newer releases.
But let's talk a bit about the TPC-DS benchmark, because the brief description in the introduction is not really detailed enough. TPC-DS is another benchmark from TPC. It's representing analytical workloads (reporting, data analysis, decision support systems and so on), so the queries are processing large amounts of data, performing aggregations (GROUP BY), various joins, etc.
It effectively extends and deprecates TPC-H benchmark, improving it in multiple ways to make it more representative of actual workloads. Firstly, it makes the schema more complex (e.g. more tables), and uses less uniform distributions of the data (which makes cardinality estimations way more difficult). It also increases the number of query templates from 22 to 99, and uses modern features like CTEs, window functions and grouping sets.
Of course, presenting all the 99 query templates here would be pointless, but one of the simpler ones looks like this:
select ca_zip ,sum(cs_sales_price) from catalog_sales ,customer ,customer_address ,date_dim where cs_bill_customer_sk = c_customer_sk and c_current_addr_sk = ca_address_sk and ( substr(ca_zip,1,5) in ('85669', '86197','88274','83405','86475', '85392', '85460', '80348', '81792') or ca_state in ('CA','WA','GA') or cs_sales_price > 500) and cs_sold_date_sk = d_date_sk and d_qoy = 2 and d_year = 2000 group by ca_zip order by ca_zip limit 100;
This particular query joins 4 tables, uses non-trivial WHERE conditions, aggregation and finally selects only results for the first 100 ZIP codes. The other templates are often way more complex.
Some of the templates are incompatible with PostgreSQL, because they rely on not-yet-implemented features (e.g. CUBE/ROLLUP). Some of the templates also seem broken, as the query generator fails on them. There are 61 queries working fine since PostgreSQL 8.4 (when CTEs and window functions were added to PostgreSQL), and 41 queries are compatible with versions since 7.4. And those 41 queries were used for this benchmark.
Note 1: Most of the remaining queries may be rewritten to make them work, but I haven't done that. Those queries were designed specifically to test those features, and the rewritten versions would benchmark the same things as the remaining queries anyway. I'm pretty sure it's possible to rewrite some of the "compatible" queries to get better performance, but I haven't done that for the same reason.
Note 2: The TPC-DS benchmark also includes tests for maintenance/management tasks (e.g.
ALTER TABLE, various DDL etc.) but I have not performed this part.
As with all benchmarks, dataset sizes really matter. I've used two sizes 1GB and 16GB, specified as the size of the CSV filed generated by the TPC tooling.
small / 1GB
large / 16GB
I haven't really spent too much time optimizing the schema - I simply used the schema provided with the TPC-DS suite, and after a few rounds of benchmarking created suitable indexes. I have used the same schema for all PostgreSQL versions, which probably discriminates the newer versions, as those might benefit from tweaking indexes to be suitable for index-only scans, for example.
The the PostgreSQL configuration was mostly default, with only minimal changes. All the versions used the same configuration:
shared_buffers = 512MB work_mem = 128MB maintenance_work_mem = 128MB effective_cache_size = 1GB checkpoint_segments = 32
Similarly to pgbench we could probably get a bit better performance by tuning the values for each version. The values used here are quite conservative already, so don't expect an effect similar to pgbench, when lowering the values resulted in significant speedup on older versions.
If you want to review the tooling I used, it's available here. It's a bit hackish (mostly a bunch of shell scripts) and certainly is not "ready to run" in various ways - it does not include the data and query generators (you can get them at tpc.org, and you'll have to modify a few paths in the scripts (to data directory etc.), but it shouldn't be difficult. Let me know by e-mail if you run into problems.
This also includes DDL (schema including indexes), PostgreSQL config files, query templates and actual queries used for the benchmark.
The first thing you often need to do is loading the data into database. The load process is very simple:
and the results for a 1GB dataset (raw size of CSV files) look like this:
Clearly, something changed in 9.0, because the VACUUM FULL step takes much longer, and indeed - that's when VACUUM FULL was rewritten to use the same implementation as CLUSTER, i.e. completely rewriting the table. On older releases it used a different approach, that was more efficient for tables with only small fraction of dead tuples (which is the case here, right after a load), but much less efficient on tables with a large portion of dead tuples (which is exactly when you want to do a CLUSTER).
That means the VACUUM FULL is rather misleading here, because it's used in exactly the context where you should not use that (and instead let autovacuum do it's job), so let's remove it from the chart.
Much better, I guess. While it took ~1000 seconds on 8.0, on 9.4 it only takes ~500 seconds - not bad, I guess. The main improvement happened in 8.1, and 8.2 was a minor regression, followed by small incremental improvements.
Let's see the performance with a larger (16GB) dataset:
Again, about 2x the speedup between 8.0 and 9.4 (just like with the small dataset), but the pattern is slightly different - both 8.1 and 8.2 improved the performance about equally (no regression in 8.2), followed by releases keeping a stable performance.
So far I've been talking about "raw size" of the datasets, i.e. size of the CSV files produced by the TPC-DS generator. But what that means for the database size? After the small (1GB) dataset, you'll get about this:
That's ~5-6GB databases - PostgreSQL 9.4 needs about 15% less compared to 8.0, which is certainly nice. About 60% of that are indexes, leaving ~2.5GB for the tables. By applying this to the 16GB dataset, it will require ~40GB on disk for the table, and additional ~60GB for the indexes.
And finally, the query performance. On the small dataset (which fits into memory), the 41 looks like this on average:
and on the large dataset, you'll get this:
Clearly, PostgreSQL 8.1 was a significant improvement. It's also possible to look at duration broken down per query (so that each query gets the same color on all versions):
The regressions on 8.1 and 8.4 - 9.1 are clearly visible - I haven't looked much into them though.
If you want look into the results, a complete set of results (including logs, EXPLAIN, EXPLAIN ANALYZE and such) for the 16GB dataset is available here. Feel free to point any inconsistencies or errors.
If you look into release notes, two major features introduced in that version should catch your eye:
PostgreSQL 9.2 was another release significantly improving performance (almost 2x compared to 9.1), most likely because of this feature (see release notes):
You can also notice that further improvement happened in 9.4, by about 10%. That's likely thanks to optimization of
There are of course many other performance-related improvements both in all the releases since 8.0, but those are related to other kinds of queries.
So, what's the conclusion?
So, in the introduction post I briefly described what was the motivation of this whole effort, what hardware and benchmarks were used, etc. So let's see the results for the first benchmark - the well known pgbench and how they evolved since PostgreSQL 7.4.
If there's a one chart in this blog postthat you should remember, it's probably the following one - it shows throughput (transactions per second, y-axis) for various numbers of clients (x-axis), for PostgreSQL releases since 7.4. bad
Note: If a version is not shown, it means "same (or almost the same) as the previous version" - for example 9.3 and 9.4 give almost the same performance as 9.2 in this particular test, so only 9.2 is on the chart.
So on 7.4 we could do up to 10k transactions per second (although the scalability was poor, and with more clients the throughput quickly dropped to ~3k tps). Since then the performance gradually improved, and on 9.2 we can do more than 70k tps, and keep this throughput even for higher client counts. Not bad ;-)
Also note that up until PostgreSQL 9.1 (including), we've seen a significant performance drop once we exceeded the number of CPUs (the machine has 8 cores in total, which is exactly the point where the number of transactions per second starts decreasing on the chart).
Of course, these are the result of just one test (read-only on medium dataset), and gains in other tests may be more modest, but it nicely illustrates the amount of improvements that happened sice PostgreSQL 7.4.
This post is a bit long, but don't worry - most of it are pretty images ;-)
If you know these pgbench basics, you may just skip to the "Results" section, presenting the results of the benchmarks.
Pgbench has two main modes - read-write and read-only (there's also a third mode, and an option to use custom scripts, but I won't talk about those).
The read-write mode executes a simple transaction, simulating a withdrawal from an account - updating the balance in "accounts" table, selecting the current balance, updating tables representing a branch and a teller, and recording a row into a history.
BEGIN; UPDATE pgbench_accounts SET abalance = abalance + :delta WHERE aid = :aid; SELECT abalance FROM pgbench_accounts WHERE aid = :aid; UPDATE pgbench_tellers SET tbalance = tbalance + :delta WHERE tid = :tid; UPDATE pgbench_branches SET bbalance = bbalance + :delta WHERE bid = :bid; INSERT INTO pgbench_history (tid, bid, aid, delta, mtime) VALUES (:tid, :bid, :aid, :delta, CURRENT_TIMESTAMP); END;
As you can see, all the conditions are on a PK column, always work with a single row, etc. Simple.
The read-only mode is even simpler - it simply executes only the "SELECT" query from the read-write mode, so it's pretty much this:
SELECT abalance FROM pgbench_accounts WHERE aid = :aid;
Again, access through PK column - very simple.
In short, scale determines size of the database as a number of rows in the main "accounts" table - the supplied value gets multiplied by 100.000 and that's how many rows in that table you get. This of course determines the size on disk, as every 100.000 rows corresponds to 15MB on disk (including indexes etc.).
When choosing the scale for your benchmark, you have three basic choices, each testing something slightly different.
The third important parameter I'd like to discuss is client count, determining the number of concurrent connections used to execute queries.
Let's assume you measure performance with a single client, and you get e.g. 1000 transactions per second. Now, what performance will you get with 2 clients? In an ideal world, you'd get 2000 tps, twice the performance with a single client.
For N clients you'd get N-times the performance of a single client, but it's clear it doesn't work like that, because sooner or later you'll run into some bottleneck. Either the CPUs will become fully utilized, the I/O subsystem will be unable to handle more I/O requests, you'll saturate network or memory bandwidth, or hit some other bottleneck.
As the number of clients is increased, we usually see three segments on the charts:
On a simple chart, it might look like this:
Of course - this is how it would look in an ideal world. In practice, the initial growth is not linear, and the throughtput in the second part is not constant but drops over time - sometimes fast (bad), sometimes gradually (better). We'll see examples of this.
For sake of clarity and brevity, I've left out the latencies from the charts in this blog post. A proper latency analysis would require a considerable amount of space (to show averages, percentiles, ...) but the results turned out to be especially dull and boring in this case. You have to trust me that the transaction rate (and implied average latency) is a sufficient description of the results in this case.
By default, all the pgbench clients (connections) are handled by a single thread - the thread submits new queries, collects results, writes statistics etc. When the queries are long, or when the number of connections is low, this is fine. But as the queries get shorter (e.g. with read-only test on small dataset), or when the number of connections grows (on many-core systems), the management overhead becomes significant.
That's why pgbench has
--jobs N (or
-j N) option, to specify the number of threads. The value has to be a divisor of number of clients, so for example with 8 clients you may use 1, 2, 4 or 8 threads, but not 5 (because 8 % 5 = 3). There are various smart strategies to set the value, but the easiest and most straightforward way is to use the same value for both options, and that's what I used for this benchmark.
pgbench tool evolves with PostgreSQL, so when new features are added to PostgreSQL, which means new
pgbench versions may not work with older PostgreSQL releases (e.g. on releases not supporting
fillfactor storage option, which is used by
pgbench when initializing tables). There are two ways to overcome this - either use
pgbench version matching the PostgreSQL release, or use the new version but patch it so that it works with older releases (by skipping features not present in that PostgreSQL version).
I've used the latter approach, mostly because I wanted to use some of the features available only in recent pgbench versions (especially
The patch I used is available here. It changes three things:
-Xoption to disable
IF EXISTSwhen dropping table (because that's not available on old releases)
-Qoption to entirely skip cleanup (because the script simply recreates the database anyway)
fillfactoron all the tables (so the default version is used)
None of the changes should have no impact on the results, because the first two just remove new syntax (which is not necessary), and after removing the explicit fillfactor option the default will be used (which is 100 on all versions).
If you're interested in the script driving the testing, it's available here. It's a bit hackish, but it should give you an idea of how the testing was done.
So, given all the possible parameters (mode, scale, client count), what combinations were benchmarked?
That gives 84 combinations in total, and for each combination there were 3 runs, 30 minutes each. That's ~126 hours per version (not including initialization and warmup), so ~6 days per PostgreSQL version. There were 12 versions benchmarked (7.4, 8.0, 8.1, 8.2, 8.3, 8.4, 9.0, 9.1, 9.2, 9.3, 9.4b2, head) so the total runtime is close to 70 days.
Hopefully that convinces you this was not a rushed unreliable benchmark.
There were only minimal changes to the configuration file:
# machine has 16GB of RAM in total shared_buffers = 2GB work_mem = 64MB maintenance_work_mem = 512MB checkpoint_segments = 64 checkpoint_completion_target = 0.9 effective_cache_size = 8GB # when executed on the SSD (otherwise 4) random_page_cost = 2
Of course, there were some tweaks (e.g. because older versions don't know
checkpoint_completion_target) but no extensive per-version tuning was performed, as that'd make the testing much more time consuming (to the point that it'd be impossible).
I however did a limited amount of tests to see how big impact may such tuning have - I'll show you a chart illustrating that later (spoiler: the older versions may be much faster with a bit of tuning, but the new ones are still faster).
So, let's see the results! I'll only show some of the charts - the rest of them is usually quite similar.
First, the chart we've already seen:
This is a read-only workload, with a dataset that completely fits into RAM. So this is CPU-bound, with locking etc. Yet PostgreSQL 9.4 is 7-20x faster than 7.4. Not bad, I guess.
The other interesting observation is that it scales linearly with the number of cores - in this case it's actually a bit better (the performance on 8 cores is more than 2x of 4 cores), most likely thanks to caching effects. During the 9.2 development this was thoroughly tested (by Robert Haas) and the conclusion was that we do scale linearly up to 64 cores (and probably beyond, but we didn't have machines with more cores).
There's also an excellent talk by Heikki Linnakangas about all the improvements done in PostgreSQL 9.2, presented at FOSDEM 2012.
This is of course dependent on workload - the improvements in locking are most visible on read-only workloads on small datasets, because that's mostly cached and should do much I/O. Once you start hitting any kind of I/O, it will probably overshadow these improvements. It's also related to improvements made in Linux kernel, for example the lseek scalability improvements committed into kernel 3.2. And finally all this is of course dependent on the exact CPU type - on older CPUs the improvements may not be as great.
Let's see how the read-only workload performs on a large dataset (thus not fitting into RAM), on an SSD drive
This is clearly I/O bound, because pgbench generates a lot of random I/O (although the SSD handles that much better than rotational drives). Because of the I/O bottleneck the difference is not as significant as with the medium dataset, but 2-4x speedup is still really nice.
There were two releases that significantly improved the performance - first 8.0, then 8.1 (and that's mostly what we get on 9.4, performance-wise).
So, what happens when we keep the large dataset, but do a read-write test?
Interesting. The 7.4 is doing rather poorly (just ~500 transactions per second on an SSD). The 8.x releases perform much better, with roughly ~2x the performance, but the throughput is mostly constant after reaching 8 clients (which is the number of CPUs in the machine).
But since 9.1, we see a quite interesting behavior - the throughput grows even after 8 clients (where it's mostly equal to 8.x), up to ~2500 tps with 32 clients. And the chart suggests it would probably grow even further with more clients (had it been part of the benchmark). This can happen because SSDs are quite good with handling parallel requests (thanks to internal architecture, which uses multiple independend channels), and since 9.1 we can saturate it much better.
Note 1: The read-write results on medium dataset are very similar to this.
If you ask how would the comparison look on traditional rotational drives, this is the answer:
For the record, these are 10k SAS drives, connected to a HP P400 controller (RAID10 on 6 drives).
It's interesting that 8.4 is slightly faster than 9.4 for higher client counts, but we're still doing better than 7.4, but clearly the I/O is a significant bottleneck that's difficult to beat at the code level.
Another interesting question is how much this depends on hardware, so here is a comparison with results from the machine with i5-2500k CPU.
Wow! On PostgreSQL 7.4 the difference is "only" 2x, but on 9.4 the i5 really crushes the Xeon, giving nearly ~4x the performance. Also, the i5 has only 4 cores while the Xeon has 8 cores (in 2 sockets), so that's ~8x the performance per socket, while the frequencies are about the same (3 GHz vs. 3.3 GHz).
Just for the record, this difference is not only thanks to the CPU - the i5 system has a better memory throughput, the SSD is not connected using a lousy PCIe controller (which limits the bandwidth), etc. But clearly, modern hardware can make a huge difference.
I also mentioned that each PostgreSQL version may have different "optimal" configuration values. That makes sense, because the development happens in the context of the current hardware. When 7.4 was released 10 years ago, most machines had only 4GB of RAM (not to mention that everything was still 32-bit and thus capped to 2GB), so there was no point in optimizing for large
shared_buffers values, for example.
So, what happens if we use much lower settings for
shared_buffers (only 128MB instead of the 2GB used in all the previous benchmarks)? This:
On 7.4, using
shared_buffers=128MB improved the performance of
shared_buffers=2GB by a factor of ~3x. On 8.0 there's still some difference, but it's much smaller (~5%). The nice thing is that 9.4b1 is faster even with the 2GB.
Note: There are also other reasons to use small
shared_buffers values on older versions (absence of
checkpoint_completion_target and usually running them on older filesystems like ext3, with poor fsync behavior).
Those were medium and large datasets, but what about the small one? The read-only results look quite similar to the medium dataset, but the read-write seems interesting on it's own:
Wow! 8.2 really improved this, boosting the performance 5x, compared to 8.1. The following releases further improved the results by ~25%, but 8.2 was clearly a huge jump forward.
And finally, let's do two more comparisons of the Xeon and i5 based systems, with read-only and read-write benchmarks on the small dataset.
The solid lines are Xeon, the dashed lines are i5. This time the Xeon actually wins over i5 (unlike with the large dataset), at least on 9.4 - having 2x the cores matters here, apparently. It's also nice to see that 9.4 fixed the performance degradation on the Xeon, with client counts higher than the number of CPU cores (on the i5 this did not happen at all).
On read-write workload, i5 wins again (despite the smaller number of cores).
Let me summarize the main points we've learned from the benchmark results:
shared_buffersetc.), because that's the context of their development.
At a time when some of the biggest names in the global defence industry have decided to end production of their well known fighter planes, Sweden’s Defense and Security company, Saab, gets going, says The Wall Street Journal.
With little in the way of a global marketing machine, Sweden has had outsize success in winning export orders. That is thanks to Saab AB’s Gripen combat jet, it adds.
WSJ interviews Saab’s Chief Executive Håkan Buskhe who points out, that unlike before, weapon buyers are becoming more cost-conscious now.
“Seven to eight years ago, no one discussed affordability when it comes to arms equipment. There was a paradigm, when I started, that if it is not expensive, it is not good. The cost component—that was not really the big deal in the past—has really caught up with each and every economy around the world. That does not mean you can sell an aircraft that is not good enough. There is no point in sitting in an aircraft that gets shot down and saying ‘at least it’s affordable’. So I think that is the key part of our success, and I believe we have a window going forward,” Buskhe says.
Read the original story here.
Please note that one may need a WSJ subscription to read the full story.
Subscribers can click below for the full story.
"tudo em nosso governo é investigado, não existe um 'engavetador geral da república'"Lava Jato: Janot pede que STF arquive investigações contra Dilma
FAB pilots perform their training flights in Gripen C
FAB pilots Gustavo Oliveira Pascotto and Ramon Santos Fórneas have entered the 'land flight without instructors’ phase of their training process, reports Ovale.com.br.
Pascotto is most impressed by the data integration capacity of the Gripen system.
"Gripen is a platform that will integrate all of our employment doctrine. The key word is integration. Everything has data fusion. It is like they have combined the performance of the Mirage, the F-5 combat systems and the system of recognition of A-1 in a single aircraft. But much better," Pascotto said.
The missions of the new phase began in late January. For these missions, the flights depart from F7 Airbase at Såtenäs in south western Sweden.
F7 Såtenäs is the prime training center for the Gripen system. Pilot trainings, including that of foreign operators, are carried out here. This year in July, F 7 will celebrate its 75th anniversary. According to the report, on an average, the Swedish Air Force trains 30 pilots every year.
"The training is intense, but Gripen is easy to use,” says Colonel Michael Cherinet, Commander of the Såtenäs Base.
Read the full story: Pilotos brasileiros iniciam nova fase de treinamento no Gripen
No Brasil, grande parte da ficção contemporânea se resume a histórias construídas numa única dimensão, histórias planas, em que os protagonistas vivem num pequeno aquário.
A caixinha de vidro brilha sob o sol, o peixinho dourado vai para lá e para cá, chega a fazer piruetas, é bem alimentado pelo narrador, mas habita um universo restrito.
Um remédio para esse problema é ler autores que fizeram o oposto: construíram um mundo complexo, com vários personagens, em que o protagonista se vê obrigado a interagir, a viver em diferentes cenários, a enfrentar situações inesperadas. Mundo que abala as certezas do protagonista, obrigando-o a rever suas decisões ou impor-se de alguma maneira, se não deseja sucumbir.
Uma história narrada dessa forma não serve apenas para convencer o leitor, mas cria uma trama que se assemelha à realidade e, portanto, oferece inúmeras possibilidades de empatia.
Quando falamos sobre narrativas que apresentem essa complexidade, quase sempre é a figura de Dom Quixote que surge em nossa imaginação. Mas quero propor a vocês um livro, infelizmente, pouco conhecido no Brasil.
Leiam O Aventuroso Simplicissimus, de Hans Jacob Christoffel von Grimmelshausen, o primeiro grande romancista da literatura alemã — disponível em língua portuguesa na elogiada tradução de Mário Luiz Frungillo —, e vocês entenderão como a literatura pode ser muito mais do que um peixinho no aquário.
Simplicissimus é um protagonista que vive em luta permanente: consigo mesmo e com a realidade. É um homem real, repleto de dúvidas, com alternâncias de humor, inseguro, jogado de um lado para o outro pela guerra. Seu cotidiano é uma infindável sucessão de alterações de curso.
Como afirmei no ensaio que escrevi há alguns anos para o Jornal Rascunho, há um brilho sedutor na inconstância, no verdadeiro caos, nos constantes equívocos que movem o protagonista.
Simplicissimus oscila entre a alegria e a tristeza, a compenetração e o arroubo guerreiro, a luxúria e o isolamento, a liberdade e a prisão, o amor e a misoginia, a busca da santidade e a pilhagem.
Mas está sempre imbuído de sinceridade, de leveza e de propósitos que, bons ou maus, nos convencem.
Simplicissimus é um pícaro perspicaz, dissimulado, encantador, que jamais se nega à autoanálise, à introspecção.
Ele nos fascina porque seus dramas não o derrotam, mas servem para impulsioná-lo a novas aventuras, sempre agarrado à vida, sem jamais conceder às dúvidas e aos temores aquela angústia quase absoluta que domina alguns personagens da literatura moderna.
Trata-se de um protagonista que aprende a rir dos acontecimentos e, principalmente, de si mesmo.
Não proponho que você use Grimmelshausen como modelo. Mas que aprenda com ele a escrever uma trama na qual as histórias secundárias servem para ampliar o núcleo da narrativa.
Aprenda com Grimmelshausen a criar um universo em que várias histórias menores acontecem de forma simultânea, contribuindo para dilatar o drama do protagonista, para aprofundar seus questionamentos.
O protagonista isolado no aquário não tem como se relacionar com o mundo — observa-o sempre como se olhasse por uma janela, sempre na defensiva, sempre fechado à vida.
Precisamos, ao contrário, de escritores engenhosos, capazes de interligar argumentos diferentes, de maneira que um exerça influência sobre o outro.
Precisamos de histórias com enredos que não estejam apenas conectados de maneira tênue, mas forcem os personagens a agir, inclusive, contra a sua própria vontade, contra as suas próprias crenças.
Só assim pode surgir uma trama convincente, em que cada personagem possui uma identidade, age realmente, estabelece relações, demonstra ou esconde seus desejos, suas intenções. Só assim pode nascer literatura.
“I’ve read what you said about Bibles being designed like dictionaries,” writes Bradford Taliaferro. “Now that I have this Bible — I get it! Reading the Bible doesn’t seem like work anymore.”
Bradford’s reaction echoes sentiments I hear all the time. People accustomed to the old dictionary-style layout of Scripture are surprised what a big difference seemingly minor changes like paragraphed text and a single-column layout make to the reading experience. The Bible that revolutionized Bradford’s reading wasn’t the radically sparse ESV Reader’s Bible. No, it was the Cambridge Clarion NASB. The Clarion is still a reference edition with chapter-and-verse numbers and cross references, and while the proportions are elegantly balanced, no one is going to mistake it for a large print Bible. Still, the single-column, paragraphed design transformed the experience: Reading the Bible doesn’t seem like work anymore.
Mark Strobel shared a similar story with me. His eleven-year-old son Max started using a single column Crossway Legacy ESV in his Christian Studies class. One night, with the Legacy still at school, Max had to do some reading in a double column thinline. Without any prompting from Dad, he volunteered these observations: “I like my new Bible [the Legacy] because it’s more like reading a book and there’s nothing that gets in the way of what I’m reading. I also like the paper better than this one. It’s thicker.”
“For Max, this is all about reader intuition,” Mark explained. “We haven’t had any conversations about book design and, as far as I know, he isn’t reading your blog under his pillow at night.”
The debate is far from new, as these mid-20th century examples show. The Oxford NEB (left) is a single-column, paragraphed edition, while the Eyre & Spottiswoode Royal Sovereign KJV is as old school as its name suggests.
Of course, for all the benefits of reader-friendly design, the fact is, there people who end up preferring the traditional two-column approach. Jason Engel, whose kindness facilitated our glimpse at the St. John’s Bible, was surprised when his time with the ESV Reader’s Bible didn’t end as expected:
I was really excited to try the ESV Readers Edition, and committed to a month to give it a work-out. About 2 weeks in, I wanted to give up on it, but felt constrained by my personal commitment. At the end of the month, I put it away and haven’t touched it since. It felt really uncomfortable reading from it. I was so relieved to get back into a double-column Bible with chapter/verse numbers and footnotes. Honestly, that response really surprised me.
Jason wasn’t skeptical about reader-friendly design. He was excited to try it. Unlike Bradford and Max, though, the experience didn’t pan out.
The thing is, reader-friendly design doesn’t begin and end with setting text in a single column. Setting text in one column doesn’t automatically make it reader-friendly, and choosing two columns doesn’t ensure unreadability, either. Two-column settings can be reader friendly. Just look at the Schuyler Quentel: by moving the cross-references to the bottom of the page and working hard to find a good ratio between column width and the number of words per line, the team at 2K/Denmark has delivered a very readable two-column reference edition. The text setting of the NIV Proclamation Bible (which I’ll be writing about soon), a favorite layout of mine over the years executed by Blue Heron Bookcraft, is a little more traditional but still congenial for reading. I wouldn’t describe either of these as “reader’s editions” in the purest sense, but they balance the twin objectives of reading and reference in a way that prevents the latter from undermining the former.
Design is a complex process where many different variables must be balanced. There is as much art to it as there is science, and with art you can break all the rules and succeed, just as you can keep them all and fail. At the simplest level, I believe that by designing Bibles to look like the kind of books we read rather than the ones we look things up in, the net effect will be a better reading experience. The intuitive response of readers like Max convinces me this is so. That’s why I’m passionate about the need for pure reader’s editions. We have only scratched the surface where such Bibles are concerned.
But reading isn’t a specialist pursuit. Every book must be as readable as possible, which means traditional reference Bibles can become much more conducive to immersive reading without sacrificing their reference function. The point is, if you’re looking for encouraging signs on the readability front, you can’t limit yourself only to novel-like text editions (as much as I love them). We are actually seeing gains in a variety of formats, single and double column, and while we have a long way to go before the prevailing culture shifts, the effect of reader-friendly design choices is being felt across the board.
Bloomberg: "Dilma É a crise brasileira"
Brazil's president doesn't have many fans right now.
Why Jews are good at money
Anti-Semites claim that Jews are habitually greedy, or even that they stand at the center of a global conspiracy – a caricature later applied to Chinese in the person of Sax Rohmer’s fictional Fu Manchu. But the fact is that Jews historically have been good at money. The Chinese make a great deal of this ability. The English journalist Clarissa Sebag-Montefiore noted this interest inThe New York Times:
“Most Chinese will think Jews are smart, clever or good at making money, and that they have achieved a great deal,” Professor Xu Xin, director of the Institute of Jewish Studies at Nanjing University (one of over half a dozen centers in
China dedicated to studying Judaism) told me …
“Revelations of Jewish People’s Wisdom,” an account on China’s largest microblog site, Sina Weibo, has nearly one-and-a-half-million fans. Its revelations include: “Make a fortune under adverse circumstances.” This logic – that the Jews are admired for their success despite their small numbers and historical oppression – has also led to a burgeoning industry of self-help books that use Jewish culture and the Talmud to preach business tips.
This sort of praise makes Jews uncomfortable, given the ugly history of European Jew-hatred. It shouldn’t. Chinese admiration of Jewish business skills carries no stigma. On the contrary: it begs an explanation.
Jews have a talent for finance, and several reasons are cited for it. During the Middle Ages Jews lent money when the Church forbade usury. Jews were outsiders, often forbidden to own land, who had to develop other means of making a living. In his 2010 book Capitalism and the Jews, Jerry Z. Muller observes that Jews had a towering advantage in international trade during the Middle Ages because they had international law: a judgment issued by a rabbinical court in Yemen would be valid for Jewish merchants on the Rhine.
Something more fundamental,though, explains Jewish success in finance. Like overseas Chinese in Asia or Greeks and Armenians in the old Levant, Jewish immigrants developed trading skills that became embedded in their culture and countless jokes (ask a Jewish first-grader to add two and two, and he’ll reply, “Is that buying or selling?”). But there is nothing uniquely Jewish about this. The same jokes are told about Greeks and Armenians.
Jews distinguished themselves in finance in an entirely different way during the Industrial Revolution. They played a central role in the new government debt markets that followed on the Napoleonic Wars and made a modern economy possible. Bankers like the Rothschild, Mendelssohn, Bleichroeder, Warburg and Seligman families pioneered the new capital markets. Bonds issued by European governments provided a secure long-term store of value, and set a precedent for financing the great projects of the era: railways, canals, and other infrastructure basic to modern industrial society. This was not the mercantile economy of the preceding two centuries, where the basic unit of capital was the bill of exchange in trade, but an industrial economy that required long-term investment to achieve lasting gains in productivity.
Something other than mere trading skills was required for an investment-driven economy, and that was long-term credit, a concept that derives from the Latin credere, “to believe”. It is not an exchange of one peasant’s eggs for another peasant’s barley, or Mexican silver for Chinese silk, but rather a commitment of the savings of whole populations to grand ventures that would pay interest because they drove growth.
Capital markets, moreover, create a kind of democracy. If the whole of society relies on the public debt as a store of value, the value all the savings of society is gauged directly or indirectly against the benchmark of public debt. But that also puts power in the hands of the market: the market has the power to tell the government whether it is doing well or badly, by selling or buying the public debt. It is not simply that the government creates a market that provides convenience and advantages to the people: it becomes dependent on the people’s faith in its policies. When that faith is shaken, as in southern Europe two years ago, and confidence flees the government debt market, the result can be catastrophic. Free capital markets require governments to win the faith of the people.
Capital markets require belief in the viability of investments, trust among counterparties, and faith in the future. It is fitting that Jews created capital markets, because the Jews invented faith. The secret of Jewish success in public finance is to be found in the Jewish encounter with the divine. The gods of the pagan world did not require faith. The gods simply were there, as much as the natural world which they personified. The “god-infested world” (Gilson) of the pagans was simply the natural world as it presented itself to man, with all its arbitrariness and cruelty. To the extent that gods demanded loyalty, it was in their capacity as the patrons of a particular policy protected by that god, for example, Athena in the case of Athens.
Nowhere in the pagan world, though, do we encounter a God who could instruct Abraham to leave his homeland and his father’s house, and betake himself to a place which God would later show him. Never do we meet a God who offers his laws (the Torah) to a people, as YHWH did at Mount Sinai, and ask that people’s free assent to accept these laws. In no other instance do we hear of a single, universal creator God who enters into a covenant of mutual obligations with humans. That is the origin of faith,emunah in Hebrew, meaning loyalty as well as belief. The Jewish concept of emunah implies not only that we conceive something to be true, but that we also must be steadfast in acting according to that truth.
That is the Jewish genius: to be able to inspire faith (or what is usually called “confidence” in markets) to make possible long-term investments in capital markets involving millions of participants. The investors in a bond or stock issue are not linked by ties of family or personal loyalty, but rather by contract, law and custom. Their obligations extend beyond the ancient loyalties of family and clan. That may seem obvious on first reflection. But most countries in the world lack functioning capital markets, because faith is absent. The public does not trust the government to enforce contracts, or the management of a company not to steal money. That is emphatically true in China, which is struggling to create modern capital markets rather than depend on state banks and shadow financing. In backward countries, trust is inconceivable outside the narrow circle of blood relations. Firms remain small because trust is restricted to family members.
That is what Chinese can learn from Jews about business. The Jews have no special aptitude for trading. But we have a special gift for promoting the rule of law and public and private institutions which promote credit, that is, faith in future outcomes and the fair treatment of market participants. In the absence of faith, there never will be enough lawyers to enforce contracts, or policemen to arrest embezzlers, or watchdogs to extirpate government corruption. Something more fundamental is required: a sense that the law is sacred, and if any of us breaks the public trust, all of us are damaged. Our rabbis of antiquity said, “All of Israel stands surety for each other.”
Adam Smith’s invisible hand isn’t enough. Capital markets require more than the interaction of self-interested individuals: they require a common sense of the sanctity of covenant, of mutual obligations between government and people, and between one individual and the next. That is why the United States of America is the most successful nation in economic history. It was founded by devout Christians who hoped to construct a new nation in emulation of ancient Israel.
Jews are no longer particularly prominent in banking, to be sure. Israelis are more interested in the frontiers of technology than in financial markets. The paranoid perception of a Jewish banking conspiracy has faded because so few of the old Jewish banking houses are still in business. The ones that remain, like Rothschild, have little influence. But the Jewish idea that contributed to modern banking remains as powerful as it was in the past. That should be of special interest to the Chinese.
Spengler is channeled by David P Goldman. He is Senior Fellow at the London Center for Policy Research and Associate Fellow at the Middle East ForumHis book How Civilizations Die (and why Islam is Dying, Too) was published by Regnery Press in September 2011. A volume of his essays on culture, religion and economics, It’s Not the End of the World – It’s Just the End of You, also appeared that fall, from Van Praag Press.
(Copyright 2015 Asia Times Holdings Ltd. All rights reserved. Please contact us about sales, syndication and republishing.)
O PT agora decidiu — e corro o risco de chegar às lágrimas antes mesmo de terminar meu post — que é a grande vítima da Operação Lava Jato. Numa resolução divulgada na sexta, o Diretório Nacional do Partido já acusou a existência de um grande complô da direita para desestabilizar Dilma e privatizar a Petrobras. Nesta quarta, Rui Falcão, presidente do partido, anunciou a disposição de processar Pedro Barusco em razão das acusações que fez.
Segundo o ex-gerente da Petrobras, que fez acordo de delação premiada, João Vaccari Neto, tesoureiro do PT, levou em propina entre US$ 150 milhões e US$ 200 milhões. “Nós vamos fazer uma interpelação cível e criminal contra esse bandido Pedro José Barusco Filho”, disse Falcão, na sede do diretório nacional do PT, em São Paulo. Segundo o petista, o mesmo acontecerá com qualquer outro que envolver o nome do partido sem provas. Entendi.
Não custa lembrar que acordos de delação premiada só trazem os benefícios esperados se o denunciante prova o que diz. O mais curioso é que, em outra frente, petistas tentam demonstrar que tudo começou — eles adoram isso! — no governo FHC. Em um dos depoimentos, Barusco afirmou que começou a receber propina em 1997.
Sob o comando de Falcão, o líder do PT na Câmara, o folclórico Sibá Machado (AC), e um grupo de deputados do partido entregaram nesta quarta à vice-procuradora-geral da República, Ela Wiecko, um pedido para que o Ministério Público investigue casos de corrupção e o pagamento de propina na Petrobras no período em que FHC era presidente (1994-2002). Diz o texto: “Causa surpresa e estupefação que, diante da extrema gravidade dos fatos relatados pelo declarante – que revelam a origem do pagamento de propinas na Petrobras – não tenha havido qualquer indagação nem aprofundamento pelos agentes de investigação”. Os deputados acusam ainda o MP de fazer uma “investigação dirigida e ilegal”, com fins “político-partidários”. Entendi: quando Barusco diz que Vaccari recebeu até US$ 200 milhões, é um bandido; quando fala que começou a receber propina em 1997, aí se torna uma pessoa confiável.
Não conheço ninguém que acredite que a corrupção na Petrobras começou com o governo do PT. É claro que não! O que é evidente é que ela se tornou um sistema nos governos petistas.
Na entrevista, Falcão ainda voltou à ladainha da suposta campanha golpista… No fim de 2005 e 2006, na esteira do mensalão, essa bobagem ainda colou. Hoje em dia, revela-se como aquilo que é e que sempre foi: uma pantomima grotesca.
Falcão deveria ser mais econômico em sua fúria planejada. Ele concedeu a entrevista antes que se divulgasse a notícia de que Ricardo Pessoa fez acordo de delação premiada. Vai saber o que sai dali, não é mesmo, Falcão?
Ricardo Pessoa, dono da construtora baiana UTC e apontado pelo Ministério Público e pela Polícia Federal como coordenador de um suposto Clube das Empreiteiras, também fez acordo de delação premiada. Era a adesão mais aguardada ao procedimento. Não custa lembrar que, entre as testemunhas de defesa arroladas por Pessoa estão Jaques Wagner, ex-governador da Bahia e atual ministro da Defesa, José De Filippi Jr., ex-tesoureiro das campanhas de Lula e Dilma e atual secretário de Saúde da prefeitura de São Paulo, e Paulo Bernardo, ex-ministro das Comunicações.
Essas testemunhas poderiam sugerir que o empresário está pronto para a briga. Vamos ver se o acordo foi feito mediante concessões à linha de apuração do Ministério Público e da Polícia Federal. O MP quer emplacar a tese do cartel, coisa a que as empreiteiras resistem. O Planalto, obviamente, vê esse esforço com bons olhos porque, de algum modo, transforma a Petrobras e os políticos nos Chapeuzinhos Vermelhos do Lobo Mau. Vocês sabem o que eu penso dessa história: não há santos por ali, mas convenham: o que podia acontecer com os representantes do contratante de obras — a Petrobras — se uma empresa se negasse a pagar a propina? Resposta: nada! E o que podia acontecer com a empresa se não soltasse a grana? Bem, ela ficava fora do jogo — e não apenas do jogo que se jogava na estatal.
Ricardo Pessoa é amigo pessoal de Lula e conhece de perto o PT. Entre os empresários, foi quem chegou mais perto da política ao tratar da bandalheira. Consta que está revoltado. Em um manuscrito de sua autoria, revelado pela revista VEJA, o empresário deixa claro que o escândalo que veio à luz é de natureza política. Não se trata apenas de um conluio de empresas assaltando o erário. Nas entrelinhas, fica claro que o coordenador da festa é o PT. Tanto é assim que o autor afirma que o Edinho Silva, tesoureiro da campanha de Dilma, está “preocupadíssimo”.
Está escrito lá: “Edinho Silva está preocupadíssimo. Todas as empreiteiras acusadas de esquema criminoso da Operação Lava-Jato doaram para a campanha de Dilma. Será que falarão sobre vinculação campanha x obras da Petrobras?”. Segundo Pessoa, a bandalheira que passou pela diretoria de Paulo Roberto Costa é “fichinha” perto de outros negócios da Petrobras que também teriam servido à coleta de propina.
Pois é… Um réu só chama como testemunha de defesa indivíduos que ele acredita possam fazer depoimentos que lhe sejam favoráveis. Então juntemos as duas pontas: Pessoa deixa claro que o esquema de corrupção é muito maior do que aquele que se investiga até agora, diz que o tesoureiro da campanha de Dilma está preocupado e, em seguida, chama uma penca de petistas para falar em seu favor. A UTC, uma empreiteira baiana, chamou Jaques Wagner, ex-governador da… Bahia! Wagner é aquele que levou para seu governo José Sérgio Grabrielli, o baiano que presidiu a Petrobras durante o período da esbórnia.
Como já afirmei aqui, dois mais dois continuam a ser quatro mesmo no governo do PT. Segundo uma fonte que conhece bem a investigação, vamos ver se a coisa se confirma, o depoimento de Pessoa “arrebenta com o governo”. Ele está magoado. Desde que caiu em desgraça, Lula, seu amigão, o deixou de lado.