Shared posts

26 Aug 04:00

Linear Regression

The 95% confidence interval suggests Rexthor's dog could also be a cat, or possibly a teapot.
25 Aug 02:53

Douglas added 'Till We Have Faces'

Till We Have Faces by C.S. Lewis Douglas gave 5 stars to Till We Have Faces (Paperback) by C.S. Lewis
bookshelves: fiction
Stupendous. World class. Top drawer.

Finished an audio version of it in August of 2016. I have read this a total of three times. Once when I was young, and I didn't like it. The second time was in 2003, and I thought it was great. This time, and greater still.
24 Aug 17:14

Leonardo commented on Leonardo's review of Cornelius Van Til: o homem e o mito

New comment on Leonardo's review of Cornelius Van Til: o homem e o mito
by John W. Robbins

Existem alguns autores que ao fazer uma análise crítica de outro autor que discordem, geram respeito e reflexão nos leitores; este não é o caso de Robbins! Alguns fatores me fizeram dar nota 2 (e não nota 1): 1) A capa é bonita. 2) Eu concordo com a crítica dele ao estilo e a forma de comunicação de Van Til. 3) Eu gosto da editora.
Para mim, esses são os únicos pontos positivos da obra.
Minhas críticas são: 1) Seu estilo irônico, zombador e desrespeitoso em alguns momentos. 2) A citação de textos de Van Til soltos e fora de contexto para criticar uma doutrina. 3) A má compreensão de Van Til em alguns pontos, principalmente quanto sua visão do conhecimento de Deus. 4) Ao criticar alguns de serem "fans" do "mito" Van Til, ele demonstra ser a mesmo coisa com relação a Clark. 5) Sua análise e critica superficial de Van Til. Creio que o próprio Clark desaprovaria essa obra.

A tonalidade da escrita se assemelha a uma criança birrenta defendendo seu super herói favorito.
18 Aug 17:35

Podemos Perder a Salvação em Cristo?

by Solano Portela
Hebreus 6.4-8

Uma pessoa me perguntou se Hebreus 6.4-6anularia a possibilidade de algum ex-cristão arrependido voltar a trilhar os caminhos do Senhor? A dúvida era: “A quem este trecho se refere?”. Uma outra questão, que surge com este texto, é a possibilidade de perda da salvação. Certamente já ouvimos muitas pessoas afirmarem que podemos perde-la. Na realidade, alguns chamam a certeza da salvação, “o orgulho dos Crentes”.

Essa noção está presente em algumas denominações de orientação teológica arminiana. Vários valorizam a manutenção de uma insegurança por “necessidade de preservar a vida cristã”.  Em meu entendimento, e assim ensina a teologia da Reforma, isso representa uma falta de compreensão do que a Bíblia ensina sobre a doutrina da salvação. No entanto, alguns reformados se surpreendem quando ouvem que a Confissão de Fé de Westminster apresenta a “certeza da salvação” como sendo algo adicional e não essencial à essência da fé real (CFW, Cap. 18, 3 e Catecismo Maior, Perguntas 81 e 172 do Catecismo Maior da CFW). A questão, então, não é se existe ou não “a certeza subjetiva”, ou pessoal, mas se o verdadeiramente salvo pode perder essa salvação. Afinal, um dos pontos principais da teologia bíblica, e da Reforma, é a Perseverança dos Santos.

Textos como Hb 6.4-6 podem nos deixar um pouco confusos, se forem estudados superficialmente, fora do contexto geral da Palavra de Deus. Precisamos, portanto, procurar entender algumas passagens bíblicas que parecem ir em sentido contrário à doutrina da Perseverança dos Santos.


1.     Duas passagens difíceis: Hebreus 6.4-6 e 2 Pedro 2.20-22 são dois trechos de difícil compreensão, mas analisemos cada um deles.


Alguns “provam o dom celestial” e caem, nos diz Hb 6.4-6: (4) É impossível, pois, que aqueles que uma vez foram iluminados, e provaram o dom celestial, e se tornaram participantes do Espírito Santo, (5) e provaram a boa palavra de Deus e os poderes do mundo vindouro,(6) e caíram, sim, é impossível outra vez renová-los para arrependimento, visto que, de novo, estão crucificando para si mesmos o Filho de Deus e expondo-o à ignomínia.


Alguns “escapam” do mundo, “conhecem”, mas voltam ao erro, nos diz 2 Pe 2.20-22: (20)  Portanto, se, depois de terem escapado das contaminações do mundo mediante o conhecimento do Senhor e Salvador Jesus Cristo, se deixam enredar de novo e são vencidos, tornou-se o seu último estado pior que o primeiro. (21) Pois melhor lhes fora nunca tivessem conhecido o caminho da justiça do que, após conhecê-lo, volverem para trás, apartando-se do santo mandamento que lhes fora dado. (22) Com eles aconteceu o que diz certo adágio verdadeiro: O cão voltou ao seu próprio vômito; e: A porca lavada voltou a revolver-se no lamaçal.


2.     As Exposição de João sobre o assunto – Precisamos estar cientes da exposição sobre a salvação que o apóstolo João faz no capítulo 10.26-28, do seu Evangelho e em suas três cartas universais, antes de abordar estes versos difíceis.


O Espírito Santo moveu João a registrar nesse trecho (João 10.26-28) o fundamento da doutrina da Perseverança dos Santos - “Ninguém as arrebatará”: (26) Mas vós não credes, porque não sois das minhas ovelhas. (27) As minhas ovelhas ouvem a minha voz; eu as conheço, e elas me seguem. (28) Eu lhes dou a vida eterna; jamais perecerão, e ninguém as arrebatará da minha mão. Somos seguros nas mãos de Cristo não por nosso próprio poder, mas pelo poder de Deus. Salvação não é apenas um ato de vontade nossa, mas a vontade é um reflexo e resposta à obra de Cristo na Cruz; de sua vitória sobre a morte, na ressurreição; e ao chamado eficaz do Espírito Santo em nossos corações.


Na sua primeira carta, João deixa claro que existem pessoas agregadas ao povo de Deus, mas que nunca fizeram parte dos verdadeiramente salvos pelo poder de Cristo. “Saíram de nós, mas não eram dos nossos”, diz ele. Em 1 João 2.18-20, temos: (18) Filhinhos, já é a última hora; e, como ouvistes que vem o anticristo, também, agora, muitos anticristos têm surgido; pelo que conhecemos que é a última hora. (19) Eles saíram de nosso meio; entretanto, não eram dos nossos; porque, se tivessem sido dos nossos, teriam permanecido conosco; todavia, eles se foram para que ficasse manifesto que nenhum deles é dos nossos. (20) E vós possuís unção que vem do Santo e todos tendes conhecimento.


Em sua segunda carta João fala daqueles que não se prendem à doutrina de Cristo. Ele se referia à pessoa que “ultrapassa a doutrina” e diz que esse não tem Deus. Em 2 Jo 9, ele escreve: Todo aquele que ultrapassa a doutrina de Cristo e nela não permanece não tem Deus; o que permanece na doutrina, esse tem tanto o Pai como o Filho. Entendemos que esse “ultrapassar” é o mesmo curso abordado por Paulo em Gálatas 1.8. Significa “ir além”, pregar um “outro evangelho”. A esses Paulo reserva palavras duras. Mesmo estando no meio do Povo de Deus, Paulo alerta seus leitores contra eles, e conclui que qualquer um que “...vos pregue evangelho que vá além do que vos temos pregado, seja anátema”. Ou seja, seja amaldiçoado aquele que ultrapassa a doutrina.


Na terceira carta, João chama atenção para o fato de que uma vida realmente transformada, realmente salva, não permanece no pecado. Aqueles que dizem ser salvos, mas não demonstram transformação de vida (e, infelizmente, sempre existem esses no meio do Povo de Deus), nunca viram a Deus. Em 3 João 11, ele fala dessa permanência na prática do mal:  Amado, não imites o que é mau, senão o que é bom. Aquele que pratica o bem procede de Deus; aquele que pratica o mal jamais viu a Deus.


3.     O contexto das passagens difíceis. Mas voltemos às nossas passagens difíceis, agora examinando o contexto imediato no qual elas são encontradas.


Hebreus 6.4-6, não pode ser lido isolado dos versos 7 e 8. Estes versos dizem: (7) Porque a terra que absorve a chuva que freqüentemente cai sobre ela e produz erva útil para aqueles por quem é também cultivada recebe bênção da parte de Deus;(8) mas, se produz espinhos e abrolhos, é rejeitada e perto está da maldição; e o seu fim é ser queimada.


Vemos nesses dois versos a harmonia do texto em Hebreus, com a parábola do semeador, encontrada em Mateus 13.18-23. O v. 22 é de especial importância. O trecho, em Hebreus, não está falando dos verdadeiramente salvos, mas dos semeados em espinhos. Eles recebem a chuva, igual à que cai na terra fértil, ou seja, “provam o dom”, no sentido de que são participantes das bênçãos, mas são sufocados pelos cuidados do mundo. Muitos aparentam seguir o evangelho; fazem parte dos ajuntamentos e atividades das igrejas; mas o coração não está regenerado. O texto em Hebreus não fala em “perda da salvação”, nem em impossibilidade de um crente cair em pecado e não ter condição de se arrepender, mas daqueles que demonstram, ao longo do tempo, que nunca foram convertidos.


Semelhantemente, 2 Pe 2.20-22, deve ser lido a partir do verso 9. Os que “escapam” do mundo conhecem, mas voltam ao erro, são contrastados com os “piedosos”. O texto de Pedro diz: (9)Porque o Senhor sabe livrar da provação os piedosos e reservar sob castigo os injustos, para o dia do juízo; (10) especialmente aqueles que, seguindo a carne, andam em imundas paixões e menosprezam qualquer governo. Atrevidos, arrogantes, não temem difamar autoridades superiores, (11) ao passo que anjos, embora maiores em força e poder, não proferem contra elas juízo infamante na presença do Senhor. (12) Esses, todavia, como brutos irracionais, naturalmente feitos para presa e destruição, falando mal daquilo em que são ignorantes, na sua destruição também hão de ser destruídos, (13) recebendo injustiça por salário da injustiça que praticam. Considerando como prazer a sua luxúria carnal em pleno dia, quais nódoas e deformidades, eles se regalam nas suas próprias mistificações, enquanto banqueteiam junto convosco; (14) tendo os olhos cheios de adultério e insaciáveis no pecado, engodando almas inconstantes, tendo coração exercitado na avareza, filhos malditos; (15) abandonando o reto caminho, se extraviaram, seguindo pelo caminho de Balaão, filho de Beor, que amou o prêmio da injustiça (16) (recebeu, porém, castigo da sua transgressão, a saber, um mudo animal de carga, falando com voz humana, refreou a insensatez do profeta). (17) Esses tais são como fonte sem água, como névoas impelidas por temporal. Para eles está reservada a negridão das trevas;(18) porquanto, proferindo palavras jactanciosas de vaidade, engodam com paixões carnais, por suas libertinagens, aqueles que estavam prestes a fugir dos que andam no erro, (19) prometendo-lhes liberdade, quando eles mesmos são escravos da corrupção, pois aquele que é vencido fica escravo do vencedor.


O trecho claramente fala de ímpios. Eles conheceram o caminho da justiça, porque aderiram fisicamente à igreja, relacionaram-se com o Povo de Deus, ouviram incontáveis exposições da Palavra. No entanto, em seu espírito e em suas obras, nunca experimentaram conversão. Por isso, nos versos 20 a 22, eles são comparados a porcos que voltam ao vômito. Viviam em um clima limpo, eivado de ensinamentos proveitosos à vida. Levam sobre si a condenação de rejeitarem a tudo isso e ao Senhor da Glória.


Conclusão:

A Palavra de Deus deixa claro, em muitas passagens, que somos salvos para sempre. Por isso Paulo ensina em Romanos que Aquele que iniciou a obra em nós é poderoso para completá-la (Filipenses 1.6). Obviamente, isso não é encorajamento para o pecado, mas motivo de ações de graças – somos salvos pelo poder de Deus e preservados por este  mesmo poder, não por nossas frágeis forças.


Os dois trechos que examinamos ainda que difíceis, são entendidos pelas explicações de João e pelo contexto imediato das passagens. Devemos confiar no preservador da nossa salvação e nunca devemos nos abalar ou permitir que dúvidas sejam colocadas em nossa cabeça.



Solano Portela

10 Aug 21:20

FSF Blogs: Support the Libre Tea Computer Card, a candidate for Respects Your Freedom certification

They write:

"Now imagine if you owned a computing device that you could easily fix yourself and inexpensively upgrade as needed. So, instead of having to shell out for a completely new computer, you could simply spend around US$50 to upgrade — which, by the way, you could easily do in SECONDS, by pushing a button on the side of your device and just popping in a new computer card. Doesn’t that sound like the way it should be?"

This project certainly sounds appealing, but only if the computer hardware is designed and configured to run software that does as much as possible to respect your freedom and ensure your control over your device. Fortunately, one option you have when backing this project is to purchase a Libre Tea Computer Card. After working closely with the developers and reviewing a sample test board, we are confident that their plans are to create a device that can achieve our Respects Your Freedom (RYF) certification. And because the project is running their crowdfunding on Crowd Supply, users can financially support them anonymously and without the use of proprietary Javascript.

The project is being developed by Luke Kenneth Casson Leighton of Rhombus-Tech and is sponsored by Christopher Waid of ThinkPenguin, a company that sells multiple RYF-certified hardware products. It is exciting to see passionate free software advocates in our community working with OEMs to produce a computer hardware product capable of achieving RYF certification. We hope that this is the first of many computing systems they are able to design and build that respect your freedom.

The Libre Tea Computer Card is built with an Allwinner A20 dual core processor configured to use the main CPU for graphics; it has 2 GB of RAM and 8 GB of NAND Flash; and it will come pre-installed with Parabola GNU/Linux-libre, an FSF-endorsed fully-free operating system.

We encourage you to back the Libre Tea Computer Card. We'll have to do another evaluation once it is actually produced to be sure it meets our certification standards, but we have high hopes. Their funding deadline is August 26th, so don't delay!

09 Aug 18:56

The People’s Code (White House blog)

by ris
US Chief Information Officer Tony Scott introduces the Federal Source Code Policy, on the White House blog. "By making source code available for sharing and re-use across Federal agencies, we can avoid duplicative custom software purchases and promote innovation and collaboration across Federal agencies. By opening more of our code to the brightest minds inside and outside of government, we can enable them to work together to ensure that the code is reliable and effective in furthering our national objectives. And we can do all of this while remaining consistent with the Federal Government’s long-standing policy of technology neutrality, through which we seek to ensure that Federal investments in IT are merit-based, improve the performance of our government, and create value for the American people." (Thanks to David A. Wheeler)
02 Aug 15:25

Simon Riggs: Thoughts on Uber’s List of Postgres Limitations

An Uber technical blog of July 2016 described the perception of “many Postgres limitations”. Regrettably, a number of important technical points are either not correct or not wholly correct because they overlook many optimizations in PostgreSQL that were added specifically to address the cases discussed. In most cases, those limitations were actually true in the distant past of 5-10 years ago, so that leaves us with the impression of comparing MySQL as it is now with PostgreSQL as it was a decade ago. This is no doubt because the post was actually written some time/years? ago and only recently published.

This document looks in detail at those points to ensure we have detailed information available for a wider audience, so nobody is confused by PostgreSQL’s capabilities.

Having said that, I very much welcome the raising of those points and also wish to show that the PostgreSQL project and 2ndQuadrant are responsive to feedback. To do this, detailed follow-ups are noted for immediate action.

These points were noted in the blog
* Poor replica MVCC support
* Inefficient architecture for writes
* Inefficient data replication
* Difficulty upgrading to newer releases

Poor replica MVCC support

“If a streaming replica has an open transaction, updates to the database are blocked if they affect rows held open by the transaction. In this situation, Postgres pauses the WAL application thread until the transaction has ended.”

This is true, though misses the point that a parameter exists to control that behaviour, so that when
hot_standby_feedback = on
the described behaviour does not occur in normal circumstances. This is supported from PostgreSQL 9.1 (2011) and above. If you’re not using it, please consider doing so.

Later, this comment leads to the conclusion “Postgres replicas … can’t implement MVCC” which is wholly incorrect and a major misunderstanding. PostgreSQL replicas certainly allow access to data with full MVCC semantics.

Inefficient architecture for writes

“If old transactions need to reference a row for the purposes of MVCC MySQL copies the old row into a special area called the rollback segment.”

“This design also makes vacuuming and compaction more efficient. All of the rows that are eligible to be vacuumed are available directly in the rollback segment. By comparison, the Postgres autovacuum process has to do full table scans to identify deleted rows.”

Moving old rows to a rollback segment adds time to the write path for UPDATEs, but that point isn’t mentioned. PostgreSQL is more efficient architecture for writes in relation to MVCC because it doesn’t need to do as many push-ups.

Later, if the workload requires that we access old rows from the rollback segment that is also more expensive. That is not always needed, yet it is very common for longer running queries to need to access older data. However, if all transactions are roughly the same short duration access to the rollback segment is seldom needed, which just happens to make benchmark results appear good while real-world applications suffer.

By contrast, PostgreSQL has multiple optimizations that improve vacuuming and compaction. First, an optimization called HOT improves vacuuming in heavily updated parts of a table (since 2007), while the visibility map ensures that VACUUM can avoid full table scans (since 2008).

Whether rollback segments help or hinder an application depend on the specific use case and it’s much more complex than this first appears.

Next, we discuss indexes…

“With Postgres, the primary index and secondary indexes all point directly to the on-disk tuple offsets.”

This point is correct; PostgreSQL indexes currently use a direct pointer between the index entry and the heap tuple version. InnoDB secondary indexes are “indirect indexes” in that they do not refer to the heap tuple version directly, they contain the value of the Primary Key (PK) of the tuple.

Comparing direct and indirect indexes we see
* direct indexes have links that go index → heap
* indirect indexes have links that go index → PK index → heap

Indirect indexes store the PK values of the rows they index, so if the PK columns are wide or contain multiple columns the index will use significantly more disk space than a direct index, making them even less efficient for both read and write (as stated in MySQL docs). Also indirect indexes have index search time >=2 times worse than direct indexes, which slows down both reads (SELECTs) and searched writes (UPDATEs and DELETEs).
Performance that is >=100% slower is understated as just a “slight disadvantage” [of MySQL].

“When a tuple location changes, all indexes must be updated.”

This is misleading, since it ignores the important Heap Only Tuple (HOT) optimization that was introduced in PostgreSQL 8.3 in 2007. The HOT optimization means that in the common case, a new row version does not require any new index entries, a point which effectively nullifies the various conclusions that are drawn from it regarding both inefficiency of writes and inefficiency of the replication protocol.

“However, these indexes still must be updated with the creation of a new row tuple in the database for the row record. For tables with a large number of secondary indexes, these superfluous steps can cause enormous inefficiencies.”

As a result of ignoring the HOT optimization this description appears to discuss the common case, rather than the uncommon case. It is currently true that for direct indexes if any one of the indexed columns change then new index pointers are required for all indexes. It seems possible for PostgreSQL to optimize this further and I’ve come up with various designs and will be looking to implement this best fairly soon.

Although they have a higher read overhead, indirect indexes have the useful property that if a table has multiple secondary indexes then an update of one secondary index does not affect the other secondary indexes if their column values remain unchanged. This makes indirect indexes useful only for the case where an application needs indexes that would be infrequently used for read, yet with a high update rate that does not touch those columns.

Thus, it is possible to construct cases in which PostgreSQL consistently beats InnoDB, or vice versa. In the “common case” PostgreSQL beats InnoDB on reads and is roughly equal on writes for btree access. What we should note is that PostgreSQL has the widest selection of index types of any database system and this is an area of strength, not weakness.

The current architecture of PostgreSQL is that all index types are “direct”, whereas in InnoDB primary indexes are “direct” and secondary indexes “indirect”. There is no inherent architectural limitation that prevents PostgreSQL from also using indirect indexes, though it is true that has not been added yet.

We’ve done a short feasibility study and it appears straightforward to implement indirect indexes for PostgreSQL, as an option at create index time. We will pursue this if the HOT optimizations discussed above aren’t as useful or possible, giving us a second approach for further optimization. Additional index optimizations have also been suggested.

Inefficient data replication

“However, the verbosity of the Postgres replication protocol can still cause an overwhelming amount of data for a database that uses a lot of indexes.”

Again, these comments discuss MySQL replication which can be characterized as Logical Replication. PostgreSQL provides both physical and logical replication. All of the benefits discussed for MySQL replication are shared by PostgreSQL’s logical replication. There are also benefits for physical replication in many cases, which is why PostgreSQL provides both logical and physical replication as options.

PostgreSQL physical replication protocol itself is not verbose – this comment is roughly the same as the “inefficient writes” discussion: if PostgreSQL optimizes away index updates then they do not generate any entries in the transaction log (WAL), so there is no inefficiency. Also, the comment doesn’t actually say what we mean by “overwhelming”. What this discussion doesn’t consider is the performance of replication apply. Physical replication is faster than logical replication because including the index pointers in the replication stream allows us to insert them directly into the index, rather than needing to search the index for the right point for insertion. Including the index pointers actually increases not decreases performance, even though the replication bandwidth requirement is higher.

PostgreSQL Logical Replication is available via 2ndQuadrant’s pglogical and will be available in PostgreSQL 10.0 in core.

MySQL “Statement-based replication is usually the most compact but can require replicas to apply expensive statements to update small amounts of data. On the other hand, row-based replication, akin to the Postgres WAL replication, is more verbose but results in more predictable and efficient updates on the replicas.”

Yes, statement-based replication is more efficient in terms of bandwidth, but even less efficient in terms of the performance of applying changes to receiving servers. Most importantly, it leads to various problems and in various cases replication may not work as expected, involving developers in diagnosing operational problems. PostgreSQL probably won’t adopt statement-based replication.

Difficulty upgrading to newer releases

“the basic design of the on-disk representation in 9.2 hasn’t changed significantly since at least the Postgres 8.3 release (now nearly 10 years old).”

This is described as if it were a bad thing, but actually it’s a good thing and is what allows major version upgrades to occur quickly without unloading and reloading data.

“We started out with Postgres 9.1 and successfully completed the upgrade process to move to Postgres 9.2. However, the process took so many hours that we couldn’t afford to do the process again. By the time Postgres 9.3 came out, Uber’s growth increased our dataset substantially, so the upgrade would have been even lengthier.”

The pg_upgrade -k option provides an easy and effective upgrade mechanism. Pg_upgrade does require some downtime, which is why 2ndQuadrant has been actively writing logical replication for some years, focusing on zero-downtime upgrade.

Although the logical replication upgrade is only currently available from 9.4 to 9.5, 9.4 to 9.6 and 9.5 to 9.6, there is more good news coming. 2ndQuadrant is working on highly efficient upgrades from earlier major releases, starting with 9.1 → 9.5/9.6. When PostgreSQL 9.1 is desupported later in 2016 this will allow people using 9.1 to upgrade to the latest versions. This is available as a private service, so if you need zero-downtime upgrade from 9.1 upwards please get in touch.

In 2017, upgrades from 9.2 and 9.3 will also be supported, allowing everybody to upgrade efficiently with zero-downtime prior to the de-supporting of those versions.

28 Jul 22:00

Markus Winand: On Uber’s Choice of Databases

A few days ago Uber published the article “Why Uber Engineering Switched from Postgres to MySQL”. I didn’t read the article right away because my inner nerd told me to do some home improvements instead. While doing so my mailbox was filling up with questions like “Is PostgreSQL really that lousy?”. Knowing that PostgreSQL is not generally lousy, these messages made me wonder what the heck is written in this article. This post is an attempt to make sense out of Uber’s article.

In my opinion Uber’s article basically says that they found MySQL to be a better fit for their environment as PostgreSQL. However, the article does a lousy job to transport this message. Instead of writing “PostgreSQL has some limitations for update-heavy use-cases” the article just says “Inefficient architecture for writes,” for example. In case you don’t have an update-heavy use-case, don’t worry about the problems described in Uber’s article.

In this post I’ll explain why I think Uber’s article must not be taken as general advice about the choice of databases, why MySQL might still be a good fit for Uber, and why success might cause more problems than just scaling the data store.

On UPDATE

The first problem Uber’s article describes in great, yet incomplete detail is that PostgreSQL always needs to update all indexes on a table when updating rows in the table. MySQL with InnoDB, on the other hand, needs to update only those indexes that contain updated columns. The PostgreSQL approach causes more disk IOs for updates that change non-indexed columns (“Write Amplification” in the article). If this is such a big problem to Uber, these updates might be a big part of their overall workload.

However, there is a little bit more speculation possible based upon something that is not written in Uber’s article: The article doesn’t mention PostgreSQL Heap-Only-Tuples (HOT). From the PostgreSQL source, HOT is useful for the special case “where a tuple is repeatedly updated in ways that do not change its indexed columns.” In that case, PostgreSQL is able to do the update without touching any index if the new row-version can be stored in the same page as the previous version. The latter condition can be tuned using the fillfactor setting. Assuming Uber’s Engineering is aware of this means that HOT is no solution to their problem because the updates they run at high frequency affect at least one indexed column.

This assumption is also backed by the following sentence in the article: “if we have a table with a dozen indexes defined on it, an update to a field that is only covered by a single index must be propagated into all 12 indexes to reflect the ctid for the new row”. It explicitly says “only covered by a single index” which is the edge case—just one index—otherwise PostgreSQL’s HOT would solve the problem.

[Side note: I’m genuinely curious whether the number of indexes they have could be reduced—index redesign in my challenge. However, it is perfectly possible that those indexes are used sparingly, yet important when they are used.]

It seems that they are running many updates that change at least one indexed column, but still relatively few indexed columns compared to the “dozen” indexes the table has. If this is a predominate use-case, the article’s argument to use MySQL over PostgreSQL makes sense.

On SELECT

There is one more statement about their use-case that caught my attention: the article explains that MySQL/InnoDB uses clustered indexes and also admits that “This design means that InnoDB is at a slight disadvantage to Postgres when doing a secondary key lookup, since two indexes must be searched with InnoDB compared to just one for Postgres.” I’ve previously written about this problem (“the clustered index penalty”) in context of SQL Server.

What caught my attention is that they describe the clustered index penalty as a “slight disadvantage”. In my opinion, it is a pretty big disadvantage if you run many queries that use secondary indexes. If it is only a slight disadvantage to them, it might suggest that those indexes are used rather seldom. That would mean, they are mostly searching by primary key (then there is no clustered index penalty to pay). Note that I wrote “searching” rather than “selecting”. The reason is that the clustered index penalty affects any statement that has a where clause—not just select. That also implies that the high frequency updates are mostly based on the primary key.

Finally there is another omission that tells me something about their queries: they don’t mention PostgreSQL’s limited ability to do index-only scans. Especially in an update-heavy database, the PostgreSQL implementation of index-only scans is pretty much useless. I’d even say this is the single issue that affects most of my clients. I’ve already blogged about this in 2011. In 2012, PostgreSQL 9.2 got limited support of index-only scans (works only for mostly static data). In 2014 I even raised one aspect of my concern at PgCon. However, Uber doesn’t complain about that. Select speed is not their problem. I guess query speed is generally solved by running the selects on the replicas (see below) and possibly limited by mostly doing primary key side.

By now, their use-case seems to be a better fit for a key/value store. And guess what: InnoDB is a pretty solid and popular key/value store. There are even packages that bundle InnoDB with some (very limited) SQL front-ends: MySQL and MariaDB are the most popular ones, I think. Excuse the sarcasm. But seriously: if you basically need a key/value store and occasionally want to run a simple SQL query, MySQL (or MariaDB) is a reasonable choice. I guess it is at least a better choice than any random NoSQL key/value store that just started offering an even more limited SQL-ish query language. Uber, on the other hand just builds their own thing (“Schemaless”) on top of InnoDB and MySQL.

On Index Rebalancing

One last note about how the article describes indexing: it uses the word “rebalancing” in context of B-tree indexes. It even links to a Wikipedia article on “Rebalancing after deletion.” Unfortunately, the Wikipedia article doesn’t generally apply to database indexes because the algorithm described on Wikipedia maintains the requirement that each node has to be at least half-full. To improve concurrency, PostgreSQL uses the Lehman, Yao variation of B-trees, which lifts this requirement and thus allows sparse indexes. As a side note, PostgreSQL still removes empty pages from the index (see slide 15 of “Indexing Internals”). However, this is really just a side issue.

What really worries me is this sentence: “An essential aspect of B-trees are that they must be periodically rebalanced, …” Here I’d like to clarify that this is not a periodic process one that runs every day. The index balance is maintained with every single index change (even worse, hmm?). But the article continues “…and these rebalancing operations can completely change the structure of the tree as sub-trees are moved to new on-disk locations.” If you now think that the “rebalancing” involves a lot of data moving, you misunderstood it.

The important operation in a B-tree is the node split. As you might guess, a node split takes place when a node cannot host a new entry that belongs into this node. To give you a ballpark figure, this might happen once for about 100 inserts. The node split allocates a new node, moves half of the entries to the new node and connects the new node to the previous, next and parent nodes. This is where Lehman, Yao save a lot of locking. In some cases, the new node cannot be added to the parent node straight away because the parent node doesn’t have enough space for the new child entry. In this case, the parent node is split and everything repeats.

In the worst case, the splitting bubbles up to the root node, which will then be split as well and a new root node will be put above it. Only in this case, a B-tree ever becomes deeper. Note that a root node split effectively shifts the whole tree down and therefore keeps the balance. However, this doesn’t involve a lot of data moving. In the worst case, it might touch three nodes on each level and the new root node. To be explicit: most real world indexes have no more than 5 levels. To be even more explicit: the worst case—root node split—might happen about five times for a billion inserts. On the other cases it will not need to go the whole tree up. After all, index maintenance is not “periodic”, not even very frequent, and is never completely changing the structure of the tree. At least not physically on disk.

On Physical Replication

That brings me to the next major concern the article raises about PostgreSQL: physical replication. The reason the article even touches the index “rebalancing” topic is that Uber once hit a PostgreSQL replication bug that caused data corruption on the downstream servers (the bug “only affected certain releases of Postgres 9.2 and has been fixed for a long time now”).

Because PostgreSQL 9.2 only offers physical replication in core, a replication bug “can cause large parts of the tree to become completely invalid.” To elaborate: if a node split is replicated incorrectly so that it doesn’t point to the right child nodes anymore, this sub-tree is invalid. This is absolutely true—like any other “if there is a bug, bad things happen” statement. You don’t need to change a lot of data to break a tree structure: a single bad pointer is enough.

The Uber article mentions other issues with physical replication: huge replication traffic—partly due to the write amplification caused by updates—and the downtime required to update to new PostgreSQL versions. While the first one makes sense to me, I really cannot comment on the second one (but there were some statements on the PostgreSQL-hackers mailing list).

Finally, the article also claims that “Postgres does not have true replica MVCC support.” Luckily the article links to the PostgreSQL documentation where this problem (and remediations) are explained. The problem is basically that the master doesn’t know what the replicas are doing and might thus delete data that is still required on a replica to complete a query.

According to the PostgreSQL documentation, there are two ways to cope with this issue: (1) delaying the application of the replication stream for a configurable timeout so the read transaction gets a chance to complete. If a query doesn’t finish in time, kill the query and continue applying the replication stream. (2) configure the replicas to send feedback to the master about the queries they are running so that the master does not vacuum row versions still needed by any slave. Uber’s article rules the first option out and doesn’t mention the second one at all. Instead the article blames the Uber developers.

On Developers

To quote it in all its glory: “For instance, say a developer has some code that has to email a receipt to a user. Depending on how it’s written, the code may implicitly have a database transaction that’s held open until after the email finishes sending. While it’s always bad form to let your code hold open database transactions while performing unrelated blocking I/O, the reality is that most engineers are not database experts and may not always understand this problem, especially when using an ORM that obscures low-level details like open transactions.”

Unfortunately, I understand and even agree with this argument. Instead of “most engineers are not database experts” I’d even say that most developers have very little understanding of databases because every developer that touches SQL needs know about transactions—not just database experts.

Giving SQL training to developers is my main business. I do it at companies of all sizes. If there is one thing I can say for sure is that the knowledge about SQL is ridiculously low. In context of the “open transaction” problem just mentioned I can conform that hardly any developer even knows that read only transactions are a real thing. Most developers just know that transactions can be used to back out writes. I’ve encountered this misunderstanding often enough that I’ve prepared slides to explain it and I just uploaded these slides for the curious reader.

On Success

This leads me to the last problem I’d like to write about: the more people a company hires, the closer their qualification will be to the average. To exaggerate, if you hire the whole planet, you’ll have the exact average. Hiring more people really just increases the sample size.

The two ways to beat the odds are: (1) Only hire the best. The difficult part with this approach is to wait if no above-average candidates are available; (2) Hire the average and train them on the job. This needs a pretty long warm-up period for the new staff and might also bind existing staff for the training. The problem with both approaches is that they take time. If you don’t have time—because your business is rapidly growing—you have to take the average, which doesn’t know a lot about databases (empirical data from 2014). In other words: for a rapidly growing company, technology is easier to change than people.

The success factor also affects the technology stack as requirements change over time. At an early stage, start-ups need out-of-the-box technology that is immediately available and flexible enough to be used for their business. SQL is a good choice here because it is actually flexible (you can query your data in any way) and it is easy to find people knowing SQL at least a little bit. Great, let’s get started! And for many—probably most—companies, the story ends here. Even if they become moderately successful and their business grows, they might still stay well within the limits of SQL databases forever. Not so for Uber.

A few lucky start-ups eventually outgrow SQL. By the time that happens, they have access to way more (virtually unlimited?) resources and then…something wonderful happens: They realize that they can solve many problems if they replace their general purpose database by a system they develop just for their very own use-case. This is the moment a new NoSQL database is born. At Uber, they call it Schemaless.

On Uber’s Choice of Databases

By now, I believe Uber did not replace PostgreSQL by MySQL as their article suggests. It seems that they actually replaced PostgreSQL by their tailor-made solution, which happens to be backed by MySQL/InnoDB (at the moment).

It seems that the article just explains why MySQL/InnoDB is a better backend for Schemaless than PostgreSQL. For those of you using Schemaless, take their advice! Unfortunately, the article doesn’t make this very clear because it doesn’t mention how their requirements changed with the introduction of Schemaless compared to 2013, when they migrated from MySQL to PostgreSQL.

Sadly, the only thing that sticks in the reader’s mind is that PostgreSQL is lousy.

If you like my way of explaining things, you’ll love my book.

Original title and author: “On Uber’s Choice of Databases” by Markus Winand.

28 Jun 23:00

The Software Heritage intiative: A comprehensive archive of Free Software code

Software Heritage initiative to create an archive of Free Software code

The Free Software Foundation Europe protects users, companies and institutions from technological abuse by promoting the use of Free Software. Now there is a project that protects the code used in Free Software itself and promised to preserve it for the future: Inria presents the Software Heritage initiative.

The importance of software in the modern world cannot be overstated. Software is at the crux of all contemporary technological development and has become essential for all areas of scientific research. Software plays a pivotal role in our daily lives, our industries and our society. Software has become the reflection of our technological, scientific and cultural progress.

However, software is prone to disappear, either because it stops being profitable, or projects get cancelled, or the code is deemed obsolete and gets erased, or is left to fade on storage that physically degrades over time.

The Software Heritage initiative is created and funded by Inria. It collects programs, applications and snippets of code distributed under free licenses from a wide variety of active and defunct sources, its aim being to protect code from sinking into oblivion. The distributed and redundant back-end hardens the system against a potentially disastrous losses of data and guarantees its availability for users.

Users can check if a certain file exists within the system and propose new sources the Software Heritage engine can explore in search of more code to store. Soon users will also be able to find out where the code originated from using the Provenance information feature, browse the stored code, run full-text searches on all files, and download the content.

The Heritage aimes to store all Free Software, in other words, software that can be used, studied, adapted and shared freely with others; and this is because the Software Heritage initiative relies on being able to share the software it stores. The Software Heritage website is designed to be a useful tool for professionals, scientists, educators and end-users. Users must be allowed to re-use the code in other products, cutting development time and costs; engineers should be able to discover how others solved certain problems; or compare the efficiency of different solutions to the same problem. And, of course, researchers must have explicit permission to study the evolution of code over time. This is only possible if the code is distributed under a Free and Open Source license.

Matthias Kirschner, President of the Free Software Foundation Europe, says: "Software is the most important cultural technology of today's society; it frames what we can and what we cannot do. Software shapes our communication and culture, our economy, education and research, as well as politics. It is important to preserve our collective knowledge about how software has influenced humankind. Collecting source code makes Software Heritage a valuable resource to understand how our society worked at any given time, and to build upon knowledge from humankind."

The Software Heritage intiative ensures today's code will be around for everybody in the future.

About Inria

Inria, the French National Institute for computer science and applied mathematics, promotes "scientific excellence for technology transfer and society". Graduates from the world's top universities, Inria's 2,700 employees rise to the challenges of digital sciences. With this open, agile model, Inria is able to explore original approaches with its partners in industry and academia and provide an efficient response to the multidisciplinary and application challenges of the digital transformation. Inria transfers expertise and research results to companies (startups, SMEs and major groups) in fields as diverse as healthcare, transport, energy, communications, security and privacy protection, smart cities and the factory of the future.

Support FSFE, join the Fellowship
Make a one time donation

01 Jul 04:00

Speed and Danger

NASCAR removed the passenger seats because drivers hated how astronauts kept riding along with them and loudly announcing "Ahh, what a nice and relaxing drive."
08 Jun 04:00

Optimization

Premature optimization is the root of all evil, so to start this project I'd better come up with a system that can determine whether a possible optimization is premature or not.
06 Jun 17:35

Federico Campoli: Another evening with PostgreSQL

Thanks to the Ferrara's University, departement of Engineering I'll attend to another evening with PostgreSQL.

This time the event will be a seminary with all talks made by me.

The first talk will cover our beloved elephant's history.

The second talk will look to the engine with  particular attention to the MVCC and the memory manager.

The final talk will cover thebackup and recovery with pgdump and the streaming replication.

The talks will be in Italian and, hopefully, I'll be able to stream them via google hangout.

Here is the event page in Italian .

The schedule is

15:00 Don’t panic! - Introduzione a PostgreSQL
15:40 Coffee break
15:50 The Ravenous Bugblatter Beast of Traal - PostgreSQL: uno sguardo dentro al motore
16:50 Coffee break
17:00 Mostly harmless - Backup & recovery in PostgreSQL
17:50 Closing remarks - Q&A


There is also a Facebook event to RSVP.



The event is Free of charge.
 
See you in Ferrara!
03 Jun 01:30

Palestra para mulheres em Brasília!

by Norma



© 2016 - Norma Braga. Todos os Direitos Reservados.
01 Jun 04:00

Map Age Guide

Does the screeching chill your blood and herald death? If yes, banshee. If no, seagull.
30 May 04:00

World War III+

I hate how the media only ever uses the first part of this quote, stripping it of its important context.
20 May 04:00

Digital Data

“If you can read this, congratulations—the archive you’re using still knows about the mouseover text”!
11 May 17:09

In Alien vs. Predator, I'm for Predator, Because He's OUR Predator

For all his faults, Trump would be an incomparably better president than Hillary.
11 May 09:58

denemo @ Savannah: Version 2.0.8 is out.

New features:

Copy and Paste
Applies to note/chord attributes
CtrlC, Ctrl-V work for these
Copied marking is highlighted
Selection changes color when copied
Improved Acoustic Feedback
Trill makes a short trill sound on entry
Copy attributes sounds
Improved Visual Feedback
Status bar notices are animated
Characters are highlighted in Lyric Verses
Directives are made more legible when cursor is on them
Cadenza Time
For un-metered music
Music can still display in “bars”
New Commands
Tuplet Positioning
Curved Tuplet Brackets
Cadenza on/off uses Cadenza Time, sets smaller note size, editable text
Notes without stems
Multi-line text annotation
Bold, Italic etc now apply to selection
A guard prevents accidental syntax collision
Updated Manual
More detail
Now indexed
Bug Fixes
Command Center search now reliable
Standalone Multi-line text with backslash editing
Pasting into measures that precede a time signature change

11 May 14:06

For Israel Independence Day: The Journey to Canaan at the Heart of American Culture

A weakness of the conservative movement is its confusion about what it means to be an American in the first place.
10 May 10:14

Two videos of the new modular Craft Camera

by Andreapazzo

Share

Introducing Craft Camera from Craft Digital Systems on Vimeo.

The recently announced new Craft Camera is a modular device that will also be available with MFT mount. Here are two videos shwowing a bit more about how this works.

Share

27 Apr 23:00

EU jeopardises its own goals in standardisation with FRAND licensing

EU jeopardises its own goals in standardisation with FRAND licensing

On 19 April, the European Commission published a communication on "ICT Standardisation Priorities for the Digital Single Market" (hereinafter 'the Communication'). The Digital Single Market (DSM) strategy intends to digitise industries with several legislative and political initiatives, and the Communication is a part of it covering standardisation. In general, the Free Software Foundation Europe (FSFE) welcomes the Communication's plausible approach for integrating Free Software and Open Standards into standardisation but expresses its concerns about the lack of understanding of necessary prerequisites to pursue that direction.

Acknowledging the importance of Free Software

The Communication starts with acknowledging the importance of Open Standards for interoperability, innovation and access to media, cultural and educational content, and promotes "community building, attracting new sectors, promoting open standards and platforms where needed, strengthening the link between research and standardisation". The latter is closely linked to the "cloud", where the Communication states that the "proprietary solutions, purely national approaches and standards that limit interoperability can severely hamper the potential of the Digital Single Market", and highlights that "common open standards will help users access new innovative services".

As a result, the Commission concludes that by the end of 2016 it intends to make more use of Free Software elements by better integrating Free Software communities into standard setting processes in the standards developing organisations.

In the Internet of Things (IoT) domain, the Communication acknowledges the EU need for "an open platform approach that supports multiple application domains ... to create competitive IoT ecosystems". In this regard, the Commission states that "this requires open standards that support the entire value chain, integrating multiple technologies ... based on streamlined international cooperation that build on an IPR ["intellectual property rights"] framework enabling easy and fair access to standard essential patents (SEPs)".

FSFE welcomes this direction taken in the Communication, as well as the Commissioner Günther Oettinger's position, highlighted in his keynote at the Net Futures 2016, that "easy reuse of standard and open components accelerates digitisation of any business or any industry sector." Furthermore, according to the Commissioner Oettinger, Free Software standards "enable transparency and build trust."

EC putting good efforts at risk

However, the attempts of the Commission to promote Open Standards and a more balanced approach towards "intellectual property rights" policies in standardisation may be seriously hampered by the Commission's stance towards FRAND licensing. In particular, the Commission sets the goal to "clarify core elements of an equitable, effective and enforceable licensing methodology around FRAND principles" which is seen as striking the right balance in standardisation and ensuring the "fair and non-discriminatory" access to standards. Furthermore, it is a well-known fact that FRAND licensing terms that in theory stand for "fair, reasonable, and non-discriminatory" terms, in practice are incompatible with most of Free Software.

In conclusion, whilst the Communication sets a positive direction towards the promotion of Open Standards and the inclusion of Free Software communities into the standardisation, this direction may be seriously limited if the Commission fails to acknowledge the incompatibility of FRAND licensing terms with Free Software licenses. This in return can in practice make a proper Free Software implementation of the standard impossible. As a result, the attempts of the Commission to achieve truly "digital single market" based on interoperability, openness and innovation will not be achieved as the significant part of innovative potential found in Free Software will be in practice excluded from standardisation.

In line with our recommendations on the DSM initiative that got well received by the Commission, FSFE believes that in order to achieve the adequate integration of Free Software communities, and the overall plausible approach towards appropriate use of Open Standards the Commission needs to avoid the harmful consequences of FRAND licensing to Free Software, and instead pursue the promotion of standards that are open, minimalistic and implementable with Free Software. These standards will give the substance to the Commission's promises to encourage Free Software communities to participate in standardisation.

Support FSFE, join the Fellowship
Make a one time donation

01 Mar 18:28

$2,000(!) off on the AG-AF100 superkit

by Andreapazzo

Share

AG-AF100

There is a $2,000 price drop on the AG-AF100 superkit sold by BHphoto (Click here).

More new deals:
Silver GX8 (Click here) and Black GX8 (Click here) for $888 only on eBay!
You save $150 on the Panasonic Vario 35-100mm lens sold on eBay (Click here).

New MFT stuff in Stock:
The new 300mm PRO lens is in Stock at Amazon US (Click here) and FocusCamera (Click here).
The Black PEN-F is also in Stock at FocusCamera (Click here).

New MFT lens preorders:
Panasonic 12-60mm f/3.5-5.6 at BHphoto (Click here) and at Adorama (Click here).
Sigma 30mm f/1.4 MFT at BHphoto (Click here) and at Adorama (Click here).

Share


23 Feb 17:16

5 razões para cursar minha Oficina de Escrita Criativa

by Rodrigo Gurgel

Nenhuma Oficina de Escrita Criativa tem o poder de transformar o aluno, num passe de mágica, em escritor. Nenhuma Oficina de Escrita Criativa pode conceder ao aluno, em poucas semanas, uma capacidade única para se expressar. Dizendo de outra forma, nenhum curso pode fazer com que o aluno ingresse, de repente, numa categoria iluminada de seres humanos.

Faço questão de sempre ressaltar essas impossibilidades, principalmente quando falo sobre o trabalho que desenvolvo em minha própria Oficina de Escrita Criativa — e quando abro inscrições para novas turmas, o que aconteceu recentemente, durante o webinário “12 Conselhos para Escritores Principiantes”.

Mas se os milagres citados no primeiro parágrafo não acontecem, por quais motivos alguém que deseja ser escritor deve cursar uma Oficina de Escrita Criativa? Para que ela serve? A seguir, apresento 5 razões:

1. Dar complexidade às narrativas

Estamos sempre compartilhando histórias, ainda que façamos isso, a maior parte do tempo, de forma inconsciente. Somos contadores de histórias; estamos sempre compondo narrativas e transmitindo-as a nossos familiares, a nossos amigos.

Em minha Oficina de Escrita Criativa levo o aluno para além dessa constatação — e faço com que ele, ao se tornar consciente dessa habilidade, compreenda como, em literatura, é possível carregar essas narrativas de tensão, humor, ironia, dramaticidade.

Esse trabalho de ensinar como uma narrativa pode ser mais complexa não se resume a meras técnicas para estimular a imaginação, mas abrange refletir sobre a condição humana, questionar a si próprio e observar a realidade com um novo olhar.

2. Exercitar o domínio da linguagem

razões para cursar uma Oficina de Escrita Criativa

É preciso transformar a habilidade para contar histórias numa prática consciente.

Transformar a habilidade para contar histórias numa prática consciente exige, portanto, um aprofundamento da autoconsciência — mas também exige maior precisão ao utilizar a linguagem, bem como o estudo dos elementos que compõem uma boa história.

Ao conhecer cada um desses elementos — em contato com textos fundamentais da literatura — e analisar de que maneira importantes escritores trabalharam, o aluno desperta para a necessidade de ter uma linguagem mais rigorosa, o que não deixa de ser uma forma de clarificar o pensamento.

Isso, aliás, nos recorda o sentido da palavra “aptidão”: não apenas uma disposição inata, mas uma habilidade que, em literatura, se aperfeiçoa à medida que estudamos e exercitamos o domínio da linguagem.

3. Criar diferentes “eus”

Observar a realidade com um novo olhar e ampliar sua autoconsciência leva o aluno a adquirir também uma compreensão mais profunda dos outros, dos seus semelhantes. Sem ela, é impossível construir narradores e personagens convincentes.

O escritor precisa saber quem narra a história que ele deseja contar e quem a vivencia: quais os valores, os preconceitos, as contradições, os sentimentos do narrador e dos personagens?

Procuro fazer, assim, com que o aluno alcance uma nova forma de empatia, por meio da qual possa viver e analisar os fatos sob diferentes perspectivas, como se carregasse consigo diferentes “eus”.

4. Reeducar a atenção

Quando estudamos os elementos acima não a partir de teorias, mas da leitura de textos fundamentais, o aluno compreende como estilos literários diversos expressam estados de espírito ou características pessoais que podem ou não ser semelhantes.

Aula após aula, o aluno é desafiado por esses grandes autores — desafiado a, conhecendo cada um deles, criar seu próprio estilo, sua própria voz.

Trata-se de um reaprendizado da leitura, de uma reeducação da atenção — um mergulho indispensável para perceber, no texto e na realidade, os pormenores que quase sempre nos escapam.

5. Cultivar a determinação

Por fim, é fundamental saber que a escrita exige disciplina, exige um comportamento metódico. Como tudo na vida, se não aprendemos a ser perseverantes, não nos desenvolvemos. É preciso ter consciência de que escrever não é fácil — e que aptidão ou talento são inúteis se não há determinação.

***

Estas 5 razões resumem o trabalho que desenvolvo em minha Oficina de Escrita Criativa. Mas você pode conhecer também o depoimento de alguns de meus alunos.

The post 5 razões para cursar minha Oficina de Escrita Criativa appeared first on Rodrigo Gurgel.

18 Feb 15:36

A Syrian Ghost Story: Lessons from Cardinal Richelieu

16 Feb 21:36

http://feedproxy.google.com/~r/EduardoMacan/~3/d2B_m1nzUto/

by Eduardo Maçan

IO FU' GIÀ QUEL CHE VOI SETE, E QUEL CH'I’ SON VOI ANCO SARETE

Pode ser tarde demais para evitar uma catástrofe na Venezuela, diz economista de Harvard

Para o venezuelano Ricardo Hausmann, não é hora de ficar em cima do muro: o país precisa de um plano crível (e isso não deve ser possível enquanto Nicolás Maduro estiver no poder)

17 Feb 05:00

Stargazing

Some of you may be thinking, 'But wait, isn't the brightest star in our sky the Sun?' I think that's a great question and you should totally ask it. On the infinite tree of possible conversations spread out before us, I think that's definitely the most promising branch.
16 Feb 01:13

Eduardo Maçan shared his post.

by Eduardo Maçan

Eduardo Maçan shared his post.

O eu do presente reavalia a previsão do eu do passado.

Abordei 15 conhecidos nos últimos 30 dias fazendo propaganda de vagas e/ou pedindo indicações. Dos 15 que abordei, 5 eu já sabia que tinham saído do país e outros 5 eu descobri que estão saindo ou planejando a saída.

E mal fez um ano. :(

Eduardo

Agora que o dólar passou de R$3 para fins práticos e uma recessão se avizinha, voltou a ser muito interessante trabalhar no exterior.

Eles já estavam vindo buscar talentos por aqui mesmo, agora então que um salário da ordem de 100 mil dólares por ano é capaz de fazer sobrar um bom pé de meia em reais, quero ver quem (bom) vai querer ficar no Brasil.

Aquisição de (verdadeiros) talentos em tecnologia: o que já era difícil vai ficar pior.

11 Feb 07:55

Chris Travers: Why Commons Should Not Have Ideological Litmus Tests

This will likely be my last post on this topic.  I would like to revive this blog on technical rather than ideological issues but there seems like a real effort to force ideology in some cases.  I don't address this in terms of specific rights, but in terms of community function and I have a few more things to say on this topic before I return to purely technical questions.

I am also going to say at the outset that LedgerSMB adopted the Ubuntu Code of Conduct very early (thanks to the suggestion of Joshua Drake) and this was a very good choice for our community.  The code of conduct provides a reminder for contributors, users, participants, and leadership alike to be civil and responsible in our dealings around the commons we create.  Our experience is we have had a very good and civil community with contributions from every walk of life and a wide range of political and cultural viewpoints.  I see  this as an unqualified success.

Lately I have seen an increasing effort to codify a sort of political orthodoxy around open source participation.  The rationale is usually about trying to make people feel safe in a community, but these are usually culture war issues so invariably the goal is to exclude those with specific political viewpoints (most of the world) from full participation, or at least silence them in public life.  I see this as extremely dangerous.

On the Economic Nature of Open Source


Open source software is economically very different from the sorts of software developed by large software houses.  The dynamics are different in terms of the sort of investment taken on, and the returns are different.  This is particularly true for community projects like PostgreSQL and LedgerSMB, but it is true to a lesser extent even for corporate projects like MySQL.  The economic implications thus are very different.

With proprietary software, the software houses build the software and absorb the costs for doing so, and then later find ways to monetize that effort.  In open source, that is one strategy among many but software is built as a community and in some sense collectively owned (see more on the question of ownerership below).

So with proprietary software, you may have limited ownership over the software, and this will be particularly limited when it comes to the use in economic production (software licenses, particularly for server software, are often written to demand additional fees for many connections etc).

Like the fields and pastures before enclosure, open source software is an economic commons we can all use in economic production.  We can all take the common software and apply it to our communities, creating value in those areas we value.  And we don't all have to share the same values to do it.  But it often feeds our families and more.

But acting as a community has certain requirements.  We have to treat eachother with humanity generally.  That doesn't mean we have to agree on everything but it does mean that some degree of civility must be maintained and cultivated by those who have taken on that power in open source projects.

On the Nature of Economic Production, Ownership and Power (Functionally Defined)


I am going to start by defining some terms here because I am using these terms in functional rather than formal ways.

Economic Production:  Like all organisms we survive by transforming our environment and making it more conducive to our ability to live and thrive.  In the interpersonal setting, we would call this economic production.  Note that understood in this way, this is a very broad definition and includes everything from cooking dinner for one's family to helping people work together.  Some of this may be difficult to value but it can (what is the difference between eating out and eating at home?  How much does a manager contribute to success through coordination?).

Ownership:  Defining ownership in functional rather than formal terms is interesting.  It basically means the right to use and direct usage of something.  Seen in this way, ownership is rarely all or nothing.  Economic ownership is the right to utilize a resource in economic production.  The extent to which one is restricted in economic production using a piece of software the less one owns it, so CAL requirements in commercial software and anti-TIVOization clauses in the GPL v3 are both restrictions on functional ownership.

Economic Power:  Economic power is the power to direct or restrict economic production.  Since economic production is required for human life, economic power is power over life itself.  In an economic order dominated by corporations, corporations control every aspect of our lives.  In places where the state has taken over from the corporations, the state takes over this as well.  But such power is rarely complete because not all economic production can be centrally controlled.

I am going to come back to these below because my hesitation on kicking people out of the community due to ideological disagreements (no matter how wrong one side may seem to be) have to do with this fear of abuse of economic power.


On Meritocracy (and what should replace it)


Meritocracy is an idea popularized by Eric Raymond, that power in a community should be given to technical merit.  In short, one should judge the code, not the person.  The idea has obvious appeal and is on the surface hyper-inclusive.  We don't have to care about anything regarding each other other than quality of code.  There is room for everyone.

More recently there has been push-back in some corners against the idea of meritocracy.  This push-back comes from a number of places, but what they have in common is questioning how inclusive it really is.

The most popular concern is that meritocracy suggests that we should tolerate people who actively make the community less welcoming, particularly for underrepresented groups. and therefore meritocracy becomes a cover for excluding the same groups who are otherwise excluded in other social dimensions, that the means of exclusion differs but who is excluded might not.

There is something to be said for the above concern, but advocates have often suggested that any nexus between community and hostile ideas is sufficient to raise a problem and therefore when an Italian Catholic expresses a view of gender based on his religion on Twitter, people not even involved in the project seek his removal from it on the grounds that the ideas are toxic.  For reasons that will become clear, that is vast overreach, and a legitimate complaint is thus made toxic by the actions of those who promote it.  And similarly toxic are the efforts by some to use social category to insist that their code should be included just to show a welcoming atmosphere.

A larger problem with meritocracy though is the way it sets up open source communities to be unbalanced, ruled by technical merit and thus not able to attract the other sorts of contributions needed to make most software successful.  In a community where technical merit is the measure by which we are judged, non-technical contributions are systematically devalued and undervalued.  How many open source communities produce software which is poorly documented and without a lot of attention to user interface?  If you devalue the efforts at documentation and UI design, how will you produce software which really meets people's needs?  If you don't value the business analysts and technical writers, how will you create economic opportunities for them in your community?  If you don't value them, how will you leverage their presence to deliver value to your own customers?  You can't if your values are skewed.

The successor to meritocracy should be economic communitarianism, i.e. the recognition that what is good for the community is economically good for all its members.  Rather than technical merit, the measure of a contribution and a contributor ought to be the value that a contribution brings the community.    Some of those will be highly technical but some will not.  Sometimes a very ordinary contribution that anyone could offer will turn the tide because only one person was brave enough to do it, or had the vision to see it as necessary.  Just because those are not technical does not mean that they are not valuable or should not be deeply respected.  I would argue that in many ways the most successful open source communities are the ones which have effectively interpreted meritocracy loosely as economic communitarianism.

On Feeling Safe in the Community


Let's face it  People need to feel safe and secure in the community regarding their physical safety and economic interests.  Is there any disagreement on this point?  If there is, please comment below.  But the community cannot be responsible for how someone feels, only in making sure that people are objectively physically and economically secure within it.  If someone feels unsafe in attending conferences, community members can help address security concerns and if someone severely misbehaves in community space, then that has to be dealt with for the good of everyone.

I don't think the proponents of ideological safety measures have really thought things through entirely.  The world is a big place and it doesn't afford people ideological safety unless they don't go out and work with people they disagree with.  As soon as you go across an international border, disagreements will spring up everywhere and if you aren't comfortable with this then interacting on global projects is probably not for you.

Worse, when it comes to conduct outside of community circles, those in power in the community cannot really act constructively most of the time.  We don't have intimate knowledge and even if we do, our viewpoints have to be larger than the current conflict.

On "Cultural Relativism:" A welcoming community for all?


One of the points I have heard over and over in discussions regarding community codes of conduct is that welcoming people regardless of viewpoint (particularly on issues like abortion, sexuality, etc) is cultural relativism and thus not acceptable.  I guess the question is not acceptable to whom?  And do we really want an ideological orthodoxy on every culture war topic to be a part of an open source project?  Most people I have met do not want this.

But the overall question I have for people who push culture war codes of conduct is "when you say a welcoming community for all, do you really mean it?  Or do you just mean for everyone you agree with?  What if the majority changes their minds?"

In the end, as I will show below, trying to enforce an ideological orthodoxy in this way does not bring marginal groups into the community but necessary forces a choice of which marginal groups to further exclude.  I don't think that is a good choice and I will go on record and say it is a choice I will steadfastly refuse to make.

A Hypothetical


Ideology is shaped by culture, and ideology of sexuality is shaped by family structures, so consequently where family structures are different, views on sexuality will be also.

So suppose someone on a community email list includes a pro-same-sex marriage email signature, something like:

"Marriage is an institution for the benefit of the spouses, not [to] bind parents to their children" -- Ted Olson, arguing for a right to same-sex marraige before the United States Supreme Court.

So a socially conservative software developer from southern India complaints to the core committee saying that this is an attack on his culture, saying that traditional Indian marriages are not real marriages.  Now, I assume most people would agree that it would be best for the core committee not to insist that the email signature be changed for someone to continue to participate.  So with such a decision, suppose the complainant changes his signature instead to read:

"If mutual consent makes a sexual act moral, whether within marriage or without, and, by parity of reasoning, even between members of the same sex, the whole basis of sexual morality is gone and nothing but misery and defect awaits the youth of the country… " -- Mohandas Gandhi

Now the first person decries the signature as homophobic and demands the Indian fellow be thrown off the email list.  And the community, if it has decided to follow the effort at ideological safety has to resolve the issue.  Which group to exclude?  The sexual minority?  Or the group marginalized through a history of being on the business end of colonialism?  And if one chooses the latter, then what does that say about the state of the world?  Should Indians, Malaysians, Catholics, etc. band together to fork a competing project?  Is that worth it as a cost?  Doesn't that hurt everyone?

On Excluding People from the Commons


In my experience, excluding people from the commons carries with it massive cost, and this is a good thing because it keeps economic power from being abused.  I have watched the impact first hand.  LedgerSMB would not even exist if this weren't an issue with SQL-Ledger.  That we are now the only real living fork of SQL-Ledger and far more active than the project we forked from is a testament to the cost.

Of course in that case the issue was economic competition and a developer who did not want to leverage community development to build his own business.  I was periodically excluded from SQL-Ledger mailing lists etc for building community documentation (he sold documentation).  Finally the fork happened beccause he wouldn't take security reports seriously.  And this is one of the reasons why I would like to push for an inclusive community.

But I also experienced economic ramifications from being excluded.  It was harder to find customers (again, the reason for exclusion was economic competition so that was the point).  In essence, I am deeply aware of the implications of kicking people out.

I have seen on email lists and tracker tickets the comparison of the goal of excluding people with problematic ideologies with McCarthyism.  The goal of McCarthyism was indeed similar, to make sure that if you had the wrong ideas you would be unable to continue a professional career.  I have had relatives who suffered because they defended the legal rights of the Communist Party during that time.  I am aware of cases where the government tried to take away their professional career (unsuccessfully).

Management of community is political and the cost of excluding someone is also political.  We already exist in some ways on the margins of the software industry.  Exclude too many people and you create your own nemesis.  That's what happened to SQL-Ledger and why LedgerSMB is successful today.

Notes on former FreeBSDGirl


One blog entry that comes from the other side of this issue is Randi Harper's piece on why she no longer will go to FreeBSD conferences and participate on IRC channels.   I am not familiar with the facts surrounding her complaints and frankly I don't have time to be so what the nature of her harassment complaint is, I will not be the judge.

There is however another side to the issue that is outside what she evidently has experience with, and that is the role of software maintainers in addressing the sorts of complaints she made.  Consequently I want to address that side and then discuss her main points at the bottom.

One thing to remember is that when people make accusations of bullying, harassment, etc. the people in charge are also the people with the least actual knowledge of what is going on.  Expecting justice from those in power in cases like this will lead, far more often than not, to feelings of betrayal.  This is not because of bad intentions but because of lack of knowledge.  This was one thing I learned navigating schoolyard bullies when I was growing up and we as project maintainers are in an even lower knowledge role than school administrators are.  Bullies are furthermore usually experts at navigating the system and take advantage of those who are not as politically adept, so the more enforcement you throw at the problem, the worse it gets.

So there is an idea that those in charge will stop people from treating eachother badly.  That has to stop because it isn't really possible (as reasonable as it sounds).  What we can do is keep the peace in community settings and that is about it.  One needs bottom up solutions, not top down ones.

So if someone came to me as a maintainer of a project alleging harassment on Twitter and demanding that an active committer be removed, that demand would probably go nowhere.  If political statements were mentioned, the response would be "do we want a political orthodoxy?"  Yet LedgerSMB has avoided these problems largely because, I think, we are a community of small businesses and therefore are used to working through disagreements and maybe because we are used to seeing these sorts of things as political.

Her main points though are worth reading and pondering.  In some areas she is perfectly right and in some areas dangerously wrong.

Randi is most right in noting that personal friction cannot be handled like a technical problem.  It is a political problem and needs to be handled as such.  I don't think official processes are the primary protection here, and planning doesn't get you very far, but things do need to be handled delicately.

Secondly, there is a difference between telling someone to stay quiet and telling someone not to be shouting publicly.   I think it is worth noting that if mediation is going to work then one cannot have people trying to undermine that in public, but people do need friends and family for support and so it is important to avoid the impression that one is insisting on total confidentiality.

Randi is also correct that how one deals with conflict is a key gauge of how healthy an open source community is.  Insisting that people be banished because of politically offensive viewpoints however does not strike me as healthy or constructive.  Insisting that people behave themselves in community spaces does.  In very rare cases it may be necessary to mediate cases that involve behavior outside that, but insisting on strict enforcement of some sort of a codified policy will not bring peace or prosperity.

More controversially I will point out that there is a point that Randi makes implicitly that is worth making explicit here, namely that there is a general tone-deafness to women's actual experiences in open source.  I think this is very valid.  I can remember a former colleague in LedgerSMB making a number of complaints about how women were treated in open source.  Her complaints included both unwanted sexual attention ("desperate geeks") and more actionably the fact that she was repeatedly asked how to attract more women to open source (she responded once on an IRC channel with "do you know how annoying that is?").  She ultimately moved on to other projects following a change in employment that moved LedgerSMB outside the scope of duties,  but one obvious lesson that those of us in open source can take from this is just to listen to complaints.  Many of these are not ones that policies can solve (you really want a policy aimed at telling people not to ask what needs to be done to attract more women to open source?) but if we listen, we can learn something.

One serious danger in the current push for more expansive codes of conduct is that it puts those who have the least knowledge in the greatest responsibility.  My view is that expansive codes of conduct, vesting greater power with maintainers over areas of political advocacy outside community fora will lead to greater, not less conflict.  So I am not keen in her proposed remedies.

How Codes of Conducts Should be Used


The final point I want to bring up here is how codes of conduct should be used.  These are not things which should be seen as pseudo-legal or process-oriented documents.  If you go this way, people will abuse the system.  It is better in my experience to vest responsibility with the maintainers in keeping the peace, not dispensing out justice, and to have codes of conduct aimed at the former, not the latter.  Justice is a thorny issue, one philosophers around the world have been arguing about for millennia with no clear resolution.

A major problem is the simple fact that perception and reality don't always coincide.  I was reminded of this controversy while reading an article in The Local about the New Years Eve sexual assaults, about work by a feminist scholar in Sweden to point out that actually men are more at risk from bodily violence than women are, and that men suffer disproportionately from crime but are the least likely to modify behavior to avoid being victimized.  The article is worth reading in light of the current issues.

So I think if one expects justice from a code of conduct, one expects too much.  If one expects fairness from a code of conduct, one expects too much.  If one expects peace and prosperity for all, then that may be attainable but that is not compatible with the idea that one has a right not to be confronted by people with dangerous ideologies.

Codes of conducts, used right, provide software maintainers with a valuable tool for keeping the peace.  Used wrong, they lead open source projects into ruin.  In the end, we have to be careful to be ideologically and culturally inclusive and that means that people cannot guarantee that they are safe from ideas they find threatening.
09 Feb 00:00

Fire From Moonlight

by xkcd

Fire From Moonlight

Can you use a magnifying glass and moonlight to light a fire?

—Rogier Spoor

At first, this sounds like a pretty easy question.

A magnifying glass concentrates light on a small spot. As many mischevious kids can tell you, a magnifying glass as small as a square inch in size can collect enough light to start a fire. A little Googling will tell you that the Sun is 400,000 times brighter than the Moon, so all we need is a 400,000-square-inch magnifying glass. Right?

Wrong. Here's the real answer: You can't start a fire with moonlight[1]Pretty sure this is a Bon Jovi song. no matter how big your magnifying glass is. The reason is kind of subtle. It involves a lot of arguments that sound wrong but aren't, and generally takes you down a rabbit hole of optics.

First, here's a general rule of thumb: You can't use lenses and mirrors to make something hotter than the surface of the light source itself. In other words, you can't use sunlight to make something hotter than the surface of the Sun.

There are lots of ways to show why this is true using optics, but a simpler—if perhaps less satisfying—argument comes from thermodynamics:

Lenses and mirrors work for free; they don't take any energy to operate.[2]And, more specifically, everything they do is fully reversible—which means you can add them in without increasing the entropy of the system. If you could use lenses and mirrors to make heat flow from the Sun to a spot on the ground that's hotter than the Sun, you'd be making heat flow from a colder place to a hotter place without expending energy. The second law of thermodynamics says you can't do that. If you could, you could make a perpetual motion machine.

The Sun is about 5,000°C, so our rule says you can't focus sunlight with lenses and mirrors to get something any hotter than 5,000°C. The Moon's sunlit surface is a little over 100°C, so you can't focus moonlight to make something hotter than about 100°C. That's too cold to set most things on fire.

"But wait," you might say. "The Moon's light isn't like the Sun's! The Sun is a blackbody—its light output is related to its high temperature. The Moon shines with reflected sunlight, which has a "temperature" of thousands of degrees—that argument doesn't work!"

It turns out it does work, for reasons we'll get to later. But first, hang on—is that rule even correct for the Sun? Sure, the thermodynamics argument seems hard to argue with,[3]Because it's correct. but to someone with a physics background who's used to thinking of energy flow, it may seem hard to swallow. Why can't you concentrate lots of sunlight onto a point to make it hot? Lenses can concentrate light down to a tiny point, right? Why can't you just concentrate more and more of the Sun's energy down onto the same point? With over 1026 watts available, you should be able to get a point as hot as you want, right?

Except lenses don't concentrate light down onto a point—not unless the light source is also a point. They concentrate light down onto an area—a tiny image of the Sun.[4]Or a big one! This difference turns out to be important. To see why, let's look at an example:

This lens directs all the light from point A to point C. If the lens were to concentrate light from the Sun down to a point, it would need to direct all the light from point B to point C, too:

But now we have a problem. What happens if light goes back from point C toward the lens? Optical systems are reversible, so the light should be able to go back to where it came from—but how does the lens know whether the light came from B or to A?

In general, there's no way to "overlay" light beams on each other, because the whole system has to be reversible. This keeps you from squeezing more light in from a given direction, which puts a limit on how much light you can direct from a source to a target.

Maybe you can't overlay light rays, but can't you, you know, sort of smoosh them closer together, so you can fit more of them side-by-side? Then you could gather lots of smooshed beams and aim them at a target from slightly different angles.

Nope, you can't do this.[5]We already know this, of course, since earlier we said that it would let you violate the second law of thermodynamics.

It turns out that any optical system follows a law called conservation of étendue. This law says that if you have light coming into a system from a bunch of different angles and over a large "input" area, then the input area times the input angle[6]Note to nitpickers: In 3D systems, this is technically the solid angle, the 2D equivalent of the regular angle, but whatever. equals the output area times the output angle. If your light is concentrated to a smaller output area, then it must be "spread out" over a larger output angle.

In other words, you can't smoosh light beams together without also making them less parallel, which means you can't aim them at a faraway spot.

There's another way to think about this property of lenses: They only make light sources take up more of the sky; they can't make the light from any single spot brighter,[7]A popular demonstration of this: Try holding up a magnifying glass to a wall. The magnifying glass collects light from many parts of the wall and sends them to your eye, but it doesn't make the wall look brighter. because it can be shown[8]This is left as an exercise for the reader. that making the light from a given direction brighter would violate the rules of étendue.[9]My résumé says étendue is my forté. In other words, all a lens system can do is make every line of sight end on the surface of a light source, which is equivalent to making the light source surround the target.

If you're "surrounded" by the Sun's surface material, then you're effectively floating within the Sun, and will quickly reach the temperature of your surroundings.[10](Very hot)

If you're surrounded by the bright surface of the Moon, what temperature will you reach? Well, rocks on the Moon's surface are nearly surrounded by the surface of the Moon, and they reach the temperature of the surface of the Moon (since they are the surface of the Moon.) So a lens system focusing moonlight can't really make something hotter than a well-placed rock sitting on the Moon's surface.

Which gives us one last way to prove that you can't start a fire with moonlight: Buzz Aldrin is still alive.

02 Feb 05:00

Backslashes

I searched my .bash_history for the line with the highest ratio of special characters to regular alphanumeric characters, and the winner was: cat out.txt | grep -o "[[(].*[])][^)]]*$" ... I have no memory of this and no idea what I was trying to do, but I sure hope it worked.